I have a self hosted setup where I have multiple server host VMs. Each host runs a full suite of services (ui, frontend, history, matching, worker). They each publish their broadcast address and all listen on that address. Each are cross talking fine it would seem. I also have two Workers, implemented via the go-sdk, running in containers on VMs. The workers communicate to the server via an Azure Load Balancer (so that the worker has a static configured URL that we don’t have to update if server hosts change). The LB is using a simple TCP health check to the GRPC port on the workers. That all seems to be working fine.
Our problem is that when one of the servers is shut down, like say for a graceful reboot, one or both of the Worker containers will experience what seems like a crash and a restart. They throw several context deadline exceeded messages, and a few “Error Frontend is not healthy yet” messages before I see container startup indications. There’s never an explicit crash or exit message that I can find. These are the last two lines of the logs:
Aug 12 16:11:59 provisioning-temporalworker a1793da26ae7[1217]: Timestamp=2025-08-12T23:11:59.469735048Z SeverityText=ERROR InstrumentationScope=temporalworker Body="failed to create schedule" deployment.environment=staging gh.sdk.name=github-telemetry-go gh.sdk.version=1.32.0 service.instance.id=d2b89284hn2hbo6eb1mg service.name=provisioning-service service.version=f6b6e1451dea9a15e2f5b01de1eb62148c9fadfd telemetry.sdk.name=opentelemetry error="context deadline exceeded"
Aug 12 16:11:59 provisioning-temporalworker dockerd[1217]: time="2025-08-12T16:11:59.537131806-07:00" level=info msg="ignoring event" container=a1793da26ae73f37d0ba9b8841a79c4eaa5c0b33f47abbc9c91c4a9003e67c69 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Now, while I appreciate that Temporal is very very good at retrying things and recovering from a lost activity, we have some activities that are “one shot” things, that are not idempotent. They can get lost when the worker crashes like this, and the workflow eventually sees this as an activity timeout, and the whole workflow fails in ways that leads to human clean up. We’re definitely working to reduce or eliminate the number of “one shot” activities. We’d ALSO like to ensure we’re doing what we can to keep our workers operating when one (of multiple) frontends is taken off line.
I’ve done multiple things to try and gracefully exit a frontend. I’m blocking the Azure LB health source for 30 seconds prior to stopping the service, so that the Azure LB considers the backend unhealthy and will not route new traffic there. Then I stop the temporal service, which appears to be doing things gracefully. I see messages about ShutdownHandler operating. e.g.:
Aug 12 16:12:37 provisioning-temporal temporal-server[2194268]: {"level":"info","ts":"2025-08-12T16:12:37.135-0700","msg":"ShutdownHandler: Evicting self from membership ring immediately","service":"matching","logging-call-at":"service.go:123"}
Aug 12 16:12:37 provisioning-temporal-d29c817.azure-eastus.github.net temporal-server[2194268]: {"level":"info","ts":"2025-08-12T16:12:37.135-0700","msg":"Current reachable members","component":"service-resolver","service":"matching","addresses":["10.0.35.212:7235"],"logging-call-at":"service_resolver.go:331"}
Aug 12 16:12:37 provisioning-temporal temporal-server[2194268]: {"level":"info","ts":"2025-08-12T16:12:37.136-0700","msg":"ShutdownHandler: Waiting for others to discover I am unhealthy","service":"matching","logging-call-at":"service.go:131"}
I’ve even enabled dynamic config for frontend that is setting frontend.ShutdownDrainDuration to 10s to increase the drain time.
Unfortunately none of that seems to prevent the worker(s) from crashing. I feel like I’m missing something fundamental here that is preventing workers from reconnecting to another frontend when one is being shutdown. I’m hopeful for some help here 