Worker does not start on kubernetes

All pods are working ony worker is restarting. It’s fresh installation, empty database with new schema.

server-version":"1.16.1

"msg":"worker starting","service":"worker","component":"worker","logging-call-at":"service.go:329"}
{"level":"fatal","ts":"2022-05-05T09:13:40.649Z","msg":"error getting system sdk client","service":"worker","error":"unable to create SDK client: get system info failed: context deadline exceeded - *serviceerror.DeadlineExceeded","logging-call-at":"factory.go:98","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Fatal\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/sdk.(*clientFactory).GetSystemClient.func1\n\t/home/builder/temporal/common/sdk/factory.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngo.temporal.io/server/common/sdk.(*clientFactory).GetSystemClient\n\t/home/builder/temporal/common/sdk/factory.go:94\ngo.temporal.io/server/service/worker.(*Service).startScanner\n\t/home/builder/temporal/service/worker/service.go:441\ngo.temporal.io/server/service/worker.(*Service).Start\n\t/home/builder/temporal/service/worker/service.go:355\ngo.temporal.io/server/service/worker.ServiceLifetimeHooks.func1.1\n\t/home/builder/temporal/service/worker/fx.go:129"}

unable to create SDK client: get system info failed: context deadline exceeded

Seems like issue setting up connection to start system workflows. Do you have TLS enabled, and can you see any connection errors in logs?

Are these errors transient (for example you only see them during redeploy or pod restart?) If so they can be ignored.

1 Like

Hi i am facing similar error while deploying temporal freshly , i am using latest helm repo version , any fix ?

{"level":"fatal","ts":"2022-05-24T11:29:03.054Z","msg":"error getting system sdk client","service":"worker","error":"unable to create SDK client: get system info failed: context deadline exceeded - *serviceerror.DeadlineExceeded","logging-call-at":"factory.go:98","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Fatal\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/sdk.(*clientFactory).GetSystemClient.func1\n\t/home/builder/temporal/common/sdk/factory.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngo.temporal.io/server/common/sdk.(*clientFactory).GetSystemClient\n\t/home/builder/temporal/common/sdk/factory.go:94\ngo.temporal.io/server/service/worker.(*Service).startScanner\n\t/home/builder/temporal/service/worker/service.go:441\ngo.temporal.io/server/service/worker.(*Service).Start\n\t/home/builder/temporal/service/worker/service.go:355\ngo.temporal.io/server/service/worker.ServiceLifetimeHooks.func1.1\n\t/home/builder/temporal/service/worker/fx.go:129"}

This is the error I am able to see through worker pod’s logs

What do you get via

kubectl get po

Can you confirm that all your pods started up ?

Just tested with latest helm repo and it deployed fine (both k8s and minikube). This could me memory related as well.

All other pods frontend,history ,admintools are running fine , just the worker pod is crashing continuously. I dont think issue is with the memory since plenty of memory is vaialble
could you share details on which k8s version and temporal version you tried out ?