Worker pod crasing

Community Support
1

Hi,

Temporal version 1.15.1
I am running the temporal server in k8s and running as individual pods. Worker pod always crashes and restarting with the below error message.

Worker pod Error logs:

{“level”:“fatal”,“ts”:“2022-05-13T07:43:03.749Z”,“msg”:“error starting scanner”,“service”:“worker”,“error”:“context deadline exceeded”,“logging-call-at”:“service.go:436”

image
image666×780 79.2 KB

I have configured the deployment.yml for worker.

tls:
frontend:
client:
rootCaFiles:
- “/etc/temporal/cacerts/root_ca.pem”
systemWorker:
certFile: “/etc/temporal/cacerts/cluster.pem”
keyFile: “/etc/temporal/cacerts/cluster.key”
client:
forceTLS: true
serverName: “server-asyncworkflow-local.apps.mt-d2.carl.gkp.jpmchase.net”
disableHostVerification: true
rootCaFiles:
- “/etc/temporal/cacerts/root_ca.pem”
publicClient:
hostPort: “frontend:7233”

{“level”:“info”,“ts”:“2022-05-15T12:37:23.199Z”,“msg”:“Current reachable members”,“service”:“worker”,“component”:“service-resolver”,“service”:“frontend”,“addresses”:[“100.127.0.217:7233”],“logging-call-at”:“rpServiceResolver.go:266”}
{“level”:“info”,“ts”:“2022-05-15T12:37:23.200Z”,“msg”:“Current reachable members”,“service”:“worker”,“component”:“service-resolver”,“service”:“matching”,“addresses”:[“100.127.0.199:7235”],“logging-call-at”:“rpServiceResolver.go:266”}
{“level”:“info”,“ts”:“2022-05-15T12:37:23.200Z”,“msg”:“Current reachable members”,“service”:“worker”,“component”:“service-resolver”,“service”:“history”,“addresses”:[“100.127.101.187:7234”],“logging-call-at”:“rpServiceResolver.go:266”}
{“level”:“info”,“ts”:“2022-05-15T12:37:23.200Z”,“msg”:“Current reachable members”,“service”:“worker”,“component”:“service-resolver”,“service”:“worker”,“addresses”:[“100.127.0.216:7239”],“logging-call-at”:“rpServiceResolver.go:266”}

worker pod member identity doesn’t come up.

image
image1475×278 30.8 KB

image
image1102×563 9.51 KB

image
image643×614 18.4 KB

Temporal-web(UI) can able to see the namespace & workflows.

What could be the reason why the worker can’t able to communicate with frontend & keep on crashing?

2

Do you by chance set up cluster ingress controller for your frontend service?

3

Yes @tihomir ingress has been configured on the cluster ip address which all the services running on the same kubernetes namespaces.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: temporal
spec:
podSelector:
matchLabels:
app: temporal
ingress:
- from:
- namespaceSelector:
matchExpressions:
- {key: gkp_namespace, operator: In, values: [ingress-contour, ingress-nginx, 109820-asyncworkflow-dev]}
#mt-d2.carl
- ipBlock:
cidr: 169.28.62.151/32
- ipBlock:
cidr: 169.28.62.252/32

4

Take a look at forum posts here and here where user was setting up tls and ingress. Maybe it’s a similar scenario, let me know.