Server-worker: Failed to poll for task

Community Support
, ,
1

when I deploy the temporal cluster by helm charts, the pod of each service is running, but the logs of worker service show some errors.

{"level":"warn","ts":"2022-12-07T10:07:30.714Z","msg":"Failed to poll for task.","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-per-ns-tq","WorkerID":"server-worker@1@temporalserver-worker-7f5499fc-q8sv6@temporal-system","WorkerType":"WorkflowWorker","Error":"error reading from server: EOF","logging-call-at":"internal_worker_base.go:297"}
{"level":"warn","ts":"2022-12-07T10:07:30.714Z","msg":"Failed to poll for task.","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-processor-parent-close-policy","WorkerID":"1@temporalserver-worker-7f5499fc-q8sv6@","WorkerType":"ActivityWorker","Error":"error reading from server: EOF","logging-call-at":"internal_worker_base.go:297"}
{"level":"warn","ts":"2022-12-07T10:07:30.714Z","msg":"Failed to poll for task.","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-per-ns-tq","WorkerID":"server-worker@1@temporalserver-worker-7f5499fc-q8sv6@temporal-system","WorkerType":"ActivityWorker","Error":"error reading from server: EOF","logging-call-at":"internal_worker_base.go:297"}
{"level":"warn","ts":"2022-12-07T10:07:30.714Z","msg":"Failed to poll for task.","service":"worker","Namespace":"temporal-system","TaskQueue":"default-worker-tq","WorkerID":"1@temporalserver-worker-7f5499fc-q8sv6@","WorkerType":"WorkflowWorker","Error":"error reading from server: EOF","logging-call-at":"internal_worker_base.go:297"}
{"level":"warn","ts":"2022-12-07T10:07:30.714Z","msg":"Failed to poll for task.","service":"worker","Namespace":"temporal-system","TaskQueue":"default-worker-tq","WorkerID":"1@temporalserver-worker-7f5499fc-q8sv6@","WorkerType":"ActivityWorker","Error":"error reading from server: EOF","logging-call-at":"internal_worker_base.go:297"}
{"level":"warn","ts":"2022-12-07T10:07:30.714Z","msg":"Failed to poll for task.","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-batcher-taskqueue","WorkerID":"1@temporalserver-worker-7f5499fc-q8sv6@","WorkerType":"WorkflowWorker","Error":"error reading from server: EOF","logging-call-at":"internal_worker_base.go:297"}

Note: other service(frontend, history, matching) does not show any logs error.
So why caused this errors and how to solve it?

2

error reading from server: EOF

Can you health check your frontend/history/matching services?

./grpc-health-probe -addr=<frontend-host>:<frontend-grpc-port> -service=temporal.api.workflowservice.v1.WorkflowService

./grpc-health-probe -addr=<matching-host>:<matching-grpc-port> -service=temporal.api.workflowservice.v1.MatchingService

./grpc-health-probe -addr=<history-host>:<history-grpc-port> -service=temporal.api.workflowservice.v1.HistoryService

Error indicates your internal worker service sdk workers (that run internal workflows on temporal-system ns) cant poll frontend (all api calls go through frontend service). Check if any pods are restarting as if they become stable this should be an intermittent issue.

(error reading from server: EOF typically indicates some service pods is/are down/restarting/are being rolled out)

3

Hi, @tihomir , thanks for your reply. According to your suggestions, I have checked all service(frontend, history, matching) by grpc-health-probe, and the status of each service is SERVING. In addition, I double-checked that all pods are running, and I try to delete worker pod, waiting for new pod to be restarted, but the new worker pod(ns temporal-system) is still show the errors that ‘Failed to poll task, ActivityWorker Error: error reading form server: EOF’

5

Hi,
Same for me:
Worker log:
{"level":"warn","ts":"2023-11-06T14:34:09.978Z","msg":"Failed to poll for task.","service":"worker","Namespace":"py-temporal-upsell-ondemand-dev","TaskQueue":"temporal-sys-per-ns-tq","WorkerID":"server-worker@1@temporalio-worker-756c5fffc7-hx4c2@py-temporal-upsell-ondemand-dev","WorkerType":"WorkflowWorker","Error":"last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.43.184.162:7233: connect: connection refused\"","logging-call-at":"internal_worker_base.go:308"}
But health probe ok all grpc ports