Hi team,
We’re trying to connect Python client(SDK version v1.10.0) to our self-hosted Temporal server(version v1.26.2) via a proxy using an authorization header. We see that it is able to connect to the worker at startup, but after exactly 60 seconds, it fails with the below error. Note that we do see some transient 503 errors while connecting to the server; but after that it’s able to connect successfully (verified in server logs).
Our issue looks similar to this this open bug: [Bug] Worker hangs after polling workflow task queue · Issue #631 · temporalio/sdk-python · GitHub
Wanted to check if it is same issue and if not we need help in understanding the fixing it.
Here’ the set of errors we see:
1.
Error: Client connection not established Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }.
Traceback (most recent call last):
File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
asyncio.run(main())
File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 157, in main
await conn.create_client()
File "/home/temporal/temporal_sample_worker_python/src/utils.py", line 105, in create_client
self.client = await Client.connect("dev.api.data.falcon.sfdcfc.net:7443",
File "/usr/local/lib64/python3.9/site-packages/temporalio/client.py", line 176, in connect
await temporalio.service.ServiceClient.connect(connect_config),
File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 209, in connect
return await _BridgeServiceClient.connect(config)
File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1117, in connect
await client._connected_client()
File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1130, in _connected_client
self._bridge_client = await temporalio.bridge.client.Client.connect(
File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/client.py", line 97, in connect
await temporalio.bridge.temporal_sdk_bridge.connect_client(
RuntimeError: Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }
2025-03-19T02:17:48.974543Z WARN temporal_sdk_core::worker::workflow::wft_poller: Error while polling for workflow tasks error=Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }
2025-03-19T02:17:48.974547Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19T02:17:48.974551Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19 02:17:48,975 - ERROR - Worker failed, shutting down
Traceback (most recent call last):
File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
act = await self._bridge_worker().poll_workflow_activation()
File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed
2025-03-19 02:17:48,975 - INFO - Beginning worker shutdown, will wait 0:00:00 before cancelling activities
Traceback (most recent call last):
File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
act = await self._bridge_worker().poll_workflow_activation()
File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
asyncio.run(main())
File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 179, in main
await run_worker(client, taskqueue_name, args.worker_type)
File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 142, in run_worker
await worker.run()
File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_worker.py", line 571, in run
raise exception
File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed