Hi Team,
I have a self hosted Temporal server running in Temporal namespace on kubernetes cluster. I have a spring boot application with the temporal client logic in a different namespace. On application start up worker bean initialization is failing as its encountering an error establishing communication with the temporal server.
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-08-29 15:12:01.677 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'compWorkflowWorker': Invocation of init method failed; nested exception is io.grpc.StatusRuntimeException: UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111
This is the error seen in the istio envoy proxy logs
[2023-08-29T18:43:36.217Z] "POST /temporal.api.workflowservice.v1.WorkflowService/GetSystemInfo HTTP/2" 200 UF upstream_reset_before_response_started{connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 5 0 2 - "-" "grpc-java-netty/1.53.0" "294df73f-fef8-41fd-ab31-c0e55f80ebe4" "temporaltest-frontend-headless.temporal.svc.cluster.local" "10.108.0.25:80" PassthroughCluster - 10.108.0.25:80 10.108.1.4:50016 - allow_any
[2023-08-29T18:43:30.336Z] "- - -" 0 - - - "-" 27872 40491 5921 - "-" "-" "-" "-" "10.86.1.26:5432" PassthroughCluster 10.108.1.4:38592 10.86.1.26:5432 10.108.1.4:38586 - -
I exec in to the pod and tried to curl the temporal frontend endpoint and I see the below response.
curl temporaltest-frontend-headless.temporal.svc.cluster.local:7233 --verbose
* Trying 10.108.0.25:7233...
* Connected to temporaltest-frontend-headless.temporal.svc.cluster.local (10.108.0.25) port 7233 (#0)
> GET / HTTP/1.1
> Host: temporaltest-frontend-headless.temporal.svc.cluster.local:7233
> User-Agent: curl/7.81.0
> Accept: */*
>
* Received HTTP/0.9 when not allowed
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed
Not sure if receiving HTTP/0.9 response is what causing the upstream server connection to close.
How can I resolve this so that communication can be established between the app and temporal server.
cc @maxim