Temporal client not able to communicate with server - upstream connect error or disconnect/reset before headers

Hi Team,

I have a self hosted Temporal server running in Temporal namespace on kubernetes cluster. I have a spring boot application with the temporal client logic in a different namespace. On application start up worker bean initialization is failing as its encountering an error establishing communication with the temporal server.

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-08-29 15:12:01.677 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'compWorkflowWorker': Invocation of init method failed; nested exception is io.grpc.StatusRuntimeException: UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111

This is the error seen in the istio envoy proxy logs

[2023-08-29T18:43:36.217Z] "POST /temporal.api.workflowservice.v1.WorkflowService/GetSystemInfo HTTP/2" 200 UF upstream_reset_before_response_started{connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 5 0 2 - "-" "grpc-java-netty/1.53.0" "294df73f-fef8-41fd-ab31-c0e55f80ebe4" "temporaltest-frontend-headless.temporal.svc.cluster.local" "" PassthroughCluster - - allow_any
[2023-08-29T18:43:30.336Z] "- - -" 0 - - - "-" 27872 40491 5921 - "-" "-" "-" "-" "" PassthroughCluster - -

I exec in to the pod and tried to curl the temporal frontend endpoint and I see the below response.

curl temporaltest-frontend-headless.temporal.svc.cluster.local:7233 --verbose     
*   Trying
* Connected to temporaltest-frontend-headless.temporal.svc.cluster.local ( port 7233 (#0)
> GET / HTTP/1.1
> Host: temporaltest-frontend-headless.temporal.svc.cluster.local:7233
> User-Agent: curl/7.81.0
> Accept: */*
* Received HTTP/0.9 when not allowed
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed

Not sure if receiving HTTP/0.9 response is what causing the upstream server connection to close.

How can I resolve this so that communication can be established between the app and temporal server.

cc @maxim

Found the issue. Port info was missing and the request was being sent on port 80 instead of 7233.