Temporal web cannot access temporal server via ssl endpoint

Shawn_Lim · June 29, 2021, 3:49pm

I passed the TEMPORAL_GRPC_ENDPOINT env variable value of example.temporal.cluster.cloud:443

The temporal cluster is fronted by an AWS ALB, enabled with gRPC. It is successfully accessed by other backend code by initializing the workflow client like this:

WorkflowServiceStubsOptions options = WorkflowServiceStubsOptions.newBuilder().setEnableHttps(true).setTarget("example.temporal.cluster.cloud:443").build();
    WorkflowServiceStubs service = WorkflowServiceStubs.newInstance(options);
    WorkflowClient client = WorkflowClient.newInstance(service);

However on the web UI, it shows “14 UNAVAILABLE: Trying to connect an http1.x server”

Ruslan · July 2, 2021, 12:40am

Hi Shawn,

i’ve been checking on the internet the possible reasons for why this could be happening with no luck so far. Will let you know if i find something.

Adding verbosity to logging may show something interesting, you can enable this with DEBUG=* GRPC_TRACE=all GRPC_VERBOSITY=DEBUG env variables when running the web

Shawn_Lim · July 5, 2021, 5:41am

Pasted my logs here after enabling the debug variables. The part on insecure channel appears suspicious to me.

establishing insecure connection…
grpc_insecure_channel_create(target=example.temporal.cluster.cloud:443, args=0x55d11e4fb820, reserved=0)


2021-07-05T05:29:08.571Z koa-router test /api/web-settings/data-converter/:port /^\/api\/web-settings\/data-converter\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05T05:29:08.571Z koa-router test /api/web-settings/data-converter/:port /^\/api\/web-settings\/data-converter\/((?:[^\/]+?))(?:\/(?=$))?$/i {

keys: [Array]

keys: [Array]

2021-07-05 13:29:08

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/signal/:signal /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/signal\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/signal/:signal /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/signal\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.571Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}


2021-07-05 13:29:08

2021-07-05T05:29:08.571Z koa-router test /api/namespaces/:namespace/task-queues/:taskQueue/pollers /^\/api\/namespaces\/((?:[^\/]+?))\/task-queues\/((?:[^\/]+?))\/pollers(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.571Z koa-router test /api/namespaces/:namespace/task-queues/:taskQueue/ /^\/api\/namespaces\/((?:[^\/]+?))\/task-queues\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}


2021-07-05 13:29:08

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/query /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/query(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/query/:queryType /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/query\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/terminate /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/terminate(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.568Z koa-router test /api/namespaces/:namespace/workflows/closed /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/closed(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.569Z koa-router test /api/namespaces/:namespace/workflows/list /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/list(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.569Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/history /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/history(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.569Z koa-router test /api/namespaces/:namespace/workflows/archived /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/archived(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.569Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/export /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/export(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.567Z koa-router test /auth/logout /^\/auth\/logout(?:\/(?=$))?$/i { keys: [] }

2021-07-05 13:29:08

2021-07-05T05:29:08.568Z koa-router test /api/namespaces /^\/api\/namespaces(?:\/(?=$))?$/i { keys: [] }

2021-07-05 13:29:08

2021-07-05T05:29:08.568Z koa-router test /api/namespaces/:namespace /^\/api\/namespaces\/((?:[^\/]+?))(?:\/(?=$))?$/i { keys: [Array] }

2021-07-05 13:29:08

2021-07-05T05:29:08.568Z koa-router test /api/namespaces/:namespace/workflows/open /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/open(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.566Z koa-router GET /actuator/health

2021-07-05 13:29:08

2021-07-05T05:29:08.567Z koa-router test /auth/sso /^\/auth\/sso(?:\/(?=$))?$/i { keys: [] }

2021-07-05 13:29:08

2021-07-05T05:29:08.567Z koa-router test /auth/sso_callback /^\/auth\/sso_callback(?:\/(?=$))?$/i { keys: [] }

2021-07-05 13:29:08

2021-07-05T05:29:08.565Z koa-session:context create session with val: undefined externalKey: undefined

2021-07-05 13:29:08

2021-07-05T05:29:08.564Z koa-session:context init from cookie

2021-07-05 13:28:57

temporal-web up and listening on port 8088

2021-07-05 13:28:57

2021-07-05T05:28:57.227Z koa-static static "/usr/app/dist" {}

2021-07-05 13:28:57

2021-07-05T05:28:57.227Z koa:application use serve

2021-07-05 13:28:57

2021-07-05T05:28:57.227Z koa:application use -

2021-07-05 13:28:57

2021-07-05T05:28:57.227Z koa:application listen

2021-07-05 13:28:57

2021-07-05T05:28:57.196Z koa:application use dispatch

2021-07-05 13:28:57

2021-07-05T05:28:57.197Z koa:application use allowedMethods

2021-07-05 13:28:57

2021-07-05T05:28:57.197Z koa:application use -

2021-07-05 13:28:57

2021-07-05T05:28:57.195Z koa:application use bound middleware

2021-07-05 13:28:57

2021-07-05T05:28:57.193Z koa:application use initialize

2021-07-05 13:28:57

2021-07-05T05:28:57.194Z koa:application use passportInitialize

2021-07-05 13:28:57

2021-07-05T05:28:57.194Z koa:application use passportAuthenticate

2021-07-05 13:28:57

2021-07-05T05:28:57.193Z koa-session session options {"key":"koa.sess","overwrite":true,"httpOnly":true,"signed":true,"autoCommit":true}

2021-07-05 13:28:57

2021-07-05T05:28:57.193Z koa:application use session

2021-07-05 13:28:57

2021-07-05T05:28:57.191Z koa:application use compress

2021-07-05 13:28:57

2021-07-05T05:28:57.184Z koa:application use bodyParser

2021-07-05 13:28:57

2021-07-05T05:28:57.183Z koa:application use -

2021-07-05 13:28:56

2021-07-05T05:28:56.232Z koa-router defined route [ 'POST' ] /api/web-settings/data-converter/:port

2021-07-05 13:28:56

2021-07-05T05:28:56.233Z koa-router defined route [ 'HEAD', 'GET' ] /api/web-settings

2021-07-05 13:28:56

2021-07-05T05:28:56.233Z koa-router defined route [ 'HEAD', 'GET' ] /api/me

2021-07-05 13:28:56

2021-07-05T05:28:56.233Z koa-router defined route [ 'HEAD', 'GET' ] /api/cluster/version-info

2021-07-05 13:28:56

2021-07-05T05:28:56.231Z koa-router defined route [ 'POST' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/signal/:signal

2021-07-05 13:28:56

2021-07-05T05:28:56.232Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/:workflowId/:runId

2021-07-05 13:28:56

2021-07-05T05:28:56.232Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/task-queues/:taskQueue/pollers

2021-07-05 13:28:56

2021-07-05T05:28:56.232Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/task-queues/:taskQueue/

2021-07-05 13:28:56

2021-07-05T05:28:56.230Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/export

2021-07-05 13:28:56

2021-07-05T05:28:56.231Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/query

2021-07-05 13:28:56

2021-07-05T05:28:56.231Z koa-router defined route [ 'POST' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/query/:queryType

2021-07-05 13:28:56

2021-07-05T05:28:56.231Z koa-router defined route [ 'POST' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/terminate

2021-07-05 13:28:56

I0705 05:28:56.230741382 1 credentials.cc:43] grpc_channel_credentials_release(creds=0)

2021-07-05 13:28:56

2021-07-05T05:28:56.228Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/history

2021-07-05 13:28:56

2021-07-05T05:28:56.229Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/archived

2021-07-05 13:28:56

2021-07-05T05:28:56.228Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/open

2021-07-05 13:28:56

2021-07-05T05:28:56.228Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/closed

2021-07-05 13:28:56

2021-07-05T05:28:56.228Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/list

2021-07-05 13:28:56

2021-07-05T05:28:56.227Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces

2021-07-05 13:28:56

2021-07-05T05:28:56.227Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace

2021-07-05 13:28:56

I0705 05:28:56.184119759 1 channel_create.cc:95] grpc_insecure_channel_create(target=example.temporal.cluster.cloud:443, args=0x55d11e4fb820, reserved=0)

I0705 05:28:56.184119759 1 channel_create.cc:95] grpc_insecure_channel_create(target=example.temporal.cluster.cloud:443, args=0x55d11e4fb820, reserved=0)

2021-07-05 13:28:56

I0705 05:28:56.184245708 1 init.cc:164] grpc_init(void)

2021-07-05 13:28:56

I0705 05:28:56.184322564 1 client_idle_filter.cc:293] (client idle filter) created with max_leisure_time = 1800000 ms

2021-07-05 13:28:56

I0705 05:28:56.184384144 1 client_channel.cc:1367] chand=0x55d11e4eb4e8: creating client_channel for channel stack 0x55d11e4eb308

2021-07-05 13:28:56

establishing insecure connection...

2021-07-05 13:28:55

2021-07-05T05:28:55.740Z koa-router defined route [ 'HEAD', 'GET' ] /sso_callback

2021-07-05 13:28:55

2021-07-05T05:28:55.740Z koa-router defined route [ 'HEAD', 'GET' ] /logout

2021-07-05 13:28:55

2021-07-05T05:28:55.736Z koa-router defined route [ 'HEAD', 'GET' ] /sso

2021-07-05 13:28:55

I0705 05:28:55.335636676 26 timer_manager.cc:250] timers not checked: expect another thread to

2021-07-05 13:28:55

I0705 05:28:55.335826959 26 timer_manager.cc:197] sleep until kicked

2021-07-05 13:28:55

I0705 05:28:55.272667907 1 completion_queue.cc:501] grpc_completion_queue_create_internal(completion_type=0, polling_type=0)

2021-07-05 13:28:55

D0705 05:28:55.271138106 1 dns_resolver.cc:294] Using native dns resolver

2021-07-05 13:28:55

I0705 05:28:55.271680978 1 timer_manager.cc:88] Spawn timer thread

2021-07-05 13:28:55

I0705 05:28:55.271839988 1 init.cc:164] grpc_init(void)

Ruslan · July 6, 2021, 10:33pm

" establishing insecure connection…" means that the Web couldn’t read mTLS environment variables

Could you double check that the mTLS env variables are passed to Web (in Optional TLS configuration variables):

Shawn_Lim · July 9, 2021, 11:58am

Hi Ruslan, so some update. I changed my LB to use NLB instead of ALB. I’m no longer doing SSL so my temporal web can now access the frontend LB address correctly.

Ruslan · July 9, 2021, 5:46pm

great to hear, thanks for the update!
Can be useful to other people

Shawn_Lim · July 10, 2021, 6:10am

The root issue appears to be that the AWS ALB is not fully compatible with gRPC. It’s downgrading to http/1.1 before forwarding to the backend target server. With NLB, it just forwards the request.

Ruslan · August 1, 2021, 3:00pm

This topic was automatically closed after 23 days. New replies are no longer allowed.

Tristan_Fletcher · August 27, 2021, 11:53pm

I have been running temporal web to a temporal server frontend behind an AWS ALB with SSL for a few months now. I am not sure of the original poster’s exact set up or issue, but it was initially really annoying to get it to work. Here are a few things I encountered that I needed to fix along the way:

Ensure you give temporal web a certificate authority that works with your ALB’s SSL cert. I made one out of amazon’s public CA certs and fed it to temporal web via env var: TEMPORAL_TLS_CA_PATH
Ensure you provide the right server name if you have the host verification enabled, this should match the server name you use in TEMPORAL_GRPC_ENDPOINT and be stored in env var TEMPORAL_TLS_SERVER_NAME
My ALB listened on 443, but I forwarded it to temporal on 7233. I recommend ensuring the target groups are set properly there so the traffic is forwarded to the correct ports.

As to the original poster’s problem, maybe there was an issue with the SSL setup or address? If you try and talk non-ssl I think the AWS ALB will downgrade you to HTTP 1.1 depending on the config of your ALB.

raghudeshu · November 30, 2021, 6:18pm

Hi Tristan,

I am currently running temporal on Kube with latest version Where do we set “TEMPORAL_WEB_TLS_CERT_PATH” and “TEMPORAL_WEB_TLS_KEY_PATH” in temporal web docker images.

Tristan_Fletcher · November 30, 2021, 6:45pm

You set them as environment variables. You also need to get those certs on your machine, probably with some kind of volume mount.

I am not familiar with the specifics of your kube deployment, but searching around for how to set environment variables and mount a volume with your certs in your deployment of kube should get you there pretty quick.

You could build your own docker images and hardcode this stuff, but that is generally unadvisable/more hassle than it’s worth.

Shuchi_Gupta · December 10, 2021, 7:12am

Hi Tristan,

Can we run frontend service using a self-signed certificate?

Topic		Replies	Views
How to enable SSL in Temporal Web? Community Support web-ui	6	1818	March 30, 2021
Enable End-to-End TLS (HTTPS to Container) in Temporal Web Community Support web-ui , tls	11	1773	August 30, 2021
Temporal External IP calling Community Support deployment	4	714	August 26, 2021
Temporal on kubernetes Community Support java-sdk , deployment	4	935	October 28, 2022
Temporal Web GRPC Connectivity Issue - AWS ECS Fargate Community Support elasticsearch , docker , aws	3	1292	December 24, 2021

Temporal web cannot access temporal server via ssl endpoint

Related Topics