Temporal web cannot access temporal server via ssl endpoint

I passed the TEMPORAL_GRPC_ENDPOINT env variable value of example.temporal.cluster.cloud:443

The temporal cluster is fronted by an AWS ALB, enabled with gRPC. It is successfully accessed by other backend code by initializing the workflow client like this:

WorkflowServiceStubsOptions options = WorkflowServiceStubsOptions.newBuilder().setEnableHttps(true).setTarget("example.temporal.cluster.cloud:443").build();
    WorkflowServiceStubs service = WorkflowServiceStubs.newInstance(options);
    WorkflowClient client = WorkflowClient.newInstance(service);

However on the web UI, it shows “14 UNAVAILABLE: Trying to connect an http1.x server”

Hi Shawn,

i’ve been checking on the internet the possible reasons for why this could be happening with no luck so far. Will let you know if i find something.

Adding verbosity to logging may show something interesting, you can enable this with DEBUG=* GRPC_TRACE=all GRPC_VERBOSITY=DEBUG env variables when running the web

Pasted my logs here after enabling the debug variables. The part on insecure channel appears suspicious to me.

establishing insecure connection…
grpc_insecure_channel_create(target=example.temporal.cluster.cloud:443, args=0x55d11e4fb820, reserved=0)


2021-07-05T05:29:08.571Z koa-router test /api/web-settings/data-converter/:port /^\/api\/web-settings\/data-converter\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05T05:29:08.571Z koa-router test /api/web-settings/data-converter/:port /^\/api\/web-settings\/data-converter\/((?:[^\/]+?))(?:\/(?=$))?$/i {

keys: [Array]

keys: [Array]

2021-07-05 13:29:08

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/signal/:signal /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/signal\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/signal/:signal /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/signal\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.571Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}


2021-07-05 13:29:08

2021-07-05T05:29:08.571Z koa-router test /api/namespaces/:namespace/task-queues/:taskQueue/pollers /^\/api\/namespaces\/((?:[^\/]+?))\/task-queues\/((?:[^\/]+?))\/pollers(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.571Z koa-router test /api/namespaces/:namespace/task-queues/:taskQueue/ /^\/api\/namespaces\/((?:[^\/]+?))\/task-queues\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}


2021-07-05 13:29:08

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/query /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/query(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/query/:queryType /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/query\/((?:[^\/]+?))(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.570Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/terminate /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/terminate(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.568Z koa-router test /api/namespaces/:namespace/workflows/closed /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/closed(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.569Z koa-router test /api/namespaces/:namespace/workflows/list /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/list(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.569Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/history /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/history(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.569Z koa-router test /api/namespaces/:namespace/workflows/archived /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/archived(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.569Z koa-router test /api/namespaces/:namespace/workflows/:workflowId/:runId/export /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/((?:[^\/]+?))\/((?:[^\/]+?))\/export(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.567Z koa-router test /auth/logout /^\/auth\/logout(?:\/(?=$))?$/i { keys: [] }

2021-07-05 13:29:08

2021-07-05T05:29:08.568Z koa-router test /api/namespaces /^\/api\/namespaces(?:\/(?=$))?$/i { keys: [] }

2021-07-05 13:29:08

2021-07-05T05:29:08.568Z koa-router test /api/namespaces/:namespace /^\/api\/namespaces\/((?:[^\/]+?))(?:\/(?=$))?$/i { keys: [Array] }

2021-07-05 13:29:08

2021-07-05T05:29:08.568Z koa-router test /api/namespaces/:namespace/workflows/open /^\/api\/namespaces\/((?:[^\/]+?))\/workflows\/open(?:\/(?=$))?$/i {

2021-07-05 13:29:08

keys: [Array]

2021-07-05 13:29:08

}

2021-07-05 13:29:08

2021-07-05T05:29:08.566Z koa-router GET /actuator/health

2021-07-05 13:29:08

2021-07-05T05:29:08.567Z koa-router test /auth/sso /^\/auth\/sso(?:\/(?=$))?$/i { keys: [] }

2021-07-05 13:29:08

2021-07-05T05:29:08.567Z koa-router test /auth/sso_callback /^\/auth\/sso_callback(?:\/(?=$))?$/i { keys: [] }

2021-07-05 13:29:08

2021-07-05T05:29:08.565Z koa-session:context create session with val: undefined externalKey: undefined

2021-07-05 13:29:08

2021-07-05T05:29:08.564Z koa-session:context init from cookie

2021-07-05 13:28:57

temporal-web up and listening on port 8088

2021-07-05 13:28:57

2021-07-05T05:28:57.227Z koa-static static "/usr/app/dist" {}

2021-07-05 13:28:57

2021-07-05T05:28:57.227Z koa:application use serve

2021-07-05 13:28:57

2021-07-05T05:28:57.227Z koa:application use -

2021-07-05 13:28:57

2021-07-05T05:28:57.227Z koa:application listen

2021-07-05 13:28:57

2021-07-05T05:28:57.196Z koa:application use dispatch

2021-07-05 13:28:57

2021-07-05T05:28:57.197Z koa:application use allowedMethods

2021-07-05 13:28:57

2021-07-05T05:28:57.197Z koa:application use -

2021-07-05 13:28:57

2021-07-05T05:28:57.195Z koa:application use bound middleware

2021-07-05 13:28:57

2021-07-05T05:28:57.193Z koa:application use initialize

2021-07-05 13:28:57

2021-07-05T05:28:57.194Z koa:application use passportInitialize

2021-07-05 13:28:57

2021-07-05T05:28:57.194Z koa:application use passportAuthenticate

2021-07-05 13:28:57

2021-07-05T05:28:57.193Z koa-session session options {"key":"koa.sess","overwrite":true,"httpOnly":true,"signed":true,"autoCommit":true}

2021-07-05 13:28:57

2021-07-05T05:28:57.193Z koa:application use session

2021-07-05 13:28:57

2021-07-05T05:28:57.191Z koa:application use compress

2021-07-05 13:28:57

2021-07-05T05:28:57.184Z koa:application use bodyParser

2021-07-05 13:28:57

2021-07-05T05:28:57.183Z koa:application use -

2021-07-05 13:28:56

2021-07-05T05:28:56.232Z koa-router defined route [ 'POST' ] /api/web-settings/data-converter/:port

2021-07-05 13:28:56

2021-07-05T05:28:56.233Z koa-router defined route [ 'HEAD', 'GET' ] /api/web-settings

2021-07-05 13:28:56

2021-07-05T05:28:56.233Z koa-router defined route [ 'HEAD', 'GET' ] /api/me

2021-07-05 13:28:56

2021-07-05T05:28:56.233Z koa-router defined route [ 'HEAD', 'GET' ] /api/cluster/version-info

2021-07-05 13:28:56

2021-07-05T05:28:56.231Z koa-router defined route [ 'POST' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/signal/:signal

2021-07-05 13:28:56

2021-07-05T05:28:56.232Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/:workflowId/:runId

2021-07-05 13:28:56

2021-07-05T05:28:56.232Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/task-queues/:taskQueue/pollers

2021-07-05 13:28:56

2021-07-05T05:28:56.232Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/task-queues/:taskQueue/

2021-07-05 13:28:56

2021-07-05T05:28:56.230Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/export

2021-07-05 13:28:56

2021-07-05T05:28:56.231Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/query

2021-07-05 13:28:56

2021-07-05T05:28:56.231Z koa-router defined route [ 'POST' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/query/:queryType

2021-07-05 13:28:56

2021-07-05T05:28:56.231Z koa-router defined route [ 'POST' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/terminate

2021-07-05 13:28:56

I0705 05:28:56.230741382 1 credentials.cc:43] grpc_channel_credentials_release(creds=0)

2021-07-05 13:28:56

2021-07-05T05:28:56.228Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/:workflowId/:runId/history

2021-07-05 13:28:56

2021-07-05T05:28:56.229Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/archived

2021-07-05 13:28:56

2021-07-05T05:28:56.228Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/open

2021-07-05 13:28:56

2021-07-05T05:28:56.228Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/closed

2021-07-05 13:28:56

2021-07-05T05:28:56.228Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace/workflows/list

2021-07-05 13:28:56

2021-07-05T05:28:56.227Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces

2021-07-05 13:28:56

2021-07-05T05:28:56.227Z koa-router defined route [ 'HEAD', 'GET' ] /api/namespaces/:namespace

2021-07-05 13:28:56

I0705 05:28:56.184119759 1 channel_create.cc:95] grpc_insecure_channel_create(target=example.temporal.cluster.cloud:443, args=0x55d11e4fb820, reserved=0)

I0705 05:28:56.184119759 1 channel_create.cc:95] grpc_insecure_channel_create(target=example.temporal.cluster.cloud:443, args=0x55d11e4fb820, reserved=0)

2021-07-05 13:28:56

I0705 05:28:56.184245708 1 init.cc:164] grpc_init(void)

2021-07-05 13:28:56

I0705 05:28:56.184322564 1 client_idle_filter.cc:293] (client idle filter) created with max_leisure_time = 1800000 ms

2021-07-05 13:28:56

I0705 05:28:56.184384144 1 client_channel.cc:1367] chand=0x55d11e4eb4e8: creating client_channel for channel stack 0x55d11e4eb308

2021-07-05 13:28:56

establishing insecure connection...

2021-07-05 13:28:55

2021-07-05T05:28:55.740Z koa-router defined route [ 'HEAD', 'GET' ] /sso_callback

2021-07-05 13:28:55

2021-07-05T05:28:55.740Z koa-router defined route [ 'HEAD', 'GET' ] /logout

2021-07-05 13:28:55

2021-07-05T05:28:55.736Z koa-router defined route [ 'HEAD', 'GET' ] /sso

2021-07-05 13:28:55

I0705 05:28:55.335636676 26 timer_manager.cc:250] timers not checked: expect another thread to

2021-07-05 13:28:55

I0705 05:28:55.335826959 26 timer_manager.cc:197] sleep until kicked

2021-07-05 13:28:55

I0705 05:28:55.272667907 1 completion_queue.cc:501] grpc_completion_queue_create_internal(completion_type=0, polling_type=0)

2021-07-05 13:28:55

D0705 05:28:55.271138106 1 dns_resolver.cc:294] Using native dns resolver

2021-07-05 13:28:55

I0705 05:28:55.271680978 1 timer_manager.cc:88] Spawn timer thread

2021-07-05 13:28:55

I0705 05:28:55.271839988 1 init.cc:164] grpc_init(void)

" establishing insecure connection…" means that the Web couldn’t read mTLS environment variables

Could you double check that the mTLS env variables are passed to Web (in Optional TLS configuration variables):

Hi Ruslan, so some update. I changed my LB to use NLB instead of ALB. I’m no longer doing SSL so my temporal web can now access the frontend LB address correctly.

1 Like

great to hear, thanks for the update!
Can be useful to other people

The root issue appears to be that the AWS ALB is not fully compatible with gRPC. It’s downgrading to http/1.1 before forwarding to the backend target server. With NLB, it just forwards the request.

2 Likes