Temporal TLS Enabled

@tihomir

Temporal-Server version: 1.16.2, ui-server: 2.2.0

I have deployed the Temporal in Kubernets by enabling the TLS with the below configuration. I can see the logs in temporal services and UI also able to communicate.

{"level":"debug","ts":"2022-09-05T11:23:19.402Z","msg":"attempted incoming TLS connection","address":"100.127.100.88:59556","server-name":"server-asyncworkflow-local-tls.jpmchase.net","logging-call-at":"localStoreTlsProvider.go:321"}
{"level":"debug","ts":"2022-09-05T11:23:19.402Z","msg":"returning TLS config for connection","address":"100.127.100.88:59556","server-name":"server-asyncworkflow-local-tls.jpmchase.net","logging-call-at":"localStoreTlsProvider.go:378"}

Temporal-server development.yaml

  tls:
    internode:
      server:
        certFile: /etc/temporal/certs/cluster.pem
        keyFile: /etc/temporal/certs/cluster.key
        requireClientAuth: false
        clientCaFiles:
          - /etc/temporal/certs/root_ca.pem
      client:
        serverName: server-asyncworkflow-local-tls.jpmchase.net
        rootCaFiles:
          - /etc/temporal/certs/root_ca.pem
    frontend:
      client:
        rootCaFiles:
          - /etc/temporal/certs/root_ca.pem
      server:
        requireClientAuth: false
        certFile: /etc/temporal/certs/cluster.pem
        keyFile: /etc/temporal/certs/cluster.key
        clientCaFiles:
          - /etc/temporal/certs/root_ca.pem
    systemWorker:
      certFile: /etc/temporal/certs/cluster.pem
      keyFile: /etc/temporal/certs/cluster.key
      client:
        forceTLS: true
        serverName: server-asyncworkflow-local-tls.jpmchase.net
        disableHostVerification: true
        rootCaFiles:
          - /etc/temporal/certs/root_ca.pem

UI-Server development.yaml (from UI i can able to see the workflows)

tls:
  caFile: /etc/temporal/certs/root_ca.pem
  certFile: /etc/temporal/certs/cluster.pem
  keyFile: /etc/temporal/certs/cluster.key
  enableHostVerification: false
  serverName: frontend:7233

I would like to connect from sdk-java client to temporal-server by passing the client key & certs.
I have followed the instructions of tls-simple and the code.

I am getting the below error, It looks to me Health check is failure.

Exception in thread “main” io.grpc.StatusRuntimeException: UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: connection termination
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:251)
at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:282)
at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61)
at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51)
at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:275)
at io.temporal.serviceclient.WorkflowServiceStubsImpl.(WorkflowServiceStubsImpl.java:182)
at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51)
at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41)
at com.jpmorgan.temporal.client.SslEnabledWorker.main(SslEnabledWorker.java:43)

I have tried same from tctl cluster health getting the below error.

tctl --ad server-asyncworkflow-local.apps.mt-d1.carl.gkp.jpmchase.net:443 --tls_ca_path certs/root_ca.pem --tls_disable_host_verification cluster health
Error: Unable to get “temporal.api.workflowservice.v1.WorkflowService” health check status.
Error Details: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection termination
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/temporalio/tctl/cli_curr.printError({0xc0005aa6c0, 0x54}, {0x2576ee0, 0xc0005bc018})
/home/tctl-builder/cli_curr/util.go:392 +0x21e
github.com/temporalio/tctl/cli_curr.ErrorAndExit({0xc0005aa6c0?, 0x25?}, {0x2576ee0?, 0xc0005bc018?})
/home/tctl-builder/cli_curr/util.go:403 +0x28
github.com/temporalio/tctl/cli_curr.HealthCheck(0x356bb80?)
/home/tctl-builder/cli_curr/clusterCommands.go:50 +0x174
github.com/temporalio/tctl/cli_curr.newClusterCommands.func1(0xc00040ba20?)
/home/tctl-builder/cli_curr/cluster.go:36 +0x19
github.com/urfave/cli.HandleAction({0x1ca14a0?, 0x21bfec0?}, 0x6?)
/root/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:526 +0x50
github.com/urfave/cli.Command.Run({{0x20dd46e, 0x6}, {0x0, 0x0}, {0xc000693850, 0x1, 0x1}, {0x211e9ca, 0x20}, {0x0, …}, …}, …)
/root/go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:173 +0x652
github.com/urfave/cli.(*App).RunAsSubcommand(0xc00078a000, 0xc00040b760)
/root/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:405 +0x91b
github.com/urfave/cli.Command.startApp({{0x20dfac1, 0x7}, {0x0, 0x0}, {0xc000693930, 0x1, 0x1}, {0x2107e2b, 0x18}, {0x0, …}, …}, …)
/root/go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:372 +0x6e7
github.com/urfave/cli.Command.Run({{0x20dfac1, 0x7}, {0x0, 0x0}, {0xc000693930, 0x1, 0x1}, {0x2107e2b, 0x18}, {0x0, …}, …}, …)
/root/go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:102 +0x808
github.com/urfave/cli.(*App).Run(0xc000569c00, {0xc00012e000, 0x8, 0x8})
/root/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:277 +0x8a7
main.main()
/home/tctl-builder/cmd/tctl/main.go:45 +0xa6

What is wrong with this?

tctl --ad server-asyncworkflow-local.apps.mt-d1.carl.gkp.jpmchase.net:443 --tls_ca_path certs/root_ca.pem --tls_disable_host_verification cluster health

Can you try specifying both client cert and key? for example:
-tls_cert_path <client.pem> --tls_key_path <client.key>

@tihomir

I have tried with client cert path & key, same error message.

image

$ tctl --ad server-asyncworkflow-local.apps.mt-d1.carl.gkp.jpmchase.net:443 --tls_ca_path root_ca.pem --tls_cert_path client/client.pem --tls_key_path client/client.key cluster health
Error: Unable to get “temporal.api.workflowservice.v1.WorkflowService” health check status.
Error Details: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection termination
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
C:/JPMC/DEV/TMP/ds/tools/go1.18/latest/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
C:/JPMC/DEV/TMP/ds/tools/go1.18/latest/src/runtime/debug/stack.go:16 +0x19
github.com/temporalio/tctl/cli_curr.printError({0xc0005be180, 0x54}, {0x3190680, 0xc00013eb18})
I:/Code/asyncworkflow/tctl/async-workflow-temporal-tctl_develop/cli_curr/util.go:392 +0x21e
github.com/temporalio/tctl/cli_curr.ErrorAndExit({0xc0005be180?, 0x25?}, {0x3190680?, 0xc00013eb18?})
I:/Code/asyncworkflow/tctl/async-workflow-temporal-tctl_develop/cli_curr/util.go:403 +0x28
> github.com/temporalio/tctl/cli_curr.HealthCheck(0x419b940?)
I:/Code/asyncworkflow/tctl/async-workflow-temporal-tctl_develop/cli_curr/clusterCommands.go:50 +0x174
github.com/temporalio/tctl/cli_curr.newClusterCommands.func1(0xc00028b080?)
I:/Code/asyncworkflow/tctl/async-workflow-temporal-tctl_develop/cli_curr/cluster.go:36 +0x19
github.com/urfave/cli.HandleAction({0x28ae9a0?, 0x2dcf6f8?}, 0x6?)
C:/JPMC/DEV/TMP/ds/data/go1.18/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:526 +0x50
github.com/urfave/cli.Command.Run({{0x2cea69b, 0x6}, {0x0, 0x0}, {0xc00058b7b0, 0x1, 0x1}, {0x2d2dbad, 0x20}, {0x0, …}, …}, …)
C:/JPMC/DEV/TMP/ds/data/go1.18/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:173 +0x652
github.com/urfave/cli.(*App).RunAsSubcommand(0xc000779dc0, 0xc00028adc0)
C:/JPMC/DEV/TMP/ds/data/go1.18/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:405 +0x91b
github.com/urfave/cli.Command.startApp({{0x2ceccca, 0x7}, {0x0, 0x0}, {0xc00058b890, 0x1, 0x1}, {0x2d164e1, 0x18}, {0x0, …}, …}, …)
C:/JPMC/DEV/TMP/ds/data/go1.18/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:372 +0x6e7
github.com/urfave/cli.Command.Run({{0x2ceccca, 0x7}, {0x0, 0x0}, {0xc00058b890, 0x1, 0x1}, {0x2d164e1, 0x18}, {0x0, …}, …}, …)
C:/JPMC/DEV/TMP/ds/data/go1.18/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:102 +0x808
github.com/urfave/cli.(*App).Run(0xc000779a40, {0xc00014c000, 0xb, 0x10})
C:/JPMC/DEV/TMP/ds/data/go1.18/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:277 +0x8a7
main.main()
I:/Code/asyncworkflow/tctl/async-workflow-temporal-tctl_develop/cmd/tctl/main.go:45 +0xa6

@tihomir Did you get a change to look on this issue?

1 Like

@tihomir - any further update on this