Hello,
I am wondering if this is a Temporal bug or if I am missing some setting.
I am trying to link clusters that are configured for mTLS. Logging shows that internode mTLS, among the services within each of the individual clusters, works fine. I can see the certs going through. The command that I am using for linking is:
./tctl --address “${TEMPORAL_CLI_ADDRESS}” admin cluster upsert-remote-cluster --frontend_address “${REMOTE_CLUSTER_FRONTEND_ADDRESS}” --enable_connection true --tls_ca_path “${TEMPORAL_TLS_CERTS_DIR}/ca.cert” --tls_cert_path “${TEMPORAL_TLS_CERTS_DIR}/client.pem” --tls_key_path “${TEMPORAL_TLS_CERTS_DIR}/client.key” --tls_disable_host_verification false --tls_server_name “${TEMPORAL_CLI_TLS_SERVER_NAME}”
The error that I am getting is:
Error: Operation AddOrUpdateRemoteCluster failed.
Error Details: rpc error: code = Unavailable desc = last connection error: connection closed before server preface received.
I am consuming temporal source, having added a custom authorizer (with claim mapper and authorizer).
I found the spot where the error is logged and modified it to give me some extra logging, so the actual output looks like this:
Error: Operation AddOrUpdateRemoteCluster failed.
Error Details: rpc error: code = Unavailable desc = last connection error: connection closed before server preface received. Address: {
“Addr”: “host.docker.internal:(remote front end port)”,
“ServerName”: “host.docker.internal:(remote front end port)”,
“Attributes”: null,
“BalancerAttributes”: null,
“Type”: 0,
“Metadata”: null
}
The logging in GetClaims() and Authorize() shows that the local front end recognizes the client certificate and allows AddOrUpdateRemoteCluster api call. The failure happens when the local front end calls the remote front end. From what I see of some stack traces that I added in the vendor code, it doesn’t appear that the AdminServiceClient consumes the client certificate.
Full error log:
Note:
- temporal-stack is the name of my wrapper around Temporal)
- I had to put dashes (-) in front of file paths in order to avoid getting them tagged as urls by this editor
{“level”:“info”,“ts”:“2023-01-19T05:25:25.042Z”,“msg”:“admin client encountered error”,“service”:“frontend”,“error”:"last connection error: connection closed before server preface received. Address: {
“Addr”: “host.docker.internal:(remote front end port)”,
“ServerName”: “host.docker.internal:(remote front end port)”,
“Attributes”: null,
“BalancerAttributes”: null,
“Type”: 0,
“Metadata”: null
}“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“metric_client.go:82”} {“level”:“error”,“ts”:“2023-01-19T05:25:25.042Z”,“msg”:“unavailable error”,“service”:“frontend”,“error”:"last connection error: connection closed before server preface received. Address: {
“Addr”: “host.docker.internal:(remote front end port)”,
“ServerName”: “host.docker.internal:(remote front end port)”,
“Attributes”: null,
“BalancerAttributes”: null,
“Type”: 0,
“Metadata”: null
}“logging-call-at”:“adminHandler.go:1793”,“stacktrace”:“go.temporal.io/server/common/log.(*zapLogger).Error
/cloud/temporal-stack/vendor/go.temporal.io/server/common/log/zap_logger.go:142
-go.temporal.io/server/service/frontend.(*AdminHandler).error
/cloud/temporal-stack/vendor/go.temporal.io/server/service/frontend/adminHandler.go:1793
-go.temporal.io/server/service/frontend.(*AdminHandler).AddOrUpdateRemoteCluster
/cloud/temporal-stack/vendor/go.temporal.io/server/service/frontend/adminHandler.go:1002
-go.temporal.io/server/api/adminservice/v1._AdminService_AddOrUpdateRemoteCluster_Handler.func1
/cloud/temporal-stack/vendor/go.temporal.io/server/api/adminservice/v1/service.pb.go:928
-go.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/caller_info.go:57
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1117
-go.temporal.io/server/common/rpc/interceptor.(*SDKVersionInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/sdk_version.go:64
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/authorization.(*interceptor).Interceptor
/cloud/temporal-stack/vendor/go.temporal.io/server/common/authorization/interceptor.go:152
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/rate_limit.go:85
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/namespace_rate_limit.go:90
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/namespace_count_limit.go:99
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/namespace_validator.go:112
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/telemetry.go:135
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1
/cloud/temporal-stack/vendor/go.temporal.io/server/common/metrics/grpc.go:66
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/rpc.ServiceErrorInterceptor
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/grpc.go:132
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/namespace_logger.go:84
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-google.golang.org/grpc.chainUnaryInterceptors.func1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1122
-go.temporal.io/server/api/adminservice/v1._AdminService_AddOrUpdateRemoteCluster_Handler
/cloud/temporal-stack/vendor/go.temporal.io/server/api/adminservice/v1/service.pb.go:930
-google.golang.org/grpc.(*Server).processUnaryRPC
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1283
-google.golang.org/grpc.(*Server).handleStream
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1620
-google.golang.org/grpc.(*Server).serveStreams.func1.2
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:922”} {“level”:“error”,“ts”:“2023-01-19T05:25:25.043Z”,“msg”:“unavailable error”,“operation”:“AddOrUpdateRemoteCluster”,“error”:"last connection error: connection closed before server preface received. Address: {
“Addr”: “host.docker.internal:(remote front end port)”,
“ServerName”: “host.docker.internal:(remote front end port)”,
“Attributes”: null,
“BalancerAttributes”: null,
“Type”: 0,
“Metadata”: null
}“logging-call-at”:“telemetry.go:280”,“stacktrace”:“go.temporal.io/server/common/log.(*zapLogger).Error
/cloud/temporal-stack/vendor/go.temporal.io/server/common/log/zap_logger.go:142
-go.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/telemetry.go:280
-go.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/telemetry.go:144
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1
/cloud/temporal-stack/vendor/go.temporal.io/server/common/metrics/grpc.go:66
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/rpc.ServiceErrorInterceptor
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/grpc.go:132
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-go.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept
/cloud/temporal-stack/vendor/go.temporal.io/server/common/rpc/interceptor/namespace_logger.go:84
-google.golang.org/grpc.chainUnaryInterceptors.func1.1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1120
-google.golang.org/grpc.chainUnaryInterceptors.func1
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1122
-go.temporal.io/server/api/adminservice/v1._AdminService_AddOrUpdateRemoteCluster_Handler
/cloud/temporal-stack/vendor/go.temporal.io/server/api/adminservice/v1/service.pb.go:930
-google.golang.org/grpc.(*Server).processUnaryRPC
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1283
-google.golang.org/grpc.(*Server).handleStream
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:1620
-google.golang.org/grpc.(*Server).serveStreams.func1.2
/cloud/temporal-stack/vendor/google.golang.org/grpc/server.go:922”}