Temporal Application is not establishing connection to Temporal Server inside Kubernetes

Community Support
,
1

With tls-simple(samples-server/tls/tls-simple at main · temporalio/samples-server · GitHub) sample(mTLS layer) connection between temporal application and worker is not establishing inside kubernetes:

It is working with local dockerized setup of temporal application and temporal server.
temporal-application(Seperate pod) is not able to make secure network communication to temporal-auto-setup(Seperate pod) inside kubernetes.
Error:
[TransportError: transport error]
To Reproduce:
I am using
1.Temporal-Application - samples-typescript/hello-world-mtls at main · temporalio/samples-typescript · GitHub.
2.Temporal-Server(image - temporalio/auto-setup:1.13.1) - Generate test certificates with generate-test-certs.sh and Start Temporal with start-temporal.sh.
3.temporal-auto-setup container(image - temporalio/auto-setup:1.13.1) is deployed in separate pod and successful connection is made with Postgres db.
4.Environment Variables for temporal-auto-setup deployment:
containers:

  • name: temporal-auto-setup
    image: temporalio/auto-setup:1.13.1
    env:
  • name: AUTO_SETUP
    value: “true”
  • name: SKIP_DEFAULT_NAMESPACE_CREATION
    value: “true”
  • name: TEMPORAL_TLS_SERVER_CA_CERT
    value: /path-to/ca.cert
  • name: TEMPORAL_TLS_SERVER_CERT
    value: /path-to/cluster.pem
  • name: TEMPORAL_TLS_SERVER_KEY
    value: /path-to/cluster.key
  • name: TEMPORAL_TLS_REQUIRE_CLIENT_AUTH
    value: “true”
  • name: TEMPORAL_TLS_FRONTEND_CERT
    value: /path-to/cluster.pem
  • name: TEMPORAL_TLS_FRONTEND_KEY
    value: /path-to/cluster.key
  • name: TEMPORAL_TLS_CLIENT1_CA_CERT
    value: /path-to/ca.cert
  • name: TEMPORAL_TLS_CLIENT2_CA_CERT
    value: /path-to/ca.cert
  • name: TEMPORAL_TLS_INTERNODE_SERVER_NAME
    value: tls-sample
  • name: TEMPORAL_TLS_FRONTEND_SERVER_NAME
    value: tls-sample
  • name: TEMPORAL_TLS_FRONTEND_DISABLE_HOST_VERIFICATION
    value: “false”
  • name: TEMPORAL_TLS_INTERNODE_DISABLE_HOST_VERIFICATION
    value: “false”
  • name: TEMPORAL_CLI_ADDRESS
    value: temporal-auto-setup.default.svc.cluster.local:7233
  • name: TEMPORAL_GRPC_ENDPOINT
    value: temporal-auto-setup.default.svc.cluster.local:7233
  • name: TEMPORAL_CLI_TLS_CA
    value: /path-to/ca.cert
  • name: TEMPORAL_CLI_TLS_CERT
    value: /path-to/cluster.pem
  • name: TEMPORAL_CLI_TLS_KEY
    value: /path-to/cluster.key
  • name: TEMPORAL_CLI_TLS_ENABLE_HOST_VERIFICATION
    value: “true”
  • name: TEMPORAL_CLI_TLS_SERVER_NAME
    value: tls-sample
  • name: TEMPORAL_TLS_SERVER_NAME
    value: tls-sample
  • name: TEMPORAL_TLS_CERT_PATH
    value: /path-to/cluster.pem
  • name: TEMPORAL_TLS_KEY_PATH
    value: /path-to/cluster.key
  • name: TEMPORAL_TLS_CA_PATH
    value: /path-to/ca.cert
  • name: TEMPORAL_TLS_ENABLE_HOST_VERIFICATION
    value: “true”
  • name: SKIP_SCHEMA_SETUP
    value: “false”
  • name: DB
    value: postgresql
  • name: DB_PORT
    value: “5432”
  • name: POSTGRES_SEEDS
    value: temporal.postgres.svc.cluster.local
  • name: POSTGRES_USER
    value: xxx
  • name: POSTGRES_PWD
    value: xxx
  • name: TEMPORAL_STORE_PASSWORD
    value: xxx
  • name: TEMPORAL_VISIBILITY_STORE_PASSWORD
    value: xxx
  • name: DYNAMIC_CONFIG_FILE_PATH
    value: “path-to/development.yaml”
  • name: SQL_TLS
    value: ‘true’
  • name: SQL_TLS_DISABLE_HOST_VERIFICATION
    value: ‘true’
  • name: SQL_TLS_ENABLED
    value: ‘true’
  • name: SQL_HOST_VERIFICATION
    value: ‘false’
    5.Environment Variables for temporal-application deployment.
    containers:
  • name: temporal-application
    image: localhost:5000/temporal-application:dev
    imagePullPolicy: Always
    env:
  • name: TEMPORAL_SERVER_ROOT_CA_CERT_PATH
    value: /path-to/ca.cert
  • name: TEMPORAL_CLIENT_CERT_PATH
    value: /path-to/client.pem
  • name: TEMPORAL_CLIENT_KEY_PATH
    value: /path-to/client.key
  • name: TEMPORAL_NAMESPACE
    value: default
  • name: TEMPORAL_ADDRESS
    value: temporal-auto-setup.default.svc.cluster.local
  • name: TEMPORAL_SERVER_NAME_OVERRIDE
    value: tls-sample
  • name: TEMPORAL_GRPC_ENDPOINT
    value: temporal-auto-setup.default.svc.cluster.local:7233
    Kubernetes deployment made through kustomize method not helm charts.
    Any help would be appreciated. (edited)
2

Hi Ajay,

Could you clarify your setup. Are you deploying two pods: one for the node.js app and one for Temporal Server?

You mentioned

2.Temporal-Server(image - temporalio/auto-setup:1.13.1) - Generate test certificates with generate-test-certs.sh and Start Temporal with start-temporal.sh

Do you use the generated client certificate and server CA to configure the node app? Can you share how you do that?

3

Hi SergeyBykov,

Sorry for the delayed response, I am new to this mTLS certificates and kubernetes networking.
Thanks for your reply.

1.Yes I am deploying two pods, one for nodejs app and one for temporal.

2.I have generated client and server certs using cmd: bash generate-test-certs.sh from this repo [https://github.com/temporalio/samples-server/tree/main/tls/tls-simple]

This is my configuration for certs generation.
cluster-cert.conf

default_bits = 4096
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[dn]
C = US
ST = WA
O = Test Cluster, Inc.
CN = temporal-auto-setup.default.svc.cluster.local
[req_ext]
subjectAltName = @alt_names
[alt_names]
DNS.1 = temporal-auto-setup.default.svc.cluster.local
IP.1 = ::1

client-cert.conf is

[req]
default_bits = 4096
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[dn]
C = US
ST = WA
O = Test Client, Inc.
CN = temporal-application.default.svc.cluster.local
[req_ext]
subjectAltName = @alt_names
[alt_names]
DNS.1 = temporal-application.default.svc.cluster.local
IP.1 = ::1
IP.2 = 127.0.0.1

I am giving env var- name: TEMPORAL_ADDRESS
value: temporal-auto-setup.default.svc.cluster.local:7233
for temporal worker application pod(nodejs).

4

Are you able to connect to the server with tctl by configuring its TLS flags?
That is the next step to take to check that the server is configured correctly before moving to the node.js configuration.

5

if I use certs from samples-server/tls/tls-simple at main · temporalio/samples-server · GitHub
temporal server starts but tctl cmd throws error

tctl --tls_ca_path=/settings/ca.cert --tls_cert_path=/settings/cluster.pem --tls_key_path=/settings/cluster.key --tls_disable_host_verification=true --tls_server_name=tls-sample cluster get-search-attributes
Error is:
Error: Unable to get search attributes.
Error Details: context deadline exceeded
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
go.temporal.io/server/tools/cli.printError({0x1dbfb3b, 0x20}, {0x20b6580, 0xc000612210})
        /temporal/tools/cli/util.go:392 +0x22a
go.temporal.io/server/tools/cli.ErrorAndExit({0x1dbfb3b, 0x20e9958}, {0x20b6580, 0xc000612210})
        /temporal/tools/cli/util.go:403 +0x28
go.temporal.io/server/tools/cli.GetSearchAttributes(0x3066fa0)
        /temporal/tools/cli/cluster_search_attributes_commands.go:39 +0xa5
go.temporal.io/server/tools/cli.newClusterCommands.func2(0xc000445340)
        /temporal/tools/cli/cluster.go:44 +0x19

Temporal server logs are:

{"level":"error","ts":"2022-01-09T09:01:26.724Z","msg":"Unable to call matching.PollWorkflowTaskQueue.","service":"frontend","wf-task-queue-name":"temporal-autosetup-ff56f7f4f-gf9jb:65ea61e2-4669-41c1-84aa-ec62824948b0","timeout":"1m9.999291679s","error":"context deadline exceeded","logging-call-at":"workflowHandler.go:812","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue\n\t/temporal/service/frontend/workflowHandler.go:812\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:532\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:118\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue\n\t/temporal/service/frontend/dcRedirectionHandler.go:528\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1088\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1113\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:89\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:87\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1118\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1090\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923"}
{"level":"error","ts":"2022-01-09T09:01:27.023Z","msg":"Error refreshing namespace cache","service":"frontend","error":"GetMetadata operation failed. Error: context deadline exceeded","logging-call-at":"registry.go:364","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/common/namespace.(*registry).refreshLoop\n\t/temporal/common/namespace/registry.go:364\ngo.temporal.io/server/internal/goro.Go.func1\n\t/temporal/internal/goro/goro.go:56"}
{"level":"error","ts":"2022-01-09T09:01:26.773Z","msg":"Unable to call matching.PollActivityTaskQueue.","service":"frontend","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","timeout":"1m9.929973333s","error":"context deadline exceeded","logging-call-at":"workflowHandler.go:1078","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollActivityTaskQueue\n\t/temporal/service/frontend/workflowHandler.go:1078\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:502\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:118\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue\n\t/temporal/service/frontend/dcRedirectionHandler.go:498\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1142\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1113\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:89\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:87\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1118\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1144\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923"}
{"level":"error","ts":"2022-01-09T09:01:26.800Z","msg":"Unable to call matching.PollWorkflowTaskQueue.","service":"frontend","wf-task-queue-name":"temporal-autosetup-ff56f7f4f-gf9jb:9ddc3872-7ad1-4cf6-9f85-d5e39f983882","timeout":"1m9.999657247s","error":"context deadline exceeded","logging-call-at":"workflowHandler.go:812","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue\n\t/temporal/service/frontend/workflowHandler.go:812\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:532\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:118\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue\n\t/temporal/service/frontend/dcRedirectionHandler.go:528\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1088\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1113\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:89\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:87\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1118\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1090\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923"}
{"level":"error","ts":"2022-01-09T09:01:26.848Z","msg":"Unable to call matching.PollWorkflowTaskQueue.","service":"frontend","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/3","timeout":"1m9.686047283s","error":"context deadline exceeded","logging-call-at":"workflowHandler.go:812","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue\n\t/temporal/service/frontend/workflowHandler.go:812\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:532\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:118\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue\n\t/temporal/service/frontend/dcRedirectionHandler.go:528\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1088\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1113\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:89\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:87\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1118\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1090\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923"}
{"level":"error","ts":"2022-01-09T09:01:26.865Z","msg":"Unable to call matching.PollActivityTaskQueue.","service":"frontend","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/2","timeout":"1m9.982422099s","error":"context deadline exceeded","logging-call-at":"workflowHandler.go:1078","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollActivityTaskQueue\n\t/temporal/service/frontend/workflowHandler.go:1078\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:502\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:118\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue\n\t/temporal/service/frontend/dcRedirectionHandler.go:498\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1142\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1113\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:89\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:87\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1118\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1144\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923"}
{"level":"error","ts":"2022-01-09T09:01:26.888Z","msg":"Unable to call matching.PollActivityTaskQueue.","service":"frontend","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/2","timeout":"1m9.918400774s","error":"context deadline exceeded","logging-call-at":"workflowHandler.go:1078","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollActivityTaskQueue\n\t/temporal/service/frontend/workflowHandler.go:1078\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:502\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:118\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue\n\t/temporal/service/frontend/dcRedirectionHandler.go:498\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1142\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1113\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:89\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:87\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1118\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1144\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923"}
{"level":"error","ts":"2022-01-09T09:01:27.091Z","msg":"Unable to call matching.PollWorkflowTaskQueue.","service":"frontend","wf-task-queue-name":"temporal-autosetup-ff56f7f4f-gf9jb:65ea61e2-4669-41c1-84aa-ec62824948b0","timeout":"1m9.990897371s","error":"context deadline exceeded","logging-call-at":"workflowHandler.go:812","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue\n\t/temporal/service/frontend/workflowHandler.go:812\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2\n\t/temporal/service/frontend/dcRedirectionHandler.go:532\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/temporal/service/frontend/dcRedirectionPolicy.go:118\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue\n\t/temporal/service/frontend/dcRedirectionHandler.go:528\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1088\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1113\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:89\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:87\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1116\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1118\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.5.1-0.20211018190919-a5f4a169cd08/workflowservice/v1/service.pb.go:1090\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923"}
{"level":"info","ts":"2022-01-09T09:02:27.346Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/3","wf-task-queue-type":"Workflow","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
{"level":"info","ts":"2022-01-09T09:02:27.346Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/3","wf-task-queue-type":"Workflow","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
{"level":"info","ts":"2022-01-09T09:03:26.288Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/1","wf-task-queue-type":"Activity","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
{"level":"info","ts":"2022-01-09T09:03:26.289Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/1","wf-task-queue-type":"Activity","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
6

if I use certs from samples-server/tls/tls-full at main · temporalio/samples-server · GitHub

Error log from pod : 
+ tctl --auto_confirm admin cluster add-search-attributes --name CustomKeywordField --type Keyword --name CustomStringField --type Text --name CustomTextField --type Text --name CustomIntField --type Int --name CustomDatetimeField --type Datetime --name CustomDoubleField --type Double --name CustomBoolField --type Bool
{"level":"error","ts":"2022-01-09T08:47:59.731Z","msg":"getSearchAttributes error","service":"frontend","error":"unable to get temporal-sys-add-search-attributes-workflow workflow state: context deadline exceeded","logging-call-at":"adminHandler.go:378","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/frontend.(*AdminHandler).getSearchAttributes\n\t/temporal/service/frontend/adminHandler.go:378\ngo.temporal.io/server/service/frontend.(*AdminHandler).GetSearchAttributes\n\t/temporal/service/frontend/adminHandler.go:362\ngo.temporal.io/server/api/adminservice/v1._AdminService_GetSearchAttributes_Handler.func1\n\t/temporal/api/adminservice/v1/service.pb.go:878\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:98\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:88\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_validator.go:113\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1121\ngo.temporal.io/server/api/adminservice/v1._AdminService_GetSearchAttributes_Handler\n\t/temporal/api/adminservice/v1/service.pb.go:880\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921"}
{"level":"error","ts":"2022-01-09T08:47:59.909Z","msg":"unavailable error","service":"frontend","error":"unable to get temporal-sys-add-search-attributes-workflow workflow state: context deadline exceeded","logging-call-at":"adminHandler.go:1670","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/service/frontend.(*AdminHandler).error\n\t/temporal/service/frontend/adminHandler.go:1670\ngo.temporal.io/server/service/frontend.(*AdminHandler).GetSearchAttributes\n\t/temporal/service/frontend/adminHandler.go:364\ngo.temporal.io/server/api/adminservice/v1._AdminService_GetSearchAttributes_Handler.func1\n\t/temporal/api/adminservice/v1/service.pb.go:878\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/temporal/common/authorization/interceptor.go:152\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1116\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_count_limit.go:98\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_rate_limit.go:88\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/rate_limit.go:83\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_validator.go:113\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:108\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1121\ngo.temporal.io/server/api/adminservice/v1._AdminService_GetSearchAttributes_Handler\n\t/temporal/api/adminservice/v1/service.pb.go:880\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921"}
{"level":"error","ts":"2022-01-09T08:47:59.940Z","msg":"unavailable error","operation":"GetSearchAttributes","error":"unable to get temporal-sys-add-search-attributes-workflow workflow state: context deadline exceeded","logging-call-at":"telemetry.go:187","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/temporal/common/rpc/interceptor/telemetry.go:187\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/telemetry.go:117\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/temporal/common/rpc/grpc.go:131\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1119\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1121\ngo.temporal.io/server/api/adminservice/v1._AdminService_GetSearchAttributes_Handler\n\t/temporal/api/adminservice/v1/service.pb.go:880\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921"}
Error: Unable to get existing search attributes.
Error Details: rpc error: code = Unavailable desc = unable to get temporal-sys-add-search-attributes-workflow workflow state: context deadline exceeded

I am using following configurations for temporal-autosetup in GKE.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: temporal-autosetup
  labels:
    app: temporal-autosetup
    # version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: temporal-autosetup
      # version: v1
  template:
    metadata:
      labels:
        app: temporal-autosetup
        # version: v1
    spec:
      
      containers:
        - name: temporal-autosetup
          image: temporalio/auto-setup:1.14.1
          # image: asia.gcr.io/wellostack/auth:latest
          # imagePullPolicy: IfNotPresent
          env:
            - name: AUTO_SETUP
              value: "true"   
            - name: SKIP_DEFAULT_NAMESPACE_CREATION
              value: "true"  
            # - name: TEMPORAL_TLS_SERVER_CA_CERT
            #   value: /settings/ca.cert
            # - name: TEMPORAL_TLS_SERVER_CERT
            #   value: /settings/cluster.pem
            # - name: TEMPORAL_TLS_SERVER_KEY
            #   value: /settings/cluster.key  
            - name: TEMPORAL_TLS_REQUIRE_CLIENT_AUTH
              value: "true"
            - name: TEMPORAL_TLS_FRONTEND_CERT
              value: /settings/cluster-internode.pem
            - name: TEMPORAL_TLS_FRONTEND_KEY
              value: /settings/cluster-internode.key
            # - name: TEMPORAL_TLS_FRONTEND_CERT_DATA  
            #   value: /settings/cluster-internode.pem
            # - name: TEMPORAL_TLS_FRONTEND_KEY_DATA
            #   value: /settings/cluster-internode.key  
            # - name: TEMPORAL_TLS_CLIENT1_CA_CERT
            #   value: /settings/ca.cert
            # - name: TEMPORAL_TLS_CLIENT2_CA_CERT
            #   value: /settings/ca.cert
            # - name: TEMPORAL_TLS_CLIENT1_CA_CERT_DATA
            #   value: /settings/ca.key
            # - name: TEMPORAL_TLS_CLIENT2_CA_CERT_DATA
            #   value: /settings/ca.key  
            - name: TEMPORAL_TLS_INTERNODE_SERVER_NAME
              value: internode.cluster-x.contoso.com
            - name: TEMPORAL_TLS_FRONTEND_SERVER_NAME
              value: internode.cluster-x.contoso.com
            - name: TEMPORAL_TLS_FRONTEND_DISABLE_HOST_VERIFICATION
              value: "false"
            - name: TEMPORAL_TLS_INTERNODE_DISABLE_HOST_VERIFICATION
              value: "false"
            - name: TEMPORAL_CLI_ADDRESS
              value: temporal-autosetup.default.svc.cluster.local:7233
            - name: TEMPORAL_GRPC_ENDPOINT
              value: temporal-autosetup.default.svc.cluster.local:7233
            - name: PUBLIC_FRONTEND_ADDRESS
              value: temporal-autosetup.default.svc.cluster.local:7233  
            # - name: TEMPORAL_CLI_TLS_CA
            #   value: /settings/ca.cert            
            # - name: TEMPORAL_CLI_TLS_CERT
            #   value: /settings/cluster.pem
            # - name: TEMPORAL_CLI_TLS_KEY
            #   value: /settings/cluster.key
            # - name: TEMPORAL_CLI_TLS_ENABLE_HOST_VERIFICATION
            #   value: "true"
            # - name: TEMPORAL_CLI_TLS_SERVER_NAME
            #   value: temporal-autosetup.temporal.svc.cluster.local
            - name: TEMPORAL_TLS_SERVER_NAME
              value: internode.cluster-x.contoso.com  
            - name: TEMPORAL_TLS_CERT_PATH
              value: /settings/cluster-internode.pem
            - name: TEMPORAL_TLS_KEY_PATH
              value: /settings/cluster-internode.key
            - name: TEMPORAL_TLS_CA_PATH
              value: /settings/server-intermediate-ca.pem
            - name: TEMPORAL_CLI_TLS_CA
              value: /settings/server-intermediate-ca.pem
            - name: TEMPORAL_CLI_TLS_CERT
              value: /settings/cluster-internode.pem
            - name: TEMPORAL_CLI_TLS_KEY
              value: /settings/cluster-internode.key
            - name: TEMPORAL_CLI_TLS_ENABLE_HOST_VERIFICATION
              value: "true"
            - name: TEMPORAL_CLI_TLS_SERVER_NAME
              value: internode.cluster-x.contoso.com
            - name: TEMPORAL_TLS_ENABLE_HOST_VERIFICATION
              value: "true"  
            # - name: DB
            #   value: CASSANDRA
            - name: env
              value: kubernetes
            - name: SKIP_SCHEMA_SETUP
              value: "false"  
            # - name: DB_PORT
            #   value: "9042"
            # - name: DB_HOST
            #   value: temporaltest1-cassandra.default.svc.cluster.local  
            # - name: CASSANDRA_USER  
            #   value: user
            # - name: CASSANDRA_SEEDS
            #   value: temporaltest1-cassandra.default.svc.cluster.local  
            - name: DB
              value: postgresql
            - name: DB_PORT
              value: "5432"
            - name: POSTGRES_SEEDS
              value: xxx
            # - name: DBNAME
            #   value: temporal  
            - name: POSTGRES_USER
              value: postgres  
            - name: POSTGRES_PWD
              value: xxx
            # - name: CASSANDRA_PASSWORD
            #   value: password
            - name: TEMPORAL_STORE_PASSWORD
              value: xxx
            - name: TEMPORAL_VISIBILITY_STORE_PASSWORD
              value: xxx
            - name: DYNAMIC_CONFIG_FILE_PATH
              value: "/settings/development.yaml"  
            - name: SQL_TLS
              value: 'true'
            - name: SQL_TLS_DISABLE_HOST_VERIFICATION
              value: 'true'  
            - name: SQL_TLS_ENABLED
              value: 'true' 
            - name: SQL_HOST_VERIFICATION
              value: 'false'   
            # - name: CASSANDRA_TLS_ENABLED
            #   value: 'true'
            # - name: CASSANDRA_HOST_VERIFICATION
            #   value: 'false'     
            # - name: WAIT_FOR_IT
            #   value: mongodb-pg-0.mongodb-pg-headless.mongodb.svc.cluster.local:27017
          ports:
            - containerPort: 7233
          volumeMounts:
            - name: settings-volume
              mountPath: /settings
              readOnly: true
            - name: temporal-autosetup-logs
              mountPath: /server/data/logs

I think there is some issue with my configurations.
Is there documentation for deployment of helm charts by editing chart values to accomodate mTLS certs ? or is there a repo where there is already edited chart values for mTLS certs ?

7

Hi all,

Note: found the (trivial) reason myself. See at the end of this item. Sorry for bothering. However, I’d like to keep the text as is, b/c someone else might find similar symptoms.

I’m also getting these error logs with “context deadline exceeded” after 70s at PollWorkflowTaskQueue and PollActivityTaskQueue in tag v1.16.1 of temporal, and I’m completely lost on what wait operation causes the 70s timeout.

{"level":"error","ts":"2022-05-02T20:04:30.602+0200","msg":"Unable to call matching.PollWorkflowTaskQueue.","service":"frontend","wf-task-queue-name":"LPOD:22bb3bbe-f6a3-4fe9-8a67-835b2898380c","timeout":"1m9.997997s","error":"context deadline exceeded","logging-call-at":"workflowHandler.go:921", ...

intermixed with these, bursted and probably only varying over all available “wf-task-queue-name”-s:

{"level":"error","ts":"2022-05-02T20:04:30.602+0200","msg":"Unable to call matching.PollActivityTaskQueue.","service":"frontend","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/3","timeout":"1m10s","error":"context deadline exceeded","logging-call-at":"workflowHandler.go:1167", ...

Background

The persistence store was virgin, I added the namespace “default”, then successfully started and finished exactly one workflow and then the temporal server went idle for multiple hours. Spontaneously, i.e. no worker processes were running, these errors were logged, so I suspect some housekeeping work of the service itself is the trigger for inspecting the task queues.

Configuration is development.yaml with logging level changed to “error” and I’m running temporal with my own SQL Server driver. The stack trace is not referring to my code-base directly, but of course it can be a DB operation causing the timeout in a statement right before the logged stack trace. Which one?

So what’s my issue here?

I’d like to debug the error and check if it’s connected with my driver code, so I want to reproduce it without waiting several hours for the housekeeping processes to trigger the faulty code. Is there a way to speed up these internal processes via configuration, effectively triggering the error situation more often? What SQL table is relevant for PollWorkflowTaskQueue and is it a DB query or a DB mutation that times out here?

Thanks for your attention. KR, Martin.

Reason was: the Windows dev machine went to sleep mode and was woken up at the exact instant of the error. Clearly a timeout was detected. So nothing to worry about. This was logged into the system log on on wake-up:

The system has resumed from sleep.
--- then ---
The system time has changed to ‎2022‎-‎05‎-‎02T18:04:30.500000000Z from ‎2022‎-‎05‎-‎02T15:32:40.564565800Z.

Change Reason: System time synchronized with the hardware clock.
Process: '' (PID 4).