I am using Temporal auto-setup: 1.18.0
PostgreSQL 13
My mtls certificates was expired and I generated new one. But still the error in Temporal Server says Authentication Handshake failed and certificate has expired.
Does the db stores any previous error’s or certificate that is getting trapped in PollWorkflowTaskQueue and getting stream of error’s ?{“level”:“error”,“ts”:“2023-01-20T17:19:15.518Z”,“msg”:“Unable to call matching.PollWorkflowTaskQueue.”,“service”:“frontend”,“wf-task-queue-name”:“temporal-autosetup-5575868868-nmdm8:fdaf7b91-bdd4-4434-96ca-2358fe56fb94”,“timeout”:“53.150265178s”,“error”:“last connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2023-01-20T17:19:12Z is after 2023-01-20T06:10:55Z"”,“logging-call-at”:“workflow_handler.go:894”,“stacktrace”:“go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:143[ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue](http://ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue)\n\t/home/builder/temporal/service/frontend/workflow_handler.go:894[ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2](http://ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2)\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:598[ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect](http://ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect)\n\t/home/builder/temporal/service/frontend/dcRedirectionPolicy.go:125[ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue](http://ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue)\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:594[ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1](http://ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1)\n\t/go/pkg/mod/go.temporal.io/api@v1.12.0/workflowservice/v1/service.pb.go:1516[ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept](http://ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept)\n\t/home/builder/temporal/common/rpc/interceptor/caller_info.go:79[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1135[ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1](http://ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1)\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:62[ngo.temporal.io/server/common/backoff.ThrottleRetryContext](http://ngo.temporal.io/server/common/backoff.ThrottleRetryContext)\n\t/home/builder/temporal/common/backoff/retry.go:194[ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept](http://ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept)\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:66[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1](http://ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1)\n\t/home/builder/temporal/common/metrics/grpc.go:66[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1](http://ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1)\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.32.0/interceptor.go:325[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor](http://ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor)\n\t/home/builder/temporal/common/rpc/grpc.go:137[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept](http://ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept)\n\t/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1140[ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler](http://ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler)\n\t/go/pkg/mod/go.temporal.io/api@v1.12.0/workflowservice/v1/service.pb.go:1518[ngoogle.golang.org/grpc.(*Server).processUnaryRPC](http://ngoogle.golang.org/grpc.(*Server).processUnaryRPC)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1301[ngoogle.golang.org/grpc.(*Server).handleStream](http://ngoogle.golang.org/grpc.(*Server).handleStream)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1642[ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2](http://ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:938”}
{“level”:“error”,“ts”:“2023-01-20T17:19:15.547Z”,“msg”:“Unable to call matching.PollWorkflowTaskQueue.”,“service”:“frontend”,“wf-task-queue-name”:“temporal-autosetup-5575868868-nmdm8:4679799a-2f29-4dda-8b0e-ee6def958986”,“timeout”:“56.848896424s”,“error”:“last connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2023-01-20T17:19:12Z is after 2023-01-20T06:10:55Z"”,“logging-call-at”:“workflow_handler.go:894”,“stacktrace”:“go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:143[ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue](http://ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue)\n\t/home/builder/temporal/service/frontend/workflow_handler.go:894[ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2](http://ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2)\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:598[ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect](http://ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect)\n\t/home/builder/temporal/service/frontend/dcRedirectionPolicy.go:125[ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue](http://ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue)\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:594[ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1](http://ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1)\n\t/go/pkg/mod/go.temporal.io/api@v1.12.0/workflowservice/v1/service.pb.go:1516[ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept](http://ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept)\n\t/home/builder/temporal/common/rpc/interceptor/caller_info.go:79[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1135[ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1](http://ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1)\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:62[ngo.temporal.io/server/common/backoff.ThrottleRetryContext](http://ngo.temporal.io/server/common/backoff.ThrottleRetryContext)\n\t/home/builder/temporal/common/backoff/retry.go:194[ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept](http://ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept)\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:66[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1](http://ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1)\n\t/home/builder/temporal/common/metrics/grpc.go:66[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1](http://ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1)\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.32.0/interceptor.go:325[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor](http://ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor)\n\t/home/builder/temporal/common/rpc/grpc.go:137[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept](http://ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept)\n\t/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1140[ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler](http://ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler)\n\t/go/pkg/mod/go.temporal.io/api@v1.12.0/workflowservice/v1/service.pb.go:1518[ngoogle.golang.org/grpc.(*Server).processUnaryRPC](http://ngoogle.golang.org/grpc.(*Server).processUnaryRPC)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1301[ngoogle.golang.org/grpc.(*Server).handleStream](http://ngoogle.golang.org/grpc.(*Server).handleStream)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1642[ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2](http://ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:938”}
{“level”:“error”,“ts”:“2023-01-20T17:19:15.600Z”,“msg”:“Unable to call matching.PollActivityTaskQueue.”,“service”:“frontend”,“wf-task-queue-name”:“/_sys/temporal-sys-tq-scanner-taskqueue-0/1”,“timeout”:“56.950606832s”,“error”:“last connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2023-01-20T17:19:12Z is after 2023-01-20T06:10:55Z"”,“logging-call-at”:“workflow_handler.go:1118”,“stacktrace”:"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:143[ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollActivityTaskQueue](http://ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollActivityTaskQueue)\n\t/home/builder/temporal/service/frontend/workflow_handler.go:1118[ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue.func2](http://ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue.func2)\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:564[ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect](http://ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect)\n\t/home/builder/temporal/service/frontend/dcRedirectionPolicy.go:125[ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue](http://ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollActivityTaskQueue)\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:560[ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler.func1](http://ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler.func1)\n\t/go/pkg/mod/go.temporal.io/api@v1.12.0/workflowservice/v1/service.pb.go:1570[ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept](http://ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept)\n\t/home/builder/temporal/common/rpc/interceptor/caller_info.go:79[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1135[ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1](http://ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1)\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:62[ngo.temporal.io/server/common/backoff.ThrottleRetryContext](http://ngo.temporal.io/server/common/backoff.ThrottleRetryContext)\n\t/home/builder/temporal/common/backoff/retry.go:194[ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept](http://ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept)\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:66[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1](http://ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1)\n\t/home/builder/temporal/common/metrics/grpc.go:66[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1](http://ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1)\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.32.0/interceptor.go:325[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor](http://ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor)\n\t/home/builder/temporal/common/rpc/grpc.go:137[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept](http://ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept)\n\t/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1138[ngoogle.golang.org/grpc.chainUnaryInterceptors.func1](http://ngoogle.golang.org/grpc.chainUnaryInterceptors.func1)\n\t/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1140[ngo.temporal.io/api/workflows](http://ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollActivityTaskQueue_Handler)
Can you give more info on your setup? Did you update the client certificates on your clients/workers?
Thank you for the reply. Yes I have updated the client certificates on clients and workers. The certificate is fine as it is working in our Dev and Pilot environment but not working in QA environment.
It has become Temporal DB issue, I tried to connect temporal autosetup server from version 1.17.5 to 1.18.5 but all server are giving TLS handshake auth-n error with this QA Temporal Database.
Does DB stores the previous error blocking states in any table if so can anyone point to it ?