Front end errors, temporal server unstable during the error window

Community Support
1

Occassionaly i see this error in my front end server, not sure what it means, can some one help…
I also suspect that the temporal server is unstable when this happens

"level":"info","ts":"2023-03-01T11:46:50.745Z","msg":"Current reachable members","service":"frontend","component":"service-resolver","service":"matching","addresses":["10.x.y.214:7235"],"logging-call-at":"rpServiceResolver.go:283"}
{"level":"info","ts":"2023-03-01T11:46:50.750Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.752Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.752Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.752Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.752Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.752Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.752Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.752Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.752Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
2

i also see this error (some kind of timeout i guess)

{"level":"error","ts":"2023-03-01T11:46:50.751Z","msg":"Unable to call matching.PollWorkflowTaskQueue.","service":"frontend","wf-task-queue-name":"my-temporal-worker-69d66bb5cf-2lrr8:bd39d521-eba2-47f9-8763-5ab4d9dbafbd","timeout":"1m9.99983253s","error":"error reading from server: EOF","logging-call-at":"workflow_handler.go:897","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:144\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue\n\t/home/builder/temporal/service/frontend/workflow_handler.go:897\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:634\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/home/builder/temporal/service/frontend/dcRedirectionPolicy.go:125\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:630\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.13.1-0.20221110200459-6a3cb21a3415/workflowservice/v1/service.pb.go:1516\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:63\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:194\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:67\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1162\ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/caller_info.go:79\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*SDKVersionInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/sdk_version.go:69\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/rate_limit.go:86\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_rate_limit.go:91\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_count_limit.go:100\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).StateValidationIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:132\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/home/builder/temporal/common/authorization/interceptor.go:153\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:142\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.1/interceptor.go:352\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).LengthValidationIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:103\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:137\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1167\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.13.1-0.20221110200459-6a3cb21a3415/workflowservice/v1/service.pb.go:1518\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1340\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1713\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:965"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"info","ts":"2023-03-01T11:46:50.751Z","msg":"matching client encountered error","service":"frontend","error":"error reading from server: EOF","service-error-type":"serviceerror.Unavailable","logging-call-at":"metric_client.go:219"}
{"level":"error","ts":"2023-03-01T11:46:50.752Z","msg":"Unable to call matching.PollWorkflowTaskQueue.","service":"frontend","wf-task-queue-name":"my-temporal-worker-69d66bb5cf-2lrr8:bd39d521-eba2-47f9-8763-5ab4d9dbafbd","timeout":"1m9.999775848s","error":"error reading from server: EOF","logging-call-at":"workflow_handler.go:897","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:144\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue\n\t/home/builder/temporal/service/frontend/workflow_handler.go:897\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue.func2\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:634\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/home/builder/temporal/service/frontend/dcRedirectionPolicy.go:125\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).PollWorkflowTaskQueue\n\t/home/builder/temporal/service/frontend/dcRedirectionHandler.go:630\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.13.1-0.20221110200459-6a3cb21a3415/workflowservice/v1/service.pb.go:1516\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:63\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:194\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:67\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1162\ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/caller_info.go:79\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*SDKVersionInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/sdk_version.go:69\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/rate_limit.go:86\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_rate_limit.go:91\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_count_limit.go:100\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).StateValidationIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:132\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/home/builder/temporal/common/authorization/interceptor.go:153\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:142\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.1/interceptor.go:352\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).LengthValidationIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:103\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:137\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1165\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1167\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.13.1-0.20221110200459-6a3cb21a3415/workflowservice/v1/service.pb.go:1518\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1340\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:1713\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.50.1/server.go:965"}