We were happily running at 1.27.2, few security fixes (CVEs)forced at to upgrade to 1.29.2
after we rolled out 1.29.2 ( we did 1.27.2 to 1.28.x to 1.29.2) helll broke loose, i see repeated errors in most pods (fort end ,matching etc)
the error looks like this
{“level”:“info”,“ts”:“2026-02-02T08:57:40.444Z”,“msg”:“matching client encountered error”,“service”:“frontend”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197”}
{“level”:“info”,“ts”:“2026-02-02T08:57:40.507Z”,“msg”:“matching client encountered error”,“service”:“frontend”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197”}
{“level”:“info”,“ts”:“2026-02-02T08:57:40.585Z”,“msg”:“matching client encountered error”,“service”:“frontend”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197”}
{“level”:“info”,“ts”:“2026-02-02T08:57:40.615Z”,“msg”:“matching client encountered error”,“service”:“frontend”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197”}
{“level”:“info”,“ts”:“2026-02-02T08:57:40.622Z”,“msg”:“matching client encountered error”,“service”:“frontend”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197”}
{“level”:“info”,“ts”:“2026-02-02T08:57:40.698Z”,“msg”:“matching client encountered error”,“service”:“frontend”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197”}
{“level”:“info”,“ts”:“2026-02-02T08:57:40.724Z”,“msg”:“matching client encountered error”,“service”:“frontend”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197”}
{“level”:“info”,“ts”:“2026-02-02T08:57:40.772Z”,“msg”:“matching client encountered error”,“service”:“frontend”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197”}
{“level”:“error”,“ts”:“2026-02-02T08:57:40.786Z”,“msg”:“Unable to call matching.PollWorkflowTaskQueue.”,“service”:“frontend”,“wf-task-queue-name”:“CLEANUP_Q”,“timeout”:“10.21421054s”,“error”:“connection error: desc = "error reading server preface: EOF"”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/service/frontend/workflow_handler.go:946”,“stacktrace”:“``go.temporal.io/server/common/log.(*zapLogger).Error``\n\t/go/src/github.com/temporalio/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).PollWorkflowTaskQueue\n\t/go/src/github.com/temporalio/temporal/service/frontend/workflow_handler.go:946\ngo.temporal.io/api/workflowservice/v1._WorkflowService_PollWorkflowTaskQueue_Handler.func1\n\t/opt/app-root/src/go/pkg/mod/go.temporal.io/api@v1.53.0/workflowservice/v1/service_grpc.pb.go:2699\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1\n\t/go/src/github.com/temporalio/temporal/common/rpc/interceptor/retry.go:38\ngo.tem…
{“level”:“error”,“ts”:“2026-02-02T08:57:40.786Z”,“msg”:“service failures”,“operation”:“PollWorkflowTaskQueue”,“wf-namespace”:“platform-test”,“grpc_code”:“Unavailable”,“error”:“connection error: desc = "error reading server preface: EOF"”,“logging-call-at”:”/go/src/github.com/temporalio/temporal/common/rpc/interceptor/telemetry.go:424",“stacktrace”:“``go.temporal.io/server/common/log.(*zapLogger).Error``\n\t/go/src/github.com/temporalio/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).logError\n\t/go/src/github.com/temporalio/temporal/common/rpc/interceptor/telemetry.go:424\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).HandleError\n\t/go/src/github.com/temporalio/temporal/common/rpc/interceptor/telemetry.go:400\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/go/src/github.com/temporalio/temporal/common/rpc/interceptor/telemetry.go:187\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n...
{“level”:“info”,“ts”:“2026-02-02T08:57:40.880Z”,“msg”:“matching client encountered error”,“service”:“frontend”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:”/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197"}
{“level”:“info”,“ts”:“2026-02-02T08:57:40.882Z”,“msg”:“matching client encountered error”,“service”:“matching”,“error”:“connection error: desc = "error reading server preface: EOF"”,“service-error-type”:“serviceerror.Unavailable”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/client/matching/metric_client.go:197”}
{“level”:“error”,“ts”:“2026-02-02T08:57:40.882Z”,“msg”:“error fetching user data from parent”,“component”:“matching-engine”,“wf-task-queue-name”:“/_sys/temporal-sys-tq-scanner-taskqueue-0/3”,“wf-task-queue-type”:“Activity”,“wf-namespace”:“temporal-system”,“error”:“connection error: desc = "error reading server preface: EOF"”,“logging-call-at”:“/go/src/github.com/temporalio/temporal/service/matching/user_data_manager.go:292”,“stacktrace”:"``go.temporal.io/server/common/log.(*zapLogger).Error``\n\t/go/src/github.com/temporalio/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/service/matching.(*userDataManagerImpl).fetchUserData.func1\n\t/go/src/github.com/temporalio/temporal/service/matching/user_data_manager.go:292\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/go/src/github.com/temporalio/temporal/common/backoff/retry.go:65\ngo.temporal.io/server/service/matching.(*userDataManagerImpl).fetchUserData\n\t/go/src/github.com/temporalio/temporal/service/matching/user_data_manager.go:32…
the error made us suspect network issues first but its not network issue/transport issue.
we rolled back to 1.27.2 and things worked fine,
then we upgraded to 1.28.x that too seems stable
so something is breaking in 1.29.x
we do not use worker versioning etc so i dont think that should be the issue
our SRE tried disabling nexus endpoints (we use grpc/http) and that too did not have any effect.
We are running out of ideas, can some one help?