Hi Temporal Team,
We have Temporal with Istio deployed currently in our server, but we are seeing an intermittent issue that sometimes the Parent Workflow called “SignupWorkflow” is unable to start a Child Workflow called “EnvCreationWorkflow”.
Parent workflow keep running for few days but child workflow is never started.
Upon checking temporal logs for that workflow run id we see some connection reset error as below.
I can see in the temporal-history log that unexpected-error-attempts is increasing for the same workflow runId most probably because temporal is trying to retry and below connection reset error is continuously coming on every attempt.
How can we solve this issue?
temporal-history log:
{
“level”: “warn”,
“ts”: “2025-09-01T05:42:23.394Z”,
“msg”: “Fail to process task”,
“shard-id”: 241,
“address”: “172.21.9.39:7234”,
“component”: “transfer-queue-processor”,
“wf-namespace-id”: “4af06cc5-331f-41ab-9477-d38aba99364b”,
“wf-id”: “dev2705102-signup”,
“wf-run-id”: “019903cc-4292-7ad1-9337-59ebfadc983f”,
“queue-task-key”: {
“FireTime”: “1970-01-01T00:00:00Z”,
“TaskID”: 19922993
},
“queue-task-type”: “TransferStartChildExecution”,
“queue-task”: “4af06cc5-331f-41ab-9477-d38aba99364b/dev2705102-signup/019903cc-4292-7ad1-9337-59ebfadc983f”,
“wf-history-event-id”: 17,
“error”: “connection error: desc = \“error reading server preface: read tcp 172.21.9.39:43864->172.21.15.41:7234: read: connection reset by peer\””,
“error-type”: “serviceerror.Unavailable”,
“unexpected-error-attempts”: 1,
“lifecycle”: “ProcessingFailed”,
“logging-call-at”: “/home/runner/work/docker-builds/docker-builds/temporal/common/log/lazy_logger.go:63”
}
For the same workflow run id the temporal frontend log is showing below error:
temporal-frontend log:
{
“level”: “error”,
“ts”: “2025-09-01T05:42:22.432Z”,
“msg”: “service failures”,
“operation”: “RespondWorkflowTaskCompleted”,
“wf-namespace”: “platform-preprod”,
“grpc_code”: “Unavailable”,
“wf-id”: “dev2705102-signup”,
“wf-run-id”: “019903cc-4292-7ad1-9337-59ebfadc983f”,
“error”: “connection error: desc = “error reading server preface: EOF””,
“logging-call-at”: “/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:423”,
“stacktrace”: “go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:154\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).logErrors\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:423\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).HandleError\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:393\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:200\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/common/rpc/interceptor.(*Redirection).handleRedirectAPIInvocation.func2\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/redirection.go:265\ngo.temporal.io/server/common/rpc/interceptor.(*NoopRedirectionPolicy).WithNamespaceRedirect\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/dc_redirection_policy.go:125\ngo.temporal.io/server/common/rpc/interceptor.(*Redirection).handleRedirectAPIInvocation\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/redirection.go:262\ngo.temporal.io/server/common/rpc/interceptor.(*Redirection).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/redirection.go:222\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/common/authorization.(*Interceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/authorization/interceptor.go:178\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServerMetricsContextInjectorInterceptor.func2\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:65\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_validator.go:132\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewFrontendServiceErrorInterceptor.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/grpc.go:178\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/grpc.go:157\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/common/rpc/interceptor.(*MaskInternalErrorDetailsInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/mask_internal_error.go:65\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1208\ngo.temporal.io/api/workflowservice/v1._WorkflowService_RespondWorkflowTaskCompleted_Handler\n\t/home/runner/go/pkg/mod/go.temporal.io/api@v1.46.0/workflowservice/v1/service_grpc.pb.go:2495\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1405\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1815\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1035”
}