ReadHistoryBranch. Close operation failed. Error: context canceled

Community Support
1

Hi Temporal community,

I’m troubleshooting intermittent WorkflowTaskTimedOut issues on a self-hosted Temporal 1.29.2 cluster running on Kubernetes.

Environment

  • Temporal Server: 1.29.2

  • Kubernetes

  • Execution store: Cassandra

  • Visibility store: MySQL 8

Initially I suspected Cassandra or MySQL, but after investigation I don’t think they’re the primary issue.

Cassandra

  • 3-node cluster

  • All nodes UN

  • Ring balanced (128 vnodes/node)

  • No streaming or hints

  • Matching service appears healthy with no task queue backlog or dispatch failures.

The error I’m consistently seeing

{
“level”: “error”,
“ts”: “2026-07-25T12:55:23.226Z”,
“msg”: “service failures”,
“operation”: “RecordWorkflowTaskStarted”,
“wf-namespace”: “default”,
“grpc_code”: “Unavailable”,
“wf-id”: “content-production-6be666f8-3995-472c-bf38-29df1447cc89”,
“wf-run-id”: “019f9829-5abc-7ee9-be02-ccbfc5837065”,
“error”: “ReadHistoryBranch. Close operation failed. Error: context canceled”,
“logging-call-at”: “/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:424”,
“stacktrace”: “go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).logError\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:424\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).HandleError\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:400\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:187\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.72.2/server.go:1217\ngo.temporal.io/server/service.GrpcServerOptionsProvider.getUnaryInterceptors.NewServerMetricsTrailerPropagatorInterceptor.func6\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:88\ngoogle.golang.org/grpc.getChainUnaryHandler.func1.getChainUnaryHandler.1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.72.2/server.go:1217\ngo.temporal.io/server/service.GrpcServerOptionsProvider.getUnaryInterceptors.NewServerMetricsContextInjectorInterceptor.func5\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:41\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.72.2/server.go:1217\ngo.temporal.io/server/common/rpc/interceptor.ServiceErrorInterceptor\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/service_error_interceptor.go:19\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.72.2/server.go:1208\ngo.temporal.io/server/api/historyservice/v1._HistoryService_RecordWorkflowTaskStarted_Handler\n\t/home/runner/work/docker-builds/docker-builds/temporal/api/historyservice/v1/service_grpc.pb.go:1639\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.72.2/server.go:1405\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.72.2/server.go:1815\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.72.2/server.go:1035”
}

From the History service:

operation: RecordWorkflowTaskStarted
grpc_code: Unavailable

error:
ReadHistoryBranch.
Close operation failed.
Error: context canceled

The stack trace originates from:

HistoryService_RecordWorkflowTaskStarted

The affected workflows eventually fail with WorkflowTaskTimedOut.

The error appears during RecordWorkflowTaskStarted, while reading workflow history.

Questions

  1. Does ReadHistoryBranch. Close operation failed. Error: context canceled indicate that the history read itself is being cancelled, or is it usually a symptom of another upstream timeout?

  2. Is this typically caused by:

    • Cassandra read latency?

    • History shard contention?

    • gRPC deadline expiration?

    • Workflow history becoming too large?

    • Something else?

  3. Are there specific History metrics or logs I should inspect next? For example:

    • Persistence latency

    • ReadHistoryBranch latency

    • Shard acquisition latency

    • Mutable state latency

    • gRPC timeout metrics

  4. Has anyone encountered this error in Temporal 1.29.x, and if so, what was the root cause?

Any guidance would be greatly appreciated. I’m trying to identify why RecordWorkflowTaskStarted is failing before focusing on the application workflows themselves.

Thanks!