Cross-region MySQL Deployment - context deadline exceeded errors in history pod

Community Support Server Deployment
, , , ,
1

Hi temporal team,

We are experiencing persistent “context deadline exceeded” errors in our Temporal deployment, specifically in the temporal-history pod. Here’s our setup:

  • Temporal v 1.21.3 on Kubernetes cluster in India
  • MySQL 8.x database deployed in Israel
  • 512 history shards

Problem: 3 system workflows have been stuck in loops with these errors:

  1. temporal-sys-history-scanner - 63% failures on createTransferTasks, 29% failures on updateExecution, 6% of failures on lockNextEventID

  2. temporal-sys-scheduler - 44% failures on createTimerTasks(this is for one of the Temporal schedule jobs we made)

  3. temporal-sys-tq-scanner - 48% failures on createTransferTasks

All errors show ‘context deadline exceeded’ after 60 seconds

Attempted fixes:

  • Increased timeouts from 15s→30s→ 60s - this helped startup of all the 512 shards (since we were first seeing similar ‘context deadline exceeded’ errors during startup), but not during execution of workflows. this is the dynamic configmap: 
    • apiVersion: v1
data:
  dynamic_config.yaml: |
    # Increase Persistence Timeouts (Critical for slow DBs)
    system.persistence.execution.timeout:
      - value: "60s"
    system.persistence.default.timeout:
      - value: "60s"
    system.persistence.visibility.timeout:
      - value: "60s"

    # Throttle Shard Acquisition (Prevents "Thundering Herd" on startup)
    history.acquireShardConcurrency:
      - value: 1
  • Modified some DB connection values in the temporal-history configmap: 
    • apiVersion: v1
data:
  config_template.yaml: |-
    log:
      stdout: true
      level: "debug,info"

    persistence:
      defaultStore: default
      visibilityStore: visibility
      numHistoryShards: 512
      datastores:
        default:
          sql:
            pluginName: "mysql"
            driverName: "mysql"
            databaseName: "pdc_temporal"
            connectAddr: "swgpu-pdc-temporal-pr-rw:3306"
            connectProtocol: "tcp"
            user: pdc_temporal_adm
            password: "{{ .Env.TEMPORAL_STORE_PASSWORD }}"
            maxConnLifetime: "5m"
            maxConns: 200
            maxIdleConns: 20
            connMaxIdleTime: "3m"
            secretName: temporal-db
        visibility:
          sql:
            pluginName: "mysql"
            driverName: "mysql"
            databaseName: "pdc_temporal"
            connectAddr: "swgpu-pdc-temporal-pr-rw:3306"
            connectProtocol: "tcp"
            user: "pdc_temporal_adm"
            password: "{{ .Env.TEMPORAL_VISIBILITY_STORE_PASSWORD }}"
            maxConnLifetime: "5m"
            maxConns: 200
            maxIdleConns: 20
            connMaxIdleTime: "3m"
            secretName: temporal-db

Sample logs for temporal-sys-history-scanner:

{"level":"error","ts":"2025-11-21T10:18:02.829Z","msg":"Update workflow execution operation failed.","shard-id":216,"address":"100.98.138.38:7234","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","wf-id":"temporal-sys-history-scanner","wf-run-id":"5986922f-bf5f-4e1a-8ec8-3405ae8827e1","store-operation":"update-wf-execution","error":"createTransferTasks failed. Error: context deadline exceeded","logging-call-at":"transaction_impl.go:512","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/service/history/workflow.updateWorkflowExecution\n\t/home/builder/temporal/service/history/workflow/transaction_impl.go:512\ngo.temporal.io/server/service/history/workflow.(*TransactionImpl).UpdateWorkflowExecution\n\t/home/builder/temporal/service/history/workflow/transaction_impl.go:190\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionWithNew\n\t/home/builder/temporal/service/history/workflow/context.go:587\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionAsActive\n\t/home/builder/temporal/service/history/workflow/context.go:460\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).updateWorkflowExecution\n\t/home/builder/temporal/service/history/timerQueueActiveTaskExecutor.go:626\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).executeWorkflowBackoffTimerTask\n\t/home/builder/temporal/service/history/timerQueueActiveTaskExecutor.go:413\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).Execute\n\t/home/builder/temporal/service/history/timerQueueActiveTaskExecutor.go:124\ngo.temporal.io/server/service/history/queues.(*executorWrapper).Execute\n\t/home/builder/temporal/service/history/queues/executor_wrapper.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).Execute\n\t/home/builder/temporal/service/history/queues/executable.go:232\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:223\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}
{"level":"error","ts":"2025-11-21T10:18:02.829Z","msg":"Fail to process task","shard-id":216,"address":"100.98.138.38:7234","component":"timer-queue-processor","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","wf-id":"temporal-sys-history-scanner","wf-run-id":"5986922f-bf5f-4e1a-8ec8-3405ae8827e1","queue-task-id":899678221,"queue-task-visibility-timestamp":"2025-11-21T00:00:00.524Z","queue-task-type":"WorkflowBackoffTimer","queue-task":{"NamespaceID":"32049b68-7872-4094-8e63-d0dd59896a83","WorkflowID":"temporal-sys-history-scanner","RunID":"5986922f-bf5f-4e1a-8ec8-3405ae8827e1","VisibilityTimestamp":"2025-11-21T00:00:00.524094Z","TaskID":899678221,"Version":0,"WorkflowBackoffType":2},"wf-history-event-id":1,"error":"createTransferTasks failed. Error: context deadline exceeded","lifecycle":"ProcessingFailed","logging-call-at":"lazy_logger.go:68","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\t/home/builder/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\t/home/builder/temporal/service/history/queues/executable.go:339\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:224\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}
{"level":"error","ts":"2025-11-21T10:18:02.829Z","msg":"Critical error processing task, retrying.","shard-id":216,"address":"100.98.138.38:7234","component":"timer-queue-processor","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","wf-id":"temporal-sys-history-scanner","wf-run-id":"5986922f-bf5f-4e1a-8ec8-3405ae8827e1","queue-task-id":899678221,"queue-task-visibility-timestamp":"2025-11-21T00:00:00.524Z","queue-task-type":"WorkflowBackoffTimer","queue-task":{"NamespaceID":"32049b68-7872-4094-8e63-d0dd59896a83","WorkflowID":"temporal-sys-history-scanner","RunID":"5986922f-bf5f-4e1a-8ec8-3405ae8827e1","VisibilityTimestamp":"2025-11-21T00:00:00.524094Z","TaskID":899678221,"Version":0,"WorkflowBackoffType":2},"wf-history-event-id":1,"attempt":268,"error":"createTransferTasks failed. Error: context deadline exceeded","operation-result":"OperationCritical","logging-call-at":"lazy_logger.go:68","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\t/home/builder/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr.func1\n\t/home/builder/temporal/service/history/queues/executable.go:263\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\t/home/builder/temporal/service/history/queues/executable.go:340\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:224\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}

Sample logs for temporal-sys-scheduler:

{"level":"error","ts":"2025-11-21T10:19:11.014Z","msg":"Update workflow execution operation failed.","shard-id":89,"address":"100.98.138.38:7234","wf-namespace-id":"0df471a6-4f48-470e-ba65-3f47c5ba7372","wf-id":"temporal-sys-scheduler:UpdatePluginScheduleId","wf-run-id":"58f536bc-4d5f-489b-9ed6-fc0738525707","store-operation":"update-wf-execution","error":"createTimerTasks failed. Error: context deadline exceeded","logging-call-at":"transaction_impl.go:512","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/service/history/workflow.updateWorkflowExecution\n\t/home/builder/temporal/service/history/workflow/transaction_impl.go:512\ngo.temporal.io/server/service/history/workflow.(*TransactionImpl).UpdateWorkflowExecution\n\t/home/builder/temporal/service/history/workflow/transaction_impl.go:190\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionWithNew\n\t/home/builder/temporal/service/history/workflow/context.go:587\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionAsActive\n\t/home/builder/temporal/service/history/workflow/context.go:460\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).updateWorkflowExecution\n\t/home/builder/temporal/service/history/timerQueueActiveTaskExecutor.go:626\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).executeUserTimerTimeoutTask\n\t/home/builder/temporal/service/history/timerQueueActiveTaskExecutor.go:184\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).Execute\n\t/home/builder/temporal/service/history/timerQueueActiveTaskExecutor.go:114\ngo.temporal.io/server/service/history/queues.(*executorWrapper).Execute\n\t/home/builder/temporal/service/history/queues/executor_wrapper.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).Execute\n\t/home/builder/temporal/service/history/queues/executable.go:232\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:223\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}
{"level":"error","ts":"2025-11-21T10:19:11.014Z","msg":"Fail to process task","shard-id":89,"address":"100.98.138.38:7234","component":"timer-queue-processor","wf-namespace-id":"0df471a6-4f48-470e-ba65-3f47c5ba7372","wf-id":"temporal-sys-scheduler:UpdatePluginScheduleId","wf-run-id":"58f536bc-4d5f-489b-9ed6-fc0738525707","queue-task-id":354418700,"queue-task-visibility-timestamp":"2025-11-21T00:00:03.635Z","queue-task-type":"UserTimer","queue-task":{"NamespaceID":"0df471a6-4f48-470e-ba65-3f47c5ba7372","WorkflowID":"temporal-sys-scheduler:UpdatePluginScheduleId","RunID":"58f536bc-4d5f-489b-9ed6-fc0738525707","VisibilityTimestamp":"2025-11-21T00:00:03.635496Z","TaskID":354418700,"EventID":8,"Version":0},"wf-history-event-id":8,"error":"createTimerTasks failed. Error: context deadline exceeded","lifecycle":"ProcessingFailed","logging-call-at":"lazy_logger.go:68","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\t/home/builder/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\t/home/builder/temporal/service/history/queues/executable.go:339\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:224\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}
{"level":"error","ts":"2025-11-21T10:19:11.014Z","msg":"Critical error processing task, retrying.","shard-id":89,"address":"100.98.138.38:7234","component":"timer-queue-processor","wf-namespace-id":"0df471a6-4f48-470e-ba65-3f47c5ba7372","wf-id":"temporal-sys-scheduler:UpdatePluginScheduleId","wf-run-id":"58f536bc-4d5f-489b-9ed6-fc0738525707","queue-task-id":354418700,"queue-task-visibility-timestamp":"2025-11-21T00:00:03.635Z","queue-task-type":"UserTimer","queue-task":{"NamespaceID":"0df471a6-4f48-470e-ba65-3f47c5ba7372","WorkflowID":"temporal-sys-scheduler:UpdatePluginScheduleId","RunID":"58f536bc-4d5f-489b-9ed6-fc0738525707","VisibilityTimestamp":"2025-11-21T00:00:03.635496Z","TaskID":354418700,"EventID":8,"Version":0},"wf-history-event-id":8,"attempt":268,"error":"createTimerTasks failed. Error: context deadline exceeded","operation-result":"OperationCritical","logging-call-at":"lazy_logger.go:68","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\t/home/builder/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr.func1\n\t/home/builder/temporal/service/history/queues/executable.go:263\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\t/home/builder/temporal/service/history/queues/executable.go:340\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:224\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:119\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:145\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:120\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"}

Could we get some advice on how to resolve this? I have deployed Temporal on 4 other K8s clusters with no issues (all the same version as this cluster), the only big difference I can think of is that the database server is in a different region, but none of the modifications I’ve tried have helped. Been stuck on this for 3 days, would really appreciate help! Thanks