WorkflowTaskTimedOut sporadic errors in v1.23.1

Community Support
,
1

Hi,

we have upgraded our server from v1.18.5 to v1.23.1 and are seeing this error in the logs with workflow tasks timing out sporadically .

{“level”:“error”,“ts”:“2024-09-04T01:30:54.976Z”,“@timestamp”:“2024-09-04T01:30:54.976Z”,“msg”:“Fail to process task”,“shard-id”:394,“address”:“127.2.1.2:7234”,“component”:“transfer-queue-processor”,“wf-namespace-id”:“a2fc6c69-cf35-4449-b1bc-f2c16536d76e”,“wf-id”:“Scheduled - Securityxxxxxxxxxx”,“wf-run-id”:“ff0f217e-43e3-4b5a-a008-543e6c45c8ed”,“queue-task-id”:2830106712,“queue-task-visibility-timestamp”:“2024-09-04T01:30:52.578Z”,“queue-task-type”:“TransferWorkflowTask”,“queue-task”:{“NamespaceID”:“a2fc6c69-cf35-4449-b1bc-f2c16536d76e”,“WorkflowID”:“Scheduled - SecurityBaselineAllWf-2024-09-04T00:00:00Z”,“RunID”:“ff0f217e-43e3-4b5a-a008-543e6c45c8ed”,“VisibilityTimestamp”:“2024-09-04T01:30:52.578861468Z”,“TaskID”:2830106712,“TaskQueue”:“7@6fd1de5c-eb5d-4468-4434-d8fa-89176-development-python-77a07dcf941e46758cee490bd7115e73”,“ScheduledEventID”:23,“Version”:0},“wf-history-event-id”:23,“error”:“context deadline exceeded”,“unexpected-error-attempts”:1,“lifecycle”:“ProcessingFailed”,“logging-call-at”:“lazy_logger.go:68”,“stacktrace”:"go.temporal.io/server/common/log.(*zapLogger).Error\n\ /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\ /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\ /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/service/history/queues/executable.go:421\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[…]).executeTask.func1\n\ /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/tasks/fifo_scheduler.go:224\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\ /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/backoff/retry.go:117\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\ /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/backoff/retry.go:143\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\ /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/backoff/retry.go:118\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[…]).executeTask\n\ /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[…]).processTask\n\ /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/tasks/fifo_sche

2

{ [-]
@timestamp: 2024-09-05T10:35:54.404Z
@version: 1
address: xxx.x.x.x:7234
app_id: xxxxx
component: transfer-queue-processor
env: dev
error: context deadline exceeded
host: wh-xxxxxxxx.svr.us.jpmchase.net
kubernetesCluster: xxxxxxxx
kubernetesNamespace: xxxxxxxx
level: error
lifecycle: ProcessingFailed
logging-call-at: lazy_logger.go:68
metaMarker: marker21
msg: Fail to process task
queue-task: { [+]
}
queue-task-id: 1138753696
queue-task-type: TransferWorkflowTask
queue-task-visibility-timestamp: 2024-09-05T10:35:53.124Z
s_sourcetype: bifrost
sealId: 89176
serverId: 100
shard-id: 349
source: gtidatabus04
stacktrace: go.temporal.io/server/common/log.(*zapLogger).Error /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/log/zap_logger.go:156 go.temporal.io/server/common/log.(*lazyLogger).Error /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/log/lazy_logger.go:68 go.temporal.io/server/service/history/queues.(*executableImpl).HandleErr /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/service/history/queues/executable.go:421 go.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1 /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/tasks/fifo_scheduler.go:224 go.temporal.io/server/common/backoff.ThrottleRetry.func1 /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/backoff/retry.go:117 go.temporal.io/server/common/backoff.ThrottleRetryContext /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/backoff/retry.go:143 go.temporal.io/server/common/backoff.ThrottleRetry /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/backoff/retry.go:118 go.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/tasks/fifo_scheduler.go:233 go.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask /jenkins/workspace/QTOOLS_mq_temporal_server_master/temporal/common/tasks/fifo_scheduler.go:211
tags: [ [+]
]
ts: 2024-09-05T10:35:54.404Z
unexpected-error-attempts: 1
wf-history-event-id: 14
wf-id: xxxxxxx Tier1FailoverPrecheckWorkflow
wf-namespace-id: 71988524-7d28-49e7-8a27-1330840f01cf
wf-run-id: 3f9e8e99-9b64-415b-a9d4-ecaecf498acd
}

3 
18

2024-09-05 UTC 10:36:03.79

WorkflowTaskStarted

Scheduled Event ID

17

Identity

31@cc5502ce-a127-4926-745a-96b3

Request ID

a78f4afa-c7b0-4f41-9b01-d53a13e95a70

History Size Bytes

2821

17
2024-09-05 UTC 10:36:03.63

WorkflowTaskScheduled

Summary
Task Queue
Start To Close Timeout

10 seconds

Attempt

1

16
2024-09-05 UTC 10:36:03.63

WorkflowTaskTimedOut

Scheduled Event ID

14

Started Event ID

15

Timeout Type

TIMEOUT_TYPE_START_TO_CLOSE

15
2024-09-05 UTC 10:35:53.62

WorkflowTaskStarted

Scheduled Event ID

14

Identity

31@cc5502ce-a127-4926-745a-96b3

Request ID

0f398faa-4679-4bf2-bbbf-7a24a0a792ff

History Size Bytes

2626