When starting worker, get the Potential Deadlock error, can anyone suggest?

Community Support
1

Getting the following error when starting a worker program:
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_worker_base.go:270 +0x106
2021/03/26 14:58:58 WARN Failed to process workflow task. Namespace default TaskQueue ml_activity WorkerID 19832@WSG126024@ WorkflowType MLDemoWorkflow WorkflowID ml_demo_cff99062-9eec-4547-ac2b-8fe6b8080f14 RunID db5bcc2f-257c-4006-8900-19aef562f2da Attempt 20 Error Potential deadlock detected: workflow goroutine “root” didn’t yield for over a second
2021/03/26 14:59:09 ERROR Workflow panic Namespace default TaskQueue ml_activity WorkerID 19832@WSG126024@ WorkflowType MLDemoWorkflow WorkflowID ml_demo_cff99062-9eec-4547-ac2b-8fe6b8080f14 RunID db5bcc2f-257c-4006-8900-19aef562f2da Attempt 21 Error Potential deadlock detected: workflow goroutine “root” didn’t yield for over a second StackTrace process event for ml_activity [panic]:
go.temporal.io/sdk/internal.(*coroutineState).call(0xc0005dd220)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_workflow.go:881 +0x1eb
go.temporal.io/sdk/internal.(*dispatcherImpl).ExecuteUntilAllBlocked(0xc0005dd1d0, 0x0, 0x0)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_workflow.go:966 +0x30b
go.temporal.io/sdk/internal.executeDispatcher(0xf19ad0, 0xc0005eed00, 0xf1d368, 0xc0005dd1d0)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_workflow.go:569 +0xb8
go.temporal.io/sdk/internal.(*syncWorkflowDefinition).OnWorkflowTaskStarted(0xc0005eec80)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_workflow.go:542 +0x51
go.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc0005e5128, 0xc000053580, 0x100, 0x0, 0x0)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_event_handlers.go:791 +0x3b8
go.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc0001a7050, 0xc0000e6090, 0x13f3880, 0xc0004aed90, 0xc0001a7050, 0x0)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_task_handlers.go:876 +0x72c
go.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc000215550, 0xc0000e6090, 0xc0005fbaa0, 0x0, 0x0, 0x0, 0x0)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_task_handlers.go:727 +0x6f9
go.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc000272ea0, 0xc0000e6090, 0x0, 0x0)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_task_pollers.go:288 +0x4c2
go.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc000272ea0, 0xcc6600, 0xc0000e6090, 0xc000251b90, 0xc0005dc5f0)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_task_pollers.go:259 +0x8b
go.temporal.io/sdk/internal.(*baseWorker).processTask(0xc000446000, 0xcc61c0, 0xc000075490)
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_worker_base.go:343 +0xc4
created by go.temporal.io/sdk/internal.(*baseWorker).runTaskDispatcher
I:/projects/golangworkspace/pkg/mod/go.temporal.io/sdk@v1.4.1/internal/internal_worker_base.go:270 +0x106
2021/03/26 14:59:09 WARN Failed to process workflow task. Namespace default TaskQueue ml_activity WorkerID 19832@WSG126024@ WorkflowType MLDemoWorkflow WorkflowID ml_demo_cff99062-9eec-4547-ac2b-8fe6b8080f14 RunID db5bcc2f-257c-4006-8900-19aef562f2da Attempt 21 Error Potential deadlock detected: workflow goroutine “root” didn’t yield for over a second
2021/03/26 14:59:13 INFO Worker has been stopped. Namespace default TaskQueue ml_activity WorkerID 19832@WSG126024@ Signal interrupt
2021/03/26 14:59:13 INFO Stopped Worker Namespace default TaskQueue ml_activity WorkerID 19832@WSG126024@

This program has been working, anyone has suggestions how to resolve this?

2

It looks like your workflow code invokes some blocking function which is not part of the Temporal SDK APIs.

3

Thanks. This is resolved, I put a 5 second sleep in the workflow code for testing purpose. So does this mean we cannot have any local execution that blocks for longer than x seconds in Temporal workflow code?

4

So does this mean we cannot have any local execution that blocks for longer than x seconds in Temporal workflow code?

Correct. That would be a bad idea as workflow uses replay for recovery and such CPU intensive task would be reexecuted multiple times. You can always use local activities to execute a code that needs to block due to IO locally.

1 Like
5

@maxim thanks for the explaination.