I have some workflows that are hanging at WorkflowTaskScheduled. When I look at the workers tab in the UI, I see that they’re all marked as X in “Workflow Task Handler”. What does this mean / is that expected? How do I debug this?
1 Like
This means that the worker has only activities registered with it and no workflows.
1 Like
@maxim What does it mean with no workflows? is it no workflows registered?
I have encountered the same case, where the workers are hung and are not processing any workflows. and the workflows are stuck in running state.
This has happened during high load. What changes do i need to do to fix this issue.
This was solved when i restarted the workers.
Usually “workers hang” happens when activities are not completing (or completing too slowly). The worker stops polling when all the activity task slots are taken.
Hello Maxim!
I got the same problem as user above.
Locally my device registered as a Workflow Task Handler, but when I run it inside k8s cluster i got this strange issue.
It works in case when I manually register workflow, register workers and activities for it and make worker factory start.
All configurations are the same
- I use yml files to auto-discover workflows and activities and start workers.
- I use same temporal instance hosted in k8s.
Can you help me?
Sorry, I don’t understand your problem. You said it works if you register workflows and activities and start the factory. What configuration doesn’t work?
So, When I run service locally, but using hosted temporal on the server with following yml configuration :
temporal:
connection:
target: ${temporal-host}
namespace: default
workers-auto-discovery:
packages:
- com.service.workflow/
- com.service.activity/
start-workers: true
and invoke workflow like this
var workflow = WorkflowUtil.buildWorkflowStubWithId(workflowClient,
SomeWorkflowClass.class, workflowDto.getWorkflowId(),
SOME_TASK_QUEUE);
return workflow.someAction(workflowDto);
everything works perfectly.
When I use same configuration, but deployed microservice I can see that my deployed service do not became a workflow task handler
And in Temporal UI inside workflow execution I can see only 3 steps:
- WorkflowExecutionStarted
- WorkflowTaskScheduled
- WorkflowExecutionTimedOut
However, when i change the way how I invoke workflow to :
var factory = WorkerFactory.newInstance(workflowClient);
var worker = factory.newWorker(SOME_TASK_QUEUE);
worker.registerWorkflowImplementationTypes(SomeWorkflowClassImpl.class);
worker.registerActivitiesImplementations(new SomeActivityImpl(objectMapper);
factory.start();
var workflow = WorkflowUtil.buildWorkflowStubWithId(workflowClient,
SomeWorkflowClass.class, workflowDto.getWorkflowId(),
SOME_TASK_QUEUE);
return workflow.someAction(workflowDto);
I’m pretty sure that I shouldn’t manually (using worker.register…) register workflows, activities and start factory, isn’t it? I guess start-workers and workers-auto-discovery properties should handle this.
P.S All logs are the same in both services during startup, all dependency versions are the same, same java version as well. Only difference, in 1st case my deployed app do not able to start workflow.
Can you bash into your pod/container where you deploy your springboot app microservice
and check if com.service.workflow is there and in correct place, and on classpath?
Seems issue is with springboot not being able to discover your workflow impls, maybe related to deployment not putting the packages on classpath, maybe file permission related? Seems activities are being discovered ok.
workers-auto-discovery:
packages:
- com.service.workflow/
- com.service.activity/
I would also mimic this sample in your app: samples-java/springboot/src/main/java/io/temporal/samples/springboot/actuator at main · temporalio/samples-java · GitHub
and then hit the actuator endpoint, it will show you what all workflows and activities have been auto-discovered and registered, could help debug this further.
Thank you, I’ve checked it.
We are deploying .jar to env, this jar includes com.service.workflow as well.
I tried to hit implement actuator endpoints and can see Workers.
Task Queue: someTaskQueue
Workflow Interface: com.service.workflow.some.SomeWorkflow
Workflow Methods: someMethod
Query Methods:
Signal Methods: someMethod2, someMethod3
Update Methods:
Update Validator Methods:
Activity Impl:
.....
But for Deployed service I can see that only Activities are registered, why?
Do you have any ideas?
Do you have any ideas?
yeah this is indeed weird, if you can see workflow methods registered would mean they were auto-discovered and registered with the worker for this task queue, so worker when it starts up should be polling for workflow tasks as well.
is there something else deployed on those k8s pods other than your spring boot app? do you have logs from this pod and see if anything is being logged during springboot startup?
since this works for you locally i think would start digging deeper into any possible differences between your local and your k8s envs