Potential deadlock detected: Exception

anil_kumble · May 28, 2021, 5:04pm

Team, May I know the reason for below exception

java.lang.IllegalStateException: Operation allowed only while eventLoop is running
	at io.temporal.internal.statemachines.WorkflowStateMachines.checkEventLoopExecuting(WorkflowStateMachines.java:904)
	at io.temporal.internal.statemachines.WorkflowStateMachines.randomUUID(WorkflowStateMachines.java:600)
	at io.temporal.internal.replay.ReplayWorkflowContextImpl.scheduleActivityTask(ReplayWorkflowContextImpl.java:214)
	at io.temporal.internal.sync.SyncWorkflowContext.executeActivityOnce(SyncWorkflowContext.java:155)
	at io.temporal.internal.sync.SyncWorkflowContext.executeActivity(SyncWorkflowContext.java:140)
	at io.temporal.internal.sync.ActivityStubImpl.executeAsync(ActivityStubImpl.java:49)
	at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:38)
	at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:62)
	at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:70)
	at com.sun.proxy.$Proxy6.composeGreeting(Unknown Source)
	at io.temporal.samples.prototest.HelloActivity$TestProtoWorkflowImpl.testProto(HelloActivity.java:72)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:324)
	at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:282)
	at io.temporal.internal.sync.WorkflowExecuteRunnable.run(WorkflowExecuteRunnable.java:53)
	at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:126)
	at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:101)
	at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021/05/28 22:22:00 [Workflow] ended
22:22:00.107 [Workflow Executor taskQueue="HelloActivityTaskQueue", namespace="default": 1] ERROR i.t.i.r.ReplayWorkflowTaskHandler - Workflow task failure. startedEventId=3, WorkflowId=HelloActivityWorkflow, RunId=665ae253-4b0e-4e45-99fa-0a5b2e15e5b4. If see continuously the workflow might be stuck.
io.temporal.internal.replay.InternalWorkflowTaskException: Failure handling event 3 of 'EVENT_TYPE_WORKFLOW_TASK_STARTED' type. IsReplaying=false, PreviousStartedEventId=3, workflowTaskStartedEventId=3, Currently Processing StartedEventId=3
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:193)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleEvent(ReplayWorkflowRunTaskHandler.java:140)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:180)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:150)
	at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithEmbeddedQuery(ReplayWorkflowTaskHandler.java:201)
	at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:114)
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:319)
	at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:279)
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: WorkflowTask: failure executing SCHEDULED->WORKFLOW_TASK_STARTED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED]
	at io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:140)
	at io.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:91)
	at io.temporal.internal.statemachines.EntityStateMachineBase.handleEvent(EntityStateMachineBase.java:63)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventImpl(WorkflowStateMachines.java:210)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:178)
	... 11 common frames omitted
Caused by: io.temporal.internal.sync.PotentialDeadlockException: Potential deadlock detected: workflow thread "workflow-method" didn't yield control for over a second. Other workflow threads:

If I call my activity method then I am getting this exception. My activity method has nothing. it has just two println statements.

Below is my activiy method

@Override
    public String composeGreeting(String greeting, String name) {

      System.out.println(
          DateTimeFormatter.ofPattern("yyyy/MM/dd HH:mm:ss").format(LocalDateTime.now())
              + " [Activity] started");

      System.out.println(
          DateTimeFormatter.ofPattern("yyyy/MM/dd HH:mm:ss").format(LocalDateTime.now())
              + " [Activity] ended");

      return greeting + " " + name + "!";
    }

If I commented calling this method then it is working.

tihomir · May 28, 2021, 6:11pm

Are you getting this while debugging?

anil_kumble · May 28, 2021, 6:24pm

Yes, I am getting this while debugging.

First I just ran the workflow that time it just started the workflow and didn’t call my activity. I see the status via UI, it was running.

I down my app server and started it,
started my workflow,
this time it didn’t even start my workflow. I went to the UI screen and stopped the previously running workflow, after I tried debugging it, it is throwing this exception.

tihomir · May 28, 2021, 6:27pm

With Java SDK, when debugging, set the TEMPORAL_DEBUG env variable to true – https://docs.temporal.io/docs/java/testing-and-debugging

Note that you should set TEMPORAL_DEBUG only when you are actually debugging (with break points in your code). When you are not debugging make sure its not set as it might hide serious problems with the workflow code.

anil_kumble · May 28, 2021, 8:11pm

But, while I am starting the workflow it doesn’t call my activity. It got stuck before calling my activity, after that only I put breakpoints fo debug.

So why my workflow code got stuck-up?

tihomir · May 28, 2021, 8:20pm

Can you show your workflow code or a reproducible test so we can understand better?

maxim · May 28, 2021, 8:49pm

@anil_kumble make sure that your code follows all the constraints.

anil_kumble · May 29, 2021, 5:37am

My code has nothing to do with,
Below find my code,

package io.temporal.samples.prototest;

import io.temporal.activity.ActivityInterface;
import io.temporal.activity.ActivityOptions;
import io.temporal.client.WorkflowClient;
import io.temporal.client.WorkflowOptions;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.worker.Worker;
import io.temporal.worker.WorkerFactory;
import io.temporal.workflow.Workflow;
import io.temporal.workflow.WorkflowInterface;
import io.temporal.workflow.WorkflowMethod;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;

public class HelloActivity {

  static final String TASK_QUEUE = "HelloActivityTaskQueue";

  static final String WORKFLOW_ID = "HelloActivityWorkflow";

  @WorkflowInterface
  public interface TestProtoWorkflow {
    @WorkflowMethod()
    String testProto(PersonContainer person);
  }

  @ActivityInterface
  public interface TestProtoActivity {
    String composeGreeting(String greeting, String name);
  }

  public static class TestProtoWorkflowImpl implements TestProtoWorkflow {

    private final TestProtoActivity activities =
        Workflow.newActivityStub(
            TestProtoActivity.class,
            ActivityOptions.newBuilder()
                .setTaskQueue(TASK_QUEUE)
                .setStartToCloseTimeout(Duration.ofSeconds(40))
                .build());

    @Override
    public String testProto(PersonContainer person) {
      System.out.println("[Workflow] started");
      String s = activities.composeGreeting("Hel", "name");
      System.out.println("[Workflow] ended");

      return s;
    }
  }

  static class TestProtoActivityImpl implements TestProtoActivity {
    @Override
    public String composeGreeting(String greeting, String name) {

      System.out.println("[Activity] started");
      System.out.println("[Activity] ended");

      return greeting + " " + name + "!";
    }
  }

  public static void main(String[] args) {

    WorkflowServiceStubs service = WorkflowServiceStubs.newInstance();
    WorkflowClient client = WorkflowClient.newInstance(service);

    WorkerFactory factory = WorkerFactory.newInstance(client);

    Worker worker = factory.newWorker(TASK_QUEUE);
    worker.registerWorkflowImplementationTypes(TestProtoWorkflowImpl.class);

    factory.start();

    TestProtoWorkflow workflow =
        client.newWorkflowStub(
            TestProtoWorkflow.class,
            WorkflowOptions.newBuilder()
                .setWorkflowId(WORKFLOW_ID)
                .setTaskQueue(TASK_QUEUE)
                .build());

    System.out.println("test");

    PersonContainer personContainer = new PersonContainer();
    List<Person> personList = new ArrayList<>();
    personList.add(new Person("1", "John", 123));
    personList.add(new Person("2", "Mathew", 124));
    personList.add(new Person("3", "John", 125));
    personContainer.setPersonList(personList);

    String val = workflow.testProto(personContainer);
    System.out.println(val);

    System.exit(0);
  }
}

If I commented the below line then it is working fine
String s = activities.composeGreeting("Hel", "name");

mleow · May 29, 2021, 9:38am

@anil_kumble You missed out registering the Activity

Worker worker = factory.newWorker(TASK_QUEUE);
    worker.registerWorkflowImplementationTypes(TestProtoWorkflowImpl.class);
+  worker.registerActivitiesImplementations(new TestProtoActivityImpl());

    factory.start();

If the worker does not have the activity registered, no work to be done; that is why it seems to be “hanging”.

anil_kumble · May 29, 2021, 10:54am

Yes, after registering it is working fine. Sorry that I forgot to register.
Thanks

Topic		Replies	Views
Randomly Getting: "java.lang.IllegalStateException: Operation allowed only while eventLoop is running" Community Support java-sdk	14	48	November 13, 2024
java.lang.IllegalStateException: Operation allowed only while eventLoop is running Community Support java-sdk , testing	5	764	September 17, 2024
Testing temporal workflows Community Support java-sdk	5	1206	September 14, 2021
Exception while executing dynamic workflow: Operation allowed only while eventLoop is running Community Support java-sdk	0	134	March 14, 2024
Exception while executing dynamic activity: Operation allowed only while eventLoop is running Community Support java-sdk , activity	3	1396	February 26, 2022

Potential deadlock detected: Exception

Related topics