Randomly Getting: "java.lang.IllegalStateException: Operation allowed only while eventLoop is running"

Community Support
1

Thanks for your help in advance! Any advises are much appreciated.

I’m getting “java.lang.IllegalStateException: Operation allowed only while eventLoop is running” Only in production, not in dev or preprod, only occasionally, seemingly random from different places. There is no Async promise or child workflows.

The exception were usually from saveEntity() invoked from different places of workflow class inherits BaseRequestWorkflow

import io.temporal.activity.ActivityOptions;



public abstract class BaseRequestWorkflow implements GenericRequestWorkflow {
   private static final Logger log = LoggerFactory.getLogger(
           BaseRequestWorkflow.class);
   private final EmailActivity emailActivity =
           Workflow.newActivityStub(
                   EmailActivity.class,
                   ActivityOptions.newBuilder()
                           .setStartToCloseTimeout(Duration.ofMinutes(10))
                           .setRetryOptions(RetryOptions.newBuilder()
                                   .setMaximumAttempts(3)
                                   .build())
                           .build());


   private final SystemActivity systemActivity =
           Workflow.newActivityStub(
                   SystemActivity.class,
                   ActivityOptions.newBuilder()
                           .setStartToCloseTimeout(Duration.ofMinutes(1))
                           .setRetryOptions(RetryOptions.newBuilder()
                                   .setMaximumAttempts(2)
                                   .build())
                           .build());


   protected RequestEntities execution;
   private List<Setting> settings;


   private Set<Long> processed = new HashSet<Long>();


   @Override
   public RequestEntities run(List<RequestEntity> input) {
       try {
           settings = systemActivity.loadSettings();
           execution = new RequestEntities();
           execution.setRequestEntityList(input);
           // systemActivity.ingest(execution.getRequestId(),
           //        WorkflowExecutionIngestorService.EventType.STARTED, "");
           traverse(execution.getRoot());
           emailActivity.emailJobStatus(execution.getUsername(), execution.getRequestId(),
                   execution.getRoot().getData().getResult());
           // systemActivity.ingest(execution.getRequestId(),
           //        WorkflowExecutionIngestorService.EventType.COMPLETED, "");
       } catch (Exception e) {
           log.error("Error during workflow execution: ", e);
       }
       return execution;
   }


   @Override
   public RequestEntities getCurrentStatus() {
       return execution;
   }


   public void saveEntity(RequestEntity entity) {
       systemActivity.saveEntity(entity);
   }

Regards

1 Like
2

Are you reusing activity stubs or any other objects across workflow instances?

3

maxim,
The activity stubs are private final per workflow instance as shown above. The workflow and activities instances are registered with springboot Config as below. Would that be an issue? It works fine mostly, except for the issue in the description.

public class WorkerConfig {
   @Autowired
   private WorkflowClient workflowClient;


   @Autowired
   Map<String, UserRequestType> userRequestTypeRegistry;


   @Autowired
   private SystemActivityImpl systemActivity;
   @Autowired
   private EmailActivityImpl emailActivity;


   @Value("${temporal.server.queue.name:default}")
   private String serverQueueName;


   @Bean
   public String serverQueueName() {
       return serverQueueName;
   }


   @Bean
   public WorkerFactory workerFactory() {
       // WorkerFactory that can be used to create workers.
       WorkerFactory factory = WorkerFactory.newInstance(workflowClient);
       Worker worker = factory.newWorker(serverQueueName);
       worker.registerWorkflowImplementationTypes(TestConfigWorkflow.class);
       for (UserRequestType type : TestConfigWorkflow.getUserRequestTypes()) {
           userRequestTypeRegistry.put(type.getType(), type);
       }
       worker.registerActivitiesImplementations(systemActivity);
       worker.registerActivitiesImplementations(emailActivity);
       factory.start();
       return factory;
   }
}
1 Like
4

What is the full stack trace of the exception?

5

Here it is. Thank you

java.lang.IllegalStateException: Operation allowed only while eventLoop is running
        at io.temporal.internal.statemachines.WorkflowStateMachines.checkEventLoopExecuting(WorkflowStateMachines.java:1313)
        at io.temporal.internal.statemachines.WorkflowStateMachines.randomUUID(WorkflowStateMachines.java:899)
        at io.temporal.internal.replay.ReplayWorkflowContextImpl.scheduleActivityTask(ReplayWorkflowContextImpl.java:202)
        at io.temporal.internal.sync.SyncWorkflowContext.executeActivityOnce(SyncWorkflowContext.java:304)
        at io.temporal.internal.sync.SyncWorkflowContext.executeActivity(SyncWorkflowContext.java:278)
        at io.temporal.internal.sync.ActivityStubImpl.executeAsync(ActivityStubImpl.java:59)
        at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:39)
        at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83)
        at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60)
        at jdk.proxy2/jdk.proxy2.$Proxy211.saveEntity(Unknown Source)
        at com.company.ipp.temporal.worker.BaseRequestWorkflow.saveEntity(BaseRequestWorkflow.java:74)
        at com.company.ipp.temporal.worker.VrlTestMachinePoolValidationProcessor.watchJob(VrlTestMachinePoolValidationProcessor.java:266)
        at com.company.ipp.temporal.worker.VrlTestMachinePoolValidationProcessor.processActivity(VrlTestMachinePoolValidationProcessor.java:196)
        at com.company.ipp.temporal.worker.BaseRequestWorkflow.process(BaseRequestWorkflow.java:208)
        at com.company.ipp.temporal.worker.BaseRequestWorkflow.traverse(BaseRequestWorkflow.java:171)
--
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:342)
        at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:317)
        at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70)
        at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135)
        at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102)
        at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107)
        at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
1 Like
6

Here’s how the workflow stub and instance are instantiated:

    public String submit(String workflowId, Class<?> workflowClass, RequestEntities execution)
            throws Exception {
        WorkflowOptions options = WorkflowOptions.newBuilder()
                .setWorkflowId(workflowId)
                .setRetryOptions(RetryOptions.newBuilder()
                        .setMaximumAttempts(0).setInitialInterval(Duration.ofSeconds(1))
                        .setBackoffCoefficient(1.0).build())
                .setTaskQueue(serverQueueName).build();
        WorkflowStub workflow = workflowClient.newUntypedWorkflowStub(
                workflowClass.getSimpleName(), options);
        // Start the workflow
        WorkflowExecution workflowExecution = workflow.start(execution.getRequestEntityList());
        return workflowExecution.getWorkflowId();
    }
7

Are you running the recent SDK version?

8 
    <temporal-sdk.version>1.23.1</temporal-sdk.version>
9

Are you sure activity stubs defined in BaseRequestWorkflow are not being reused across multiple executions? Do you get same error if you create the stub in saveEntity method each time instead of reusing?

10

All activity stubs are private fields (shown below) of in the Workflow class, which is submitted with the submit() method above by workflowClient. The Exception happens not often and there is not a clear pattern how to reproduce it. The activity components themselves are instantiated by sprintboot framework.

Should the activity stubs be created differently, not like this?

public abstract class BaseRequestWorkflow implements GenericRequestWorkflow {
    private final SystemActivity systemActivity =
            Workflow.newActivityStub(
                    SystemActivity.class,
                    ActivityOptions.newBuilder()
                            .setStartToCloseTimeout(Duration.ofMinutes(1))
                            .setRetryOptions(RetryOptions.newBuilder()
                                    .setMaximumAttempts(2)
                                    .build())
                            .build());
11

Could you create a stand alone replication of the problem? Then we could try to troubleshoot.

12

it’s difficult to create a standalone case that reproduces the symptom. Could you instruct on how to write code to detect if the eventLoop is running in the current thread/context? It would help me troubleshooting. Thanks

13

One more thing. Do you run with the recent version of the SDK? I remember that some of very old clients could have this issue.

14 
 <temporal-sdk.version>1.23.1</temporal-sdk.version>
15

Would you try upgrading to the latest to see if this problem persists?

16 
    <temporal-sdk.version>1.24.2</temporal-sdk.version>

Yes, it still occurs occasionally which really bugs me. It clearly runs within the context a workflow. I wonder if any debugging info can be put in to reveal more information.

17

The latest Java SDK version is v1.28.0 so I would recommend trying that.

To help debug we would need a stand alone replication of the problem. You could try logging the current thread name Thread.currentThread().getName() to confirm you are in a workflow thread.

18

I added debugging which shows it was always in the thread of a temporal workflow

public void saveEntity(RequestEntity entity) {
    log.info("Check running in thread: {}", Thread.currentThread().getName());
    systemActivity.saveEntity(entity);
}

2025-03-05 04:11:05,719 INFO i.t.i.l.ReplayAwareLogger.info 241 - Check running in thread: workflow-method-TestConfigRequest-78-2025-03-05-04:11:03.360-e0036051-aa24-4f0a-800b-1189115ea58e
2025-03-05 04:11:05,730 INFO i.t.i.l.ReplayAwareLogger.info 241 - Check running in thread: workflow-method-TestConfigRequest-90-2025-03-05-04:11:01.919-343c898a-31ab-4f0a-8d44-68f0f911eef2
2025-03-05 04:11:06,031 INFO i.t.i.l.ReplayAwareLogger.info 241 - Check running in thread: workflow-method-TestConfigRequest-78-2025-03-05-04:11:03.360-e0036051-aa24-4f0a-800b-1189115ea58e
2025-03-05 04:11:06,039 INFO i.t.i.l.ReplayAwareLogger.info 241 - Check running in thread: workflow-method-TestConfigRequest-90-2025-03-05-04:11:01.919-343c898a-31ab-4f0a-8d44-68f0f911eef2
2025-03-05 04:11:06,081 INFO i.t.i.l.ReplayAwareLogger.info 241 - Check running in thread: workflow-method-TestConfigRequest-78-2025-03-05-04:11:03.360-e0036051-aa24-4f0a-800b-1189115ea58e
2025-03-05 04:11:06,082 ERROR i.t.i.l.ReplayAwareLogger.error 431 - Request:78 Exception when processing entity:6911
java.lang.IllegalStateException: Operation allowed only while eventLoop is running
at io.temporal.internal.statemachines.WorkflowStateMachines.checkEventLoopExecuting(WorkflowStateMachines.java:1360)
at io.temporal.internal.statemachines.WorkflowStateMachines.randomUUID(WorkflowStateMachines.java:948)
at io.temporal.internal.replay.ReplayWorkflowContextImpl.scheduleActivityTask(ReplayWorkflowContextImpl.java:202)
at io.temporal.internal.sync.SyncWorkflowContext.executeActivityOnce(SyncWorkflowContext.java:304)
at io.temporal.internal.sync.SyncWorkflowContext.executeActivity(SyncWorkflowContext.java:278)
at io.temporal.internal.sync.ActivityStubImpl.executeAsync(ActivityStubImpl.java:59)
at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:39)
at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83)
at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60)

19

java version is still 17. I wonder if there is a reason to switch to Java 1.28.0.