Temporal Workflow failing

The workflow task fails due to the following error.

{
  "message": "java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@669ab3b5[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@4b1c6cb0[Wrapped task = io.temporal.worker.ActiveThreadReportingExecutor$$Lambda$1585/0x00007fe1dba33818@6be7fe0a]] rejected from java.util.concurrent.ThreadPoolExecutor@14c45dad[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 348]",
  "source": "JavaSDK",
  "stackTrace": "io.temporal.internal.sync.WorkflowThreadImpl.start(WorkflowThreadImpl.java:247)\nio.temporal.internal.sync.WorkflowThreadImpl.runUntilBlocked(WorkflowThreadImpl.java:299)\nio.temporal.internal.sync.DeterministicRunnerImpl.runUntilAllBlocked(DeterministicRunnerImpl.java:229)\nio.temporal.internal.sync.SyncWorkflow.eventLoop(SyncWorkflow.java:147)\nio.temporal.internal.replay.ReplayWorkflowExecutor.eventLoop(ReplayWorkflowExecutor.java:67)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler$StatesMachinesCallbackImpl.eventLoop(ReplayWorkflowRunTaskHandler.java:360)\nio.temporal.internal.statemachines.WorkflowStateMachines.eventLoop(WorkflowStateMachines.java:490)\nio.temporal.internal.statemachines.WorkflowStateMachines.access$500(WorkflowStateMachines.java:56)\nio.temporal.internal.statemachines.WorkflowStateMachines$WorkflowTaskCommandsListener.workflowTaskStarted(WorkflowStateMachines.java:954)\nio.temporal.internal.statemachines.WorkflowTaskStateMachine.handleCompleted(WorkflowTaskStateMachine.java:127)\nio.temporal.internal.statemachines.WorkflowTaskStateMachine.handleStarted(WorkflowTaskStateMachine.java:117)\nio.temporal.internal.statemachines.FixedTransitionAction.apply(FixedTransitionAction.java:46)\nio.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:148)\nio.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:102)\nio.temporal.internal.statemachines.EntityStateMachineBase.handleEvent(EntityStateMachineBase.java:68)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleSingleEvent(WorkflowStateMachines.java:277)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:234)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:208)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:224)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:208)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:156)\nio.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:131)\nio.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:96)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:407)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:317)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:259)\nio.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105)\njava.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\njava.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\njava.base/java.lang.Thread.run(Unknown Source)\n",
  "encodedAttributes": null,
  "cause": {
    "message": "Task java.util.concurrent.FutureTask@669ab3b5[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@4b1c6cb0[Wrapped task = io.temporal.worker.ActiveThreadReportingExecutor$$Lambda$1585/0x00007fe1dba33818@6be7fe0a]] rejected from java.util.concurrent.ThreadPoolExecutor@14c45dad[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 348]",
    "source": "JavaSDK",
    "stackTrace": "java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)\njava.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)\njava.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)\njava.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)\nio.temporal.worker.ActiveThreadReportingExecutor.submit(ActiveThreadReportingExecutor.java:48)\nio.temporal.internal.sync.WorkflowThreadImpl.start(WorkflowThreadImpl.java:234)\nio.temporal.internal.sync.WorkflowThreadImpl.runUntilBlocked(WorkflowThreadImpl.java:299)\nio.temporal.internal.sync.DeterministicRunnerImpl.runUntilAllBlocked(DeterministicRunnerImpl.java:229)\nio.temporal.internal.sync.SyncWorkflow.eventLoop(SyncWorkflow.java:147)\nio.temporal.internal.replay.ReplayWorkflowExecutor.eventLoop(ReplayWorkflowExecutor.java:67)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler$StatesMachinesCallbackImpl.eventLoop(ReplayWorkflowRunTaskHandler.java:360)\nio.temporal.internal.statemachines.WorkflowStateMachines.eventLoop(WorkflowStateMachines.java:490)\nio.temporal.internal.statemachines.WorkflowStateMachines.access$500(WorkflowStateMachines.java:56)\nio.temporal.internal.statemachines.WorkflowStateMachines$WorkflowTaskCommandsListener.workflowTaskStarted(WorkflowStateMachines.java:954)\nio.temporal.internal.statemachines.WorkflowTaskStateMachine.handleCompleted(WorkflowTaskStateMachine.java:127)\nio.temporal.internal.statemachines.WorkflowTaskStateMachine.handleStarted(WorkflowTaskStateMachine.java:117)\nio.temporal.internal.statemachines.FixedTransitionAction.apply(FixedTransitionAction.java:46)\nio.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:148)\nio.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:102)\nio.temporal.internal.statemachines.EntityStateMachineBase.handleEvent(EntityStateMachineBase.java:68)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleSingleEvent(WorkflowStateMachines.java:277)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:234)\nio.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:208)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:224)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:208)\nio.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTask(ReplayWorkflowRunTaskHandler.java:156)\nio.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:131)\nio.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:96)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handleTask(WorkflowWorker.java:407)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:317)\nio.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:259)\nio.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105)\njava.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\njava.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\njava.base/java.lang.Thread.run(Unknown Source)\n",
    "encodedAttributes": null,
    "cause": null,
    "applicationFailureInfo": {
      "type": "java.util.concurrent.RejectedExecutionException",
      "nonRetryable": false,
      "details": null
    }
  },
  "applicationFailureInfo": {
    "type": "io.temporal.internal.sync.WorkflowRejectedExecutionError",
    "nonRetryable": false,
    "details": null
  }
}

The reason seems to be not clear. Could someone help to figure this out?

Is it related to this configuration

    workflow-cache:
      max-instances: 10
      max-threads: 10

10 for workflow thread pool is very low as threads are needed for signals and other callbacks.

What would be the ideal number of threads and what is defined here as max-instances?

It depends on the number of threads each workflow needs to execute. The ideal number would be the number of threads per workflow multiplied by the number of parallel workflow tasks being executed, plus threads used by cached workflows.

1 Like