`PotentialDeadlockException` when running Temporal workflow replay tests exclusively in GitHub Actions CI environment (Linux runners). The same tests pass consistently on local macOS development machines

Aniket_Gupta · November 7, 2025, 3:07pm

We are getting the following error each time on CI (GitHub Actions), meanwhile, when I run the test case locally, it works fine.

Error:

WorkLifeCycleWorkflowReplayTest.test_work_lifecycle_workflow_non_deterministic_detection

query failure for workflow_id: "workflow_id_in_replay"
run_id: "run_id_in_replay"
, queryType=__replay_only, args=Optional.empty, error=io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event 63 of type 'EVENT_TYPE_WORKFLOW_TASK_COMPLETED' during replay. {WorkflowTaskStartedEventId=9223372036854775807, CurrentStartedEventId=62}
	at io.temporal.internal.statemachines.WorkflowStateMachines.createEventProcessingException(WorkflowStateMachines.java:426)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:334)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleEvent(WorkflowStateMachines.java:293)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.applyServerHistory(ReplayWorkflowRunTaskHandler.java:249)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleWorkflowTaskImpl(ReplayWorkflowRunTaskHandler.java:231)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler.handleDirectQueryWorkflowTask(ReplayWorkflowRunTaskHandler.java:204)
	at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTaskWithQuery(ReplayWorkflowTaskHandler.java:129)
	at io.temporal.internal.replay.ReplayWorkflowTaskHandler.handleWorkflowTask(ReplayWorkflowTaskHandler.java:100)
	at io.temporal.internal.worker.QueryReplayHelper.queryWorkflowExecution(QueryReplayHelper.java:96)
	at io.temporal.internal.worker.QueryReplayHelper.queryWorkflowExecution(QueryReplayHelper.java:66)
	at io.temporal.internal.worker.SyncWorkflowWorker.queryWorkflowExecution(SyncWorkflowWorker.java:223)
	at io.temporal.worker.Worker.replayWorkflowExecution(Worker.java:483)
	at io.temporal.testing.WorkflowReplayer.replayWorkflowExecution(WorkflowReplayer.java:178)
	at com.dmg.work.workflow.replay.BaseWorkflowReplayTest.validateReplay(BaseWorkflowReplayTest.java:83)
	at com.dmg.work.workflow.replay.WorkLifeCycleWorkflowReplayTest.test_work_lifecycle_workflow_non_deterministic_detection(WorkLifeCycleWorkflowReplayTest.java:52)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.executeNonConcurrentTasks(ForkJoinPoolHierarchicalTestExecutorService.java:155)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.RuntimeException: WorkflowTask: failure executing STARTED->WORKFLOW_TASK_COMPLETED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED, SCHEDULED->WORKFLOW_TASK_STARTED]
	at io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:163)
	at io.temporal.internal.statemachines.StateMachine.handleHistoryEvent(StateMachine.java:103)
	at io.temporal.internal.statemachines.EntityStateMachineBase.handleEvent(EntityStateMachineBase.java:84)
	at io.temporal.internal.statemachines.EntityStateMachineInitialCommand.handleEvent(EntityStateMachineInitialCommand.java:70)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleSingleEvent(WorkflowStateMachines.java:477)
	at io.temporal.internal.statemachines.WorkflowStateMachines.handleEventsBatch(WorkflowStateMachines.java:332)
	... 62 more
Caused by: io.temporal.internal.sync.PotentialDeadlockException: [TMPRL1101] Potential deadlock detected. Workflow thread "child workflow completion callback" didn't yield control for over a second. {detectionTimestamp=1761646381997, threadDumpTimestamp=1761646382010} 

child workflow completion callback
	at java.base@11.0.29/java.lang.String.indexOf(String.java:1578)
	at app//org.jacoco.agent.rt.internal_4a7f17c.asm.Type.getArgumentTypes(Type.java:308)
	at app//org.jacoco.agent.rt.internal_4a7f17c.asm.commons.AnalyzerAdapter.<init>(AnalyzerAdapter.java:159)
	at app//org.jacoco.agent.rt.internal_4a7f17c.asm.commons.AnalyzerAdapter.<init>(AnalyzerAdapter.java:121)
	at app//org.jacoco.agent.rt.internal_4a7f17c.core.internal.flow.ClassProbesAdapter$2.visitEnd(ClassProbesAdapter.java:85)
	at app//org.jacoco.agent.rt.internal_4a7f17c.asm.ClassReader.readMethod(ClassReader.java:1518)
	at app//org.jacoco.agent.rt.internal_4a7f17c.asm.ClassReader.accept(ClassReader.java:744)
	at app//org.jacoco.agent.rt.internal_4a7f17c.asm.ClassReader.accept(ClassReader.java:424)
	at app//org.jacoco.agent.rt.internal_4a7f17c.core.instr.Instrumenter.instrument(Instrumenter.java:91)
	at app//org.jacoco.agent.rt.internal_4a7f17c.core.instr.Instrumenter.instrument(Instrumenter.java:109)
	at app//org.jacoco.agent.rt.internal_4a7f17c.CoverageTransformer.transform(CoverageTransformer.java:92)
	at java.instrument@11.0.29/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
	at java.instrument@11.0.29/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument@11.0.29/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
	at java.base@11.0.29/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base@11.0.29/java.lang.ClassLoader.defineClass(ClassLoader.java:1022)
	at java.base@11.0.29/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base@11.0.29/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base@11.0.29/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base@11.0.29/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base@11.0.29/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base@11.0.29/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base@11.0.29/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	at java.base@11.0.29/java.lang.Class.getDeclaredMethods0(Native Method)
	at java.base@11.0.29/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
	at java.base@11.0.29/java.lang.Class.getMethodsRecursive(Class.java:3307)
	at java.base@11.0.29/java.lang.Class.getMethod0(Class.java:3293)
	at java.base@11.0.29/java.lang.Class.getMethod(Class.java:2106)
	at app//com.google.protobuf.GeneratedMessageV3.getMethodOrDie(GeneratedMessageV3.java:1995)
	at app//com.google.protobuf.GeneratedMessageV3.access$1000(GeneratedMessageV3.java:79)
	at app//com.google.protobuf.GeneratedMessageV3$FieldAccessorTable$SingularFieldAccessor$ReflectionInvoker.<init>(GeneratedMessageV3.java:2348)
	at app//com.google.protobuf.GeneratedMessageV3$FieldAccessorTable$SingularFieldAccessor.<init>(GeneratedMessageV3.java:2418)
	at app//com.google.protobuf.GeneratedMessageV3$FieldAccessorTable$SingularStringFieldAccessor.<init>(GeneratedMessageV3.java:3071)
	at app//com.google.protobuf.GeneratedMessageV3$FieldAccessorTable.ensureFieldAccessorsInitialized(GeneratedMessageV3.java:2137)
	at app//com.dmg.data_services.v1.JobConditions$Builder.internalGetFieldAccessorTable(JobConditions.java:2468)
	at app//com.google.protobuf.GeneratedMessageV3$Builder.hasField(GeneratedMessageV3.java:731)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeField(JsonFormat.java:1652)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1500)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1458)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.parseFieldValue(JsonFormat.java:2018)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeOneofField(JsonFormat.java:1708)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeField(JsonFormat.java:1667)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1500)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1458)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.parseFieldValue(JsonFormat.java:2018)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeField(JsonFormat.java:1669)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1500)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1458)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.parseFieldValue(JsonFormat.java:2018)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeField(JsonFormat.java:1669)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1500)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1458)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.parseFieldValue(JsonFormat.java:2018)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeOneofField(JsonFormat.java:1708)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeField(JsonFormat.java:1667)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1500)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1458)
	at app//com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1340)
	at app//com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:472)
	at app//com.dmg.work.config.temporal.ProtobufPayloadConverter.fromData(ProtobufPayloadConverter.java:124)
	at app//io.temporal.common.converter.PayloadAndFailureDataConverter.fromPayload(PayloadAndFailureDataConverter.java:95)
	at app//io.temporal.common.converter.PayloadAndFailureDataConverter.fromPayloads(PayloadAndFailureDataConverter.java:133)
	at app//io.temporal.internal.sync.SyncWorkflowContext.lambda$executeChildWorkflow$450d5096$1(SyncWorkflowContext.java:787)
	at app//io.temporal.internal.sync.SyncWorkflowContext$$Lambda$1025/0x00000001008ec040.apply(Unknown Source)
	at app//io.temporal.internal.sync.CompletablePromiseImpl.lambda$thenApply$2df5ef44$1(CompletablePromiseImpl.java:211)
	at app//io.temporal.internal.sync.CompletablePromiseImpl$$Lambda$911/0x000000010086f040.apply(Unknown Source)
	at app//io.temporal.internal.sync.CompletablePromiseImpl.lambda$handle$6a2a7e3d$1(CompletablePromiseImpl.java:220)
	at app//io.temporal.internal.sync.CompletablePromiseImpl$$Lambda$912/0x000000010086f440.apply(Unknown Source)
	at app//io.temporal.internal.sync.CompletablePromiseImpl.lambda$then$16b0e4cc$1(CompletablePromiseImpl.java:269)
	at app//io.temporal.internal.sync.CompletablePromiseImpl$$Lambda$913/0x000000010086e840.apply(Unknown Source)
	at app//io.temporal.internal.sync.CompletablePromiseImpl.invokeHandlers(CompletablePromiseImpl.java:279)
	at app//io.temporal.internal.sync.CompletablePromiseImpl.complete(CompletablePromiseImpl.java:168)
	at app//io.temporal.internal.sync.SyncWorkflowContext.lambda$executeChildWorkflow$3(SyncWorkflowContext.java:766)
	at app//io.temporal.internal.sync.SyncWorkflowContext$$Lambda$1051/0x0000000100929040.run(Unknown Source)
	at app//io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:103)
	at app//io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107)
	at app//io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:54)
	at app//io.temporal.worker.ActiveThreadReportingExecutor$$Lambda$866/0x000000010082c440.run(Unknown Source)
	at java.base@11.0.29/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base@11.0.29/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base@11.0.29/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base@11.0.29/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base@11.0.29/java.lang.Thread.run(Thread.java:829)

 Other workflow threads: [
workflow-method-workflow_id_in_replay-run_id_in_replay
	at java.base@11.0.29/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@11.0.29/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
	at java.base@11.0.29/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
	at app//io.temporal.internal.sync.WorkflowThreadScheduler.yieldLocked(WorkflowThreadScheduler.java:57)
	at app//io.temporal.internal.sync.WorkflowThreadContext.yield(WorkflowThreadContext.java:90)
	at app//io.temporal.internal.sync.WorkflowThreadImpl.yield(WorkflowThreadImpl.java:393)
	at app//io.temporal.internal.sync.WorkflowThread.await(WorkflowThread.java:47)
	at app//io.temporal.internal.sync.CompletablePromiseImpl.getImpl(CompletablePromiseImpl.java:85)
	at app//io.temporal.internal.sync.CompletablePromiseImpl.get(CompletablePromiseImpl.java:75)
	at app//io.temporal.internal.sync.ChildWorkflowStubImpl.execute(ChildWorkflowStubImpl.java:90)
	at app//io.temporal.internal.sync.ChildWorkflowInvocationHandler.invoke(ChildWorkflowInvocationHandler.java:91)
	at app//com.sun.proxy.$Proxy135.executeWorkflow(Unknown Source)
	at app//com.dmg.work.worklifecycle.workflow.impl.WorkLifeCycleWorkflowImpl.triggerMarketplaceLifecycleChildWorkflow(WorkLifeCycleWorkflowImpl.java:146)
	at app//com.dmg.work.worklifecycle.workflow.impl.WorkLifeCycleWorkflowImpl.execute(WorkLifeCycleWorkflowImpl.java:70)
	at java.base@11.0.29/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base@11.0.29/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base@11.0.29/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base@11.0.29/java.lang.reflect.Method.invoke(Method.java:566)
	at app//io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:381)
	at app//io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:353)
	at app//io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:71)
	at app//io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:143)
	at app//io.temporal.internal.sync.SyncWorkflow$$Lambda$881/0x0000000100850840.run(Unknown Source)
	at app//io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:103)
	at app//io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107)
	at app//io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:54)
	at app//io.temporal.worker.ActiveThreadReportingExecutor$$Lambda$866/0x000000010082c440.run(Unknown Source)
	at java.base@11.0.29/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base@11.0.29/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base@11.0.29/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base@11.0.29/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base@11.0.29/java.lang.Thread.run(Thread.java:829)
]

	at io.temporal.internal.sync.WorkflowThreadContext.runUntilBlocked(WorkflowThreadContext.java:261)
	at io.temporal.internal.sync.WorkflowThreadImpl.runUntilBlocked(WorkflowThreadImpl.java:302)
	at io.temporal.internal.sync.DeterministicRunnerImpl.runUntilAllBlocked(DeterministicRunnerImpl.java:229)
	at io.temporal.internal.sync.SyncWorkflow.eventLoop(SyncWorkflow.java:215)
	at io.temporal.internal.replay.ReplayWorkflowExecutor.eventLoop(ReplayWorkflowExecutor.java:105)
	at io.temporal.internal.replay.ReplayWorkflowRunTaskHandler$StatesMachinesCallbackImpl.eventLoop(ReplayWorkflowRunTaskHandler.java:410)
	at io.temporal.internal.statemachines.WorkflowStateMachines.eventLoop(WorkflowStateMachines.java:722)
	at io.temporal.internal.statemachines.WorkflowStateMachines.access$700(WorkflowStateMachines.java:53)
	at io.temporal.internal.statemachines.WorkflowStateMachines$WorkflowTaskCommandsListener.workflowTaskStarted(WorkflowStateMachines.java:1321)
	at io.temporal.internal.statemachines.WorkflowTaskStateMachine.handleCompleted(WorkflowTaskStateMachine.java:139)
	at io.temporal.internal.statemachines.FixedTransitionAction.apply(FixedTransitionAction.java:46)
	at io.temporal.internal.statemachines.StateMachine.executeTransition(StateMachine.java:159)
	... 67 more

*Environment:**
- Temporal SDK: `1.26.2` (Java)
- Spring Boot: `2.6.8`
- JUnit 5 
- CI: GitHub Actions (Linux, on-prem runners)
- Local: macOS (Apple Silicon/Intel)

## The Issue

### Test Code (Sanitized)

**Base Test Class:**
```java
@Slf4j
public abstract class BaseWorkflowReplayTest {
    
    protected TestWorkflowEnvironment testEnv;
    private static final boolean IS_CI_ENVIRONMENT = 
        Boolean.parseBoolean(System.getProperty("CI", "false"));
    
    @BeforeEach
    void setupTestEnvironment() {
        if (IS_CI_ENVIRONMENT) {
            log.info("Running in CI environment - using CI-optimized configuration");
        }
        
        // Create WorkflowClientOptions with custom protobuf data converter
        WorkflowClientOptions clientOptions = WorkflowClientOptions.newBuilder()
                .setDataConverter(ProtobufPayloadConverter.createDataConverter())
                .build();
        
        // Create TestEnvironmentOptions with the custom client options
        TestEnvironmentOptions testOptions = TestEnvironmentOptions.newBuilder()
                .setWorkflowClientOptions(clientOptions)
                .build();
        
        // Create TestWorkflowEnvironment with custom options
        testEnv = TestWorkflowEnvironment.newInstance(testOptions);
    }
    
    @AfterEach
    void tearDown() {
        if (testEnv != null) {
            testEnv.close();
        }
    }
    
    protected void validateReplay(Class<?> workflowImpl, String eventHistoryResource) 
            throws Exception {
        long startTime = System.currentTimeMillis();
        log.info("Validating replay for workflow: {} (CI: {})", 
                workflowImpl.getSimpleName(), IS_CI_ENVIRONMENT);
        
        try {
            // Load event history JSON and convert to WorkflowExecutionHistory
            String eventHistoryJson = loadEventHistoryFromResource(eventHistoryResource);
            WorkflowExecutionHistory workflowExecutionHistory = 
                WorkflowExecutionHistory.fromJson(eventHistoryJson);
            
            // Create a Worker with the custom data converter
            io.temporal.worker.Worker worker = testEnv.newWorker("test-task-queue");
            worker.registerWorkflowImplementationTypes(workflowImpl);
            
            // Execute replay validation - THIS IS WHERE IT FAILS IN CI
            WorkflowReplayer.replayWorkflowExecution(workflowExecutionHistory, worker);
            
            long duration = System.currentTimeMillis() - startTime;
            log.info("Replay successful (duration: {}ms)", duration);
        } catch (Exception e) {
            long duration = System.currentTimeMillis() - startTime;
            log.error("Replay failed after {}ms (CI: {})", duration, IS_CI_ENVIRONMENT, e);
            throw e;
        }
    }
}
```

**Concrete Test Class:**
```java
@Slf4j
@Execution(ExecutionMode.SAME_THREAD)  // Force sequential execution
@ResourceLock("temporal-workflow-replay") // Prevent parallel execution
public class MyWorkflowReplayTest extends BaseWorkflowReplayTest {
    
    @Test
    @DisplayName("Should detect non-deterministic changes")
    public void test_workflow_non_deterministic_detection() throws Exception {
        // This test fails in CI with PotentialDeadlockException
        validateReplay(MyWorkflowImpl.class, "workflow-events.json");
    }
}
```

**Workflow Implementation (Sanitized):**
```java
@Slf4j
@WorkflowImpl(taskQueues = "my-task-queue")
@RequiredArgsConstructor
public class MyWorkflowImpl implements IMyWorkflow {
    
    private WorkflowStateDto workflowState;
    
    // Activity stub with proper configuration
    private final IMyActivity myActivity = Workflow.newActivityStub(
        IMyActivity.class, 
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofMinutes(1))
            .setRetryOptions(RetryOptions.newBuilder()
                .setMaximumAttempts(3)
                .setBackoffCoefficient(2.0)
                .setInitialInterval(Duration.ofSeconds(2))
                .setMaximumInterval(Duration.ofSeconds(30))
                .build())
            .build()
    );
    
    @Override
    public void execute(MyWorkflowRequest request) {
        try (MDCCloseable ignored = MDC.putCloseable("workflowType", "my-workflow")) {
            log.debug("Starting workflow");
            
            // Call activity to create child workflow request
            ChildWorkflowRequest childRequest = myActivity.createChildWorkflowRequest(request);
            
            // Configure child workflow options
            ChildWorkflowOptions childOptions = fetchChildWorkflowOptions(request, "child-task-queue");
            
            if (Objects.isNull(childOptions)) {
                log.error("Child workflow options are null");
                throw new WorkflowException("Unable to configure child workflow");
            }
            
            // Create and execute child workflow
            IChildWorkflow childWorkflow = Workflow.newChildWorkflowStub(
                IChildWorkflow.class, 
                childOptions
            );
            
            ChildWorkflowResponse response = childWorkflow.execute(childRequest);
            log.info("Child workflow executed, response: {}", response);
            
            // Process response via activity
            myActivity.processResponse(response, request.getId(), workflowState);
            
        } catch (ActivityFailure | ChildWorkflowFailure ex) {
            log.error("Error executing activity/child workflow: {}", ex.getMessage(), ex);
            throw ex;
        } catch (Exception ex) {
            log.error("Error in workflow: {}", ex.getMessage(), ex);
            throw ex;
        }
    }
    
    private ChildWorkflowOptions fetchChildWorkflowOptions(
            MyWorkflowRequest request, String taskQueue) {
        // Uses Temporal's Workflow.getInfo() and Duration - all deterministic
        return ChildWorkflowOptions.newBuilder()
            .setWorkflowId(generateWorkflowId(request))
            .setTaskQueue(taskQueue)
            .setWorkflowExecutionTimeout(Duration.ofMinutes(30))
            .build();
    }
}
```

What We’ve Verified

Workflow is deterministic:

No Thread.sleep(), System.currentTimeMillis(), UUID.randomUUID(), Math.random()
No new Date(), LocalDateTime.now(), Instant.now()
All timing uses Duration API
All I/O delegated to activities
Child workflows created via Workflow.newChildWorkflowStub()
Activity stubs created via Workflow.newActivityStub()

Event history is valid:

Tests pass locally with same event history JSON
Successfully replays on macOS consistently

Attempted Solutions

Verified workflow determinism
Still failing in CI
Test runs successfully each time on local but on CI it is breaking each time.

Questions

Is there a known issue with WorkflowReplayer.replayWorkflowExecution() in resource-constrained environments?
- CI runners have different CPU/memory characteristics vs local machines
Should we configure TestWorkflowEnvironment differently for CI?
- Are there timeouts or thread pool settings we should adjust?
Is the custom ProtobufPayloadConverter potentially causing issues?
- We use protobuf for workflow payloads - could this interact poorly with replay in CI?
Are there JVM flags or test container configuration changes we should make for CI?
- Different GC behavior, heap settings, or containerization in CI?
Could JaCoCo code coverage instrumentation interfere with Temporal’s workflow replay?
- JaCoCo is active during tests in CI (though we have exclusions configured)

Expected vs Actual

Expected: Replay test should pass in CI just as it does locally

Actual: PotentialDeadlockException thrown exclusively in CI environment during WorkflowReplayer.replayWorkflowExecution()

Any insights on what might cause environment-specific replay failures would be greatly appreciated! Has anyone else experienced similar issues with Temporal replay tests in CI environments?

antonio.perez · November 10, 2025, 5:40pm

Hi

The DataConverter (ProtobufPayloadConverter.fromData) is doing a lot of protobuf reflection and class loading, which is slowed down even more by JaCoCo instrumentation. This resource constrained environments can contribute to triggering the deadlock detector.

Is there a known issue with WorkflowReplayer.replayWorkflowExecution() in resource-constrained environments?

it is not only replay workflow, any code doing CPU-bound calls or overloaded workers

github.com/temporalio/rules

rules/TMPRL1101.md

main

# TMPRL1101 - Deadlock detected during workflow run

## Cause

Workflow code took too long to yield to Temporal.

## Description

When an SDK detects that workflow code is running too long (often either 1 or 2 seconds) before waiting on Temporal, a
deadlock detection error can occur. This can happen for multiple reasons:

* Code is stuck in a busy loop (e.g. an infinite for loop)
* Code is doing heavy CPU-bound calls in a workflow (e.g. running heavy cryptography)
* Code is accidentally blocking the coroutine with a synchronous/non-deterministic call (e.g. blocking sleep)
* Worker is overloaded which means CPU is blocked.
* Data conversion is taking too long (e.g. a codec is calling an external encryption service)

For healthy execution of the system, workflow code should only run for a very short time before deferring back to
Temporal. In most cases this is only a few milliseconds.

This file has been truncated. show original

Should we configure TestWorkflowEnvironment differently for CI?

You can change the default deadlock timeout

    Worker worker = factory.newWorker(TASK_QUEUE, 
            WorkerOptions.newBuilder().setDefaultDeadlockDetectionTimeout(..).build());

or disable it in your data converters with


WorkflowUnsafe.deadlockDetectorOff(() -> {
    // your blocking or IO code here
});

Is the custom ProtobufPayloadConverter potentially causing issues?
I think so
Are there JVM flags or test container configuration changes we should make for CI?
More resources is the only thing that comes to mind
Could JaCoCo code coverage instrumentation interfere with Temporal’s workflow replay?
I think so as per the stack trace

Topic		Replies	Views
WorkflowReplayer.replayWorkflowExecution throwing NonDeterministicException when reading workflow id from Workflow Community Support java-sdk	5	650	May 8, 2024
PotentialDeadlockException, when does this happen? Community Support java-sdk	7	870	September 28, 2024
How to write junit test cases in java Community Support java-sdk	6	1190	August 17, 2023
Potential deadlock detected Community Support java-sdk	4	4670	December 2, 2022
Non Determinism Error Community Support java-sdk	13	1039	May 30, 2023

`PotentialDeadlockException` when running Temporal workflow replay tests **exclusively in GitHub Actions CI environment** (Linux runners). The same tests pass consistently on local macOS development machines

What We’ve Verified

Attempted Solutions

Questions

Expected vs Actual

Related topics

`PotentialDeadlockException` when running Temporal workflow replay tests exclusively in GitHub Actions CI environment (Linux runners). The same tests pass consistently on local macOS development machines