Java SDK test environment not shutting down properly when using heartbeating

Community Support
,
1

I’m testing a workflow that executes exactly one activity and does nothing else. This activity uses heartbeating. We’re on version 1.13 of the Java SDK.

At the end of the test, I call shutdown on the TestEnvironment followed by awaitTermination with a 5 second timeout. The awaitTermination call waits the 5 seconds, then times out. The following log line is also emitted:

io.tem.int.wor.ShutdownManager : Wait for a graceful shutdown of SyncActivityWorker{namespace=workflow-unit-test, taskQueue=test:zoolander:DEFAULT, identity=13812@st-jeffschoner2}#heartbeatExecutor timed out, fallback to shutdownNow()

There are more errors logged (that do not affect the test) of the form:

Jul 26, 2022 5:11:45 AM io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=7, target=directaddress:///17552345-d334-4bf0-974e-49002c2f8506} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.

If I comment out the heartbeat call in the activity, the awaitTermination call nearly instantly returns. Because we use a 5 second timeout, this behavior has the effect of adding 5 seconds to every test method, which substantially slows down otherwise fast tests.

There are other workflows with heartbeating activities that we test, that don’t exhibit this waiting behavior or the additional log line, but it’s not clear to me what might be meaningfully different with this workflow or activity.

Any ideas of what might be causing this, or ways to mitigate it?

3

@jeffschoner there is an open issue Cached activity heartbeat prevents worker from a quick shutdown · Issue #1257 · temporalio/sdk-java · GitHub that might be related to the problem you are facing.

4

Thanks for the pointer. If I set setMaxHeartbeatThrottleInterval on the WorkerOptions I pass to the TestEnvironment.newWorker, to a smaller duration, runtimes are dramatically improved. I believe this will effectively eliminate heartbeat throttling against the test service, but I wouldn’t expect this to be a problem since it is all running locally.