Activity doesn't detect own timeout

Community Support
,
1

Hi.

One of the Temporal use-cases for me is task execution. It’s when you have a one-liner workflow which synchroniously calls activity with some business logic inside. I understand that it’s not a primary use-case, but anyway. I use Java.

I have a problem with activity timeout. Let’s say I call an activity with start to close timeout set to 1 sec. Activity running longer, 3 seconds for example. Workflow fails on ui and client side after 1 sec (as expected), but on a worker side it will execute until it finishes.

Another example, developer made a mistake and one of the instructions inside activity starts to perform much slower (let’s say database fetch for example). Worker thread will continue execution even after activity timeout and together with excessive retry policy can blocks multiple threads on multiple workers. It ultimately may stop queue from processing.

I tried heartbeat but it doesn’t fail activity (at least with clear message). Moreover it sounds weird for me to insert a heartbeat between every n lines of code to know that activity is timed out.

2024-05-27 19:54:26.804  INFO --- c.p.playground.perf.PingWorkflowImpl     : Workflow started
2024-05-27 19:54:26.847  INFO --- c.p.playground.perf.PingActivityImpl     : Activity start
2024-05-27 19:54:27.853  INFO ---  c.p.playground.perf.PingActivityImpl     : tick
2024-05-27 19:54:28.866  INFO --- c.p.playground.perf.PingActivityImpl     : tick
2024-05-27 19:54:28.866 ERROR --- c.p.playground.perf.PingWorkflowImpl     : workflow failed

io.temporal.failure.ActivityFailure: Activity with activityType='Ping' failed: 'Activity task timed out'. scheduledEventId=5, startedEventId=6, activityId=ed2c8ddd-c9fa-39ef-a874-68e8a75f93a4, identity='', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
Caused by: io.temporal.failure.TimeoutFailure: message='activity StartToClose timeout', timeoutType=TIMEOUT_TYPE_START_TO_CLOSE

2024-05-27 19:54:28.867  INFO --- c.p.playground.perf.PingWorkflowImpl     : Workflow finished
2024-05-27 19:54:29.871  INFO --- c.p.playground.perf.PingActivityImpl     : tick
2024-05-27 19:54:29.872 ERROR --- c.p.playground.perf.PingActivityImpl     : activity failed

io.temporal.client.ActivityNotExistsException: WorkflowId=f3d24b43-5d6e-4d8b-907f-ae8e2207b287, RunId=9261ed2b-8a76-40d2-8e28-21cd26d5e089, ActivityType=Ping, ActivityId=ed2c8ddd-c9fa-39ef-a874-68e8a75f93a4
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: workflow execution already completed
2024-05-27 19:54:29.873  INFO --- c.p.playground.perf.PingActivityImpl     : Activity end

For my use case it’s crucial to terminate abstract activity timely according to timeout. What am I doing wrong?

Thanks.

2

Java doesn’t provide a clean way to terminate a thread. So, the Temporal SDK cannot do much about activity code, which takes longer than its timeout. You have to implement your activity logic to not execute longer than the timeout. To get a timeout value for an activity use:

Activity.getExecutionContext().getInfo().getStartToCloseTimeout()
3

Thanks @maxim,

One of the popular ideas to implement this is to wrap code into thread (future) and terminate it after some timeout. Obviously it’s not ideal, but it’s something.

ActivityExecutionContext will be wiped for new thread, but we want to have an ability to use it (for hearbeating for example). API to propagate it to newly created thread is private.

It doesn’t look like SDK designed to do something like this smoothly, but I feel that this is very popular request.

Any advices/ideas how it can be integrated?

4

Would you point me to the Java documentation explaining how to safely terminate a Java thread?