Random Activity Timeouts

Community Support
1

We are noticing random activities timing out without anything being executed. Here’s a sample event stream.

Screenshot 2024-01-29 at 6.09.59 PM
Screenshot 2024-01-29 at 6.09.59 PM1775×114 41.1 KB

In the first line of the activity, we have added a logger message but don’t see the log getting posted.
The activity StartToClose timeout is 1 hour.

Am I missing something here?

2

There are failure scenarios when an activity task is lost before starting execution on the worker. Temporal has a hard guarantee that the activity task cannot be lost before the timer for its attempt has started. So, timeouts are the way to deal with all sorts of activity task delivery and activity worker failures.

The solution in your case is either to lower the StartToClose timeout to some shorter value or specify a short Heartbeat timeout. If you specify the heartbeat timeout, then the activity must call the heartbeat periodically without exceeding the specified timeout.

3

@maxim thank you for the response. Could you kindly elaborate on the scenarios where an activity task can end up in this state? Seems like its quite random and we did not notice this when there was not much load on the workers.

With regards to the potential solutions:

  1. Having a short StartToClose timeout is not feasible since we cannot really predict the duration for which the activity will run but want to have a hard stop at 1 hour.
  2. Heartbeats seem like a good way to indicate whether an activity is up and running. However, for our case - the activity makes a blocking call which runs for varied amounts of time. If we were to set a short heartbeat timeout, we will end up with a heartbeat timeout in the blocking step.

Is there any technique we can use in our scenario?

4

Wow, 1 hour blocking call?

In this case, you can heartbeat from a different thread. Usually, we don’t recommend doing this as it does not detect the stuck main activity thread, but in your case, it is the only option.

Could you kindly elaborate on the scenarios where an activity task can end up in this state?

There are many scenarios. For example, the activity implementation function is invoked, and the process crashes before it makes any progress.