What happens if my system is backlogged and scheduleToStart timeout happens for a task? is it just for metric purposes?
I have a long running activity with startToCloseTimeout=30 mins. Currently since i dont do any heartbeating, Do i need to set heartbeat timeout as well?
Do you have any recommendation/java examples for how to heartbeat for a long running process? I cannot hearbeat in the middle, so i may need to wrap it with another thread that heartbeats.
What happens if my system is backlogged and scheduleToStart timeout happens for a task? is it just for metric purposes?
The activity is going to fail and the timeout is going to be delivered to the workflow for handling. In the majority of the situations, we recommend not setting scheduleToStart at all (which defaults it to the workflow run timeout) to avoid failures in intermittent failure scenarios. The mean use for a short scheduleToStart timeout is to detect that a certain host is down in the scenarios when an activity is routed to a specific host.
I have a long running activity with startToCloseTimeout=30 mins. Currently since i dont do any heartbeating, Do i need to set heartbeat timeout as well?
Temporal doesn’t require using a heartbeat for long-running activities. But by using it you get the following benefits:
Without the heartbeat if an activity worker restarts the Temporal service is going learn about the failure only through the StartToClose timeout. So in your case the activity is not going to be restarted or handled by the workflow up to 30 minutes. When the heartbeat is enabled the timeout is going to happen after a heartbeat timeout which is usually pretty short.
Activity cancellation is supported only for activities that heartbeat. The cancellation request is delivered in the response to the heartbeat call. So if you need to cancel a long-running activity ensure that it heartbeats.
Do you have any recommendation/java examples for how to heartbeat for a long running process? I cannot hearbeat in the middle, so i may need to wrap it with another thread that heartbeats.
Currently wrapping it in another thread is the best option in your case. I don’t think we have a sample for this. If you have a lot of parallel activities consider using a Java timer to avoid using too many threads just for hreartbeating.
In this case I assume the activity task will be retried as per the activity retry options?
For my usecase i place a limit on maxConcurrentExecutions for activities because of resource limitations and an increased scheduleToStart is expected when there is a new batch of requests. So we are fine with activities not getting scheduled for 15-30 mins and would rather it doest timeout.
ScheduleToStart timeout is not retried as it doesn’t make sense to retry it. The retry would take the task from the task queue and put it back into the same queue.