I have a long-running activity which code can sometimes generate errors that should lead to a restart (for example, the activity detects that something has broken on the current worker, such as loss of network access to the resources the activity works with). Currently, before detecting such a situation, I save the activity’s intermediate work result in the code, send a heartbeat, and then generate an error.
In this scenario, will the heartbeat throttling limits apply, and is it possible for the heartbeat not to be delivered to the server before the error occurs? If the heartbeat might not be delivered to the server in such cases, what would you recommend using to forcibly save the activity’s state before an error, in order to resume work from the point of failure upon the next restart?