We have a few use-cases of long running activity. We wanted to make activity resumable in case of worker pods re-deployments.
You can freely record Heartbeats as often as you want (e.g. once a minute, or every time a loop iterates). The SDKs throttle the Heartbeats that get sent back to the Server anyway. Ref
Based on the above two points, I can see there might be possible uncertainty window here, where heartbeat sent by the activity might get lost in the event pod redeployment happens and graceful shutdown triggers. Is this assumption right ? Or does temporal’s SDK takes care of clearing off the heartbeat batch before killing the application in case of graceful shutdown ?
I believe in case of a graceful shutdown, the heartbeat is sent. But it doesn’t ensure that heartbeat is recorded as not all shutdowns are graceful. So data loss is indeed possible.