Handling activity retry timeouts during automated manifest verification workflows

I am currently setting up a Temporal workflow to automate build verification and asset synchronization for our PojavLauncher release pipeline. The workflow runs on a schedule to validate distribution packages, ping mirror servers, and ensure the latest client profiles are properly synced.

The problem occurs inside an activity responsible for polling update files directly from our website under unpredictable network loads. When the remote endpoint experiences intermittent rate limiting or gateway timeouts, the activity fails to fail over gracefully to our fallback activity, causing the entire execution history to bloat with continuous retry attempts despite having defined a maximum retry limit.

Could this be related to how activity timeouts interact with the underlying HTTP client context, or should I be handling this pattern using child workflows instead? Any insights into properly structuring retry intervals and heartbeat timeouts for external API health checks would be very helpful.

You shouldn’t go for a child workflow for this. You say max retry limit is not obeyed, but are those retries from same workflow execution? Maybe executions are overlapping or they are fired one after other in very short time or you haven’t set retry options correctly.

For the fallback, put a finite Schedule-to-Close timeout on the polling Activity. It bounds the whole Activity execution, including retries and backoff. Set Start-to-Close for one attempt, and set finite maximum attempts if you also want a retry-count limit. The workflow can catch the final Activity failure and invoke the fallback, but it won’t get that failure while Temporal is still retrying. Allow enough overall time for the attempts and backoff you want, since the overall timeout can expire first.

Choose the initial interval, backoff coefficient and maximum interval based on how quickly the endpoint recovers and how much load it can tolerate. Longer delays reduce pressure but postpone fallback. Set the HTTP client’s own deadline within the per-attempt timeout so a stalled request returns promptly. A heartbeat timeout is useful for a long-running Activity that actually heartbeats; it isn’t a substitute for an HTTP deadline. Check the failure type and effective Activity options in the execution history too, especially if the observed retries don’t match the configured budget.