Hey everyone, I am currently building a Temporal workflow designed to automate hardware bench provisioning, specifically coordinating the flashing of raw disk images across multiple connected drives using custom workers. We chose Temporal to manage the state tracking and automated retries, but I am running into a roadblock where the activity executing the raw disk write command blocks indefinitely and fails to emit a heartbeat.
While troubleshooting the CLI parameters and standard exit codes mentioned on the website for the disk imaging tool, it looks like bad sectors or sudden I/O dropouts cause the external utility to freeze completely without returning a standard error. Temporal eventually triggers an activity timeout as expected, but the automatic retry fails right away because the operating system still holds a file lock on the target device handle from the orphaned process.
Has anyone tackled a reliable cancellation and zombie process termination pattern inside Temporal activities when dealing with low-level disk tools? I would love some insight on how to cleanly tear down the external process and release the drive handles before the workflow schedules the next attempt.
Heartbeating is the key piece here. Run the disk tool as a child process from the activity and heartbeat on a timer while it runs, with a short HeartbeatTimeout, so Temporal notices a frozen tool well before StartToClose expires. When the activity gets cancelled or times out, kill the whole process group (not just the parent) in a finally/cleanup block and wait for it to exit before returning. For stubborn device handles, a cleanup step at the start of each retry that checks for and kills leftover processes holding the device can make the next attempt safe.
There’s a separate ownership issue across attempts: a Temporal timeout or cancellation doesn’t prove that the old OS process has stopped or released the physical device. A retry can run on another worker while the old process is still alive. Temporal’s Activity idempotency guidance also notes that an activity can complete its side effect and then be retried if the worker fails before reporting completion.
Consider a durable per-device lease with an ownership generation, enforced by the component that controls access to the device. A retry should acquire ownership before touching the device, and a stale owner must be prevented from issuing further I/O. Lease expiry alone isn’t sufficient if an already-running raw disk process can continue writing without checking a fence. If the old process cannot be proven stopped and access cannot be fenced, keep the device quarantined rather than hand it to a new attempt. Release ownership only after the process has exited and the handle is confirmed released, or after an independently enforced fence makes further access by the old owner impossible. Treat the device’s contents as potentially partial and reconcile them before retrying.