Idempotency across Workflow Reset: operation identity and durable evidence

Hi Temporal community,

I’m working on FCI (Formal Conservative Interposition), a research project about safe recovery before external actions. I would appreciate help identifying the strongest standard Temporal-based design to use as a comparison.

I have read the discussions on external-call idempotency, authorization/execution drift, and the recent Truth Engine experiment. I understand that an intermediary cannot guarantee exactly-once effects against a non-idempotent target that offers no reliable reconciliation.

My question concerns recovery across a Workflow Reset, rather than ordinary Activity retries.

Consider a logical business operation K:

  1. An Activity requests an external effect.
  2. The target may commit it, but the response is lost before Activity completion is recorded.
  3. An operator resets the Workflow to a point before that Activity.
  4. The new execution reaches the operation again.

For this scenario, Temporal’s persistence is not restored from a backup, and the target’s state remains intact. The reset is not assumed to erase the original execution’s history or undo external effects.

We want to test two outcomes:

  • If K already committed, recovery produces no second effect.
  • If K never committed, recovery can eventually complete it once the required services are available and authorization remains valid.

Would a business-operation ID preserved across runs, combined with target-side atomic deduplication and result lookup, be the recommended baseline?

In particular:

  • How would you preserve K across Reset while distinguishing a retry from a genuinely new business operation?
  • What atomicity and retention guarantees must the target provide?
  • Is there an existing sample or failure-injection test covering this sequence, including an earlier Activity attempt that may still be running?

If the target lacks those guarantees, identifying precisely when recovery must remain blocked or require reconciliation would also be valuable.

The goal is a reproducible comparison, including the possibility that standard mechanisms already provide everything this scenario requires.

Thank you for any corrections or references.

Alexandre Couret

Yeah, your downstream API has to be idempotent. If you reset to a workflow task before this activity was scheduled, new run that reset creates will schedule it again.
You can reset both closed and running executions. If you reset a running one while this activity is still pending (running, not completed yet), your downstream API can get called twice for the same request, possibly at the same time. A reset of a closed execution also calls it again just later.

If your downstream API allows you to pass a unique token as part of the request, pass that token in as workflow input, pass it to the activity as activity input, and send it with the request.
This should be unique per business operation but stay the same across retries and resets, so dont generate it inside the workflow (for example with a random UUID or from the run ID as its diff after reset).
API has to dedupe per token and remember the token long enough to cover a later reset (so longer than your namespace retention period set for namespace).

If the API cant do this, your activity code can first check whether the work was already done before making the request.
Note that check alone is not safe if two calls can run at the same time, as in the running-exec-reset case. Both can see “not done” and both make the request. In that case you might need to add a “human in loop” type of decision in your business logic if possible or always take the last result or similar.

Thank you, Tihomir. Your explanation is very helpful, particularly the possibility of concurrent downstream calls after resetting a running execution.

We are investigating a related boundary: what happens when the downstream effect has committed, but the durable evidence used for deduplication has been lost or rolled back independently of that effect?

In that situation, neither a stable operation token nor a check-before-call appears sufficient unless the downstream system retains authoritative evidence across the relevant recovery scenarios.

Our current approach is to distinguish three outcomes: confirmed prior commitment, confirmed absence of commitment, and insufficient evidence. In the third case, we would refuse automatic replay rather than risk duplicating an irreversible effect.

Does this distinction align with the recovery practices you recommend for Temporal integrations? Are there established patterns for handling this uncertainty when the downstream service cannot guarantee durable idempotency?

Thank you again for your insights.

Hi Tihomir,

Thank you for your clear explanation, especially the possibility of overlapping downstream calls after resetting a running Workflow. We will treat a stable business-operation key, passed in as input, and durable atomic deduplication at the receiver as the strongest conventional baseline. This is a downstream contract, not a guarantee that Workflow Reset itself can provide.

I have also opened a separate discussion about the stronger disaster-recovery case in which an application-side idempotency ledger is restored independently of an effect that survives:
https://community.temporal.io/t/recovery-after-an-idempotency-ledger-rollback-authoritative-evidence-and-replay-safety/19893

I would like to avoid conflating that with ordinary Workflow Reset, and ask a narrower question about what evidence actually authorizes retry when the receiver cannot supply the full idempotency contract.

Suppose operation K is stable across runs, the Activity receives no final response, and a downstream lookup returns NOT_FOUND. An older Activity attempt may still be executing. As you point out, check-before-call is not safe under concurrency: the older attempt might commit after the lookup. In practice, what recipient-side contract would make NOT_FOUND actionable? Would this require a linearizable operation registry coupled atomically to the effect, or could a receiver-enforced fencing generation combined with a reliable negative lookup suffice? I am assuming that cancellation or termination of a Temporal run alone cannot revoke a request already executing at the downstream service.

There is a related safety-versus-progress question. Reserving K durably before dispatch may prevent duplication, but a crash before the actual effect can leave an operation reserved without ever having completed. If no authoritative outcome can be recovered, is the recommended policy to retain an explicit UNKNOWN / RECONCILIATION_REQUIRED state rather than retry automatically? Are there established Temporal integration patterns for reconciling this state without either silently duplicating an effect or leaving every uncertain operation blocked indefinitely?

We are building a controlled failure-injection comparison that counts actual downstream commits separately from HTTP requests, and reports safe refusals separately from successful recovery. Our cases include lost acknowledgments, live Workflow Reset with an older attempt still active, crashes before and after commitment, and (as a distinct, stronger scenario) rollback of an application-side ledger. A standard Temporal integration and our FCI research prototype will receive the same external evidence and trust assumptions. We do not regard conservative refusal alone as an FCI-specific advantage.

If you know of a reference implementation or production pattern that captures this boundary, I would be very interested in using it as our baseline. Any correction to the failure model would be equally valuable.

Thank you again for your help.

Best regards,
Alexandre Couret

Yeah, you’re correct that terminating or resetting the workflow can’t cancel a pending activity. Reset on a running workflow terminates the old run, and the activity attempt keeps running on the worker.

Cancelling the workflow is a better option.
If workflow code receives cancel request, and pending activity heartbeats, it learns about the cancellation on its next heartbeat.
It can then stop before making the downstream call, or, if the call was already made, run logic to undo it (if the downstream service supports that). Note this still cant stop a downstream call that’s already in flight.

Activities learn about that they timed out similarly, when they comlete/fail or on next heartbeat after timeout, as you mentioned via NotFound response you mentioned.

If activity asks downstream if work was done and gets “not found” from it, answer is only safe to act on if this downstream guarantees an older attempt cant still land afterwards.
If downstream cant guarantee that the outcome is unknown.
In that case dont retry activity, but fail it with non-retryable application failure, run activity that notifies you (human in the loop pattern) and have workflow await signal from human decision on what to do.

Hi Tihomir,

Thank you for your 1 November explanation of cancellation, in-flight downstream calls and the human-in-the-loop pattern when an outcome remains unknown. We are treating those existing practices as the starting point for our FCI discovery study.

May I ask a practical follow-up? In integrations where the receiver cannot supply the full idempotency contract, does manual reconciliation represent a significant operational burden, or is it usually an acceptable exception? I would value public references or general experience about the frequency, investigation effort and team responsible, rather than another hypothetical failure scenario. No customer-specific information is needed.

A short written reply would be very helpful; alternatively, would someone from the relevant team be available for a 20-minute conversation?

General, non-confidential examples are sufficient. I am not requesting customer data, internal documents, source code or access to systems. This is a discovery study, not a product demonstration or a claim that your current safeguards are insufficient. If the existing approach fully addresses the issue, that is equally useful feedback.

Please feel free to direct me to the appropriate colleague. If you prefer no further contact, please let me know.

Kind regards,
Alexandre Couret
Independent research and development — France