Preliminary investigation into idempotent signals

Temporal recommends making activities idempotent.

An inherent concern of any distributed system is that we might make a request to an external system, but then have a crash or a network failure before we get the response back of whether the request was received. At this point we don’t know whether the request did go through or not: the crash or failure might have occurred before or after the request was received. We can retry on failures (and in general will need to), but that might cause the request to be received twice.

If the request we’re making is to pay someone $1,000, we’d rather not have that be processed twice :wink:

One way that API’s handle this, and payment API’s in particular, is that the API may support an idempotency key. For an example, see Idempotent requests | Stripe API Reference

If the service receives another request with the same idempotency key as a previous request, the duplicate request can be simply ignored. Then we get “exactly once delivery”: we retry on failures so that a temporary failure doesn’t cause a request to be lost, but retries don’t cause any requests to be duplicated either.

What if we’re going the other way? We have an API, and we want to offer the client of our API the opportunity to pass us an idempotency key. Suppose our API sends a signal to one of our workflows. If a second request comes in with the same idempotency key we want it to be ignored.

@maxim pointed me to api/temporal/api/workflowservice/v1/request_response.proto at b4bdd8035cd1883aa96cbad5bb0e582850feea5f · temporalio/api · GitHub. The request_id can be used to deduplicate signals.

My understanding is that the Temporal SDK’s can use the request_id to automatically deduplicate retries from the same client.

I don’t think the Java SDK is doing this yet, at least as of 1.25.2, as far as I can tell.

Down in sdk-java/temporal-sdk/src/main/java/io/temporal/internal/client/RootWorkflowClientInvoker.java at master · temporalio/sdk-java · GitHub I can set the requestId, and over in the Temporal service temporal/service/frontend/workflow_handler.go at main · temporalio/temporal · GitHub I can see the request id coming through. I’m not seeing the unmodified Java SDK send a request id though.

The request id appears to be working as expected: if I send two signals with the same request id, only one signal is delivered to the workflow.

If I’m in the right place, the deduplication code appears to be here: temporal/service/history/api/signalworkflow/api.go at main · temporalio/temporal · GitHub

The request id doesn’t appear to be saved in the workflow event history; at least I’m not seeing it in the Temporal UI or with “temporal workflow show --output json --workflow-id …” In the code, the request id isn’t stored in the Workflow Execution Signaled Event, but is stored in the workflow’s “mutable state” (I’m not sure what that is): temporal/service/history/api/signalworkflow/api.go at main · temporalio/temporal · GitHub

I don’t know what the preferred API would be in the client SDK to send an idempotent signal. In the Java SDK when using an untyped workflow stub, the WorkflowStub interface has a signal method. If perhaps we might like to add an idempotentSignal method, that could take a signal name, the idempotent key, and then the signal arguments. It seems like it would be an easy task to thread that through to the internals where the request id would be set. I don’t know what we’d want to do with the typed workflow stubs to include a idempotency key option.