I have a workflow signal that is invoked in an at-least once manner. To support this, I want my workflow to deduplicate identical signals before processing them.
Consider the following example (Kotlin):
val signalQueue: WorkflowQueue<Signal> = Workflow.newWorkflowQueue()
val signalHistory: MutableSet<Signal> = mutableSetOf()
@SignalMethod
override fun signal(signal: Signal) {
if (signalHistory.add(signal)) {
log.info("Publishing new signal to the queue")
signalQueue.put(signal)
} else {
log.warn("Ignoring duplicate signal")
}
}
(Here I am just relying on equals() to determine duplicates but I could also use an explicit idempotency key.)
My question: Is there a risk that a Temporal worker could effectively lose a signal if it crashed after persisting the state of signalHistory? Or would the boolean returned by the add() call also be guaranteed to be persisted?
And perhaps a broader, conceptual question is whether Temporal workflow instructions are guaranteed to be executed at-least-once or exactly-once? And how does this apply to functions that may trigger multiple machine instructions? How are such functions replayed in the event of a worker crash?
Is there a risk that a Temporal worker could effectively lose a signal if it crashed after persisting the state of signalHistory ?
Signals are written in workflow history if request is accepted by Temporal server. After a worker crash, execution history replay would replay each received signal and worker would invoke your signal handler to put the workflow state back to the original before the crash.
whether Temporal workflow instructions are guaranteed to be executed at-least-once or exactly-once ?
If you are asking for activity invocation, it’s done at least once (by default). Activities have retry policy where you can set max attempts to 1 to disable retries which would make your activity invocations at most one time.
For workflows, conceptually your workflow method is going to get executed exactly once. Under the hood (since workflow executions are not tied to a particular worker) workers replay workflow history (when needed) in order to reconstruct your workflow state so your workflow classes can be instantiated many times and your workflow method execute many times.
I guess I’m struggling to understand what level of granularity is used to persist the workflow history. If I call a Java method that results in two machine instructions, and the worker crashes in between the two instructions, what happens? Or if I call a method that in-turn calls two Java methods, and the worker crashes in between, what happens? Can Temporal resume execution mid-function?
If a Worker crashes the work will be placed in a new Worker (if available) that will replay the event history until the point the execution was left, starting from the very beginning.
If the activity was completed successfully, it wouldn’t be executed again, the activity result will be retrieved from the event history.
If I call a Java method that results in two machine instructions, and the worker crashes in between the two instructions, what happens? Or if I call a method that in-turn calls two Java methods, and the worker crashes in between, what happens?
What do you mean with Java method? Do you mean a method inside the workflow?
When the new worker replays the event history, the workflow code/methods will be called again with the same parameters (if it has any). The workflow state (variables) is recreated when the Worker replays the workflow history, and every variable value will be restored to the value that it had before the crash.
Keep in mind that your code has to be deterministic. Please see
Thank you! I think the important concept to realize is that the workflow state is NOT restored via serializing a persisted state, it’s restored via replaying persisted history events (which then results in a deterministic state), so there should be no risk in the example I provided.