Exception on compensation

In my project, we are starting to add some error handling to our workflows and we came up using the Saga mechanism and adding compensations whenever it was needed.

private var sagaOpts = new Saga.Options.Builder().setParallelCompensation(true).build();
final var saga = new Saga(SAGA_OPTS);
try {
      activity.operation1();
      saga.addCompensation(activity::operation1Compenstation);

      activity.operation2();
      saga.addCompensation(activity::operation2Compenstation);

       ...
} catch (ActivityFailure e) {
       saga.compensate();
       throw e;
}

After some testing, we tested what happens if the compensations fail (configuring the saga options to run in parallel and also to not continue on error).

What we are seeing is that after the compensation fails and all retries are reached, in the Temporal UI appears a new event WorkflowTaskFailed in the History tab.

io.temporal.internal.replay.InternalWorkflowTaskException: Failure handling event 27 of 'EVENT_TYPE_WORKFLOW_TASK_STARTED' type. IsReplaying=false, PreviousStartedEventId=27, workflowTaskStartedEventId=27, Currently Processing StartedEventId=27
....
Caused By: java.lang.RuntimeException: WorkflowTask: failure executing SCHEDULED->WORKFLOW_TASK_STARTED, transition history is [CREATED->WORKFLOW_TASK_SCHEDULED]
...
Caused By: io.temporal.workflow.Saga$CompensationException: Exception from saga compensate 

The issue is the workflow stays in RUNNING state forever, and I have a couple questions:

  • Is this the expected behavior?
  • Is there any documentation I can read regarding this behavior? I wasn’t able to find any.

By default, workflows get blocked instead of failing on any exceptions that don’t extend TemporalException. This is done to avoid failing long-running workflows on unexpected bugs/failures. This gives the chance for fixing the bug without dealing with potentially millions of failed workflows.

If you want the workflow to fail on Saga$CompensationException you have two options.

The first one is to catch this exception and rethrow it as ApplicationFailure (which always fails a workflow).

The second one is two add the Saga$CompensationException to the list of exceptions that fail the workflow. This list is specified through the WorkflowImplementationOptions.

1 Like