Update activity and/or workflow inputs?

Hi,

I’m currently deploying Temporal on production, and someone from my team raised an interesting question about a common feature we’re used to have with our old homemade workflow orchestrator that I can’t find in Temporal.

Let’s imagine a workflow with 5 activities.
The last activity failed because there is a missing input parameter in the workflow.

It should have been checked before launching the activities or the workflow but it’s too late and the workflow is already 80% done.

Is there a way to change the input parameters for the workflow, or for the failed activity to inject the missing input ?
I don’t want to start a new workflow execution from scratch as I don’t want to execute again the first four activities, I’d prefer the restart the failed activity with proper input.

I had a look into workflow replayer and thought about modifying the workflow history json file but I definitely don’t feel comfortable about this :smile:

Thanks

1 Like

The last activity failed because there is a missing input parameter in the workflow.

Activities have automatic retries. You can update/change your activity code to fix the error if possible, and then redeploy workers without breaking workflow determinism. Your updated activity code would then be executed on the next scheduled activity retry.

You mention this activity failed is it because you restricted activity retries in config?
You should be able to also change activity options (config) without breaking workflow determinism and it would be applied to activity invocations which have not be performed yet.

Is there a way to change the input parameters for the workflow, or for the failed activity to inject the missing input ?

If your workflows have already completed you can fix the activity code to account for the missing workflow input and then use reset feature to reset the workflow to the WorkflowTaskStarted event right before the activity invocation in your history.
Note that this will create a new execution.
Reset would replay your workflow history up to the reset point, so your already executed activities would not be re-executed again, and it then creates a new workflow execution that continues exec of your workflow code from the defined reset point onwards.

I had a look into workflow replayer

WorkflowReplayer is only used for debugging and backwards compatibility testing and yes I would not manipulate the json history, just use the reset feature mentioned if possible.

Thanks for the answer !

I’ll try with a simple example to help you understand my need.

Let’s imagine a simple workflow made of 3 activities. The first two ones performs some database operations, and the last activity send an email to an email address taken from the workflow input parameters.

When writing the workflow, the developer forgot to check the email input.

When reaching activity 3, after successfully ended activity 1&2, the last activity failed because of a NonRetryable error: the email address is missing from the workflow input, so for the activity input.
Let’s say we are in production and we have 10 workflow executions failed with customers waiting for the execution to complete.

The obvious fix for the future workflow executions is to check the input before starting any activity (or even better, before launching the workflow).

But how to fix the already launched and failed workflow executions ? We’re talking here about some RUN operations to fix a failed activity.

Fixing the activity code to allow empty inputs does not fit to my need as I don’t have a “default value” for email address. In our current run operations, I’d look into each failed workflow execution to determine manually to proper value to set for email address.

In our homemade workflow orchestrator, we are manually going into the database to update the input parameters of the step (“activity-like” concept), so the next run of the step will have the correct input.

I don’t want to restart a new workflow with the same input as I don’t want to execute again activity 1&2.
I had a look on the reset feature, it fits the need to reset the workflow without executing again activity 1&2, but I can’t change the worklow inputs with a reset.

Regarding how Temporal is working, it does not look easy to change inputs in an existing workflow execution, and the need I have is probably due to some old (bad ?) habits we have with our current homemade workflow orchestrator, but members of my team are pushing this feature as a mandatory one to properly move to Temporal.

We do plan to add the ability to override activity inputs and other options through an operator interface. But we don’t have ETA on this feature.

1 Like

In our homemade workflow orchestrator, we are manually going into the database to update the input parameters of the step (“activity-like” concept), so the next run of the step will have the correct input.

For now maybe an idea is to catch ActivityFailure in your workflow code and then wait for a signal with the updated/fixed values and then run the activity again using those as input.

And the signal workaround @tihomir proposed can be implemented generically as an interceptor.

Thanks for both answers.

Do you have a public roadmap to follow this kind of feature requests ?
I’ll not rely on it to go further with Temporal as I think it’s more a question of workflow design and going out of old habits we have with our current workflow orchestrator, but I’d love to follow the upcoming features.

Thanks for pointing out this workaround with signal usage, I’ll try to make a POC from this :slight_smile: