Update activity and/or workflow inputs?

Hi,

I’m currently deploying Temporal on production, and someone from my team raised an interesting question about a common feature we’re used to have with our old homemade workflow orchestrator that I can’t find in Temporal.

Let’s imagine a workflow with 5 activities.
The last activity failed because there is a missing input parameter in the workflow.

It should have been checked before launching the activities or the workflow but it’s too late and the workflow is already 80% done.

Is there a way to change the input parameters for the workflow, or for the failed activity to inject the missing input ?
I don’t want to start a new workflow execution from scratch as I don’t want to execute again the first four activities, I’d prefer the restart the failed activity with proper input.

I had a look into workflow replayer and thought about modifying the workflow history json file but I definitely don’t feel comfortable about this :smile:

Thanks

1 Like

The last activity failed because there is a missing input parameter in the workflow.

Activities have automatic retries. You can update/change your activity code to fix the error if possible, and then redeploy workers without breaking workflow determinism. Your updated activity code would then be executed on the next scheduled activity retry.

You mention this activity failed is it because you restricted activity retries in config?
You should be able to also change activity options (config) without breaking workflow determinism and it would be applied to activity invocations which have not be performed yet.

Is there a way to change the input parameters for the workflow, or for the failed activity to inject the missing input ?

If your workflows have already completed you can fix the activity code to account for the missing workflow input and then use reset feature to reset the workflow to the WorkflowTaskStarted event right before the activity invocation in your history.
Note that this will create a new execution.
Reset would replay your workflow history up to the reset point, so your already executed activities would not be re-executed again, and it then creates a new workflow execution that continues exec of your workflow code from the defined reset point onwards.

I had a look into workflow replayer

WorkflowReplayer is only used for debugging and backwards compatibility testing and yes I would not manipulate the json history, just use the reset feature mentioned if possible.

Thanks for the answer !

I’ll try with a simple example to help you understand my need.

Let’s imagine a simple workflow made of 3 activities. The first two ones performs some database operations, and the last activity send an email to an email address taken from the workflow input parameters.

When writing the workflow, the developer forgot to check the email input.

When reaching activity 3, after successfully ended activity 1&2, the last activity failed because of a NonRetryable error: the email address is missing from the workflow input, so for the activity input.
Let’s say we are in production and we have 10 workflow executions failed with customers waiting for the execution to complete.

The obvious fix for the future workflow executions is to check the input before starting any activity (or even better, before launching the workflow).

But how to fix the already launched and failed workflow executions ? We’re talking here about some RUN operations to fix a failed activity.

Fixing the activity code to allow empty inputs does not fit to my need as I don’t have a “default value” for email address. In our current run operations, I’d look into each failed workflow execution to determine manually to proper value to set for email address.

In our homemade workflow orchestrator, we are manually going into the database to update the input parameters of the step (“activity-like” concept), so the next run of the step will have the correct input.

I don’t want to restart a new workflow with the same input as I don’t want to execute again activity 1&2.
I had a look on the reset feature, it fits the need to reset the workflow without executing again activity 1&2, but I can’t change the worklow inputs with a reset.

Regarding how Temporal is working, it does not look easy to change inputs in an existing workflow execution, and the need I have is probably due to some old (bad ?) habits we have with our current homemade workflow orchestrator, but members of my team are pushing this feature as a mandatory one to properly move to Temporal.

We do plan to add the ability to override activity inputs and other options through an operator interface. But we don’t have ETA on this feature.

1 Like

In our homemade workflow orchestrator, we are manually going into the database to update the input parameters of the step (“activity-like” concept), so the next run of the step will have the correct input.

For now maybe an idea is to catch ActivityFailure in your workflow code and then wait for a signal with the updated/fixed values and then run the activity again using those as input.

And the signal workaround @tihomir proposed can be implemented generically as an interceptor.

Thanks for both answers.

Do you have a public roadmap to follow this kind of feature requests ?
I’ll not rely on it to go further with Temporal as I think it’s more a question of workflow design and going out of old habits we have with our current workflow orchestrator, but I’d love to follow the upcoming features.

Thanks for pointing out this workaround with signal usage, I’ll try to make a POC from this :slight_smile:

I’d also love to know how to achieve this.

One example is that the workflow relies on another service, which had a bug and returned the wrong response for a period of time. We don’t want to restart the whole workflow, which has completed half-way. Resetting to just before this activity is also not as ideal, because the system we rely on has some side effects.

Instead, we want to simply modify the inputs for the affected workflows and do a reset. Is there a way to achieve this?

Resetting to just before this activity is also not as ideal, because the system we rely on has some side effects.

Is it something you can handle with your activity code / business (workflow) logic / retries / rate limiting? Would like to get more info on the use case

we want to simply modify the inputs for the affected workflows and do a reset . Is there a way to achieve this?

Given the previous question, seems the validity of your third party service depends on the initial activity inputs?

I can give a real example we’re trying to resolve:

  1. Service A calls service B (third party) to OpenAccount
  2. Due to a bug, some incorrect input was provided to B, which we reconcile and fix through other means (eg some operators fix it in their own system). Or it could just be that B returned an incorrect response at that time, saying that the account can’t be opened, when it in fact did open. In both cases, retrying the call to the third party service B is not ideal, as they are not idempotent
  3. Ideally, we would modify the input to the next activity (which contained the incorrect response from B) and rerun the step

I have a similar case. Mine is very similar to Zhiwei’s, but in my case I’m storing a reference to an upstream request ID.

Imagine a workflow, one part of which is doing 1000 requests to an upstream system. Each request is both expensive and may take a long time (days), so the task is split into two activities: 1. Make the requests 2. Poll for results. That way I don’t rerun them if the worker dies. But say a single request fails and I need to intervene by re-requesting it.

It would be slick if I could restart the polling activity with new arguments rather than restarting this whole workflow.

For now maybe an idea is to catch ActivityFailure in your workflow code and then wait for a signal with the updated/fixed values and then run the activity again using those as input.

I think this is a workable solution, but it would be nice to have the pattern documented with a code sample (if it isn’t already) with any potential pitfalls.

You can fail/complete the activity through API/CLI with a special result that would indicate to the workflow to retry it with different arguments.

Hi @maxim

Please help with example.

I am trying to run workflow with input to workflow “1”. After that I have create activity if input is “1” then throw the exception.

Then using signal I have change the input value with “2”. But activity still receiving the value “1”. Below are the screenshot for better understanding.

How do i update the activity input without affecting the previous activities?

Below is the workflow.

Below is the implemented activity code.

Below is the frontend

I could see the updated value in query.

In your case the activity is retried by the service. So it is invoked only once by the workflow. So changing the workflow field is not going to affect its arguments.

You must cancel the activity and retry it from the workflow when a signal is received.

Is there way to retry. As I can cancel it using cancellation scope. But how to start it again. Unable to find code.

You can code it youself as a loop or use Workflow.retry.

@PRATIK_NAIK were you able to figure it out? It would be super helpful for my use case

I also have a similar usecase.

We use temporal to orchestrate a graph of microservice calls. The input to the workflow is provided by the user via our own frontend.
Each activity is idempotent in our case.

We want to expose the user the ability to retry their request from a specific step by allowing them to edit their supplied input values (in case they made an error) and re-run the workflow.

I understand that since each activity is idempotent, re-run can be just modeled as rerunning the entire workflow again, but it would be neat if we can just modify and an existing workflow (for which lets say one activity is infinitely retrying), update the input values of the activity and let the workflow complete.