Recommendations on implementing a composable optimization pipeline as Temporal workflow

Community Support
1

Hi,

we are using Temporal workflows for key processes in our company and we love it.
A team of data scientists developed a solution to an optimization problem very important to the business.
We’d like to use Temporal to industrialize the proof-of-concept created by the team.

The details of the optimization are of course irrelevant, but there are few aspects that I need to describe to illustrate our challenges in using Temporal for this.

Challenging aspects
The approach used by the team consists of a set of optimization stages that are assembled into an optimization pipeline.
These stages are interchangeable and can be re-arranged.
Different problem configurations are best solved by different pipeline compositions.
Each stage produces an intermediate solution and if the quality decreases too much compared to the previous stage, the pipeline halts.
In this way the team can investigate, do some changes (like swapping some stages in the sequence) and then start a new pipeline using the intermediate solution built by the second-last stage that executed, in order to verify that their changes have actually resolved the issue.

A possible approach
I’ve skecthed the following approach.
I would pass the pipeline defintion (sequence of stages) as an input to the workflow.

The workflow code would be a loop that at each iteration

  1. checks the stage to execute and calls the corresponding activity/child-workflow
  2. computes metrics on the quality of the intermediate solution generated by the current iteration
  3. halts if the quality of the intermediate solution drops by a certain % compared to the previous iteration

The worklfow ends when there are no more stages to execute.

Issues with this approach

  1. some stages in the pipeline split the problem in smaller instances; in a way we could say that the pipeline allows for nested stages, but it’s very unlikely that it goes beyond 2 levels of nesting
  2. since workflow input and history can’t change, the scenario where the pipeline is changed to fix an issue gets a bit more complicated

Issue 2 can probably be resolved by extending the workflow input with a solution to start from: in the first run, this input solution would be an empty initial solution, while issue resolution scenario the input solution would be the intermediate solution produce by the second last executed stage.

Issue 1 is much harder for me to figure out a solution.
What do you recommend/suggest?

2
  1. Sounds like a good candidate for composite stage abstraction. Consider using child workflows to implement it.
  2. Starting the workflow with new specification stages can solve this. It is unclear how much state is preserved in the workflow vs outside. One option is not to fail the workflow, but let it jump to the appropriate stage based on an external update/signal.