Workflow with variable activities dependencies

Community Support
1

Hello, I am new to Temporal and I am trying to implement with Temporal a workflow whose goal is to configure several services on behalf of a user.
The services are configured by both synchronous and async activities, depending on the service itself and, most importantly, some service configuration activity might require as input the output of a different service configuration.
Example: services A, B, C; given input “alpha” A requires B.out, C requires both A.out and B.out

To make it more complex, these dependencies are not “hard coded” because, depending on the input passed to the workflow, B might not require A.out and, in some cases there might even be a different, even opposite, dependency: given input “beta” B requires A.out, C requires both B.out and A.out

Given that it’s not really possible to code these dependencies in the workflow, my idea is to implement the workflow as a set of child workflows, one for each service A, B, C, …, all running in parallel with the same input.
If a service child workflow detects (inside an activity) that it misses some input, than it will fail in non recoverable way, otherwise will complete.
The parent workflow will gather the outputs of all succeeded child workflow and, in case of a child workflow failure, reschedule the failed workflow passing the consolidated output as the new input to it.
This approach should provide a sort of an “eventually consistent” workflow which execute several times the child workflows until they all succeed.

A slightly different solution could be this: the input of each child workflow is not passed as a parameter but is instead queried by the activity (or by the children workflow logic, before starting the activity) as a Query to the main workflow state, that acts as a global state which is set at the beginning of the workflow as “input” and mutated by the child workflows activities as a result of the configuration tasks (via Signal or directly mutating the state as a result of child workflows successful completion).

This second option seems more complex in terms of state handling, also I am afraid of not properly using the Temporal API and, most importantly, the overall approach to designing robust workflows.

2

Sorry, I don’t understand “it’s not really possible to code these dependencies in the workflow”. Workflow is code, so if there is a way to describe these rules then there is a way to code them. I don’t think child workflows will help here.

3

Sorry I did not explain myself correctly, what I meant is that the workflow will not be statically defined as a fixed set of activities always executed in a fixed order and in a fixed number of times.
My doubt is about coding the workflow in a way that is compatible with the Temporal model.
I am implementing the workflow with the query status approach and it seems to actually work. I am also using one child workflow to handle each service configuration as a more observable entity in the temporal console, but it is possibly not really necessary to do that.

the workflow looks like this:

run:
  - loop:
    - save "input" and "output" (output is initially empty) as workflow state, queryable by activities
    - start child workflow for service A
    - start child workflow for service ...
    - start child workflow for service Z
    - gather outputs from all all children
    - did outputs changed since last iteration?
      - yes: continue loop
      - no: exit loop (as no more changes are required and status is "stable")

each child workflow is a mostly sequential execution of activities, all of them are idempotent.

4

Sorry, I still don’t understand why you use child workflows and status query activities here.

What is the reason that doesn’t allow specifying this algorithm inside a single workflow? Note that workflow can be as dynamic as needed. There is nowhere requirement that it should be “statically defined.” For example, you can read a DSL document to execute dynamically.

5

I think that I just misunderstood the limits in the Temporal workflow model.
I was thinking that the workflow to be deterministic must also had been “static” (or, in my understanding: with no branches on conditions related to the results of an activity) but this is not the case: to achieve determinism the workflow must be able to be executed more than once (“replay”) given the same input but it can absolutely do branching on activity results, did I get it right?

6

Yes, it looks like you misunderstood what “deterministic” means. The workflow code can be as complex as necessary. You can use the full power of a programming language of your choice. Note that it includes complex object graphs, not only functions.

1 Like