Computation Graph Orchestration

Community Support
, , ,
1

Hello, I have a computation graph as following

image
image2044×1608 132 KB

Arrows represent input-output requirements between tasks. Nevermind the start and end nodes, they are dummy placeholders.

Each task depends on some other task’s output to use as input (e.g. task5 is using task4’s outputs, calculates its results, then task7 will use outputs of task5 and so on…)

def task_N(*args, **kwargs):
    # load task dependencies as inputs
    ...
    # run task with inputs
    ...
    # save task result
    ...

Moving the implementation from python celery to temporalio (using python again). In celery we have achieved this with chaining and grouping tasks,

How should we implement such orchestration logic in temporalio? Is there any similar structure in temporalio? Does it makes sense to use signals?

2

You could pass in your dependency graph as input (dsl represented as json for example) to workflow, or read it via activity from a store if needed on workflow exec start. Could build your data structure from it and then execute activities (tasks) based on the graph.
How large can graphs get (max number of possible tasks)?

3

Max number of tasks on a graph is about 10-15.

I didn’t get your example though, how does one activity (task) waits for its predecessor tasks to finish first?

4

hello @cya ,

I am not a python expert but I think you can do something like this, it might be a better way.



        task1result = workflow.execute_activity(
            task1,
            "input",
            start_to_close_timeout=timedelta(seconds=10),
        )

        result1 = await task1result

        result = await asyncio.gather(workflow.execute_activity(
            task2,
            result1,
            start_to_close_timeout=timedelta(seconds=10),
        ), workflow.execute_activity(
            task3,
            "task3",
            start_to_close_timeout=timedelta(seconds=10),
        ))

        result2 = result[0]
        result3 = result[1]

....

here are some examples that might be useful

let us know if it helps