Dependencies handling between activities

Hello everyone,

we have a financial calculation engine which is running on Spark at the moment. Due huge overhead caused by Spark, we are moving the implementation to the Prefect Core. To be honest, I’m not big fan of Python and would rather user Go for this, so I found Temporal. The biggest concern in our tasks/activities is, that some tasks have many dependencies to other tasks, which needs to be calculated before another task can start. I looked briefly through Go SDK documentation, but was not able to find an example or a solution, how to handle such dependencies in Temporal. In prefect we have custom task, where we use set_dependency function from Prefect. Is there any other possibilities in Temporal where I can specify, on which other activity is another depends on?

Best

Besides the execution engine and the choice of languages the biggest difference between Prefect/Airflow and Temporal is the programming model.

Prefect doesn’t execute your code as workflow. The code generates DAG of tasks which in turn is executed by the engine.

Temporal does execute your code as workflow directly. It means that it doesn’t need special tasks to implement orchestration of practically any complexity. There are multiple ways to achieve what you want. The simplest one would be just executing all these tasks asynchronously and then waiting for all of them to complete before calling the next task. Something like:

    // Use Futures to implement split/join
	var futures []workflow.Future
	for i := 1; i <= totalBranches; i++ {
		future := workflow.ExecuteActivity(ctx, Activity1, activityInput)
		futures = append(futures, future)
	}
	// accumulate results
	for _, future := range futures {
		var singleResult string
		err = future.Get(ctx, &singleResult)
		if err != nil {
			return
		}
		result = append(result, singleResult)
	}
    // Next activity
    workflow.ExecuteActivity(ctx, Activity2, activityInput)

What is the maximum number of parallel branches do you expect in your case?

Hi Maxim,

thank you for such fast reply.

Although I think I understand now the difference between Prefect/Airflow and Temporal better now, I’m still not quite sure about the granularity of the implementation. What I mean by granularity, in Prefect we have one Flow with many tasks. These tasks are representing Items which we have to calculate. In total we have about 260 different Items and ca. 150 of them have different amount of dependencies to each other. Because of this dependencies between some of them, like you explained before, Prefect generates a DAG and executes the tasks. So we don’t have to care about the execution order, only to specify on which items current items relies.

If I understand your suggestion correctly, we would run all dependencies of a specific item first and then calculate it and call the another one? Is my understanding correct?

Thanks!

Temporal doesn’t put any restrictions on how tasks are chained. If your use case is better modeled through a DAG you can implement it as a DAG by modeling your workflow as a bunch of linked objects. If you use case is better implemented as a code that calls these tasks as a sequence of functions as in the example I posted then you can follow that approach.

The point is that it is just code and you implement your workflow in the way that fits your use case the best.

Understood. Thank you very much for your help! :slightly_smiling_face:

Best