we have a financial calculation engine which is running on Spark at the moment. Due huge overhead caused by Spark, we are moving the implementation to the Prefect Core. To be honest, I’m not big fan of Python and would rather user Go for this, so I found Temporal. The biggest concern in our tasks/activities is, that some tasks have many dependencies to other tasks, which needs to be calculated before another task can start. I looked briefly through Go SDK documentation, but was not able to find an example or a solution, how to handle such dependencies in Temporal. In prefect we have custom task, where we use set_dependency function from Prefect. Is there any other possibilities in Temporal where I can specify, on which other activity is another depends on?
Besides the execution engine and the choice of languages the biggest difference between Prefect/Airflow and Temporal is the programming model.
Prefect doesn’t execute your code as workflow. The code generates DAG of tasks which in turn is executed by the engine.
Temporal does execute your code as workflow directly. It means that it doesn’t need special tasks to implement orchestration of practically any complexity. There are multiple ways to achieve what you want. The simplest one would be just executing all these tasks asynchronously and then waiting for all of them to complete before calling the next task. Something like:
// Use Futures to implement split/join
var futures []workflow.Future
for i := 1; i <= totalBranches; i++ {
future := workflow.ExecuteActivity(ctx, Activity1, activityInput)
futures = append(futures, future)
}
// accumulate results
for _, future := range futures {
var singleResult string
err = future.Get(ctx, &singleResult)
if err != nil {
return
}
result = append(result, singleResult)
}
// Next activity
workflow.ExecuteActivity(ctx, Activity2, activityInput)
What is the maximum number of parallel branches do you expect in your case?
Although I think I understand now the difference between Prefect/Airflow and Temporal better now, I’m still not quite sure about the granularity of the implementation. What I mean by granularity, in Prefect we have one Flow with many tasks. These tasks are representing Items which we have to calculate. In total we have about 260 different Items and ca. 150 of them have different amount of dependencies to each other. Because of this dependencies between some of them, like you explained before, Prefect generates a DAG and executes the tasks. So we don’t have to care about the execution order, only to specify on which items current items relies.
If I understand your suggestion correctly, we would run all dependencies of a specific item first and then calculate it and call the another one? Is my understanding correct?
Temporal doesn’t put any restrictions on how tasks are chained. If your use case is better modeled through a DAG you can implement it as a DAG by modeling your workflow as a bunch of linked objects. If you use case is better implemented as a code that calls these tasks as a sequence of functions as in the example I posted then you can follow that approach.
The point is that it is just code and you implement your workflow in the way that fits your use case the best.
Is there any restriction on the total number of branches that we can specify while trying to execute a few activities in parallel with each other inside the workflow?
The only hard restriction is the events’ history size. I believe by default it is 50k events and 50MB of payloads. But it also depends on other factors. For example, if you start 1k short-lived activities they will get contention on completion. A single workflow serializes all updates and each activity completion is a DB operation. For 1k updates to the same workflow might take a long time to process if executed at the same time. From the other point of view, 1k activities that don’t try to complete at the same time should be absolutely fine.
