What are the pros and cons of Temporal with respect to Prefect?

Tech Comparisons
1

Hi,
First off, am super impressed with Temporal and am considering it as a backend in a product we’re developing. We’re looking to use it for a mix of business process automation combined with ETL for pulling and training ML models.

So this would require bootable workflows and variable persistence, a requirement met by Temporal.

I’m just wondering if there’s any use cases that Temporal doesn’t cover that Prefect/Airflow does.

Because as of now I’m fairly convinced with Temporal

3

Hi Abhik, welcome to the community!

I’m biased and haven’t used Prefect, so take this with a grain of salt :smiling_face: And please correct me if I have something wrong about Prefect.

  • Temporal is a general-purpose tool: it makes your code run reliably.
  • My impression of Prefect is it’s for data pipelines in Python.

Prefect doesn’t sound like the right tool for business process automation, so I’d go with Temporal for that. Then the question is whether you’d like the simplicity of a single tool (in which case use Temporal for both business processes and data pipelines) or want to consider a second tool. If the latter, then the question is whether you prefer Temporal or Prefect for data pipelines.

Prefect

There may be some things Prefect has that make it nicer DX for its purpose—maybe it has an ecosystem of data connectors, or built-in blob storage between steps, or built-in notifications. If you chose to use Temporal for data processing, you could still do those things—you’d just need to code them yourself.

Temporal

  • Better lifecycle management
  • Polyglot: not only can you use languages besides Python, but you can also mix them: write a Workflow in one language and its Activities in three other languages.
  • Signals and Queries
  • Lower latency

Looks like Prefect 2.0 is code-based instead of dag-based, so this just applies to Prefect 1.0:

  • Greater flexibility: can run arbitrary code, versus Prefect uses code to define a dag like Airflow. The more you customize Airflow dags, the more you logic you have to put in string templates. In Temporal you can express everything with code.
    • This also makes Temporal very easy to test as you can apply traditional software testing practices unlike with Prefect.

And compared to Airflow (not sure if Prefect is the same in these regards), Temporal is:

  • Much easier to manage deployments for Temporal Workflows vs Airflow dags. With Temporal, Workflows and Activities are bundled in the Worker and there can be multiple of those while in Airflow there’s only one worker type so if you have multiple projects using the same Airflow deployment, they can’t have conflicting dependencies.

Here is an example Airflow :arrow_right: Temporal migration.

See also this thread.

2 Likes