Suggestion for blog post about covering celery problems

Show & Tell
1

Hi. This is a post about Celery problems in HN. I think it’s a good topic/opportunity for a blog post for telling that how every of those problem is solved (or also exists) in Temporal.

2

Here’s a quick heading-by-heading Temporal response to the Celery post:

  • Celery prefetches jobs - Temporal workers get one at a time only asking if there’s room
  • Celery loses jobs by default - Temporal does not
  • Celery’s retry defaults are bad - Temporal does exponentially retry by default
  • No transactional job enqueuing - Temporal workflows are inherently transactional (but the activities may not be and requires idempotency management on user’s part)
  • Canvas, cords, and friends encourage brittle pipelines - Temporal primitives are enough to not need helpers to overcome
  • API isn’t Pythonic - Temporal Python SDK was explicitly developed to be Pythonic
  • Config isn’t type safe - Everything in Temporal Python SDK is as type-safe as the most modern Python type hints allow
  • Doesn’t encourage safe evolution of tasks - Temporal has entire versioning strategies to help here (and are actively improving them)
  • Jobs aren’t interruptible - Temporal workflows can be interrupted at any time
  • Difficult to disable jobs gone haywire - Schedules can be paused, workflows can be terminated, but “haywire” workflows by default in Temporal are “suspended” until code fix is deployed
  • Doesn’t support async - We are fully async (we leverage asyncio significantly)
  • Automated testing is difficult - We have full test framework including time-skipping
  • Serialization issues are common - Our serialization mechanism is well documented and goes way above the json.dumps/json.loads the standard library comes with, and is highly customizable
  • Monitoring story isn’t great - We export metrics (OTel and Prom) from worker side and export metrics from server side
  • Type checking - We do this heavily (the most extreme/modern typing I’ve seen in any Python project to be honest)

And noted at the bottom of the post:

Also worth looking into the new hotness, Temporal, but it has its own learning curve.

We may indeed make a blog post about this, thanks!

1 Like