Could Temporal serve a queue scheduler use case?

Community Support
1

We are currently using many SQS queues for processing various async flows. On top of the queue workers, we have our own homegrown queue scheduling system that essentially plays the role of reading the messages on the queues and determining which worker to send them to be processed.

Is this a use case that Temporal could help with? Is this something that others have used it for? Thanks!

2

The short answer is yes. Temporal is a technology that makes implementing async flows trivial.

But what you described sounds like a specific solution. What is your use case? What is the actual problem you are solving?

3

We have messages that are written to many queues - and we have written custom logic for determining when to process the messages (i.e. when we actually execute the code on the message)

In an ideal world, we don’t need this custom logic for determining how and when to process the messages (what I refer to as queue scheduling).

We essentially want to separate out the logic for the scheduling from the code that actually is needed for executing the queue messages. And, we want to be able to have things like persistence and retries, etc.

We are considering exploring Lambdas for this, but we have enjoyed using Temporal and were wondering if it could fit this purpose well

4

Sorry, you again talk about the solution which is using messages and queues. What is the actual use case that you are trying to solve?

5

It has become clear that our custom scheduling logic is less than optimal, and has resulted in behavior whereby a slowdown in the processing of messages of one queue is blocking the processing of messages to another queue

6

Yes, Temporal can serve as a queue scheduler effectively. Temporal is a platform for building and running resilient, scalable microservices, and it’s particularly well-suited for complex workflows and tasks that need to be reliable and fault-tolerant. Here’s how Temporal can be utilized in a queue scheduler use case:
Workflow Management

  • Scheduling Workflows: Temporal allows you to define workflows in code, which means you can schedule tasks or entire workflows to be executed at a later time or on a recurring basis.
  • Stateful Execution: Temporal workflows are stateful, which means they can remember where they left off in case of a failure or restart. This is crucial for scheduling tasks that might span over a long period or need to be retried.
7

Sorry for being pedantic. But queues are a specific way to solve some business use case. What is the use case you are trying to solve with queues? I"m asking because Temporal eliminates the need for using queues in your application in most cases.

8

No problem, the business use case for queues here is for triggering async flows that are not time sensitive and where we are okay with retries of failures (i.e. idempotency is maintained).

9

Thank you, it does look to fit the bill - what i am leaning towards is:

  1. have our current individual SQS queues map to Temporal workflows
  2. have separate Task Queues according to business “domains”

but the part I am thinking through is whether we:
a. have separate groups of workers registered to each Task Queue to maintain isolation OR
b. share workers across Task Queues?

Any feedback/thoughts on this?

10
  1. Why individual SQS queues only? Have you considered implementing the whole async flow as a workflow? This would allow the complete elimination of external queues.
  2. I would start from a single task queue unless you want to deploy different “business domains” as separate services.
11
  1. Sorry should have been more clear, what I meant was that we would replace each SQS queue with a workflow
  2. This is helpful, but I am curious of the reasoning behind this
12

This is helpful, but I am curious of the reasoning behind this

A single worker (which also means a single task queue) per process avoids shared resources like memory and threads. So, performance tuning is more straightforward. The code is simpler as well.

13

Actually - just got out of a brainstorm with the team and was wondering if there could be a world in which the Task Queue is backed by/replaced with SQS queues

Context here is that we have some concerns around ripping out our SQS queue functionality completely as this is managed for us/easy to maintain

14

It is not possible and not necessary. Temporal task queues are also managed for you by the Temporal cluster.

15

Yes the problem here is that, we want to avoid having to ramp up developers on Temporal as they already have the SQS knowledge and we have all of the SQS observability setup for them. So, was wondering if there is a world in which we can mix and match here, such that developers still only have to “know” about SQS, but under the hood it is Temporal that is serving as the engine that determines how and when we process those messages coming into SQS.

It is a similar use case to the one described here where you recently added a response

This could mean that Temporal is not the solution for this, but wanted to make sure we did our due diligence as we have enjoyed playing around with the technology

16

Temporal is a new developer experience. But you lose most of its benefits by locking developers out of Temporal. The whole idea of Temporal is that you don’t need to think about queues, databases and state machines. You just write code that implements your use case.

This could mean that Temporal is not the solution for this, but wanted to make sure we did our due diligence as we have enjoyed playing around with the technology

You can use Temporal as a state machine that orchestrates messages between queues. And it is OK to start from this in a legacy system. However, I believe that keeping queues is decremental to your architecture in the long term.

BTW. I led the backend for SQS at some point, as well as other messaging systems. So I’m talking from real experience at many companies.