As we are delving deeper into Temporal and the type of problems we can solve through it, a few application level architecture questions are popping up so as to take an informed decision to converge on a common solution. As such the boundary line seems so hazy, I wanted to get an expert opinion from the community.
Can Temporal be a replacement for all common queuing / messaging solution which typically are being solved through SQS (in AWS), in RabbitMQ (mostly on-premise) or say, even Kafka? While we understand that Temporal queue is an internal core component with very little scope for user customization, I see that Temporal queues can be used to programmatically solve most queuing patterns, except when durability of the messages /payload is needed. Is that a right understanding?
Whats the maximum amount of payload Temporal queues can persist without affecting the scale and it’s effective functioning?
Can Temporal queues guarantee ordering of messages?
How about requeuing of messages on processing failures?
Dead letter queue implementation using Temporal queues?
Apologies for a long list of queries but an insight into these will really help in choosing the right solution.
, I see that Temporal queues can be used to programmatically solve most queuing patterns, except when durability of the messages /payload is needed. Is that a right understanding?
Temporal state is always durable. So I don’t understand why you are saying “except when durability needed”.
I don’t think implementing “queueing patterns” with Temporal is a valid goal. You need to think about implementing your business application using either queues or Temporal Durable Execution abstraction. Internally Temporal uses task queues, but this is more like an implementation detail.
The Temporal goal is not to be a direct replacement for existing queues. It is a different way to design an implement your application in which you don’t need to think about queues at all.
If you have a specific application/use-case in mind that is currently based on queues we can discuss how Temporal can solve it.
My statement “except when durability needed” was on the payload durability as to what would happen if the payload on a Temporal queue increases due to message not being able to consume at the expected rate by the workers. Will it affect the overall scalability of my Temporal server? As opposed to this, in SQS kind of solutions, I don’t have t worry about those intricacies.
Our majority use cases are jobs=>>workers model where workers pick up messages from SQS queues, do the processing. The queues have priorities associated with them where we assign more number of workers to do the processing. In case of processing failures, workers either re-enqueue the job or eventually land up in a DLQ.
My statement “except when durability needed” was on the payload durability as to what would happen if the payload on a Temporal queue increases due to message not being able to consume at the expected rate by the workers. Will it affect the overall scalability of my Temporal server? As opposed to this, in SQS kind of solutions, I don’t have t worry about those intricacies.
Temporal works fine in the presence of backlogs given that DB has enough disk capacity.
Our majority use cases are jobs=>>workers model where workers pick up messages from SQS queues, do the processing. The queues have priorities associated with them where we assign more number of workers to do the processing. In case of processing failures, workers either re-enqueue the job or eventually land up in a DLQ.
Sorry, this is not a use case. This is a piece of a solution that implements a use case. Temporal is a much higher level and doesn’t require thinking in jobs.