Hi temporal team
Currently our POC framework uses saga orchestration. Our system has nearly 100 microservice, with load up to nearly 1000TPS. We have learned about temporal framework, it is very useful to implement saga but we can’t find load information when applying this framework. Do you have any load tests or someone who has tested them can share them for us to use as a basis for evaluation?
The second problem when learning about taskqueue we found a term related to task queue partition. So how is this partition mechanism different from kafka partition ?
Hi @minhmv
For the first question, Temporal provides the Maru load simulator which is specific to workflows written using the Go SDK.
As for scalability, there are production use cases which have hundreds of millions open workflows at any time.
There are also some load/performance-testing threads that might help here and here.
@tihomir thanks for your reply,
I have a question about taskqueue what is the partition in taskqueue, is it the same as partition on kafka, how is the consumer message mechanism of taskqueue ?
Temporal task queues are different from Kafka. Kafka partition is a stream and can have only one consumer. So Kafka doesn’t allow to have more consumer processes than partitions for the same consumer group. Temporal task queues are real queues. Partitions are used internally and not exposed to consumers. And a single task queue can have any number of consumers independently of the number of partitions.
@maxim thanks a lot.
@maxim
I want to learn how taskqueue works ?
do you have any documentation about message storage mechanism, load capacity, scalability of taskqueue ?
Thanks.
We don’t have such documentation yet.
-
Messages are stored in the same DB the rest of the data is stored. In SQL databases they are stored in task_queues table.
-
A single task queue can keep up with the rest of the load if configured with enough partitions. Our load tests frequently use a single task queue to saturate pretty large clusters.