Hello people! I was talking with @markmark and a question arose. My team and I are looking to be able to orchestrate the process of one of our products. On the days of greatest flow, we have a concurrency of between 300 and 800 requests per second. Currently we have an orchestrator that generates many post-execution problems. The question arose if with temporal.io + mysql we could bear this scale of requests. We are playing with cadence and temporal, but until now only my local doubts or problems have been solved thanks to the temporal community and also the tools and documentation that they provide us. What do you think?
2 Likes
I think this is a very interesting question. I think the answer depends on how many steps/activities you are expecting from each workflow and how long they run on average. I’ve tagged Maxim here as he will have a more practical answer.
1 Like
Agree with Ryland that more info is needed.
What is “800 requests per second”? Is it 800 signals per second or 800 workflow starts per second with each workflow executing 1000 activities?
Hello.
Giving a little insight on what we currently have. We use a massive SQLServer and process out custom workflow with 10 activities on average. In our highest request period we have had to process between 300 and 800 requests per second(one unit of workflow execution) for 2 to 10 continuous hours (sometimes this is massive and we even cant handle to process all activities at once, we need to make priorities in our queues). A run should ideally end in less than 2 minutes. Currently we have queue and retry issues (not all activities can process the same amount of request per second), and SQLServer performance issues where we have had to constantly optimize queries to improve. We are concerned with the costs of using a MYSQL cluster (Ideally lower our SQLServer cost) and if we are going to be able to exceed these requests per second (We make stress test continuously to improve or workflow execution limits). Our business depend on this process we need to select the right database, mysql or cassandra
1 Like
Such load is absolutely achievable with Temporal without sacrificing reliability. Most of our experience running at scale is on top of Cassandra. We would need to do some performance test with MySQL to understand what type of DB instance is needed for such rates.