I am pretty new to Temporal and exploring it to see if it could meet my team needs.
Essentially we are developing a scheduler-like platform, where end users have a list of tasks, each task with a desired frequency of execution (default 24h).
Users can change their tasks frequencies from time to time (from daily to weekly, or from weekly to every 2 days for example).
We tried creating batch schedules (1.5M) on our dev cluster but we’re pretty sure it is not the right way to do it, cluster was very slowed down.
We also found this great article which could potentially fit our needs, but it looks like infinite loops are discouraged in favor of ContinueAsNew to limit events history size.
These tasks can be of any duration (between seconds and hours) with a limit according to the frequency of course to not have overlapping tasks.
Would Temporal fit our needs ? If it is, what would be the recommended way, according to best practices ?
Temporal is a great fit for your use case. Check out the schedule feature documentation and let us know if you have any questions.
We tried creating batch schedules (1.5M) on our dev cluster but we’re pretty sure it is not the right way to do it, cluster was very slowed down.
I’m not sure what are the “batch schedules”. Note that a cluster is usually provisioned to handle a certain number of actions per second. So if your cluster can support 500 actions per second and you try to start 1.5 million in a minute you can overload it. Cluster doesn’t have a problem running a very large number of schedules in parallel given that it can support the rate of their execution.
Well i’m not a native english speaker so I might have mispronounced what I had in mind.
What I called “batch schedules” is that we created 1.5M schedules within a loop. It will be mandatory at the beginning to initialize every customer tasks, but at a lower scale, ~300k tasks.
Cluster absorbed all schedules creations under ~3h30, which is not too bad considering the number of tasks.
Once every task was scheduled successfully, single update / deletion / creation is pretty smooth. We still have some difficulties listing all tasks with the Go SDK (~10-12min, but cmon, we’re talking about a million and a half tasks !) but it is not an essential need.
Thank you again for your answer @maxim, it comforted my team on confirming our approach and on keeping Temporal as our platform.
I see. In this case, rate limit the schedule creation calls. For example, if your cluster is provisioned to handle 100 actions per second you don’t want to try to start 1000 schedules per second.