We are trying to rebuild our OMS using temporal. The system receives the orders, Shards them based on some properties, and processes the orders for each shard collected over time. If the batch is taking time to process because of the load on the downstream services we will fall back to process the orders one at a time till downstream scales up or stabilizes.
We want to test temporal before we migrate over from the current system.
Sharding will be based on geolocation and vehicle properties. There are around 100 distinct locations and 4 vehicle properties and each property can have 1-10 distinct values . These properties can be added/removed based on that the number of shards may increase/decrease.
Sharding is done in this way because downstream service requires batch of orders to have similar properties
Since shards can be any combination of geo and vehicle properties number of shards increases over time can go up to 33000 shards and more in the worst-case scenario. To begin with, the number of shards will be around 300-500. Shards can be dynamically added/removed. If shards increase or decrease orders will be rebalanced from the next cycle.
Estimated Min: 100 (Geo location) Estimated Max: 100 (Geo location) x 2 (Vehicle Property 1) x 5 (Vehicle Property 2) x 3 (Vehicle Property 3) x 11 (Vehicle Property 4) = 33000 Number of shards will increase around 10-20% each year
- The current OMS would enqueue the orders in RabbitMQ.
- An Order collector workflow will be running which will be collecting them and pushing them to a data store.
- Order collector workflow will also trigger a periodic workflow for each shard as and when it gets the order for that shard. We will generate a deterministic Workflow ID for each shard. If the workflow id exists temporal should reject the ExecuteWorkflow request.
- This periodic workflow will read all the current open orders and pass the ID(s) to a child workflow.
- If mode is batch, Periodic workflow waits for the child workflow execution to complete/times out and it continues as new to process the next set of orders in the same shard.
- If processing mode is not batch it will trigger a single workflow for each order and periodic workflow will not wait for the execution of child workflows to complete and pick the next set of orders for processing.
- The child workflow calls several activities which involves calling multiple external services.
- The Activities will only be doing the external calls and the processing would be done in the workflow itself.
- Each activity will have its own timeouts and retry policies.
- If in between an order is canceled, we would send a signal to the periodic workflow for the particular shard handling the order, which in turn will signal the child workflow and remove it from the list of current processing orders.
- Workers will be running on VMs and will be scaled as when required. The Temporal server will be running on Kubernetes with auto scale enabled.
We have attached the flow diagram as well to make this more clear.
- The Order throughput ranges between 5K-10K orders per minute.
- Timeout for each batch processing would be around 5 sec
- Assuming each shard takes a minimum of 1 sec there can be 60 cycles per minute per shard
- Input to each activity would possibly range between 4MB to 10MB.
We are looking to scale by 10-20% each year at the minimum.
- Are we using temporal the correct way to solve our problem?
- Will temporal be able to scale based on the projected throughput?
- Will the amount of data being passed around workflows and Activities pose a problem? What are the alternatives?
- Is the use of signals correct?
- Will temporal be able to scale to the number of parallel workflows that would be created as per our implementation? Else what is the alternative?