Currently trying to figure out if Temporal is a good fit for our use case. A common scenario we need to support is, poll a third-party server for a list of IDs. A list of hundreds to thousands of IDs may be returned. Then, perform a series of tasks for each returned ID.
Depends on what kind of entity we are processing, there are two modes of operation we need to support.
- Fire-and-forget. Once we kick off the processing steps for each of the IDs, we don’t need to report back what happened to each of them.
- Request-response. We need to collect the results of each of the spawned executions and collect aggregate statistics.
How well does Temporal fit into the desired use case for the two above scenarios? I read somewhere else in this forum that the throughput for a single workflow is limited. Perhaps, then, for the fire-and-forget case, what we should do is have a single workflow that concludes when the IDs are emitted, and then another workflow that picks up a single ID and applies the processing steps? Then, there would be thousands of workflow instances of the latter type, which could achieve good throughput.
If that is indeed the correct design for the fire-and-forget scenario, then what should be the design for the request-response scenario? Perhaps similar to the afore scenario, but with a third workflow at the very end, listening for the results of the second workflow for all received IDs, and collect their results?
Or perhaps is this not the right way to think about workflows, and there is a better model?