Creating Multiple Child Workflows

I want to use Temporal for a data ingestion pipeline. The data is provided in an excel file. The file can contain up to 10000 entries. The parent workflow will parse the file. For each entry, a child workflow will be created that will do the end-to-end ingestion for that specific entry. All these child workflows will run in parallel.

What would be the most efficient way to implement this?

  • Since the file can be huge (up to 10000 entries), the parsing can take long. Should parsing be done in an activity?
  • Is there any batch mode to create child workflows? Will it be efficient if we iterate over 10000 entries and create child workflows one by one?

I recommend what I call iterator workflow pattern. The iterator workflow:

  1. Executes getNextEntries activity that loads a single page (let’s say 500) entries from CVS file.
  2. Executes child workflow per entry, consider running some number of children in parallel depending on your use case.
  3. Calls continue as new to reset workflow history passing pagination token for the getNextEntries call to the next run as a parameter.

This way a workflow can process a file of unlimited size sequantially.

1 Like