Creating Multiple Child Workflows

Community Support
1

I want to use Temporal for a data ingestion pipeline. The data is provided in an excel file. The file can contain up to 10000 entries. The parent workflow will parse the file. For each entry, a child workflow will be created that will do the end-to-end ingestion for that specific entry. All these child workflows will run in parallel.

What would be the most efficient way to implement this?

  • Since the file can be huge (up to 10000 entries), the parsing can take long. Should parsing be done in an activity?
  • Is there any batch mode to create child workflows? Will it be efficient if we iterate over 10000 entries and create child workflows one by one?
1 Like
2

I recommend what I call iterator workflow pattern. The iterator workflow:

  1. Executes getNextEntries activity that loads a single page (let’s say 500) entries from CVS file.
  2. Executes child workflow per entry, consider running some number of children in parallel depending on your use case.
  3. Calls continue as new to reset workflow history passing pagination token for the getNextEntries call to the next run as a parameter.

This way a workflow can process a file of unlimited size sequantially.

2 Likes
3

I was using the following example to execute child workflows: https://github.com/temporalio/samples-go/blob/main/child-workflow-continue-as-new/parent_workflow.go
Seems using Get(), the workflow waits for the child-workflow to complete before starting the next child-workflow.

Does this mean using Branch workflow inside child-workflow which returns the future to execute some number of activities in parallel. For example, processing 500 entries in one batch with 5 child-workflows
each child-workflow executing 100 activities concurrently.

4

< Does this mean using Branch workflow inside child-workflow which returns the future to execute some number of activities in parallel. For example, processing 500 entries in one batch with 5 child-workflows
each child-workflow executing 100 activities concurrently.

Yes, exactly that.

BTW. I’ve added the relevant batch samples to the Java samples repository.

1 Like