Parent workflow trigger Async child workflows

Community Support
1

On one hand, because Child Workflow Executions have their own Event Histories, they are often used to partition large workloads into smaller chunks. For example, a single Workflow Execution does not have enough space in its Event History to spawn 100,000 Activity Executions. But a Parent Workflow Execution can spawn 1,000 Child Workflow Executions that each spawn 1,000 Activity Executions to achieve a total of 1,000,000 Activity Executions.

However, because a Parent Workflow Execution Event History contains Events that correspond to the status of the Child Workflow Execution, a single Parent should not spawn more than 1,000 Child Workflow Executions.

I want to create a schedule to trigger a workflow that will fetch N amount of records that will be used by other workflow, the amount of record is around 200k, so I had the idea to create parent workflow that triggers child workflow to process each record, I do not need to wait or register events related with the child workflows in the parent workflow.

base in the documentation

a single Parent should not spawn more than 1,000 Child Workflow Executions, could I trigger all (200k) child workflows without have issues in the storage of the parent?

Anyway I was thinks to trigger the events from a external service as a k8s cronjob but i want to try to figure out if i can do this triggering using temporal.

2

Do you want to process all 200k records in parallel? Or do you want to process them in chunks or using a sliding window?

3

Initially In parallel sound good, certain activities involve downstream http calls and storage interactions so probably there will be some kind of rate limiting at worker or task queue level

My concern was initially because I wanted to keep the sync process isolated per account in that way each account will execute a workflow and then take the advantages of the workflow retries, also if something happens probably will affect just one account or a sub set of them. so I found that a parent workflow are limited by the storage and this indirectly limit the amount of child to create then I was looking the best approach to achieve this without hacks.

For now I decided to use “hierarchical pattern” that’s basically to create 1000 child workflows and each of then kind create 1M of child that is enough for now without sature the event history limit per workflow.