Amount of levels in a workflow-child-workflow relation
When creating the diagram for this one workflow I was testing, one question got into my mind and I posted it on Slack:
Hello everyone, how’s it going? So I came to understand that there’s a limit of 2000 child workflows a given workflow can have. Is this per child-level? like I could have a tree of child workflows each having at most 2000 child in it?
We got a nice interaction from the community so I didn’t want to lose this discussion due to history limits on Slack. James Watkins provided amazing reasoning about the functioning of “tree-like” workflow architecture, I put them below:
Temporal indeed enforce a few limits. Temporal Cloud also add some more limits, which helps avoid situations where some customer’s usage pattern may become disruptive for other users (aka. noisy neighbors issues). These limits are described in this doc page. Among the limits enforced by Temporal Cloud, there is indeed the well known “maximum of 50K History Events for a single Workflow”, though we strongly recommend staying below 10K. There is also a limit of a maximum of 2000 concurrently running child workflows . That is for a same parent Workflow. Similar constraints also applies to the number of pending Activities, pending outgoing signals, and pending workflow cancellation; each of these constraint applies individually to each Workflow.
So putting all of this together, it would indeed be technically possible to have a parent Workflow that spans 2000 child Workflows, and then each child workflow would themselves span 2000 “grand child” Workflows, and then each of these grand child Workflow could run 2000 activities. You would then have 8 billions activities running concurrently.Well… theoretically concurrent, as, in practice, you will obviously get throttled by various other limits, such as your Worker fleet capacity at handling Workflow Tasks and Activity Tasks, or the maximum number of actions per second allowed on your Temporal Cloud namespace (defaults to 200 action per second per namespace, with spikes up to 400; these can be raised with a support ticket).
I made another question:
That’s a very interesting explanation James! I don’t think I’ll need all of that hahaha but it’s important to know the possible pathways! You wrote 2000 concurrent so in a batch-like orchestration I can have more than 2.000 children, just not at the same time, is it correct to think like this?
Is there any “preferences” in terms of “have at most x-deep levels” in your workflow relations or you will doom yourself?
James responded:
That’s exact. However, iteratively running more than 2000 child Workflows from a single parent will likely make that Workflow’s History grow near or above to the 10K max events per Workflow recommendation.To understand why, consider that, assuming that the parent Workflow is waiting for its Children to complete (ie. the child are not “abandoned” by completing the parent Workflow), then each child getting started will cause three events in the parent’s Workflow History: first the
StartChildWorkflowExecutioncommand, then theChildWorkflowExecutionStartedevent immediately after, and eventually theChildWorkflowExecutionCompletedevent.More over, some of these events will cause Workflow Tasks to be scheduled, adding another 3 events per Workflow Task (scheduled,startedandcompleted). Now, the exact number of Workflow Tasks that will be required to run through all the events mentioned before is hard to predict since several events may get batched in a single Workflow Task, but just the sake of estimation, let assume an average of one Workflow Task per Child Workflow…So, to summarize, we find out that 2000 child workflows implies 6000 events related to the child workflows themselves, and ~6000 events for the Workflow Tasks related to these events. That’s a total of ~12000 events in the parent Workflow History, and I have skipped some other potential events.
I just wanted the register here in case someone else is also reasoning about this! Hope it helps.
Arthur Pires.