Waiting for all async child workflows to complete, when triggered from an activity

Community Support
,
1

Hi,

Would there be a way for an activity to trigger multiple async child workflows and also allow the parent workflow to wait for all these child workflows to complete?

I know that if I trigger the child workflows directly from the parent workflow - without an activity - I can do something like Promise.allOf(promises).get() to get it to wait, but I’m not sure about the approach when there is an activity triggering the child workflows

For some background:

  • I have a potentially a large amount of items to process. Thus to avoid potentially overloading the workflow’s state/history size, I’m using an activity to read in each of the items to process from a file (rather than an object of the parent workflow).

  • I need to wait for all these child workflows to complete first before the parent workflow can continue on with its other tasks.

I’d also be good if there is a way to enforce a limit to the number of concurrently running child workflows running at a time.

Interested to hear any advice or suggestions!

1 Like
2

In your case, I would use option 2 from this post.

1 Like
3

Thanks Maxim for pointing me in that direction - will look that ContinueAsNew approach for the items processing.

A few things if you could clarify about the approach:

  1. If I open the file in the workflow (not Activity) I’d need to handle any retry logic myself?
  2. I’d need to keep track of where the workflow is up to with a counter
  3. And I’d need to pass the counter information through when calling ContinueAsNew so the new workflow knows where to start reading from in the file
4
  1. If I open the file in the workflow (not Activity) I’d need to handle any retry logic myself?

Any external API calls from workflow are not allowed. Only open files from activities.

Correct

5

Right, I see. I had an attempt to implement this, but realised I’m still a bit unclear.

So I’ve now got: parent workflow → ‘iterating’ workflow (calls an activity to read items from file, uses ContinueAsNew) → child workflow (processes each item).

These are the 2 possible methods I can see - wondering what your thoughts were:

a. Activity reads file and passes all the items back to calling ‘iterating’ workflow. ‘Iterating’ workflow then triggers the child workflows.

or

b. Activity reads file, and then from this activity it triggers the child workflows.

My concern with b) is that I’m not sure sure how an activity can trigger and then wait for the child workflows to complete? Does the waiting happen in the ‘iterating’ workflow or triggering activity?

Most of the Java examples are for a ‘workflow’ calling child workflows and not an ‘activity’ calling child workflows.

Thanks!

6

a. Activity reads file and passes all the items back to calling ‘iterating’ workflow. ‘Iterating’ workflow then triggers the child workflows.

Ideally, the activity is going to read a chunk of items from the file and pass the offset of the last record to the next iteration of the workflow. This way the file can be very large and doesn’t need to be returned as a single activity result.

b. Activity reads file, and then from this activity it triggers the child workflows. I’m not sure sure how an activity can trigger and then wait for the child workflows to complete? Does the waiting happen in the ‘iterating’ workflow or triggering activity?

When using Java SDK an activity can call a workflow stub synchronously. This will block it until the workflow is completed.

1 Like