How to execute child workflow one at a time

Community Support
1

I have a use case where I need to run the childworkflow one at a time and it should not depend on parent workflows closure.
I implemeted something very similar to AsyncChildWorkflow, but the only change, in application.yaml I added max-concurrent-workflow-task-executors: 1

  temporal:
    connection:
      enable-https: true
    workers:
      - task-queue: ParentWorkflowTaskQueue
        workflow-classes:
          - com.company.abc.ParentWorkflowImpl
        activity-beans:
          - someActivityImpl
      - task-queue: ChildWorkflowTaskQueue
        capacity:
          max-concurrent-workflow-task-executors: 1
        workflow-classes:
          - com.company.abc.ChildWorkflowImpl
        activity-beans:
          - someActivityImpl

Now when I dont add max-concurrent-workflow-task-executors: 1 for the child workflow task queue everything runs as expected which is child workflow keeps on running while the parent closes but only child workflow runs in parallel (which I dont want to happen)

But when I add it - the childworkflow is picked very slowly. Lets say - I have 5 worflows in running state then the second child workflow is picked 10-15 minutes later after the first one completes.

How I have created the child worklow

ChildWF childWf =
        Workflow.newChildWorkflowStub(
            ChildWF.class,
            ChildWorkflowOptions.newBuilder()
                .setTaskQueue("ChildWorkflowTaskQueue")
                .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON)
                .build());
    Async.function(childWf::doSomeLargeActivity);
    Promise<WorkflowExecution> childExecution = Workflow.getWorkflowExecution(childWf);
    logger.info("Ending parent workflow");
    return childExecution.get();

I also tried with max-concurrent-activity-executors. But the activties are not in order. It runs perfectly one at a time, but the activities in the multiple workflows run in any order - like activity-1 of workflow-execution2 → activity-1 of workflow-execution4 → activity-2 of workflow-execution2 and so on

2

I’d try adding a workflow in the middle. The parent would start the middle workflow, which would be the one that is set to PARENT_CLOSE_POLICY_ABANDON. The middle workflow would execute the child workflows one at a time in the desired order.

3

Hey, can you give one pseudo example? I didnt really get it

4

Well, maybe I misunderstand what you’re trying to do, but what I thought you meant was:

  1. You want to execute a series of child workflows, one at a time.

  2. You want to trigger that execution from a parent workflow.

  3. However, you don’t want the parent workflow to have to wait for the child workflows. You want the parent workflow to be able to finish without affecting the child workflows.

So, if that’s what you want to do, create a workflow to do #1. This is what I called the “middle” workflow. When the middle workflow executes, it runs the child workflows you want to run, one at a time, in the order you want to run them.

Now the parent workflow can start the middle workflow without waiting for it to finish. The middle workflow runs, the parent workflow is free to terminate, and the middle workflow runs the desired child workflows one at a time.

But maybe you wanted something different?

5

Hey, The solution you have suggested works in case there were a series of child workflows to execute, however we actually only want to invoke 1 child workflow from the parent workflow. Hence in this case, having a middle workflow does not work.
Can you please suggest some other solution which can help us to mitigate the issue of child workflow being picked after 10-15 mins?

6

Would you describe the “issue of child workflow being picked after 10-15 mins”?

7

Sure.

Lets assume there is a parent workflow which invokes only one child workflow after doing a few activities. And I started 5 parent workflows. So these parent workflow will generate the child workflow. Which means I have total 5 child workflows in queue.

For the child workflow I have added the task queue something like this -

- task-queue: ChildWorkflowTaskQueue
        capacity:
          max-concurrent-workflow-task-executors: 1
        workflow-classes:
          - com.company.abc.ChildWorkflowImpl
        activity-beans:
          - someActivityImpl

On adding max-concurrent-workflow-task-executors: 1 I am assuming that the childworklow will be executed one at a time.

Problem -
After the first child workflow execution is completed, ideally the second child workflow should start the execution. But the second child workflow starts some 10-15-20 minutes (totally random) after the first child workflow is completed. The third starts some hours later after the second child workflow is complete.

Note - I got this problem when I introduced max-concurrent-workflow-task-executors: 1 to a simple task queue (no parent child logic).

8

On adding max-concurrent-workflow-task-executors: 1 I am assuming that the childworklow will be executed one at a time.

This is an incorrect assumption. Temporal doesn’t provide a way to limit the number of parallel workflows, and this configuration limits a worker’s throughput, not the number of workflows it can run.

Why do you want to limit the number of parallel workflows? The more common requirement is to limit the number of parallel activities per worker.

9

Let me see if I understand:

You have a parent workflow that can be executed multiple times.

Each execution of the parent workflow needs to start the execution of a child workflow, without waiting for the child workflow to finish.

Only one instance of the child workflow should be executing at a time. If multiple parent workflows have started instances of the child workflow, those need to be queued so that each child execution will wait for its turn.

10

yes that is correct

11

How about using signals?

Instead of a child workflow, have a long running execution workflow that waits for and responds to signals by running the execution that you would have had the child workflow run. If it receives a signal while it’s still processing a previous signal, it would queue it up to run after it has finished its current execution.

What had been the parent workflow, the initiating workflow, would then send a signal to the execution workflow to execute or queue the task it needs run.

Since the execution workflow is now long running, it should periodically call continue-as-new to avoid the workflow history getting too large. Any queued tasks waiting to be executed can be included in the workflow parameters passed to the next iteration of the workflow.