ContinueAsNew and workflow child handles

Community Support
1

Hi there.

I have a workflow that works more or less like this:

[Workflow("indexing_orchestrator")]
public class IndexingOrchestratorWorkflow
{
    private List<(ChildWorkflowHandle, Task)> ongoingWorkflows = new();

    [WorkflowInit]
    public IndexingOrchestratorWorkflow(IndexingOrchestratorWorkflowInput input)
    {
        //?!
    }

    [WorkflowRun]
    public async Task RunAsync(IndexingOrchestratorWorkflowInput _)
    {
        while (!Workflow.ContinueAsNewSuggested)
        {
            IndexingRetrieveRequestsActivityOutput newRequests;
            //execute an activity to get new requests from somewhere

            foreach (var request in newRequests.Requests)
            {
                var wfHandle = await Workflow.StartChildWorkflowAsync(...,
                    new()
                    {
                        Id = ...,
                        ParentClosePolicy = ParentClosePolicy.Abandon,
                    });

                this.ongoingWorkflows.Add((wfHandle, wfHandle.GetResultAsync()));
            }

            
            // wait for any to finish
            await WorkflowWrapper.WhenAnySafe(this.ongoingWorkflows.Select(t =>
            {
                t.Item2.ConfigureAwait(true);
                return t.Item2;
            }));

            this.ongoingWorkflows.RemoveAll(t => t.Item2.IsCompleted);
        }

        await Workflow.WaitConditionAsync(() => Workflow.AllHandlersFinished);

        // must continue as new if requests are ongoing
        if (this.ongoingWorkflows.Count > 0)
        {
            var newInput = //???;
            throw Workflow.CreateContinueAsNewException((IndexingOrchestratorWorkflow x) => x.RunAsync(newInput));
        }
    }

So the idea is that we have a parent workflow (an orchestrator) that has, at any given time, a maximum amount of child workflows running simultaneously. When one of them finishes, it starts a new one and so forth.

Since this is a long-running workflow, we obviously need to support ContinueAsNew, which is fine. The problem is: how to I “transfer” the ongoing workflows to the next instance of the orchestrator workflow?

I can store the IDs, but the only thing I can do with them (I think) is to call GetExternalWorkflowHandle, but that handle doesn’t allow me to wait on them.

In other words: with a workflow ID, am I able to get the handle for said workflow from another workflow? If not, is what I’m doing incorrect?

Thank you in advance.

2

This is not supported today. Child workflows are subject to the parent close policy when the parent run is closed, which is what continue as new does (closes the current run, starts anew).

You may need workflows that are not tied to a specific run. Sure Abandon parent close policy for children can technically do this, but you can just as easily start a workflow from an activity too. Regardless, the common pattern here is to have the other workflow signal back to its “starting” workflow ID. So when you start these other workflows, can pass the workflow ID to signal back to upon completion, and have those other workflows use GetExternalWorkflowHandle to signal back with the result. Using just workflow ID without a run ID will signal to whichever one is running (e.g. the after-continue-as-new one).

Just make sure in the overarching one that receives these signals, it doesn’t continue as new while still processing any signals or they will be lost. Can have something like await Workflow.WaitConditionAsync(() => Workflow.AllHandlersFinished) just before continuing as new. There is technically the possibility that if you have long-running signal (i.e. other-workflow-complete) processing, you can get completions in too frequently to ever have a chance to continue as new, so keep that in mind and carryover signals-to-process to the continued-as-new one for processing if needed.

Alternatively, consider whether you need child workflows vs just inlining whatever that child may do. This may not be possible due to history size or code separation or complexity, but for many it is possible.

3

Here is a sliding window batch sample that uses the abandoned child signaling the parent approach: samples-java/core/src/main/java/io/temporal/samples/batch/slidingwindow at main · temporalio/samples-java · GitHub

I don’t think this one was ported to .NET yet.

4

We need to be able to use the child-workflows in various scenarios, so we can’t just take their code and put it inside the “main” workflow. It would break other workflows or we would need to duplicate code.

As for having the child-workflow notify the parent, we went with this approach and it works, but there’s a huge downside: the child-workflow needs to have logic and be aware of this, it can’t simply do its thing… It breaks a bit our modular design.

@Chad_Retz since you said “This is not supported today”, and if I can, I would like to suggest some API that would allow this type of scenario. The GetExternalWorkflowHandle is almost there, we just need the part where we can wait on it.

Well, regardless, thank you for your help.

5

We do plan to add ability to wait for a workflow started by other workflow. No ETA.