Unhandled Command - Typescript SDK

Community Support
1

Hi there,
I wanted to figure out what this error means. The master workflow shows as failed because of it. Below is the error and screenshot.
Unhandled Command

The Workflow Task failed because there are new available events since the last Workflow Task started. A retry Workflow Task has been scheduled and the Workflow will have a chance to handle those new events.

Context
This is a scheduled workflow which runs at an interval of every 5 mins, where master workflow starts 22 child workflows in parallel and finishes the execution. Master workflow does not wait for children to finish. Child workflows finish on their own once they receive data from third party API call, the data is processed and stored in db.

The failure we see here is random. If I observe the timeline, I can see dots where it seemed to me that child workflow execution has not started. I am not sure what could be causing this. Can you please help me figure out what could be the problem?

We are using 1 dedicated worker for this job.

SDK used: Typescript

Temporal configuration:
Master workflow execution timeout - 4 mins
Child workflow execution timeout - 10 mins
Activity timeout - 8 mins
Activity retries - 3
No. of activities - 1

image
image2330×741 55 KB

2
3

Hey @maxim, Thanks for quick response.
I already went through the post earlier but could not figure out what could be the problem in my case. I believe the child workflows from previous run took more time which caused current workflow to fail because child workflow already exists. Is it so?

How can I reduce from happening this?

There are 2 child workflows which failed to start with cause - WorkflowAlreadyExists

Attaching the event history screenshot for giving more context.

image
image1920×3910 586 KB

4

I believe the child workflows from previous run took more time which caused current workflow to fail because child workflow already exists. Is it so?

How can I reduce from happening this?

Don’t assign child workflow IDs at all or assign them so that they don’t collide. Or make sure that the previous child workflow invocation is completed before starting the new one.

6

Ok. So one thing is sure that we want to assign child workflow IDs.

  1. If we assign them unique workflow IDs, let’s say we append date time, won’t they overlap with each other?
  2. How do we check if the previous child workflow invocation is completed or not?

I still want to keep the child workflows behavior async from master/parent workflows.

7

It is hard to recommend without understanding the use case. What exactly are you trying to achieve here?

8

Yeah. Let me explain. So basically we are using scheduler to run our workflows. This particular workflow that I have shared has a schedule set to runs every 5 mins. Scheduler starts master workflow.

Below is what Master workflow does:

  • There is an activity defined which get a list of properties from database
  • It loops through each and initiates child workflow for each property asynchronously in parallel
  • Once all the child workflows have initiated, master workflow finishes the execution
  • Here is the temporal configuration for master workflow
    • ScheduleOverlapPolicy.SKIP is set on master workflow
    • workflowExecutionTimeout is 3 mins

Below is what Child workflow does:

  • It calls database to get additional information, makes a call to an external API
  • Results from API gets processed and stored in database, completes execution
  • Here is the temporal configuration for child workflow
    • workflowExecutionTimeout is 3 mins
    • parentClosePolicy is set to PARENT_CLOSE_POLICY_ABANDON

Here, master workflow does not wait for child workflows to finish. It just makes sure that all the child workflows should be initialized.

Activity configuration:

  • startToCloseTimeout is 8m
  • maximumAttempts to retry are 3

There are 22 child workflows running in parallel. Here, external API call could take more than 5 mins to get the results. This behavior is uncertain. Since we don’t have any control over external API we can’t do anything about it.

What do you think? What can we improve here?

9

I would not complete the parent workflow until all the children are completed. Then, I would prohibit overlaps in the schedule.

10

Thanks @maxim for the suggestion. So you think this is the best way to prohibit overlaps in the schedule? Otherwise we could run into this kind of issues!!

11

This makes the schedule linked to the children. Otherwise they are independent and can step on each other.

1 Like