# Event corruption with child workflows? (Java SDK)

**URL:** <https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180>\
**Category:** Community Support\
**Tags:** java-sdk\
**Created:** [October 16, 2021, 5:24pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180 "2021-10-16T17:24:43Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![sdonovan](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@sdonovan](https://community.temporal.io/u/sdonovan)\
**Post date:** [October 16, 2021, 5:24pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/1 "2021-10-16T17:24:43Z")

</div>

Greetings. We’ve run across a problem you might find interesting. If we launch N workflows, then (i) wait for them to start and (ii) then wait for them to complete, the SDK reports event corruption during processing. The specific exception is:

> `Caused by: java.lang.IllegalStateException: COMMAND_TYPE_START_CHILD_WORKFLOW_EXECUTION doesn't match EVENT_TYPE_START_CHILD_WORKFLOW_EXECUTION_INITIATED with EventId=5 `

Code workflow code below. Note, the child-workflow does nothing. I can upload full Spring/project source if it’s helpful.

We’re using Java SDK 1.4.0 against Temporal 1.12.3 (latest docker-compose using MySQL). It launches 200 child workflows. We’ve also seen the problem launching 50 workflows if those workflows do something else (e.g. send signals). Figured you’d want the simpler code.

```
    final int numChildren = 200;
    final Map<String, Promise<Void>> childWorkflows = new HashMap<>();
    final List<Promise<WorkflowExecution>> childStartPromises = new ArrayList<>();

    // Launch all child workflows.
    for (int i = 0; i < numChildren; i++) {
        final String childWorkflowId = UUID.randomUUID().toString() + "-child-" + i;

        final ChildWorkflow child =
            Workflow.newChildWorkflowStub(
                ChildWorkflow.class,
                ChildWorkflowOptions.newBuilder()
                    .setWorkflowId(childWorkflowId)
                    .setCancellationType(WAIT_CANCELLATION_COMPLETED)
                    .setTaskQueue("test")
                    .setWorkflowTaskTimeout(Duration.ofSeconds(60))
                    .build());

        childWorkflows.put(childWorkflowId, Async.procedure(child::execute));
        childStartPromises.add(Workflow.getWorkflowExecution(child));
    }

    // Wait for all child workflows to start.
    Promise.allOf(childStartPromises).get();

    // Wait for all child workflows to complete.
    final List<Promise<Void>> childEndPromises = new ArrayList<>(childWorkflows.values());
    Promise.allOf(childEndPromises).get();
}

```

Thanks - any assistance would be much appreciated!

Sean

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [October 16, 2021, 5:40pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/2 "2021-10-16T17:40:51Z")

</div>

Hi @sdonovan , looking at the workflow code, it does not look deterministic, specifically:

> [@sdonovan](#):
>
> ```auto
> final String childWorkflowId = UUID.randomUUID().toString() + "-child-" + i;
> 
> ```

see constraints [here](https://docs.temporal.io/docs/java/workflows/#workflow-implementation-constraints) . You should use

`Workflow.randomUUID().toString()`

instead.

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [October 16, 2021, 5:47pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/3 "2021-10-16T17:47:57Z")

</div>

Another thing to look into is looping over unordered collections. This can also cause non-deterministic behavior during replay. Make sure that you use ordered collections.

Some other rules that don’t apply to your code but just to add:

- Don’t use explicit synchronization in your workflow code.

- You can use non-static fields in your workflow definition without having to worry about isolation issues.

- For static fields use [io.temporal.workflow.WorkflowLocal](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/WorkflowLocal.html) or [io.temporal.workflow.WorkflowThreadLocal](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/workflow/WorkflowThreadLocal.html) depending on your use case You can use Atomic variables, but not really needed.

- Don’t use synchronized lists, as that will break workflow determinism

---

<div class="post-metadata">

**Author:** ![sdonovan](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@sdonovan](https://community.temporal.io/u/sdonovan)\
**Post date:** [October 16, 2021, 6:02pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/4 "2021-10-16T18:02:47Z")

</div>

Just replaced `HashMap` with `LinkedHashMap`, and used `Workflow.randomUUID()` and . . . it’s working.

Thank you!

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [October 16, 2021, 6:15pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/5 "2021-10-16T18:15:01Z")

</div>

Glad it’s working. I also think you can simplify your code:

```auto
final int numChildren = 200;
    List<Promise<Void>> childPromises = new ArrayList<>();
    // Launch all child workflows.
    for (int i = 0; i < numChildren; i++) {
      final String childWorkflowId = Workflow.randomUUID().toString() + "-child-" + i;

      final ChildWorkflow child =
          Workflow.newChildWorkflowStub(
              ChildWorkflow.class,
              ChildWorkflowOptions.newBuilder()
                  .setWorkflowId(childWorkflowId)
                  .setCancellationType(ChildWorkflowCancellationType.WAIT_CANCELLATION_COMPLETED)
                  .setTaskQueue("test")
                  .setWorkflowTaskTimeout(Duration.ofSeconds(60))
                  .build());

      childPromises.add(Async.procedure(child:: execute));
    }

    Promise.allOf(childPromises).get();

```

parent workflow should wait until all children have completed in this case.

---

<div class="post-metadata">

**Author:** ![sdonovan](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@sdonovan](https://community.temporal.io/u/sdonovan)\
**Post date:** [October 16, 2021, 6:16pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/6 "2021-10-16T18:16:22Z")

</div>

Thanks, it’s OK. In our case, we specifically need to know when the child workflows have started, such that we can send signals to them.

---

<div class="post-metadata">

**Author:** ![sp13](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sp13](https://community.temporal.io/u/sp13)\
**Post date:** [October 16, 2021, 9:07pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/7 "2021-10-16T21:07:03Z")

</div>

I am getting a different error with same scenario for the below code. I am using a list to collect the executions and workflow results, so I hope determinism is not a problem here.

```auto
List<Promise<WorkflowExecution>> executionResults = new ArrayList<>();
List<Promise<Void>> results = new ArrayList<>();

for(int i=0 ;i< 150; i++) {
    ChildWorkflow childWf =
            Workflow.newChildWorkflowStub(ChildWorkflow.class, ChildWorkflowOptions.newBuilder()
                    .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON)
                    .setWorkflowId("workflow-"+i)
                    .build());

    results.add(Async.procedure(childWf::execute));
    executionResults.add(Workflow.getWorkflowExecution(childWf));

}

//wait for all childworkflows to get spawned
Promise.allOf(executionResults).get();
//wait for all childworkflows to complete
Promise.allOf(results).get();

```

If you examine the events are after elapsed time 6s, you can see a `WorkflowTaskTimedOut` with `ScheduleToStart`  
I believe this belongs to `Promise.allOf(executionResults).get();`

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [October 16, 2021, 11:27pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/8 "2021-10-16T23:27:59Z")

</div>

> [@sp13](#):
>
> I am getting a different error with same scenario for the below code.

What is the error you are getting? `ParentClosePolicy` doesn’t apply here as you are waiting for all child workflows to complete.

---

<div class="post-metadata">

**Author:** ![sp13](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sp13](https://community.temporal.io/u/sp13)\
**Post date:** [October 16, 2021, 11:28pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/9 "2021-10-16T23:28:50Z")

</div>

I am seeing a workflowtasktimeout. ScheduletoStart

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [October 16, 2021, 11:29pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/10 "2021-10-16T23:29:38Z")

</div>

Is this the last event in the history?

---

<div class="post-metadata">

**Author:** ![sp13](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sp13](https://community.temporal.io/u/sp13)\
**Post date:** [October 16, 2021, 11:30pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/11 "2021-10-16T23:30:40Z")

</div>

No, it happens in an event at around 6s

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [October 16, 2021, 11:31pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/12 "2021-10-16T23:31:17Z")

</div>

It is benign then. Is workflow completed OK?

---

<div class="post-metadata">

**Author:** ![sp13](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sp13](https://community.temporal.io/u/sp13)\
**Post date:** [October 16, 2021, 11:32pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/13 "2021-10-16T23:32:01Z")

</div>

Workflow completed all okay.

---

<div class="post-metadata">

**Author:** ![sp13](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sp13](https://community.temporal.io/u/sp13)\
**Post date:** [October 16, 2021, 11:42pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/14 "2021-10-16T23:42:11Z")

</div>

I don’t know if it is by design - it schedules, fails and reschedule again until it success in this case. Just wanted to see if this is normal.

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [October 16, 2021, 11:45pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/15 "2021-10-16T23:45:31Z")

</div>

Yes, it is by design. It looks like the worker had failed or was busy. So the workflow task that was scheduled just for that worker wasn’t picked up fast enough and was rescheduled to be picked up by another worker. See the [post](https://community.temporal.io/t/different-tasklist-names-in-the-same-workflow/142/2) for the details of this mechanism.

---

<div class="post-metadata">

**Author:** ![sp13](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sp13](https://community.temporal.io/u/sp13)\
**Post date:** [October 16, 2021, 11:55pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/16 "2021-10-16T23:55:04Z")

</div>

//When a task in a host specific task queue times out it is immediately rescheduled to a shared task list for other hosts to pick up.//  
Now all the puzzle pieces fit for me. Thanks very much.  
Since there was a task failure in sticky queue, temporal rescheduled and replayed the workflow which had a bug in determinism and caused the other issue.

---

<div class="post-metadata">

**Author:** ![sp13](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sp13](https://community.temporal.io/u/sp13)\
**Post date:** [October 17, 2021, 12:02am UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/17 "2021-10-17T00:02:50Z")

</div>

I did a sysout and saw that workflow execution is indeed replayed.  
Thinking about it, it was my local mac just executing this workflow. Worker was not too busy. If this is going to be a common case, worker cache will be under-utilized.

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [October 17, 2021, 12:27am UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/18 "2021-10-17T00:27:54Z")

</div>

It is certainly not a common case unless something is broken in your local setup.

---

<div class="post-metadata">

**Author:** ![sp13](https://avatars.discourse-cdn.com/v4/letter/s/aca169/32.png) [@sp13](https://community.temporal.io/u/sp13)\
**Post date:** [October 18, 2021, 2:30pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/19 "2021-10-18T14:30:29Z")

</div>

This is what I am trying with docker compose v1.12. Not sure what is wrong in the setup. If you get a chance, please give it a try. It seems to me that Promise.get() is scheduled way up in the chain of events and timeout here is inevitable unless prmoise.get() is scheduled at appropriate time.

> <https://github.com/sp13ceg/samples-java/blob/master/src/main/java/io/temporal/samples/hello/ParentChildWorkflow.java>

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [October 18, 2021, 4:18pm UTC](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180/20 "2021-10-18T16:18:21Z")

</div>

I’m able to run the code you posted without any problem.

[Next page](https://community.temporal.io/t/event-corruption-with-child-workflows-java-sdk/3180.md?page=2)