# Workflow.Sleep caused Non-deterministic error

**URL:** https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495
**Category:** Community Support
**Tags:** go-sdk, general-impl
**Created:** [September 15, 2023, 6:59am UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495 "2023-09-15T06:59:15Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![Bryan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/bryan/32/3245_2.png) [@Bryan](https://community.temporal.io/u/Bryan)
#### Post date: [September 15, 2023, 6:59am UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/1 "2023-09-15T06:59:15Z")

</div>

Hi, our workflow is currently facing a non-deterministic error. The code uses workflow.Sleep() to implement below logic:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/e/ea8c58b94eaa37448768b15c60893181fc10d491.png)

It will sleep for 6 hours, then execute an activity, then sleep again if success condition is not met.  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/1/1a04b7091f34ef39b9f4dd1cd2ca7f3e06b7a3d2.png)

After 6 hours, timer is fired:  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/6/6d89b02db28869b26b15f6f8ea784e729d1e5b48.png)

The function works as expected.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/2/2c207e193f70f50276b33dcd8608fcb5a9c67c2d.png)

A sleep timer started at Aug 10th 6:30:00 pm. It is expected to be fired 6 hours later, which is Aug 11th 12:30:00. However, a problem occurred when switching the DB to another site (where cassandraDB data is stored). The activity occurred during the sleep period. After the activity completed, the timer was fired at Aug 11th 1:02:46 am (32 mins delay):  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/3/38891b750d3d63ae47eacd018467b0f851502131.png)

Then, it got WorkflowTaskTimedOut error:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/d/dca2fbe186d6d17ed1cdecd3bf5e19b66e5b51cf.png)

The next timer started at Aug 11th 1:02:59 am, and fired at Aug 11th 7:02:59 am. Then, it got WorkflowTaskFailed:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/a/a2a107af723078dde7a34927796f721d0300339a.png)

> PanicError: unknown command CommandType: Timer, ID: 384, possible causes are nondeterministic workflow definition code or incompatible change in the workflow definition  
> process event for PSB\_WEB\_WF\_QUEUE [panic]:  
> [go.temporal.io/sdk/internal.panicIllegalState(...)](http://go.temporal.io/sdk/internal.panicIllegalState(...))  
> /build/vendor/go.temporal.io/sdk/internal/internal\_decision\_state\_machine.go:409  
> [go.temporal.io/sdk/internal.(\*commandsHelper).getCommand](http://go.temporal.io/sdk/internal.(*commandsHelper).getCommand)(0xc040de3360, 0xc000000004, 0xc0005b7028, 0x3, 0x0, 0x0)

So, there was no code change, only DB migration, which caused delay in the sleep timer, and then NDE.  
My question is, for this scenario, can the transaction still continue? Or is it a loss and can only be terminated?

Thank you.

---

<div class="post-metadata">

### Author: ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)
#### Post date: [September 15, 2023, 6:52pm UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/2 "2023-09-15T18:52:37Z")

</div>

How do you calculate the delay? You have to use workflow.Now(…) to get the current time.

---

<div class="post-metadata">

### Author: ![Bryan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/bryan/32/3245_2.png) [@Bryan](https://community.temporal.io/u/Bryan)
#### Post date: [September 17, 2023, 6:36pm UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/3 "2023-09-17T18:36:20Z")

</div>

Hi Maxim. The delay is calculated using subtraction. Below is the implementation:

```auto
deliveryHasIncident := false
subProcessDurationLeft := time.Duration(2 * 24 * 60) * time.Minute
sleepDuration := time.Duration(6 * 60) * time.Minute

for true {
  status = executeActivity()

  if status == SUCCESS {
    break
  } else {
    if subProcessDurationLeft <= 0 {
      deliveryHasIncident = true
      break
    } else if subProcessDurationLeft < sleepDuration {
      workflow.Sleep(ctx, subProcessDurationLeft)
    } else {
      workflow.Sleep(ctx, sleepDuration)
    }

    subProcessDurationLeft = subProcessDurationLeft - sleepDuration
  }
}

```

Do you have any suggestion on how to implement the sleep function?

---

<div class="post-metadata">

### Author: ![Quinn\_Klassen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/quinn_klassen/32/3948_2.png) [@Quinn\_Klassen](https://community.temporal.io/u/Quinn_Klassen)
#### Post date: [September 20, 2023, 4:00am UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/4 "2023-09-20T04:00:09Z")

</div>

Could you share the full history of the workflow? Given your description of the workflow I don’t understand what events occurred between ID 384 and 391.

---

<div class="post-metadata">

### Author: ![Bryan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/bryan/32/3245_2.png) [@Bryan](https://community.temporal.io/u/Bryan)
#### Post date: [September 20, 2023, 5:23pm UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/5 "2023-09-20T17:23:17Z")

</div>

Below is the history:

 ![NDE](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/9/941ac123fa68abc4276563b3e2775e8fcabe38ed.png)

---

<div class="post-metadata">

### Author: ![Quinn\_Klassen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/quinn_klassen/32/3948_2.png) [@Quinn\_Klassen](https://community.temporal.io/u/Quinn_Klassen)
#### Post date: [September 20, 2023, 6:03pm UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/6 "2023-09-20T18:03:16Z")

</div>

Thank you for sharing the history. It looks like your history is corrupted, the TimerFired event at event 389 thinks the timer was started at event ID 384, but event 384 is a WorkflowTaskScheduled event not a timer event. That bad TimerFired event is written by the server. Your DB migration, was the DB that backs your temporal cluster yes?

---

<div class="post-metadata">

### Author: ![Bryan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/bryan/32/3245_2.png) [@Bryan](https://community.temporal.io/u/Bryan)
#### Post date: [September 21, 2023, 8:15am UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/7 "2023-09-21T08:15:18Z")

</div>

Yes that is correct. The history is corrupted because of the workflow.Sleep() implementation? Without it, the history should be fine and Temporal auto-recovers when DB is up again right?

If a transaction history is corrupted, can it still be recovered?

---

<div class="post-metadata">

### Author: ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)
#### Post date: [September 25, 2023, 11:23pm UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/8 "2023-09-25T23:23:08Z")

</div>

I guess that it is corrupted due to DB not being consistent.

> switching the DB to another site

Cassandra cross-cluster replication is known to be not consistent and should not be used with Temporal.

---

<div class="post-metadata">

### Author: ![Bryan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/bryan/32/3245_2.png) [@Bryan](https://community.temporal.io/u/Bryan)
#### Post date: [September 26, 2023, 5:43am UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/9 "2023-09-26T05:43:51Z")

</div>

I see. In my case, this transaction cannot be saved anymore and can only be terminated?

---

<div class="post-metadata">

### Author: ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)
#### Post date: [September 26, 2023, 4:29pm UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/10 "2023-09-26T16:29:19Z")

</div>

You can reset this workflow to the point before the corruption.

---

<div class="post-metadata">

### Author: ![Bryan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/bryan/32/3245_2.png) [@Bryan](https://community.temporal.io/u/Bryan)
#### Post date: [September 27, 2023, 8:12am UTC](https://community.temporal.io/t/workflow-sleep-caused-non-deterministic-error/9495/11 "2023-09-27T08:12:36Z")

</div>

I see. I just realized that there is reset feature. Thank you.
