# Workflow/Activity Getting Stuck on Worker Restart

**URL:** <https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349>\
**Category:** Community Support\
**Tags:** cadence\
**Created:** [August 3, 2020, 6:26pm UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349 "2020-08-03T18:26:20Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![chandanbhattad](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chandanbhattad/32/149_2.png) [@chandanbhattad](https://community.temporal.io/u/chandanbhattad)\
**Post date:** [August 3, 2020, 6:26pm UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/1 "2020-08-03T18:26:20Z")

</div>

Whenever I restart cadence workflow and activity workers - the workflows and activities which were running before worker restart **get stuck.**  
For eg: Workflow gets stuck on “ **Activity Started** ” phase

I have to manually terminate the workflow and submit it again to resume working.

**What should be done to let activities resume after workers are restarted?**

---

<div class="post-metadata">

**Author:** ![chandanbhattad](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chandanbhattad/32/149_2.png) [@chandanbhattad](https://community.temporal.io/u/chandanbhattad)\
**Post date:** [August 3, 2020, 6:28pm UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/2 "2020-08-03T18:28:52Z")

</div>

Both workflow and activity code is in python

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 3, 2020, 6:34pm UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/3 "2020-08-03T18:34:17Z")

</div>

Activities rely on timeouts for retries. Make sure that `StartToClose` timeout is set in the activity options to some reasonable value. If activity is long-running, set `Heartbeat` timeout, and make sure that the activity heartbeats more frequently than that timeout.

---

<div class="post-metadata">

**Author:** ![chandanbhattad](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chandanbhattad/32/149_2.png) [@chandanbhattad](https://community.temporal.io/u/chandanbhattad)\
**Post date:** [August 3, 2020, 6:36pm UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/4 "2020-08-03T18:36:08Z")

</div>

I have set the activity timeout to 5 hours – whereas the activity takes only 1 minute.  
While the workflow is in “Activity Started” state and a I restart workers – the workflow gets stuck at “Activity Started” phase and doesn’t move forward.

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 3, 2020, 6:40pm UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/5 "2020-08-03T18:40:05Z")

</div>

It behaves exactly as designed. When worker goes down the service is going to wait 5 hours to restart the activity. Change the timeout to a couple of minutes, for example and the activity will be restarted in two minutes.

In the future we might implement _worker heartbeat_, then all activities that run on a worker are going to be restarted immediately on worker failure.

---

<div class="post-metadata">

**Author:** ![chandanbhattad](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chandanbhattad/32/149_2.png) [@chandanbhattad](https://community.temporal.io/u/chandanbhattad)\
**Post date:** [August 4, 2020, 4:55am UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/6 "2020-08-04T04:55:05Z")

</div>

Thanks @maxim

---

<div class="post-metadata">

**Author:** ![chandanbhattad](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chandanbhattad/32/149_2.png) [@chandanbhattad](https://community.temporal.io/u/chandanbhattad)\
**Post date:** [August 4, 2020, 6:39am UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/7 "2020-08-04T06:39:09Z")

</div>

@maxim what is the best way to periodically record hearbeat in activity code?  
just putting  
“**Activity.heartbeat(“heartbeat”)**”  
in different parts of activity code feels very weird.

I tried to spawn a thread from main activity thread.

```
def heartbeat_fn():
    while True:
        Activity.heartbeat("heartbeat")
        time.sleep(2)

class ActivityImpl:
    def activity1(self, parameters: dict):
        x = threading.Thread(target=heartbeat_fn)
        x.daemon = True
        x.start()
       .... <main_activity_code>

```

But it gives below error:

```
Activity.heartbeat("heartbeat")
  File "/usr/local/lib/python3.7/site-packages/cadence/activity.py", line 109, in heartbeat
ActivityContext.get().heartbeat(details)
  File "/usr/local/lib/python3.7/site-packages/cadence/activity.py", line 73, in get
return current_activity_context.get()
LookupError: <ContextVar name='current_activity_context' at 0x7fdde93cdd70>
```

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 4, 2020, 4:44pm UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/8 "2020-08-04T16:44:40Z")

</div>

Having heartbeat in the actual activity implementation protects against code being stuck on something. I’ve seen production outages when an activity was completely deadlocked, but happily heartbeat from another thread.

If you still want to do it reverse your implementation. Heatbeat from the main activity thread as it relies on thread local and use different thread to implement the rest of the logic.

---

<div class="post-metadata">

**Author:** ![mohits](https://avatars.discourse-cdn.com/v4/letter/m/bbe5ce/32.png) [@mohits](https://community.temporal.io/u/mohits)\
**Post date:** [March 15, 2021, 6:43am UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/9 "2021-03-15T06:43:44Z")

</div>

@maxim Is there any plan to implement the worker heartbeat in near future as you mentioned in above?If yes, Can i know the ETA ?

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [March 16, 2021, 6:09pm UTC](https://community.temporal.io/t/workflow-activity-getting-stuck-on-worker-restart/349/10 "2021-03-16T18:09:36Z")

</div>

No immediate plans for worker heartbeat feature.