# Temporal client failure when server restarts

**URL:** <https://community.temporal.io/t/temporal-client-failure-when-server-restarts/10161>\
**Category:** Community Support\
**Tags:** python-sdk\
**Created:** [November 21, 2023, 3:46pm UTC](https://community.temporal.io/t/temporal-client-failure-when-server-restarts/10161 "2023-11-21T15:46:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Benedito\_Marques](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/benedito_marques/32/3588_2.png) [@Benedito\_Marques](https://community.temporal.io/u/Benedito_Marques)\
**Post date:** [November 21, 2023, 3:46pm UTC](https://community.temporal.io/t/temporal-client-failure-when-server-restarts/10161/1 "2023-11-21T15:46:16Z")

</div>

Hi everyone.  
I’m using temporalio python SDK version **0.1b2** and I see that if Temporal Server restarts, the error bellow appears on client, and pending workflows are not executed anymore by this client:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/9/98491d69be4af5618fbbcfb3e03ce3b115d07476.png)

So, how to restart the client/application, or retry connection automatically, if connection with server is lost in any moment?

---

<div class="post-metadata">

**Author:** ![craigd](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/craigd/32/3166_2.png) [@craigd](https://community.temporal.io/u/craigd)\
**Post date:** [December 7, 2023, 6:02pm UTC](https://community.temporal.io/t/temporal-client-failure-when-server-restarts/10161/2 "2023-12-07T18:02:53Z")

</div>

The SDK is at 1.4.0. You should upgrade!

I’d be interested to know what the recommended way of autoreconnecting as with the latest client SDK as if the Temporal server is restarted the client does not autoreconnect.

With a bit of trial and error trying to work out how to solve this, I found that the act of calling `check_health()` on the Temporal service client revives the connection after the Temporal server restarts so I am using this:

```auto
import asyncio
import logging
from typing import Optional
from temporalio.client import Client

TEMPORAL_SERVER = "localhost:7233"
TEMPORAL_NAMESPACE = "default"

logger = logging.getLogger( __name__ )

class TemporalClientManager:
    def __init__ (self) -> None:
        self.client: Optional[Client] = None
        self.client_health_check_period = 10

    async def connect(self) -> None:
        # Start task to periodically reconnect to Temporal if connection drops
        asyncio.create_task(self._keep_alive())
        # Make initial connection
        logger.info(f"Connecting to Temporal server {TEMPORAL_SERVER}")
        self.client = await Client.connect(
            target_host=TEMPORAL_SERVER,
            namespace=TEMPORAL_NAMESPACE,
        )

    async def _keep_alive(self) -> None:
        while True:
            # If disconnected, act of checking health appears to reconnect client
            await asyncio.sleep(self.client_health_check_period)
            if await self.is_connected():
                logger.debug(
                    f"Connection to Temporal server '{TEMPORAL_SERVER}' is alive"
                )
            else:
                logger.error(
                    f"Connection to Temporal server '{TEMPORAL_SERVER}' failed"
                )

    async def is_connected(self) -> bool:
        try:
            return await self.client.service_client.check_health()
        except Exception as e:
            logger.warning(f"Failed to check Temporal health: {e}")
            return False

temporal_client_mgr = TemporalClientManager()

```

I am not confident this is a good solution as I periodically get a “transport error” when running the health check:

17:51:39.903 | DEBUG | \_keep\_alive:95 | 🐞 Connection to Temporal server ‘localhost:7233’ is alive  
17:51:49.914 | WARNING | is\_connected:108 | ⚠ Failed to check Temporal health: transport error  
17:51:49.915 | ERROR | \_keep\_alive:100 | ❌ Connection to Temporal server ‘localhost:7233’ failed  
17:51:59.927 | DEBUG | \_keep\_alive:95 | 🐞 Connection to Temporal server ‘localhost:7233’ is alive  
.  
.  
.  
17:57:10.360 | DEBUG | \_keep\_alive:95 | 🐞 Connection to Temporal server ‘localhost:7233’ is alive  
17:57:20.364 | WARNING | is\_connected:108 | ⚠ Failed to check Temporal health: transport error  
17:57:20.364 | ERROR | \_keep\_alive:100 | ❌ Connection to Temporal server ‘localhost:7233’ failed  
17:57:30.378 | DEBUG | \_keep\_alive:95 | 🐞 Connection to Temporal server ‘localhost:7233’ is alive  
17:57:40.401 | DEBUG | \_keep\_alive:95 | 🐞 Connection to Temporal server ‘localhost:7233’ is alive

---

<div class="post-metadata">

**Author:** ![craigd](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/craigd/32/3166_2.png) [@craigd](https://community.temporal.io/u/craigd)\
**Post date:** [December 7, 2023, 6:09pm UTC](https://community.temporal.io/t/temporal-client-failure-when-server-restarts/10161/3 "2023-12-07T18:09:56Z")

</div>

It looks like there is an open issue for this the client not auto-reconnecting:

> <https://github.com/temporalio/sdk-python/issues/397>
>
> \### Describe the bug
> 
> There are reports of clients not auto-reconnecting on se…rver shutdown/restart. Confirm that clients for direct use and for worker use are properly auto-reconnecting. This may be solved by #396, but confirm.

---

<div class="post-metadata">

**Author:** ![Chad\_Retz](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/chad_retz/32/1396_2.png) [@Chad\_Retz](https://community.temporal.io/u/Chad_Retz)\
**Post date:** [December 7, 2023, 7:12pm UTC](https://community.temporal.io/t/temporal-client-failure-when-server-restarts/10161/4 "2023-12-07T19:12:09Z")

</div>

> [@craigd](#):
>
> I’d be interested to know what the recommended way of autoreconnecting as with the latest client SDK as if the Temporal server is restarted the client does not autoreconnect.

High-level client calls should automatically retry if they fail because of a connection is no longer available (as opposed to low level calls on one of the service fields, but you can set retry on those too). 1.4.0 now has keep alive built in by default (30s interval, 15s timeout). Which situations are you making a call after server restart that is failing? Is it a high level call like start workflow?

EDIT: I just updated/closed [the GH issue](https://github.com/temporalio/sdk-python/issues/397) with a test I performed confirming worker and client both recover

---

<div class="post-metadata">

**Author:** ![craigd](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/craigd/32/3166_2.png) [@craigd](https://community.temporal.io/u/craigd)\
**Post date:** [December 8, 2023, 9:43am UTC](https://community.temporal.io/t/temporal-client-failure-when-server-restarts/10161/5 "2023-12-08T09:43:46Z")

</div>

@Chad_Retz - please accept my apologies. I added my workaround when I was running 1.3.0 but didn’t retest on 1.4.0 because I found [#397](https://github.com/temporalio/sdk-python/issues/397) open.

I only require autoreconnect to work for high-level calls so after reading your reply I removed my workaround code and found version 1.4.0 does reconnect automatically.

Thanks for the quick response.