# Unexpected health check behavior when connecting to multiple clusters

**URL:** <https://community.temporal.io/t/unexpected-health-check-behavior-when-connecting-to-multiple-clusters/5538>\
**Category:** Community Support\
**Tags:** java-sdk, multicluster, healthcheck\
**Created:** [August 1, 2022, 7:35pm UTC](https://community.temporal.io/t/unexpected-health-check-behavior-when-connecting-to-multiple-clusters/5538 "2022-08-01T19:35:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nathan\_shields](https://avatars.discourse-cdn.com/v4/letter/n/bc79bd/32.png) [@nathan\_shields](https://community.temporal.io/u/nathan_shields)\
**Post date:** [August 1, 2022, 7:35pm UTC](https://community.temporal.io/t/unexpected-health-check-behavior-when-connecting-to-multiple-clusters/5538/1 "2022-08-01T19:35:09Z")

</div>

I’m working on some code that manages connections to multiple different temporal clusters, and I only see this issue when I’m connecting to two+ clusters. The health check works fine when connecting to a single cluster only.

Unexpected test behavior, using local temporalite clusters:

1. Start cluster, call health check, returns healthy ✅
2. Turn off cluster, call health check, returns unhealthy ✅
3. Turn on cluster, wait 5-90 seconds (same result expected no matter how long I wait), call health check, returns unhealthy ❌
4. Leave cluster on, call health check 2nd (or more) times, returns healthy ✅

Test code + comments/instructions, run against 2 local temporalite clusters (ports 7233, 7234):

```java
        WorkflowFacade primary = supplier.getHealthyFacade().orElseThrow();
        // Turn off primary cluster
        WorkflowFacade fallback = supplier.getHealthyFacade().orElseThrow();
        assertNotEquals(primary, fallback);
        // Turn the primary cluster on again
        // another == fallback, where it should be primary, because the health check on primary failed.
        WorkflowFacade another = supplier.getHealthyFacade().orElseThrow();
        // This assertion fails
        assertEquals(primary, another);

```

Error from the health check that ran on line `WorkflowFacade another = ...`, which is step 3:

```auto
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.46.0.jar:1.46.0]
...
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7233
Caused by: java.net.ConnectException: Connection refused

```

Here’s my health check code:

```java
return ListenableFuturesExtra.toCompletableFuture(getService().futureStub()
                    .getClusterInfo(GetClusterInfoRequest.newBuilder().build()))
                    .thenApplyAsync(GetClusterInfoResponse::isInitialized)
                    .orTimeout(2000, TimeUnit.MILLISECONDS);

```

Any ideas why after the cluster is turned off and back on, the first health check call returns unhealthy?

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [August 3, 2022, 6:37pm UTC](https://community.temporal.io/t/unexpected-health-check-behavior-when-connecting-to-multiple-clusters/5538/3 "2022-08-03T18:37:10Z")

</div>

> Unexpected test behavior, using local temporalite clusters

Are you using [temporalite from the temporal repo](https://github.com/temporalio/temporalite)?

> Any ideas why after the cluster is turned off and back on, the first health check call returns unhealthy?

Tried to reproduce and was getting

`DEADLINE_EXCEEDED: deadline exceeded after 4.960510536s.` for about 10 seconds before health check returned success.

Temporalite is experimental and this could be a bug, do you mind [opening issue](https://github.com/temporalio/temporalite/issues) and post your full error trace? Thanks.

---

<div class="post-metadata">

**Author:** ![nathan\_shields](https://avatars.discourse-cdn.com/v4/letter/n/bc79bd/32.png) [@nathan\_shields](https://community.temporal.io/u/nathan_shields)\
**Post date:** [August 3, 2022, 10:14pm UTC](https://community.temporal.io/t/unexpected-health-check-behavior-when-connecting-to-multiple-clusters/5538/4 "2022-08-03T22:14:40Z")

</div>

> [@tihomir](#):
>
> Are you using [temporalite from the temporal repo](https://github.com/temporalio/temporalite)?

I had been using the [DataDog repo](https://github.com/DataDog/temporalite), which I cannot find anymore. So I moved over to the temporal repo you linked and reproduced the issue.

> [@tihomir](#):
>
> Temporalite is experimental and this could be a bug, do you mind [opening issue](https://github.com/temporalio/temporalite/issues) and post your full error trace? Thanks.

Done! [https://github.com/temporalio/temporalite/issues/101](https://github.com/temporalio/temporalite/issues/101)