# Is port change necessary for multiple deployments on a single cluster

**URL:** <https://community.temporal.io/t/is-port-change-necessary-for-multiple-deployments-on-a-single-cluster/6325>\
**Category:** Community Support\
**Created:** [October 26, 2022, 8:44am UTC](https://community.temporal.io/t/is-port-change-necessary-for-multiple-deployments-on-a-single-cluster/6325 "2022-10-26T08:44:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yang\_Yu](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yang_yu/32/743_2.png) [@Yang\_Yu](https://community.temporal.io/u/Yang_Yu)\
**Post date:** [October 26, 2022, 8:44am UTC](https://community.temporal.io/t/is-port-change-necessary-for-multiple-deployments-on-a-single-cluster/6325/1 "2022-10-26T08:44:40Z")

</div>

Hi,

I am seen issue of cross connection between different server instances here:

> <https://github.com/temporalio/temporal/issues/1234>
>
> \*\*Is your feature request related to a problem? Please describe.\*\*
> 
> I run mult…iple separate Temporal clusters within a single k8s cluster. Each Temporal cluster has its own separate set of frontend, history, and matching services as well as persistence. Let's say I am running two Temporal clusters called "A" and "B" in a single k8s cluster. Note that in my setup, there are no networking restrictions on pods within the k8s cluster -- any pod may connect to any other pod if the IP address is known.
> 
> I recently encountered a problem where it appeared that a frontend service from Temporal cluster A was talking to a matching service from Temporal cluster B. This happened during a time where the pods in both of the Temporal clusters were getting cycled a lot due to some AZ balancing automation. It also happens that this particular k8s cluster is configured in such a way that pod IP address reuse is more likely than usual.
> 
> Both Temporal cluster A and B are running 3 matching nodes each. However, I saw this log line on Temporal cluster A's frontend service:
> \`\`\`
> {"level":"info","ts":"2021-01-27T00:34:15.414Z","msg":"Current reachable members","service":"frontend","component":"service-resolver","service":"matching","addresses":"\[100.123.207.80:7235 100.123.65.65:7235 100.123.120.28:7235 100.123.60.187:7235 100.123.17.255:7235 100.123.203.172:7235\]","logging-call-at":"rpServiceResolver.go:266"}
> \`\`\`
> This is saying that Temporal cluster A's frontend service is seeing 6 matching nodes, three from A and three from B. Yikes.
> 
> I believe what led to this is something like:
> 1. A matching pod in cluster A gets replaced, releasing its IP address. This IP address remains in cluster A's \`cluster\_metadata\` table.
> 2. A matching pod is created in cluster B re-using this IP address.
> 3. An event occurs that causes a frontend in cluster A to re-read the the node membership for its matching nodes. It finds the original matching node's IP address still in the table and it can still connect to it even though it is actually now a matching node in cluster B.
> 4. Through this matching node in cluster B the the other cluster B matching nodes are located.
> 
> My fix for this is to make sure that each Temporal cluster has its own set of membership ports for each service. This would have prevented the discovery process in cluster A from seeing the pods in cluster B since it would be trying to connect on a different port.
> 
> \*\*Describe the solution you'd like\*\*
> 
> It may be possible to prevent this by including a check that a given node is indeed part of the correct cluster before adding it to the ring.
> 
> \*\*Describe alternatives you've considered\*\*
> 
> I don't believe our k8s environment has an easy way to prevent this using networking restrictions.

And the suggestion server configuration is at here:

> **[Temporal Server self-hosted production deployment | Temporal Documentation](https://docs.temporal.io/server/production-deployment/#faq-multiple-deployments-on-a-single-cluster)**
>
> The information in this page is being dispersed into Knowledge base articles, Cluster concept guide, and the Cluster deployment guide.

My question is:  
Is the configuration “Cluster membership ports should be different for each deployment” necessary? My understanding is the clustermembership is tracked in database, so if we always use a different database for each server, would the service of one server still be able to connect to another service in a different server?

The reason to ask is we have multiple server deployments, and having same set of ports across all servers simplifies many tasks.

regards,  
Yu Yang

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [October 31, 2022, 2:57am UTC](https://community.temporal.io/t/is-port-change-necessary-for-multiple-deployments-on-a-single-cluster/6325/4 "2022-10-31T02:57:21Z")

</div>

> so if we always use a different database for each server, would the service of one server still be able to connect to another service in a different server?

Your thinking is correct, each service writes its membership info to the cluster\_membership table of the persistence-\>defaultStore you set in static config.

---

<div class="post-metadata">

**Author:** ![Yang\_Yu](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yang_yu/32/743_2.png) [@Yang\_Yu](https://community.temporal.io/u/Yang_Yu)\
**Post date:** [November 4, 2022, 1:18am UTC](https://community.temporal.io/t/is-port-change-necessary-for-multiple-deployments-on-a-single-cluster/6325/5 "2022-11-04T01:18:08Z")

</div>

So as long as we make sure each server has a different database, then these server can use the same set of ports, is that correct?

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [November 21, 2022, 1:43pm UTC](https://community.temporal.io/t/is-port-change-necessary-for-multiple-deployments-on-a-single-cluster/6325/6 "2022-11-21T13:43:08Z")

</div>

Yes, you would have to have some sort of network isolation between the clusters so you dont run into the “port already in use” type of issues but generally you are correct.
