We have discovered that the wrong clusterInformation made its way into the cluster rpcAddress which only affects the new scheduler feature. Unfortunately Temporal only load the clusterMetadata section of the static config once in a lifetime and after that any change in the clusterMetadata do not reflect. tctl does not have a feature to change the local cluster metadata either, it only applies to the remote clusters.
Could someone please suggest how we can change the cluster_address recorded in the cluster_metadata_info table for the local cluster?
What’s the server version you are deploying?
Take a look at this forum post that might help, let me know.
It is 1.17.4.
I’ve seen the post and what I understood the only way forward is to stop the services, delete the data from the metadata table and let temporal create a new metadata on startup. I wonder if removing the data is safe enough and won’t break the cluster?
We have deleted the local cluster record from the cluster_metadata_info table and recycled the deployment. This re-created the metadata record with the updated endpoint and didn’t break the XDC.
After recycling the remote cluster metadata it is also nice to delete/add the remote cluster as cluster_id GUID is changing up on the metadata re-initialisation.