Search Attributes API Failing after upgrading from 1.22 to 1.24 Temporal Server

Community Support Server Deployment
1

Hi,

I upgraded my temporal server from 1.22 to 1.24 Temporal Server. I deleted my existing MySQL 8 tables and used the new DDL scripts, but now I am seeing this issue when I go into the UI.

API: v1/namespaces/link-and-provisioning/search-attributes?

Error Log: {“level”:“error”,“ts”:“2024-09-11T18:39:23.168Z”,“msg”:“service failures”,“operation”:“OperatorListSearchAttributes”,“wf-namespace”:“link-and-provisioning”,“error”:“Unable to get namespace link-and-provisioning info with error: context deadline exceeded”,“logging-call-at”:“telemetry.go:411”,“stacktrace”:"go.temporal.io/server/common/log.

This is a weird error as this name space definitely exist because when the search-attributes times out, i get a 200 success when it is able to fetch the link-and-provisioning namespace.

2

Hey @tihomir is it possible to get the Temporal team to look at this? We are only seeing this issue for the OperatorListSearchAttributes operation. Everything else is working fine.

3

Does this happen all the time for your or intermittently?
Can you run
tctl adm cl d
and look at your ring members see if anything stands out (as in should not be there). asking because have seen this happen before when you have multiple cluster members join together due to some membership misconfig

4

This happens everytime.

We are self-hosting each service inside their own pods and currently running single replicas for each pod.

Let me run my temporal server locally and get back to you.

5

I got this info:

{
“supportedClients”: {
“temporal-cli”: “\u003c2.0.0”,
“temporal-go”: “\u003c2.0.0”,
“temporal-java”: “\u003c2.0.0”,
“temporal-php”: “\u003c2.0.0”,
“temporal-server”: “\u003c2.0.0”,
“temporal-typescript”: “\u003c2.0.0”,
“temporal-ui”: “\u003c3.0.0”
},
“serverVersion”: “1.24.2”,
“membershipInfo”: {
“currentHost”: {
“identity”: “172.17.0.4:7233”
},
“reachableMembers”: [
“172.17.0.4:6933”,
“172.17.0.2:6934”,
“172.17.0.3:6935”
],
“rings”: [
{
“role”: “frontend”,
“memberCount”: 1,
“members”: [
{
“identity”: “172.17.0.4:7233”
}
]
},
{
“role”: “history”,
“memberCount”: 1,
“members”: [
{
“identity”: “172.17.0.2:7234”
}
]
},
{
“role”: “matching”,
“memberCount”: 1,
“members”: [
{
“identity”: “172.17.0.3:7235”
}
]
},
{
“role”: “worker”
}
]
},
“clusterId”: “3448c440-ac9e-4c27-94a0-72b01d0bcdd3”,
“clusterName”: “active”,
“historyShardCount”: 200,
“persistenceStore”: “mysql8”,
“visibilityStore”: “mysql8”,
“failoverVersionIncrement”: “10”,
“initialFailoverVersion”: “1”,
“isGlobalNamespaceEnabled”: true
}