HTTP 503 when UI tries to 'DescribeNamespace'

I upgraded my cluster from 1.16.x to 1.22.0, and I have configured a default jwt authorizer. I am also using postgres12 driver for my cluster.

After logging into the temporal web UI, a network call is made to 'api/v1/namespaces/{namespace}/search-attributes? which returns HTTP 503.

This is the error response

  "code": 14,
  "message": "Unable to get namespace info with error: {namespace}",
  "details": [

Looking through the code, I see that operator service’s ListSearchAttributes gets executed which then makes a call to the frontend service.

The code then flows through here where it errors out.

What I think is happening is that NewLocalFrontendClientWithTimeout creates a client without any Authorizer provider and that’s why it is failing.

One other thing I know is that the code enters this code block as I am seeing that metric coming through in my dashboard.

Some more additional context which could be relevant:
The per namespace workers that gets initialized are also crashing with the following error stack trace:
I believe this is also failing due to the same issue but I might be wrong.

"parsed":{"error":"Request unauthorized.","level":"fatal","logging-call-at":"service.go:501","msg":"error starting scanner","service":"worker","stacktrace":"*zapLogger).Fatal\n\t/home/builder/temporal/common/log/zap_logger.go:180\*Service).startScanner\n\t/home/builder/temporal/service/worker/service.go:501\*Service).Start\n\t/home/builder/temporal/service/worker/service.go:396\[...].func1\n\t/go/pkg/mod/\*Lifecycle).runStartHook\n\t/go/pkg/mod/\*Lifecycle).Start\n\t/go/pkg/mod/\*App).start.func1\n\t/go/pkg/mod/\*App).withRollback\n\t/go/pkg/mod/\*App).start\n\t/go/pkg/mod/\\n\t/go/pkg/mod/","ts":"2023-09-11T19:15:28.569Z"},"stream":"stdout","time":"2023-09-11T19:15:28.570010675Z"}

Is there any configuration that I am missing on my end? Any help or pointers is appreciated!

Related post/topic: Temporal Worker unable to talk to Internal Frontend, receiving Request Unauthorized

You should probably use internal-frontend: there’s no (easy) way to get the calls from worker and from operator service to send a valid JWT for the claim mapper + authorizer to pick up.