I upgraded my cluster from 1.16.x to 1.22.0, and I have configured a default jwt authorizer. I am also using postgres12 driver for my cluster.
After logging into the temporal web UI, a network call is made to 'api/v1/namespaces/{namespace}/search-attributes? which returns HTTP 503.
This is the error response
{
"code": 14,
"message": "Unable to get namespace info with error: {namespace}",
"details": [
]
}
Looking through the code, I see that operator service’s ListSearchAttributes gets executed which then makes a call to the frontend service.
The code then flows through here where it errors out.
What I think is happening is that NewLocalFrontendClientWithTimeout creates a client without any Authorizer provider and that’s why it is failing.
One other thing I know is that the code enters this code block as I am seeing that metric coming through in my dashboard.
Some more additional context which could be relevant:
The per namespace workers that gets initialized are also crashing with the following error stack trace:
I believe this is also failing due to the same issue but I might be wrong.
"parsed":{"error":"Request unauthorized.","level":"fatal","logging-call-at":"service.go:501","msg":"error starting scanner","service":"worker","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Fatal\n\t/home/builder/temporal/common/log/zap_logger.go:180\ngo.temporal.io/server/service/worker.(*Service).startScanner\n\t/home/builder/temporal/service/worker/service.go:501\ngo.temporal.io/server/service/worker.(*Service).Start\n\t/home/builder/temporal/service/worker/service.go:396\ngo.uber.org/fx/internal/lifecycle.Wrap[...].func1\n\t/go/pkg/mod/go.uber.org/fx@v1.20.0/internal/lifecycle/lifecycle.go:80\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/fx@v1.20.0/internal/lifecycle/lifecycle.go:256\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/fx@v1.20.0/internal/lifecycle/lifecycle.go:216\ngo.uber.org/fx.(*App).start.func1\n\t/go/pkg/mod/go.uber.org/fx@v1.20.0/app.go:679\ngo.uber.org/fx.(*App).withRollback\n\t/go/pkg/mod/go.uber.org/fx@v1.20.0/app.go:661\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/fx@v1.20.0/app.go:678\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/fx@v1.20.0/app.go:782","ts":"2023-09-11T19:15:28.569Z"},"stream":"stdout","time":"2023-09-11T19:15:28.570010675Z"}
Is there any configuration that I am missing on my end? Any help or pointers is appreciated!