HTTP 503 when UI tries to do anything with grpc

Howdy all!

I’ve got a self hosted temporal instance at $domain/$path/

I’ve got an interesting issue where on every page load, the client tries to load from $domain/$path/api/v1/namespaces/default/search-attributes and gets a 503 error for its trouble (after maybe a 20 second timeout).

message: "Unable to get namespace default info with error: received context error while waiting for new LB policy update: context deadline exceeded"

This happens for both default and temporal-system, though I get different errors if I try a namespace that doesn’t exist.

The page eventually loads and it seems like the cluster is up. I can hit it from the admin-tools just fine.

It seems like this post:

But I don’t have an Authorizer set up at all.

When I start a workflow (that doesn’t exist, just testing), I see…

Uncaught (in promise) 
Object { statusCode: 504, statusText: "", response: Response, message: "context deadline exceeded", report: false }
​
message: "context deadline exceeded"
​
report: false
​
response: Response { type: "basic", url: "$site$path/api/v1/namespaces/default/task-queues/abc?taskQueueType=2", redirected: false, … }
​
statusCode: 504
​
statusText: ""
​
<prototype>: Object { … }
7.Cp8cJTm6.js:1:68714

It’s a little weird.

In other setups, if I trigger a workflow that doesn’t exist on a task queue that doesn’t exist, the UI still works.

This problem persists when I set the environment variable TEMPORAL_ALLOW_NO_AUTH = "true"

It also persists when I use an internal-frontend and point the UI and other services at it instead of the main frontend.