Temporal UI shows 500 on Nexus

We have enabled Nexus with dynamic config on our internal Temporal cluster. Seems like registry is working fine, we can create Nexus endpoints and run workers through them via temporal cli, but when we try to reach Nexus from UI, we get 500


temporal-web pod shows the following log only

 {"time":"2025-05-02T12:20:15.631641843Z","id":"4228358f58845ef863cb3a6fd2fd5e12","remote_ip":"reducted","host":"reducted","method":"GET","uri":"/api/v1/namespaces?","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","status":200,"error":"","latency":5225767 │
│ {"time":"2025-05-02T12:20:15.634805252Z","id":"5482b4bb332671933a867fc3e39ff3ba","remote_ip":"reducted","host":"reducted","method":"GET","uri":"/api/v1/cluster-info?","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","status":200,"error":"","latency":59998 │
│ {"time":"2025-05-02T12:20:15.646325755Z","id":"2356c8052756ab760e48ffc7c695ef59","remote_ip":"reducted","host":"reducted,"method":"GET","uri":"/api/v1/system-info?","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","status":200,"error":"","latency":317104 │
│ {"time":"2025-05-02T12:24:30.429772065Z","id":"9e49b9e6c8d3d453a7495edb85a8933f","remote_ip":"reducted","host":"reducted","method":"GET","uri":"/api/v1/nexus/endpoints?name=","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","status":500,"error":"","latenc │
│ {"time":"2025-05-02T12:24:30.44599729Z","id":"934037aeb27a928983095dae9a22cb93","remote_ip":"reducted","host":"reducted","method":"GET","uri":"/_app/version.json","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","status":200,"error":"","latency":2400135," │
│ {"time":"2025-05-02T12:24:30.458456818Z","id":"9ce6a4fa9170212d59e93038a1b6fe85","remote_ip":"reducted","host":"reducted","method":"GET","uri":"/_app/version.json","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","status":200,"error":"","latency":643115," │
│                                                                                                                                                                                                                                                                                                                                                                          │

The dynamic config we have used to activate Nexus is

  # This is for Temporal Nexus
    dynamicConfig:
      system.enableNexus:
        - value: true
      component.nexusoperations.callback.endpoint.template:
        - value: https://127.0.0.1:7243/namespaces/{{.NamespaceName}}/nexus/callback
      component.callbacks.allowedAddresses:
        - value:
          - Pattern: "*"
            AllowInsecure: true
  # end

PS:host and IP info is reducted
Anyone has an idea what is wrong here?

The same thing is happening to me while the Nexus operation is still running, after the operation finishes, UI works fine.
Using the latest UI (2.39.0), with this server config:

limit.maxIDLength:
  - value: 255
    constraints: {}
system.forceSearchAttributesCacheRefreshOnRead:
  - value: true # Dev setup only. Please don't turn this on in production.
    constraints: {}
system.enableNexus:
  - value: true
component.nexusoperations.callback.endpoint.template:
  # The URL must be publicly accessible if the callback is meant to be called by external services.
  # When using Nexus for cross namespace calls, the URL's host is irrelevant as the address is resolved using
  # membership. The URL is a Go template that interpolates the `NamepaceName` and `NamespaceID` variables.
  - value: https://0.0.0.0:7243/namespaces/{{.NamespaceName}}/nexus/callback
component.callbacks.allowedAddresses:
  # This list is a security mechanism for limiting which callback URLs are accepted by the server.
  # Attackers may leverage the callback mechanism to force the server to call arbitrary URLs.
  # The config below is only recommended for development, tune this to your requirements.
  - value:
      - Pattern: "*"
        AllowInsecure: true