Custom search attributes lost after 1.19 -> 1.20

Hello! I’ve hit a issue with the upgrade path from 1.19 → 1.20. Hoping the community can help me out.

Cluster Description:

Temporal Server 1.19
Admin-tools 1.20
Temporal UI Version 2.10.3
Elastic Search Version: 7.10 
PostgreSQL: 11.17

The server is configured with 3 Custom search attributes:

Bool, Datetime, Text.

I followed the upgrade path where I upgraded admin-tools to 1.20. Ran the temporal-sql-tool and re ran the Elastic search curl commands listed here.

But when I upgrade the server to 1.20 I lose my custom search attributes. I also can’t re add them using tctl.

Example: 

tctl admin cluster add-search-attributes --name AwaitingInvestingOps --type Bool
You are about to add search attributes [AwaitingInvestingOps:Bool]. Continue? Y/N y

Error: Unable to add search attributes.
Error Details: rpc error: code = Unavailable desc = Workflow temporal-sys-add-search-attributes-workflow returned an error: workflow execution error (type: temporal-sys-add-search-attributes-workflow, workflowID: temporal-sys-add-search-attributes-workflow, runID: 0ea37bf1-81a6-43f4-b9f6-1c8fb86c1130): unable to execute activity: AddESMappingFieldActivity: activity error (type: AddESMappingFieldActivity, scheduledEventID: 5, startedEventID: 6, identity: 1@temporal-fos-server-worker@): unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): no permissions for [] and User [name=user_name, backend_roles=[], requestedTenant=null] [type=security_exception] (type: wrapError, retryable: true): unable to execute activity.

If I downgrade back to 1.19 I can view/add/delete custom search attributes. I read the Release notes on 1.20 but it did not seem to apply to my use-case ( maybe I am wrong).

Has anyone run into this before?

Thanks!

Managed too create a Partial reproduction of the issue. It seems if the namespace is set to global this occurs. I reproduced it locally using the docker-compose repo