Hello! I’ve hit a issue with the upgrade path from 1.19 → 1.20. Hoping the community can help me out.
Cluster Description:
Temporal Server 1.19
Admin-tools 1.20
Temporal UI Version 2.10.3
Elastic Search Version: 7.10
PostgreSQL: 11.17
The server is configured with 3 Custom search attributes:
Bool, Datetime, Text.
I followed the upgrade path where I upgraded admin-tools to 1.20. Ran the temporal-sql-tool and re ran the Elastic search curl commands listed here.
But when I upgrade the server to 1.20 I lose my custom search attributes. I also can’t re add them using tctl.
Example:
tctl admin cluster add-search-attributes --name AwaitingInvestingOps --type Bool
You are about to add search attributes [AwaitingInvestingOps:Bool]. Continue? Y/N y
Error: Unable to add search attributes.
Error Details: rpc error: code = Unavailable desc = Workflow temporal-sys-add-search-attributes-workflow returned an error: workflow execution error (type: temporal-sys-add-search-attributes-workflow, workflowID: temporal-sys-add-search-attributes-workflow, runID: 0ea37bf1-81a6-43f4-b9f6-1c8fb86c1130): unable to execute activity: AddESMappingFieldActivity: activity error (type: AddESMappingFieldActivity, scheduledEventID: 5, startedEventID: 6, identity: 1@temporal-fos-server-worker@): unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): no permissions for [] and User [name=user_name, backend_roles=[], requestedTenant=null] [type=security_exception] (type: wrapError, retryable: true): unable to execute activity.
If I downgrade back to 1.19 I can view/add/delete custom search attributes. I read the Release notes on 1.20 but it did not seem to apply to my use-case ( maybe I am wrong).
Has anyone run into this before?
Thanks!