Custom search attributes lost after 1.19 -> 1.20

Community Support Server Deployment
1

Hello! I’ve hit a issue with the upgrade path from 1.19 → 1.20. Hoping the community can help me out.

Cluster Description:

Temporal Server 1.19
Admin-tools 1.20
Temporal UI Version 2.10.3
Elastic Search Version: 7.10 
PostgreSQL: 11.17

The server is configured with 3 Custom search attributes:

Bool, Datetime, Text.

I followed the upgrade path where I upgraded admin-tools to 1.20. Ran the temporal-sql-tool and re ran the Elastic search curl commands listed here.

But when I upgrade the server to 1.20 I lose my custom search attributes. I also can’t re add them using tctl.

Example: 

tctl admin cluster add-search-attributes --name AwaitingInvestingOps --type Bool
You are about to add search attributes [AwaitingInvestingOps:Bool]. Continue? Y/N y

Error: Unable to add search attributes.
Error Details: rpc error: code = Unavailable desc = Workflow temporal-sys-add-search-attributes-workflow returned an error: workflow execution error (type: temporal-sys-add-search-attributes-workflow, workflowID: temporal-sys-add-search-attributes-workflow, runID: 0ea37bf1-81a6-43f4-b9f6-1c8fb86c1130): unable to execute activity: AddESMappingFieldActivity: activity error (type: AddESMappingFieldActivity, scheduledEventID: 5, startedEventID: 6, identity: 1@temporal-fos-server-worker@): unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): no permissions for [] and User [name=user_name, backend_roles=[], requestedTenant=null] [type=security_exception] (type: wrapError, retryable: true): unable to execute activity.

If I downgrade back to 1.19 I can view/add/delete custom search attributes. I read the Release notes on 1.20 but it did not seem to apply to my use-case ( maybe I am wrong).

Has anyone run into this before?

Thanks!

2

Managed too create a Partial reproduction of the issue. It seems if the namespace is set to global this occurs. I reproduced it locally using the docker-compose repo

3

Can you share the repro steps and your config (visibilityStore, etc)?

4

We are also facing the same issue. We upgraded from 1.19 to 1.20. We had one custom search attribute which was global. We were using elasticsearch already. After upgrading to 1.20, the attribute was no longer there (used tctl to verify that). It also caused new workflow start to fail with the error that custom search attribute didn’t exist. We downgraded to 1.19 again & after few minutes it started working again.

5

Think this may be related to fix here.

6

Hello,

Sorry the late reply. I was just about to post the reproduction steps :sweat_smile: .

Thanks for the link to the fix! I will check it out and report back!

7

Hello we are facing the same or a very similar issue upgrading from 1.18.5 to 1.20.1. The solution posted is already included in 1.20.1 image (I guess) and it is not fixing our issue it looks like.

Below is the stacktrace we’re getting on when trying to add the attributes in this way:

tctl --auto_confirm admin cluster add-search-attributes --name myAttribute --type Keyword

Error: Unable to get existing search attributes.
Error Details: rpc error: code = Unavailable desc = unable to get mapping from Elasticsearch: elastic: Error 403 (Forbidden)
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/temporalio/tctl/cli_curr.printError({0x1a91863, 0x29}, {0x1e4b040, 0xc000012e20})
/home/builder/tctl/cli_curr/util.go:393 +0x21e
github.com/temporalio/tctl/cli_curr.ErrorAndExit({0x1a91863?, 0x1e6fef0?}, {0x1e4b040?, 0xc000012e20?})
/home/builder/tctl/cli_curr/util.go:404 +0x28
github.com/temporalio/tctl/cli_curr.AdminAddSearchAttributes(0xc0002c5a20)
/home/builder/tctl/cli_curr/admin_cluster_search_attributes_commands.go:60 +0x145
github.com/temporalio/tctl/cli_curr.newAdminClusterCommands.func1(0xc0002c5a20?)
/home/builder/tctl/cli_curr/admin.go:458 +0x19
github.com/urfave/cli.HandleAction({0x178efa0?, 0x1b28a08?}, 0x15?)
/go/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:526 +0x7f
github.com/urfave/cli.Command.Run({{0x1a5a465, 0x15}, {0x0, 0x0}, {0xc000413a30, 0x1, 0x1}, {0x1a6d02e, 0x1c}, {0x0, …}, …}, …)
/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:173 +0x67b
github.com/urfave/cli.(*App).RunAsSubcommand(0xc00047c8c0, 0xc0001af760)
/go/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:405 +0xe87
github.com/urfave/cli.Command.startApp({{0x1a3a235, 0x7}, {0x0, 0x0}, {0xc000413bf0, 0x1, 0x1}, {0x1a72e92, 0x1e}, {0x0, …}, …}, …)
/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:378 +0xb7f
github.com/urfave/cli.Command.Run({{0x1a3a235, 0x7}, {0x0, 0x0}, {0xc000413bf0, 0x1, 0x1}, {0x1a72e92, 0x1e}, {0x0, …}, …}, …)
/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:102 +0x845
github.com/urfave/cli.(*App).RunAsSubcommand(0xc0004dddc0, 0xc0001af600)
/go/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:405 +0xe87
github.com/urfave/cli.Command.startApp({{0x1a35889, 0x5}, {0x0, 0x0}, {0xc000413ba0, 0x1, 0x1}, {0x1a54925, 0x13}, {0x0, …}, …}, …)
/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:378 +0xb7f
github.com/urfave/cli.Command.Run({{0x1a35889, 0x5}, {0x0, 0x0}, {0xc000413ba0, 0x1, 0x1}, {0x1a54925, 0x13}, {0x0, …}, …}, …)
/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:102 +0x845
github.com/urfave/cli.(*App).Run(0xc0004dda40, {0xc00003e1e0, 0x1d, 0x1e})
/go/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:277 +0xb87
main.main()
/home/builder/tctl/cmd/tctl/main.go:47 +0xa7

8

Sorry for the late reply. Upgraded to 1.20.1 and I am sill getting a 403 when trying to add custom search attributes.

I am investigating if my ES schema commands are incorrect.

    # Taken from https://temporal.io/blog/auto-setup
    ES_SERVER="${ES_SCHEME}://${ES_SEEDS%%,*}:${ES_PORT}"
    SETTINGS_URL="${ES_SERVER}/_cluster/settings"
    SETTINGS_FILE="/etc/temporal/schema/elasticsearch/visibility/cluster_settings_${ES_VERSION}.json"
    TEMPLATE_URL="${ES_SERVER}/_template/temporal_visibility_v1_template"
    SCHEMA_FILE="/etc/temporal/schema/elasticsearch/visibility/index_template_${ES_VERSION}.json"
    INDEX_URL="${ES_SERVER}/temporal_visibility_v1_dev"
    curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${SETTINGS_URL}" -H "Content-Type: application/json" --data-binary "@${SETTINGS_FILE}" --write-out "\n"
    curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${TEMPLATE_URL}" -H 'Content-Type: application/json' --data-binary "@${SCHEMA_FILE}" --write-out "\n"
    curl --user "${ES_USER}":"${ES_PWD}" -X PUT "${INDEX_URL}" --write-out "\n"
10

The above was not the fix.

Actual fix:

11

Don’t work at 1.20.4.

Return same error
Error Details: rpc error: code = Unavailable desc = Workflow temporal-sys-add-search-attributes-workflow returned an error: workflow execution error (type: temporal-sys-add-search-attributes-workflow, workflowID: temporal-sys-add-search-attributes-workflow, runID: 334c2292-7057-4277-a7f9-fbca20f33645): unable to execute activity: AddESMappingFieldActivity: activity error (type: AddESMappingFieldActivity, scheduledEventID: 5, startedEventID: 6, identity: 1@temporal-worker-7d5dfc9dcc-l79v2@): unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): no permissions for [] and User [name=temporal, backend_roles=[], requestedTenant=null] [type=security_exception] (type: wrapError, retryable: true): unable to execute activity.

12

@Dmytro_Orlenko Can you share how you set up Temporal and your config?

13

My guess is that you’re using a config that it’s setting up dual visibility.
I’d suggest to check your config, and modify it to only use advancedVisibilityStore config key to set up your visibility (ie., remove the config keys visibilityStore and secondaryVisibilityStore if they are present).