Multi-Cluster Replication config changes corrupting database

Community Support
,
1

I did a poc to set up a multi cluster env and i was able to do it successfully configure the same the details of what i tried are documented here → I tried this as a brand new setup.

Now, i am trying to update an existing setup ( my dev env) which has few namespaces and lots of workflows, i am encoutering issues and i think i endedup corrupting the temporal db.

Can some one let me know the exact steps to change the multi cluster replication settings?

To start with my multi cluster config was the default one like this

clusterMetadata:
  enableGlobalNamespace: false
  failoverVersionIncrement: 10
  masterClusterName: "active"
  currentClusterName: "active"
  clusterInformation:
    active:
      enabled: true
      initialFailoverVersion: 0
      rpcAddress: "127.0.0.1:7233"
  #replicationConsumer:
  #type: kafka

I attempted to chage this to


clusterMetadata:
       clusterInformation:
         primary:
           enabled: true
           initialFailoverVersion: 1
           rpcAddress: dns:///temporal-dr-primary.mydomain.com:7233
           rpcName: frontend
         secondary:
           enabled: true
           initialFailoverVersion: 2
           rpcAddress: dns:///temporal-dr-secondary.mydomain.com:7233
           rpcName: frontend
       currentClusterName: primary
       enableGlobalNamespace: true
       failoverVersionIncrement: 100
       masterClusterName: primary
       replicationConsumer:
         type: rpc

With these changes i attempted to upgrade and brought up new pods…
the new pods fails to startup and i see various differnt errors like

{"level":"info","ts":"2022-01-07T12:17:50.080Z","msg":"Updated dynamic config","logging-call-at":"file_based_client.go:142"}
Unable to start server. Error: could not build arguments for function "go.temporal.io/server/temporal".ServerLifetimeHooks (/temporal/temporal/fx.go:777): failed to build temporal.Server: could not build arguments for function "go.temporal.io/server/temporal".glob..func1 (/temporal/temporal/server_impl.go:80): failed to build *temporal.ServerImpl: could not build arguments for function "go.temporal.io/server/temporal".NewServerFxImpl (/temporal/temporal/server_impl.go:84): could not build value group *temporal.ServicesMetadata[group="services"]: could not build arguments for function "go.temporal.io/server/temporal".HistoryServiceProvider (/temporal/temporal/fx.go:251): failed to build config.Persistence: received non-nil error from function "go.temporal.io/server/temporal".ApplyClusterMetadataConfigProvider (/temporal/temporal/fx.go:532): error while backfiling cluster metadata: %!w(<nil>)

since the default cluster name was active already, i attempted to set the cluster name back to stand by and used the following configuration

clusterMetadata:
       clusterInformation:
         active:
           enabled: true
           initialFailoverVersion: 1
           rpcAddress: dns:///temporal-dr-primary.mydomain.com:7233
           rpcName: frontend
         standby:
           enabled: true
           initialFailoverVersion: 2
           rpcAddress: dns:///temporal-dr-secondary.mydomain.com:7233
           rpcName: frontend
       currentClusterName: primary
       enableGlobalNamespace: true
       failoverVersionIncrement: 100
       masterClusterName: primary
       replicationConsumer:
         type: rpc

this time around i got the below error

t[07:12 pm] Ramani, Natarajan
{"level":"warn","ts":"2022-01-07T12:25:52.405Z","msg":"Failed to save cluster metadata.","component":"metadata-initializer","error":"SaveClusterMetadata encountered version mismatch, expected 0 but got 1.","cluster-name":"standby","logging-call-at":"fx.go:584"}

also after setting EnableGlobalNamespace to true i ALWAYS see these errors in the logs

{"level":"error","ts":"2022-01-07T12:25:52.408Z","msg":"Supplied configuration key/value mismatches persisted cluster metadata. Continuing with the persisted value as this value cannot be changed once initialized.","component":"metadata-initializer","key":"clusterMetadata.EnableGlobalNamespace","ignored-value":true,"value":false,"logging-call-at":"fx.go:608","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider\n\t/temporal/temporal/fx.go:608\nreflect.Value.call\n\t/usr/local/go/src/reflect/value.go:543\nreflect.Value.Call\n\t/usr/local/go/src/reflect/value.go:339\ngo.uber.org/dig.defaultInvoker\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:439\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:912\ngo.uber.org/dig.paramSingle.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:240\ngo.uber.org/dig.paramObjectField.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:396\ngo.uber.org/dig.paramObject.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:323\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:903\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:458\ngo.uber.org/dig.paramObjectField.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:396\ngo.uber.org/dig.paramObject.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:323\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:903\ngo.uber.org/dig.paramSingle.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:240\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:903\ngo.uber.org/dig.paramSingle.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:240\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*Container).Invoke\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:587\ngo.uber.org/fx.(*App).executeInvoke\n\t/go/pkg/mod/go.uber.org/fx@v1.14.2/app.go:873\ngo.uber.org/fx.(*App).executeInvokes\n\t/go/pkg/mod/go.uber.org/fx@v1.14.2/app.go:846\ngo.uber.org/fx.New\n\t/go/pkg/mod/go.uber.org/fx@v1.14.2/app.go:594\ngo.temporal.io/server/temporal.NewServerFx\n\t/temporal/temporal/fx.go:97\ngo.temporal.io/server/temporal.NewServer\n\t/temporal/temporal/server.go:58\nmain.buildCLI.func2\n\t/temporal/cmd/server/main.go:162\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224\nmain.main\n\t/temporal/cmd/server/main.go:52\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255"}

this apart I also get errors stating the cluster rpc address cannot be changed once initialized.

{"level":"error","ts":"2022-01-07T12:25:52.409Z","msg":"Supplied configuration key/value mismatches persisted cluster metadata. Continuing with the persisted value as this value cannot be changed once initialized.","component":"metadata-initializer","key":"clusterInformation.RPCAddress","ignored-value":{"Enabled":true,"InitialFailoverVersion":1,"RPCAddress":"dns:///temporal-dr-primary.mydomain.com:7233"},"value":{"Enabled":true,"InitialFailoverVersion":1,"RPCAddress":"127.0.0.1:7933"},"logging-call-at":"fx.go:702","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/temporal.loadClusterInformationFromStore\n\t/temporal/temporal/fx.go:702\ngo.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider\n\t/temporal/temporal/fx.go:658\nreflect.Value.call\n\t/usr/local/go/src/reflect/value.go:543\nreflect.Value.Call\n\t/usr/local/go/src/reflect/value.go:339\ngo.uber.org/dig.defaultInvoker\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:439\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:912\ngo.uber.org/dig.paramSingle.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:240\ngo.uber.org/dig.paramObjectField.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:396\ngo.uber.org/dig.paramObject.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:323\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:903\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:458\ngo.uber.org/dig.paramObjectField.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:396\ngo.uber.org/dig.paramObject.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:323\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:903\ngo.uber.org/dig.paramSingle.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:240\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:903\ngo.uber.org/dig.paramSingle.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:240\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*Container).Invoke\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:587\ngo.uber.org/fx.(*App).executeInvoke\n\t/go/pkg/mod/go.uber.org/fx@v1.14.2/app.go:873\ngo.uber.org/fx.(*App).executeInvokes\n\t/go/pkg/mod/go.uber.org/fx@v1.14.2/app.go:846\ngo.uber.org/fx.New\n\t/go/pkg/mod/go.uber.org/fx@v1.14.2/app.go:594\ngo.temporal.io/server/temporal.NewServerFx\n\t/temporal/temporal/fx.go:97\ngo.temporal.io/server/temporal.NewServer\n\t/temporal/temporal/server.go:58\nmain.buildCLI.func2\n\t/temporal/cmd/server/main.go:162\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224\nmain.main\n\t/temporal/cmd/server/main.go:52\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255"}
{"level":"error","ts":"2022-01-07T12:25:52.409Z","msg":"Supplied configuration key/value mismatches persisted cluster metadata. Continuing with the persisted value as this value cannot be changed once initialized.","component":"metadata-initializer","key":"clusterInformation","ignored-value":{"active":{"Enabled":true,"InitialFailoverVersion":1,"RPCAddress":"127.0.0.1:7933"},"standby":{"Enabled":true,"InitialFailoverVersion":2,"RPCAddress":"dns:///temporal-dr-secondary.mydomain.com:7233"}},"value":{"Enabled":true,"InitialFailoverVersion":2,"RPCAddress":"dns:///temporal-dr-secondary.mydomain.com:7233"},"logging-call-at":"fx.go:716","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\ngo.temporal.io/server/temporal.loadClusterInformationFromStore\n\t/temporal/temporal/fx.go:716\ngo.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider\n\t/temporal/temporal/fx.go:658\nreflect.Value.call\n\t/usr/local/go/src/reflect/value.go:543\nreflect.Value.Call\n\t/usr/local/go/src/reflect/value.go:339\ngo.uber.org/dig.defaultInvoker\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:439\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:912\ngo.uber.org/dig.paramSingle.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:240\ngo.uber.org/dig.paramObjectField.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:396\ngo.uber.org/dig.paramObject.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:323\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:903\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:458\ngo.uber.org/dig.paramObjectField.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:396\ngo.uber.org/dig.paramObject.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:323\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:903\ngo.uber.org/dig.paramSingle.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:240\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*node).Call\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:903\ngo.uber.org/dig.paramSingle.Build\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:240\ngo.uber.org/dig.paramList.BuildList\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/param.go:196\ngo.uber.org/dig.(*Container).Invoke\n\t/go/pkg/mod/go.uber.org/dig@v1.13.0/dig.go:587\ngo.uber.org/fx.(*App).executeInvoke\n\t/go/pkg/mod/go.uber.org/fx@v1.14.2/app.go:873\ngo.uber.org/fx.(*App).executeInvokes\n\t/go/pkg/mod/go.uber.org/fx@v1.14.2/app.go:846\ngo.uber.org/fx.New\n\t/go/pkg/mod/go.uber.org/fx@v1.14.2/app.go:594\ngo.temporal.io/server/temporal.NewServerFx\n\t/temporal/temporal/fx.go:97\ngo.temporal.io/server/temporal.NewServer\n\t/temporal/temporal/server.go:58\nmain.buildCLI.func2\n\t/temporal/cmd/server/main.go:162\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224\nmain.main\n\t/temporal/cmd/server/main.go:52\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255"}

in temporal database cluster_metadata and cluster_metadata_info , i see my old and new cluster details,

when i do tctl adm cl d from my old pods which are still running, i see my old cluster details alone (not the secondary/standby details), how ever in database, i see entries for both.

I guess i endedup corrupting the db.

Can some one suggest a way out pl?

2

i am trying all this on temporal 1.14.0

3

With the 1.14 release and future 1.15 release, we are moving cluster metadata loading from yaml static config to API dynamic config. To make this process airstream, we initialize new cluster metadata from yaml config once during startup.

If you already start the service with a cluster metadata (secondary or standby) and later update some critical fields in the cluster metadata. It gives you the errors you mentioned above.

There are two options for you as you are experiment the multi-cluster setup.

Option 1: Update cluster metadata from admin tctl

./tctl adm cl upsert-remote-cluster --frontend_address <rpcAddress> --enable_connection <true/false>

This will make a call to the rpcAddress endpoint to fetch the target cluster metadata. You need to make sure the enableGlobalNamespace is true, the initialFailoverVersion is not change, and the failoverVersionIncrement values are equal. (If not, you have to do delete it first, see option 2)

Option 2: Delete the invalid cluster metadata first, use admin tctl or yaml to set the correct cluster metadata.

./tctl adm cl  remove-remote-cluster --cluster <cluster name>

Note: make sure to do this operation on both clusters.

4

Thanks @yux will try this out and let you know.

5

Is the gloabl namespace true/false still at static config? i dont see tctl command to enable disable this.

by default the current cluster name will be active right? is there a way to change this name?
e.g i may want to call the currently running cluster primary, then how to go about it?

should i add , a cluster name primary, alllow data to replicate there, and remove the current setup (active)?

6

between i am not able to update the clusters with these command i get the below error

tctl  adm cl urc  --frontend_address dns:///temporal-dr-standby.mydomain.com:7233
Error: Operation AddOrUpdateRemoteCluster failed.
Error Details: rpc error: code = InvalidArgument desc = Cannot update current cluster metadata from rpc calls
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
go.temporal.io/server/tools/cli.printError({0x1de6776, 0x2a}, {0x20ddf00, 0xc00000e0c0})
        /temporal/tools/cli/util.go:391 +0x22a
go.temporal.io/server/tools/cli.ErrorAndExit({0x1de6776, 0x210f278}, {0x20ddf00, 0xc00000e0c0})
        /temporal/tools/cli/util.go:402 +0x28
go.temporal.io/server/tools/cli.AdminAddOrUpdateRemoteCluster(0xc00041d080)
        /temporal/tools/cli/adminClusterCommands.go:61 +0x147
go.temporal.io/server/tools/cli.newAdminClusterCommands.func5(0xc00041d080)
        /temporal/tools/cli/admin.go:519 +0x19
github.com/urfave/cli.HandleAction({0x19c7000, 0x1e4ca38}, 0x15)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:526 +0x50
github.com/urfave/cli.Command.Run({{0x1db14df, 0x15}, {0x0, 0x0}, {0xc00011cea0, 0x1, 0x1}, {0x1e11c67, 0x3f}, {0x0, ...}, ...}, ...)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:173 +0x652
github.com/urfave/cli.(*App).RunAsSubcommand(0xc0006ffa40, 0xc00041cdc0)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:405 +0x9ec
github.com/urfave/cli.Command.startApp({{0x1d929fb, 0x7}, {0x0, 0x0}, {0xc00011cfd0, 0x1, 0x1}, {0x1dc5df8, 0x1e}, {0x0, ...}, ...}, ...)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:372 +0x6e9
github.com/urfave/cli.Command.Run({{0x1d929fb, 0x7}, {0x0, 0x0}, {0xc00011cfd0, 0x1, 0x1}, {0x1dc5df8, 0x1e}, {0x0, ...}, ...}, ...)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:102 +0x808
github.com/urfave/cli.(*App).RunAsSubcommand(0xc0006ff880, 0xc00041cc60)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:405 +0x9ec
github.com/urfave/cli.Command.startApp({{0x1d8e1b0, 0x5}, {0x0, 0x0}, {0xc00011cf80, 0x1, 0x1}, {0x1dab4da, 0x13}, {0x0, ...}, ...}, ...)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:372 +0x6e9
github.com/urfave/cli.Command.Run({{0x1d8e1b0, 0x5}, {0x0, 0x0}, {0xc00011cf80, 0x1, 0x1}, {0x1dab4da, 0x13}, {0x0, ...}, ...}, ...)
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:102 +0x808
github.com/urfave/cli.(*App).Run(0xc0006ff6c0, {0xc00003a1e0, 0x6, 0x6})
        /go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:277 +0x705
7

the errors were because the default customer name was active every where. after changing the cluster name in my standby i was able to do adm cl urc

However after ding adm cl urc, the command was successful
but when i do tctl adm cl d, i still do not see the second cluster details,

not sure why the urc is behaving like no op
is this a bug ? or something wrong with my configuration??

8

So in nutshell my problem now is

a) if you specific multiple cluster details in clusterMetaData only the currentcluster is getting considered
b) if you add additional cluster details using tctl adm cl urc --frontend_address , its behaving like no op.

so there is no way to configure/add standby/fail over cluster details information either thoruhg static config or through tctl

is there a guide for 1.14.x xdc setup? these things were working in 1.12.x and seems to have broken recently.

9

The tctl adm cl d shows current cluster metadata by default. You can add a option:

tctl adm cl d --cluster <cluster name>

Let me know if you can see the remote cluster metadata

10

i think since the default cluster name is “active” , my clusters were failing to communcate with each other , once i changed

Thanks , i am able to see the additional cluster details with --cluster

11

However, i still dont see the data syncing up between clusters.
Also, in my standby cluster, when i try to add the active cluster details , i get this error

tctl adm cl urc --fad dns:///temporal-dr-test-primary.mydomain.com:7233
Error: Operation AddOrUpdateRemoteCluster failed.
Error Details: rpc error: code = InvalidArgument desc = Cannot add remote cluster as global namespace is not supported
('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)

But the Gloab namesape is enabled and i see it through the cluster descriptin too .

{
 ...
 ...
  "clusterId": "55ba77cd-e636-4fc2-a9c0-ecc4d03d1cb3",
  "clusterName": "standby",
  "historyShardCount": 512,
  "persistenceStore": "mysql",
  "visibilityStore": "mysql",
  "failoverVersionIncrement": "100",
  "initialFailoverVersion": "2",
  "isGlobalNamespaceEnabled": true
}
12

You are correct. This case is not well handled. This error means the active cluster is still set the IsGlobalNamespaceEnabled as False. But I believe you have changed it to True.

To unblock you, you can clean up the cluster_metadata_info table in active cluster and start the services. Or you can also clean up the DB and initialize the active cluster again.

At the same time, I will make a change to cover this use.

13

Thanks @yux i tried cleaning up the cluster_metadata_info and cluster_metadata tables manaully using delete from query, and after bring up the active cluster, i was able to register the active cluster to standby… however, i still do not see the data replicating :frowning: there seems to be something broken here…

Also, at namespace level i am not able to change the cluster definations , refer the other post on namespace update