Temporal cannot create keyspace when Replication Factor is more than 1

Hi All,

Got error when creating temporal keyspace and this error only happens when Replication Factor is more than 1 (if 1 then it’s ok), the error is: Unable to setup CQL schema. {“error”: “Operation timed out - received only 1 responses.”, “logging-call-at”: “handler.go:61”}

Please help, thanks

The complete log below:

  • temporal-cassandra-tool --ep create -k temporal --rf 3
    2022-01-09T12:23:53.597Z Validating connection to cassandra cluster. {“logging-call-at”: “cqlclient.go:111”}
    2022-01-09T12:23:53.828Z Connection validation succeeded. {“logging-call-at”: “cqlclient.go:117”}
    2022-01-09T12:23:53.828Z Creating Keyspace temporal using SimpleStrategy with RF=3. {“logging-call-at”: “cqlclient.go:157”}
  • temporal-cassandra-tool --ep -k temporal setup-schema -v 0.0
    2022-01-09T12:23:53.860Z Validating connection to cassandra cluster. {“logging-call-at”: “cqlclient.go:111”}
    2022-01-09T12:23:53.956Z Connection validation succeeded. {“logging-call-at”: “cqlclient.go:117”}
    2022-01-09T12:23:53.956Z Starting schema setup {“config”: {“SchemaFilePath”:"",“InitialVersion”:“0.0”,“Overwrite”:false,“DisableVersioning”:false}, “logging-call-at”: “setuptask.go:57”}
    2022-01-09T12:23:53.956Z Setting up version tables {“logging-call-at”: “setuptask.go:67”}
    2022-01-09T12:23:59.002Z Setting initial schema version to 0.0 {“logging-call-at”: “setuptask.go:118”}
    2022-01-09T12:24:01.016Z Unable to setup CQL schema. {“error”: “Operation timed out - received only 1 responses.”, “logging-call-at”: “handler.go:61”}
    2022/01/09 12:24:01 Loading config; env=docker,zone=,configDir=config
    2022/01/09 12:24:01 Loading config files=[config/docker.yaml]
    {“level”:“error”,“ts”:“2022-01-09T12:24:01.133Z”,“msg”:“Unable to read dynamic config file. Continue with default settings but the ERROR MUST BE FIXED before the next upgrade”,“error”:“unable to read dynamic config: dynamic config file: /etc/temporal/config/dynamicconfig: read /etc/temporal/config/dynamicconfig: is a directory”,“logging-call-at”:“main.go:140”,“stacktrace”:“go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\nmain.buildCLI.func2\n\t/temporal/cmd/server/main.go:140\ngithub.com).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224\nmain.main\n\t/temporal/cmd/server/main.go:50\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255”}
    {“level”:“info”,“ts”:“2022-01-09T12:24:01.133Z”,“msg”:“Starting server for services”,“value”:[“history”,“matching”,“frontend”,“worker”],“logging-call-at”:“server.go:123”}
    Unable to start server. Error: cassandra schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: failed to get current schema version from cassandra

additional info, I noticed that for Replication Factor = 1, Temporal can create TABLE temporal.tasks (after TABLE temporal.schema_version creation), however when Replication Factor = 3, it does not have TABLE temporal.tasks so it might be fail to create TABLE temporal.tasks?

Please help.

Thanks

Update, this already solved, turns out the chrony (time) is not sync between cassandra nodes.