Temporal cannot create keyspace when Replication Factor is more than 1

Community Support
,
1

Hi All,

Got error when creating temporal keyspace and this error only happens when Replication Factor is more than 1 (if 1 then it’s ok), the error is: Unable to setup CQL schema. {“error”: “Operation timed out - received only 1 responses.”, “logging-call-at”: “handler.go:61”}

Please help, thanks

The complete log below:

  • temporal-cassandra-tool --ep create -k temporal --rf 3
    2022-01-09T12:23:53.597Z Validating connection to cassandra cluster. {“logging-call-at”: “cqlclient.go:111”}
    2022-01-09T12:23:53.828Z Connection validation succeeded. {“logging-call-at”: “cqlclient.go:117”}
    2022-01-09T12:23:53.828Z Creating Keyspace temporal using SimpleStrategy with RF=3. {“logging-call-at”: “cqlclient.go:157”}
  • temporal-cassandra-tool --ep -k temporal setup-schema -v 0.0
    2022-01-09T12:23:53.860Z Validating connection to cassandra cluster. {“logging-call-at”: “cqlclient.go:111”}
    2022-01-09T12:23:53.956Z Connection validation succeeded. {“logging-call-at”: “cqlclient.go:117”}
    2022-01-09T12:23:53.956Z Starting schema setup {“config”: {“SchemaFilePath”:"",“InitialVersion”:“0.0”,“Overwrite”:false,“DisableVersioning”:false}, “logging-call-at”: “setuptask.go:57”}
    2022-01-09T12:23:53.956Z Setting up version tables {“logging-call-at”: “setuptask.go:67”}
    2022-01-09T12:23:59.002Z Setting initial schema version to 0.0 {“logging-call-at”: “setuptask.go:118”}
    2022-01-09T12:24:01.016Z Unable to setup CQL schema. {“error”: “Operation timed out - received only 1 responses.”, “logging-call-at”: “handler.go:61”}
    2022/01/09 12:24:01 Loading config; env=docker,zone=,configDir=config
    2022/01/09 12:24:01 Loading config files=[config/docker.yaml]
    {“level”:“error”,“ts”:“2022-01-09T12:24:01.133Z”,“msg”:“Unable to read dynamic config file. Continue with default settings but the ERROR MUST BE FIXED before the next upgrade”,“error”:“unable to read dynamic config: dynamic config file: /etc/temporal/config/dynamicconfig: read /etc/temporal/config/dynamicconfig: is a directory”,“logging-call-at”:“main.go:140”,“stacktrace”:“go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:142\nmain.buildCLI.func2\n\t/temporal/cmd/server/main.go:140\Ngithub.com).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163\Ngithub.com).RunContext\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313\Ngithub.com).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224\nmain.main\n\t/temporal/cmd/server/main.go:50\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255”}
    {“level”:“info”,“ts”:“2022-01-09T12:24:01.133Z”,“msg”:“Starting server for services”,“value”:[“history”,“matching”,“frontend”,“worker”],“logging-call-at”:“server.go:123”}
    Unable to start server. Error: cassandra schema version compatibility check failed: unable to read DB schema version keyspace/database: temporal error: failed to get current schema version from cassandra
2

additional info, I noticed that for Replication Factor = 1, Temporal can create TABLE temporal.tasks (after TABLE temporal.schema_version creation), however when Replication Factor = 3, it does not have TABLE temporal.tasks so it might be fail to create TABLE temporal.tasks?

Please help.

Thanks

3

Update, this already solved, turns out the chrony (time) is not sync between cassandra nodes.