Temporal-cassandra-tool hangs when running validate-health

I’m trying to provision a Cassandra instance for use by Temporal, but the temporal-cassandra-tool seems to be hanging when I run validate-health . The only message it prints is “2021/10/13 21:58:54 validating connection to cassandra cluster”

This Cassandra cluster is provisioned via Datastax Astra. From the same container I’m running the temporal-cassandra-tool, I can confirm that cqlsh can connect and interact with the cluster. The image I’m using for this container is “temporalio/admin-tools:1.12.3”

I’m configuring temporal-cassandra-tool with the following environment variables:

CASSANDRA_PASSWORD=<REDACTED>
CASSANDRA_ENABLE_TLS=true
CASSANDRA_USER=<REDACTED>
CASSANDRA_KEYSPACE=temporal_default
CASSANDRA_TLS_CA=/workdir/ca.crt
CASSANDRA_HOST=<REDACTED>-us-east1.db.astra.datastax.com
CASSANDRA_TLS_KEY=/workdir/key
CASSANDRA_TLS_SERVER_NAME=<REDACTED>-us-east1.db.astra.datastax.com
CASSANDRA_TLS_CERT=/workdir/cert
CASSANDRA_PORT=29042

Any suggestions here would be greatly appreciated!

EDIT 1:

After some more digging, it appears gocql is discovering some internal nodes addresses from Astra and is trying to connect to them. If I force gocql to only connect to the initial addresses, things run smoothly.

Here is a gist demonstrating my modified client config: CQL Connect · GitHub

I don’t know much about Cassandra, is this a misconfiguration on the Astra side, or does Temporal’s client config need to be updated to allow this filtering?

Server config exposes “disableInitialHostLookup” via config: temporal/config.go at master · temporalio/temporal · GitHub
which is used by gocql client: temporal/client.go at master · temporalio/temporal · GitHub

It’s not currently exposed by temporal-cassandra-tool, added issue: Expose disableInitialHostLookup for temporal-cassandra-tool · Issue #2054 · temporalio/temporal · GitHub

Setting disableInitialHostLookup to true in the server config tho should help you reduce the “hang time” of the temporal-cassandra-tool call.