Cadence connecting with AWS Keyspace

sandeep_M · August 4, 2020, 8:16pm

I am trying out AWS keyspace service for cadence cassandra store. When trying to using the cadence-cassandra-tool for schema migration i see two issues

Partitioner Issue:
Error: update the token ring due to error: Unsupported partitioner ‘com.amazonaws.cassandra.DefaultPartitioner’
Consistency level:
Error: consistency level ANY is not supported for this operation. Supported consistency levels are: ONE, LOCAL_QUORUM, LOCAL_ONE

Steps to reproduce the behavior:

docker run --rm -it ubercadence/server:0.11.0 bash
export the following env variable for cadence-cassandra-tool:
export CASSANDRA_HOST=
export CASSANDRA_PORT=
export CASSANDRA_USER=
export CASSANDRA_PASSWORD=
export CASSANDRA_ENABLE_TLS=
export CASSANDRA_TLS_VERIFY_HOST=
export CASSANDRA_TLS_CA=
export CADENCE_KEYSPACE=cadence
export VISIBILITY_KEYSPACE=cadence_visibility
cadence-Cassandra-tool setup-schema -v 0.0

Additional context:

I could not find any pointers for Partitioner issue. AWS documents mentions that they support gocql but only basic example provided, which does not talk about setting the partitioner. Based on the research i have done, i think gcql will only accepts following https://github.com/gocql/gocql/blob/cd92c541a0b5013df76fd0f980d978b76839f9a9/token.go#L156
and not sure how that gets abstracted for AWS Keyspace.

For consistency level issue, AWS Keyspace does not support Any and only support ONE, LOCAL_QUORUM, LOCAL_ONE as show in the error. When i checked in cadence-cassandra-tool i see that defaultconsistency as ALL as seen here

github.com

uber/cadence/blob/01d2d1a475827257861b3e39846357d6f3e9def9/tools/cassandra/cqlclient.go#L64


		TLS         *auth.TLS
	}
)

var errNoHosts = errors.New("Cassandra Hosts list is empty or malformed")
var errGetSchemaVersion = errors.New("Failed to get current schema version from cassandra")

const (
	defaultTimeout       = 30    // Timeout in seconds
	cqlProtoVersion      = 4     // default CQL protocol version
	defaultConsistency   = "ALL" // schema updates must always be ALL
	defaultCassandraPort = 9042
	systemKeyspace       = "system"
)

const (
	readSchemaVersionCQL        = `SELECT curr_version from schema_version where keyspace_name=?`
	listTablesCQL               = `SELECT table_name from system_schema.tables where keyspace_name=?`
	listTypesCQL                = `SELECT type_name from system_schema.types where keyspace_name=?`
	writeSchemaVersionCQL       = `INSERT into schema_version(keyspace_name, creation_time, curr_version, min_compatible_version) VALUES (?,?,?,?)`
	writeSchemaUpdateHistoryCQL = `INSERT into schema_update_history(year, month, update_time, old_version, new_version, manifest_md5, description) VALUES(?,?,?,?,?,?,?)`

Any help would be greatly appreciated.

Thank you.

maxim · August 4, 2020, 8:30pm

It looks like Keyspaces don’t support ALL as the schema is updated asynchronously. You can try to change the consistency level in the cassandra-tool to something else and see if it is going to work.

I found this discussion related to the partitioner.

sandeep_M · August 4, 2020, 10:22pm

Hi Maxim,

Thanks for your quick response. I am following the same thread on amazon and it doesn’t have any info that i can try out. I will check if AWS support can help on this.

Regarding the changing the consistency level in the cassandra-tool , i get following error when trying to build locally with make bins. Can you please let me know if there is any additional required for building the tools.

git submodule update --init --recursive
thriftrw --plugin=yarpc --pkg-prefix=github.com/uber/cadence/.gen/go/ --out=.gen/go idls/thrift/cadence.thrift
Failed to generate code: failed to generate code for “C:\go\src\cadence\idls\thrift\cadence.thrift”: failed to generate code for “idl embedding”: could not parse generated code: cadence.go:9:28: illegal character U+0072 ‘r’ in escape sequence (and 4 more errors):
package thriftrw

            // ThriftModule represents the IDL file used to generate this package.
            var ThriftModule = &thriftreflect.ThriftModule {
                    Name: "cadence",
                    Package: "github.com\uber\cadence\.gen\go\cadence",
                    FilePath: "cadence.thrift",
                    SHA1: "e6751c109262ef7584dbec4c934a7d8e2dbe8123",
                    Includes: []*thriftreflect.ThriftModule {
                                            shared.ThriftModule,
                                    },
                    Raw: rawIDL,
            }

Thank you for your help.

maxim · August 4, 2020, 11:13pm

It looks like a Windows specific issue. Do you have access to some Unix based machine?

Also would you try to build Temporal to see if it has a similar problem?

sandeep_M · August 5, 2020, 1:08am

tools build for cadence and temporal is good on Mac(GOOS=darwin and GOARCH=amd64) but fail on windows and linux_amd64 . Attached the images in next comment due to post limitation

sandeep_M · August 5, 2020, 1:10am

maxim · August 5, 2020, 2:20am

Thanks! Could you file an issue with details? It will help us to prioritize fixing this.

111 · June 15, 2021, 12:34pm

Hi guys,
Here you discussed building problems on Windows.
Now I’m experiencing the same one:
(Failed to generate code: failed to generate code for “D:\cadence\idls\thrift\admin.thrift”: failed to generate code for “idl embedding”: could not parse generated code: admin.go:9:28: illegal character U+0072 ‘r’ in escape sequence (and 3 more errors)"

So I suppose it wasn’t resolved but maybe there is some workaround you found or link to appropriate ticket ?

Topic		Replies	Views
Cassandra read/write consistency level for cadence Community Support cadence	2	901	September 11, 2020
Error update schema version Community Support	2	1351	August 7, 2020
Config Server and Web cadence Community Support cadence	2	1609	August 11, 2020
Config cadence to production Community Support cadence	1	772	August 10, 2020
Gocql: no hosts available in the pool Community Support cassandra , cadence	5	2930	May 2, 2023

Cadence connecting with AWS Keyspace

Related topics