Hi, we have a Temporal installed in k8s from the helm chart. Everything worked fine, then temporal service pods began to fail during the startup with the message:
cassandra schema version compatibility check failed: unable to read cassandra schema version keyspace/database: temporal error: Failed to get current schema version from cassandra
Pods that were already running, continued to operate. Only the newly created pods crashed. All the Init containers finished execution successfully.
I had to fully recreate a Temporal DB: deleted keyspaces and performed schema setup with tctl. After this, all the crashing pods started successfully.
Do you have any ideas what could have caused this? Which diagnostic should I do to obtain more data if this repeats?