Elasticsearch and Cassandra visibility

Community Support
,
1

We installed Temporal using Helm Chart. We wanted to use Elasticsearch for Visibility. During the installation, we found that if we don’t give the Cassandra Visibility parameters, the installation does not go through. We specified both Elasticsearch and Cassandra details for Visibility. The expectation was, Visibility data will be inserted into Elasticseach and the Cassandra table will be ignored. But we found that the Visibility data is getting inserted in both Cassandra and Elasticseach. Also, we saw that not all the records are going to Cassandra (found 29 while Elasticsearch had all the 100 Workflow details).

Not sure if this is the expected behavior or there was some problem in the installation. We wanted to know,

  • If we want to use Elasticsearch, do we still need to provide the Cassandra details for Visibility? If not, what changes are required during Helm Chart installation?
  • If we still need to provide the Cassandra details, can we stop writes in Cassandra and insert data only in Elasticsearch?
2

Older version of Temporal required Cassandra visibility to be configured even if it is not in use. Starting from 1.9.2 this requirement is removed and you can omit it. Helm is not updated yet. It would be great if you open an issue for it.

There is a support for dual visibility writing mode when Temporal writes visibility records to both Cassandra and Elasticsearch. This is primarily for the live migration from Cassandra to Elasticsearch. It is controlled by system.advancedVisibilityWritingMode setting in dynamic config. When it is set to dual then Temporal will write to both visibility stores. If you don’t need this just set it to on.

There is also sampling for visibility records in Cassandra. It means that Temporal can drop some records when writing for Cassandra. It is also controlled by the bunch of dynamic config settings but I wouldn’t go deeper here because I am about to remove this feature and replace it with proper throttling.

3

Thanks, @alex for sharing the details.

When we install the Helm Chart (without making any changes in the dynamic config), I can see the following line in the Matching service log:

{"level":"info","ts":"2021-05-26T12:14:55.103Z","msg":"Get dynamic config","name":"system.advancedVisibilityWritingMode","value":"on","default-value":"on","logging-call-at":"config.go:79"}

Looks like advancedVisibilityWritingMode is set to the correct value and still Visibility data is getting inserted to both Elasticsearch and Cassandra.

4

That might be a bug. Let me investigate it.

5

Thanks, @alex. Here is the version we are testing with.

bash-5.0$ tctl --version
tctl version 1.8.0

6

This is just tctl version. You can get actual server version with temporal-server --version command. But I assume you have the same 1.8.0 there.

I’ve just run different tests on our current master and release/1.8.x branches. I can’t reproduce this. temporal_visibility keyspace stays empty for me and I don’t see any calls to cassandra visibility when advanced visibility is enabled. I would recommend to use latest 1.9.2 version with removed cassandra visibility configuration. In this case you will know for sure that Temporal doesn’t touch visibility database.

Also I should mention that our helm charts are for testing purpose only. They are not ready for production.