Hey all – wondering if you could point me to some docs (or quickly explain here) that describe how to set up and enable ElasticSearch for workflow visibility indexing/searching. Thanks!
Samar will be able to give better details, but it’s worth noting that we have a Helm chart which handles this:
You can enable ElasticSearch integration with Temporal by providing the correct configuration in service config for Temporal Server. There are two main configs you need understand to be able to configure ElasticSearch as the visibility store.
Update the persistence section in config so Temporal server can connect to ElasticSearch store for visibility needs. First specify the key for your datastore under ‘advancedVisibilityStore’ key:
Also specify ElasticSearch connection information under datastores with the key specified for advancedVisibilityStore:
datastores: ... es-visibility: elasticsearch: url: scheme: "http" host: "127.0.0.1:9200" indices: visibility: temporal-visibility-dev
Next configure the kafka topic which is going to be used for shipping visibility records for workflow executions to processor running on the worker role. ESProcessor on worker role consumes these messages from Kafka topic and writes them to ElasticSearch in a batched write.
applicationskey in the config with
visibilitysection which contains the name for main and dlq Kafka topic:
kafka: ... applications: visibility: topic: temporal-visibility-dev dlq-topic: temporal-visibility-dev-dlq
Also specify the config section on with connection information to Kafka broker and mapping for each topic to Kafka broker.
kafka: ... clusters: test: brokers: - 127.0.0.1:9092 topics: temporal-visibility-dev: cluster: test temporal-visibility-dev-dlq: cluster: test
Here is how the entire Kafka config should look like.
kafka: tls: enabled: false clusters: test: brokers: - 127.0.0.1:9092 topics: temporal-visibility-dev: cluster: test temporal-visibility-dev-dlq: cluster: test applications: visibility: topic: temporal-visibility-dev dlq-topic: temporal-visibility-dev-dlq
And finally in case you want to use Temporal helm chart which does all of that for you.