How to set up ElasticSearch?

Community Support
1

Hey all – wondering if you could point me to some docs (or quickly explain here) that describe how to set up and enable ElasticSearch for workflow visibility indexing/searching. Thanks!

3 Likes
3

Samar will be able to give better details, but it’s worth noting that we have a Helm chart which handles this:

4

Hey @skrul,

You can enable ElasticSearch integration with Temporal by providing the correct configuration in service config for Temporal Server. There are two main configs you need understand to be able to configure ElasticSearch as the visibility store.

  1. Persistence
    Update the persistence section in config so Temporal server can connect to ElasticSearch store for visibility needs. First specify the key for your datastore under ‘advancedVisibilityStore’ key:
    advancedVisibilityStore: es-visibility
    Also specify ElasticSearch connection information under datastores with the key specified for advancedVisibilityStore:
datastores:
    ...
    es-visibility:
      elasticsearch:
        url:
          scheme: "http"
          host: "127.0.0.1:9200"
        indices:
          visibility: temporal-visibility-dev
  1. Kafka
    Next configure the kafka topic which is going to be used for shipping visibility records for workflow executions to processor running on the worker role. ESProcessor on worker role consumes these messages from Kafka topic and writes them to ElasticSearch in a batched write.
    First specify applications key in the config with visibility section which contains the name for main and dlq Kafka topic:
kafka:
  ...
  applications:
    visibility:
      topic: temporal-visibility-dev
      dlq-topic: temporal-visibility-dev-dlq

Also specify the config section on with connection information to Kafka broker and mapping for each topic to Kafka broker.

kafka:
  ...
  clusters:
    test:
      brokers:
        - 127.0.0.1:9092
  topics:
    temporal-visibility-dev:
      cluster: test
    temporal-visibility-dev-dlq:
      cluster: test

Here is how the entire Kafka config should look like.

kafka:
  tls:
    enabled: false
  clusters:
    test:
      brokers:
        - 127.0.0.1:9092
  topics:
    temporal-visibility-dev:
      cluster: test
    temporal-visibility-dev-dlq:
      cluster: test
  applications:
    visibility:
      topic: temporal-visibility-dev
      dlq-topic: temporal-visibility-dev-dlq

And finally in case you want to use Temporal helm chart which does all of that for you.

2 Likes