Beginner Question : Using Elastic Search

Community Support
1

Our Project is starting on Temporal.
It is not clear from the documentation why Elastic search is to be used.

  1. Is it being used as long-term storage for the Workflow data?
    ( Whenever Workflow data is updated, Temporal sends a POST to Elasticsearch so as to update the ES data )

  2. Is a Kafka installation needed for using ES ? ( An old post seems to be suggesting that it is needed: How to set up ElasticSearch? - #4 by samar )

  3. Our interest is mainly in the long-term storage of our business objects that will get updated as the result of various workflows. ( Temporal related data and the ability to debug Temporal Workflows is not really important at this point )… is it better if we maintain our business data independent of Temporal-ES integration?

Any help will be much appreciated

2
  1. Is it being used as long-term storage for the Workflow data?
    ( Whenever Workflow data is updated, Temporal sends a POST to Elasticsearch so as to update the ES data )

No, workflow data is never stored in ES. It only stores so-called “visibility records” which are used to list workflows based on various properties we call “search attributes”.

  1. Is a Kafka installation needed for using ES ? ( An old post seems to be suggesting that it is needed: How to set up ElasticSearch? - #4 by samar )

It was needed some time ago. The recent releases of Temporal don’t use Kafka for anything.

  1. Our interest is mainly in the long-term storage of our business objects that will get updated as the result of various workflows. ( Temporal related data and the ability to debug Temporal Workflows is not really important at this point )… is it better if we maintain our business data independent of Temporal-ES integration?

Yes, we recommend storing this business data independently from ES. Another option is to have an always open workflow that maintains that particular entity data. It is hard to give a concrete recommendation without knowing the specifics of your use case.

3

Thank you Maxim.
I thought Temporal will remove the WF data after a ( configurable ) retention period.
If WF data itself is removed, what use is the ability to search for them from ES ? ( How can we debug if WF data itself is gone ? )

( Sorry like I mentioned, we are just getting started with temporal )

4

Visibility records are removed from ES after the retention period as well.

5

Thank you again Maxim.
( Sorry ) Im still confused. I thought the very purpose of ES is to act as a long term storage ( along with the ability to run more complicated queries ) .
But if teh data inside ES will live only as long as teh data inside Temporal, it is not serving as a long term storage.

Could you please point to the Temporal documentation that describes the purpose of ES

6

Temporal is not designed to act as a long term storage for historical data. ES is used to index open and recently (up to retention period) completed workflows.