Fetching Data from Query Method vs Elastic Search

Community Support
1

Use Case
I have a Workflow (W) which has an Object (O)
For a given Workflow ID (WFID), I want to fetch the Object (O)

Approach 1

  1. Store the Object in a class variable and expose it via a query method.
  2. Use the Temporal Service Library to query.

Approach 2

  1. Store the Object (Convert to JSON String) as a search attribute (integrated with Elastic Search)
  2. Use Elastic Search to Query for the objects json string. (The Object’s Json String contains the workflow ID)

Questions

  1. Which approach is faster and effecient?
  2. When should one prefer Elastic Search over a query method.?
2
  1. Which approach is faster and efficient?

The Approach 1.

  1. When should one prefer Elastic Search over a query method.?

When you need to find workflows by some custom or built-in attribute. For example, get a list of workflows that failed yesterday by customer id.

3

Maxim,
Why is the approach 1 faster and efficient?

4

Because approach 2 relies on an eventually consistent index stored in ElasticSearch which is not designed for high read rates. It is also not designed to store large amounts of data (which objects stored as JSON tend to be).

Approach 1 relies on primary Temporal DB and is fully consistent.

5

Maxim,

If my primary goal is to push data to elastic search and not really to search workflows based on attributes, should i really be using search attributes?

My team found search attributes as an easy way to push data to elastic search.
But as search attributes only supports primitives, we are forced to convert complex objects to json string.
With this approach we are converting structed data (Object) to unstructed data (json string) and pushing it to elastic search.

Wouldn’t writing our own code and pushing the structed data (Object) to elastic code be better?

I’m not an expert in elastic search, Isn’t it better to store structed data when ever possible in elastic search instead of a big json string?

6

I’m confused. Elastisearch is an indexing/search technology. I’ve never seen it being used to push complex objects for storage only. What is the use case that you are trying to solve?

7

Let me clarify

We have two use cases

Use Case 1
Push data to elastic search and use it to identify/search workflows.
These are small bits of information. And we are rightly using search attributes to push these bits of information to elastic search.

Use Case 2
We have complex objects. These objects will be searched and queried in elastic search but the intent of search is not really to find the workflow which contains that object.

  • Approach 1 - Convert this complex object into json string and push it to elastic search via search attributes.

  • Approach 2 - Write our own code to push this complex object to elastic search

Question
While approach 1 makes is easier to push data to elastic search, but is forces us to convert it to a json string (Unstructed data).
With the approach 2, while it takes some coding to push the data to elastic search, we will be able to retain the structure of the object.

Which approach should i take if I’m looking for better search over additional work? Is it approach 2?

8

Given the previous replies by Maxim imho it would be approach 2. Search attributes seem not to be fitting your use case.

9

Agree with @tihomir. The “Use Case 2” looks like an application level store that has very different requirements from the extended visibility Temporal provides. I would go with “Approach 2”.