Ability to add custom Object Search Attribute

Community Support
1

Currently, the type of search attribute that can be added is any of the following:

string
keyword
int
double
bool
datetime

Upserting is nice and simple:

attr1 := map[string]interface{}{
    "CustomIntField": 1,
    "CustomBoolField": true,
}
workflow.UpsertSearchAttributes(ctx, attr1)

Now let’s say we want our workflow to contain some custom Object attribute, like a histogram:

{
  "my_histogram" : {
      "values" : [0.1, 0.2, 0.3, 0.4, 0.5], 
      "counts" : [3, 7, 23, 12, 6] 
   }
}

It is not currently possible to pass this object through to elasticsearch via Temporal.

2

Temporal currently only support Binary field type for memo payload. Can you describe your use case? Why do you want to store custom objects inside Elastic Search document? Do you expect to search on these custom objects?

3

I’d like to store these custom objects inside the workflow’s document so that I can use advanced ES DSL query techniques to retrieve the list of workflows. So to answer the second question, and present another feature request, I would need the ability to query using raw DSL.

I can describe the use-case in more detail via slack if needed.

4

Hey @pauldemarco,
Have you looked at ListWorkflowExecutions, ScanWorkflowExecutions, and CountWorkflowExecutions exposed by Temporal server? They already allow you to specify advanced queries using SQL like syntax. Under the covers Temporal uses SearchWithDSL capabilities of ES Client to issue advanced queries to elastic search backend. Here are few example of queries it already supports:

List All Closed Executions

./tctl wf list -psa
       WORKFLOW TYPE       |                      WORKFLOW ID                       |                RUN ID                |    TASK QUEUE     | START TIME | EXECUTION TIME | END TIME |                 SEARCH ATTRIBUTES
  SearchAttributesWorkflow | search_attributes_e7aa9e23-6e65-4d59-8052-19bdd0b177ca | efacc806-fd23-4c4c-92c1-19069823a8c0 | search-attributes | 10:40:00   | 10:40:00       | 10:41:12 | CustomKeywordField=Update2
                           |                                                        |                                      |                   |            |                |          | CustomIntField=2 CustomDoubleField=3.14
                           |                                                        |                                      |                   |            |                |          | CustomDatetimeField=2019-01-01T00:00:00-08:00
                           |                                                        |                                      |                   |            |                |          | CustomBoolField=true
                           |                                                        |                                      |                   |            |                |          | BinaryChecksums=[f98984a93518de6dac73f53d28446891]
                           |                                                        |                                      |                   |            |                |          | CustomStringField=String field is for text. When
                           |                                                        |                                      |                   |            |                |          | query, it will be tokenized for partial match.
                           |                                                        |                                      |                   |            |                |          | StringTypeField cannot be used in Order By
  SearchAttributesWorkflow | search_attributes_c2967c6b-56c6-402b-a030-ccd0baf90ffe | f321f9cf-b2a8-42e5-b886-28eecca608fc | search-attributes | 13:48:38   | 13:48:38       | 13:48:40 | CustomDatetimeField=2019-01-01T00:00:00-08:00
                           |                                                        |                                      |                   |            |                |          | CustomBoolField=true
                           |                                                        |                                      |                   |            |                |          | BinaryChecksums=[f98984a93518de6dac73f53d28446891]
                           |                                                        |                                      |                   |            |                |          | CustomStringField=String field is for text.
                           |                                                        |                                      |                   |            |                |          | When query, it will be tokenized for partial
                           |                                                        |                                      |                   |            |                |          | match. StringTypeField cannot be used in Order
                           |                                                        |                                      |                   |            |                |          | By CustomKeywordField=Update2 CustomIntField=2
                           |                                                        |                                      |                   |            |                |          | CustomDoubleField=3.14

List Workflow Using System Defined Attribute (WorkflowId)

./tctl wf list -q 'WorkflowId="search_attributes_e7aa9e23-6e65-4d59-8052-19bdd0b177ca"'
       WORKFLOW TYPE       |                      WORKFLOW ID                       |                RUN ID                |    TASK QUEUE     | START TIME | EXECUTION TIME | END TIME
  SearchAttributesWorkflow | search_attributes_e7aa9e23-6e65-4d59-8052-19bdd0b177ca | efacc806-fd23-4c4c-92c1-19069823a8c0 | search-attributes | 10:40:00   | 10:40:00       | 10:41:12

List Workflows Using Custom attributes

./tctl wf list -q 'CustomIntField=2'
       WORKFLOW TYPE       |                      WORKFLOW ID                       |                RUN ID                |    TASK QUEUE     | START TIME | EXECUTION TIME | END TIME
  SearchAttributesWorkflow | search_attributes_e7aa9e23-6e65-4d59-8052-19bdd0b177ca | efacc806-fd23-4c4c-92c1-19069823a8c0 | search-attributes | 10:40:00   | 10:40:00       | 10:41:12
  SearchAttributesWorkflow | search_attributes_c2967c6b-56c6-402b-a030-ccd0baf90ffe | f321f9cf-b2a8-42e5-b886-28eecca608fc | search-attributes | 13:48:38   | 13:48:38       | 13:48:40

Count Workflows Using Custom Query:

./tctl wf count -q 'CustomIntField=2'
2

Order By Query:

./tctl wf list -q 'order by StartTime asc'
       WORKFLOW TYPE       |                      WORKFLOW ID                       |                RUN ID                |    TASK QUEUE     | START TIME | EXECUTION TIME | END TIME
  SearchAttributesWorkflow | search_attributes_c2967c6b-56c6-402b-a030-ccd0baf90ffe | f321f9cf-b2a8-42e5-b886-28eecca608fc | search-attributes | 13:48:38   | 13:48:38       | 13:48:40
  SearchAttributesWorkflow | search_attributes_e7aa9e23-6e65-4d59-8052-19bdd0b177ca | efacc806-fd23-4c4c-92c1-19069823a8c0 | search-attributes | 10:40:00   | 10:40:00       | 10:41:12

You can run a lot of advanced queries using these capabilities already supported by Temporal. Would love to hear details about your use case which is not covered by the 3 APIs I linked above. Feel free to reach out to me on slack if that is easier for you to discuss use case.

1 Like