Seeing exception when using Elasticsearch + MySQL

Community Support
, ,
1

Using the temporal helm charts with MySQL and elasticsearch.enabled=true, I’m seeing a “search_phase_execution_exception” when using Temporal Web:

image
image2488×200 20.9 KB

This stack trace pops up in the logs for elasticsearch-master-0:

{“type”: “server”, “timestamp”: “2020-07-07T18:01:11,425Z”, “level”: “DEBUG”, “component”: “o.e.a.s.TransportSearchAction”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “All shards failed for phase: [query]”, “cluster.uuid”: “5p7x6DWoRBOMkXWlKRewTg”, “node.id”: “6bA93ZRJTdSCLPRcIUgw2A” ,
“stacktrace”: [“org.elasticsearch.index.query.QueryShardException: No mapping found for [StartTime] in order to sort on”,
“at org.elasticsearch.search.sort.FieldSortBuilder.resolveUnmappedType(FieldSortBuilder.java:506) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:399) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:159) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.search.SearchService.parseSource(SearchService.java:858) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.search.SearchService.createContext(SearchService.java:678) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:631) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:428) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.search.SearchService.access$200(SearchService.java:135) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.7.0.jar:7.7.0]”,
“at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]”,
“at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]”,
“at java.lang.Thread.run(Thread.java:832) [?:?]”] }

I can also verify the keys are setup properly by running:

bash-5.0# tctl --namespace default cl get-search-attr
±----------------------±-----------+
| KEY | VALUE TYPE |
±----------------------±-----------+
| BinaryChecksums | Keyword |
| CloseTime | Int |
| CustomBoolField | Bool |
| CustomDatetimeField | Datetime |
| CustomDoubleField | Double |
| CustomIntField | Int |
| CustomKeywordField | Keyword |
| CustomStringField | String |
| ExecutionStatus | Int |
| ExecutionTime | Int |
| HistoryLength | Int |
| NamespaceId | Keyword |
| RunId | Keyword |
| StartTime | Int |
| TaskQueue | Keyword |
| TemporalChangeVersion | Keyword |
| WorkflowId | Keyword |
| WorkflowType | Keyword |
±----------------------±-----------+

I’m not sure where to go from here, perhaps there’s an Elasticsearch mapping template that hasn’t been applied? Any help would be appreciated.

2 Likes
3

Hi, Paul!

Thank you for trying this and for the report.

Let me try to reproduce this problem.

Do you think you could share the full command line you used to run helm install, and the version of temporal helm chart that you are using (e. g. the output of git log -n 1 --pretty=oneline in temporal-helm-charts repo)?

Thank you!
Mark.

4

In the meantime, I was able to deploy the latest default Temporal Helm Chart (Cassandra) from @markmark (0.26.0) by applying the following changes to the elasticsearch config:

elasticsearch:
  enabled: true
  replicas: 3
  persistence:
    enabled: false
  imageTag: 6.8.8
  antiAffinity: soft

The addition of antiAffinity allows the elasticsearch cluster to run on a single node (for my dev server).

2 Likes
5

The cassandra deployment seems to be working fine for me now, although it does take longer to start but at least I can get back to development. In the case it does go down again and I’m forced to use MySQL, I’ll be sure to report the steps I take to this thread.

Thanks!

2 Likes
6

Ok, great, thank you, please don’t hesitate to ping if you run into problems. (and I will plan to test our mysql / ES scenario a bit more).

1 Like