This is most likely ES version mismatch. How do you run Temporal? docker-compose files are configured to use v7 but docker image itself use v6 by default. ES_VERISION env needs to be set to v7 similar to this. Also proper index template needs to be used for v7.
Are you using external ES or want to deploy ES using helm? If you want to deploy it you should not use -f values/values.elasticsearch.yaml because this file configures cluster to use ES which is already deployed in the same kube cluster (don’t ask me why). And if you just remove this line, it will use defaults from values.yaml which will deploy ES and create index.
You don’t need to do anything manually to create index or its template.
Hi @alex, I am using an external ES cluster and have configured the details in values.elasticsearch.yaml before running helm install.
I can also see the index temporal-visibility-dev created in the external ES. When I execute a workflow, I can see an entry like the following getting inserted in ES:
Ok, this is what I suspected. For some reason you don’t have index template and it wasn’t applied when temporal-visibility-dev index was created. And this where all other problems comes from.
To fix this I would suggest you to drop your current index and recreate it with two curl commands the same way as we do it for server development.
And it would be great to investigate why did it happen. Am I right that you have:
enabled: false
external: true
in your values/values.elasticsearch.yaml file?
Another thing to check is ENABLE_ES env in any server container. It should be true.
Honestly, I don’t know what happened. If ENABLE_ES env is set to true then server should create index template and index during start up but it didn’t. The easiest fix for you would be to just drop the index and create it manually.
It would be great if, instead of manually creating index, you do another deploy using same helm command and send me startup logs. Unfortunately it is not possible to say which instance actually created the index, so I will need these startup logs from every role.
Btw, I am going to change this startup logic soon.
Thanks, @alex. I dropped the index and recreated it manually after creating the template. Things are working fine now.
Before doing that, I also ran the helm command again to understand if this was an intermittent issue. But the same problem happened and I have captured the startup logs. How can I share the logs with you?
Also, if I don’t want to use Kafka, what changes should I make to the helm command and different values.yaml files?
When Kafka is not used, does Temporal write to Elasticsearch synchronously? Does it have any effect on performance?
Hi @alex, I had a few queries regarding the visibility feature with Elasticsearch.
I am using v1.7.0. I have installed it using Helm chart. As per documentation, Kafka is used by default for versions below v1.8.0. When I give any random Kafka host, I can still see that the installation goes through and data is inserted to Elasticsearch. Not sure how is this happening if the Kafka host itself does not exist?
How do I disable the usage of Kafka for v1.7.0? Does Temporal write to Elasticsearch synchronously when Kafka is disabled?
In the link, the index name is hard-coded as temporal-visibility-dev. Does it mean if I use any other name, I have to create it manually?
In the index, I see only workflow details like the following? Does visibility UI get the other information like activity details from Cassandra?
For default namespace, retention period is unknown and data is not written to Elasticsearch once the Workflow completes. Is that a bug or the expected behavior?
I got the Temporal version from the Temporal UI? Is there any command or some other means of getting the Temporal version?
I believe docs says that starting 1.8.0 Kafka is no longer supported but default was changed in 1.7.0 and its release notes explicitly says this.
So it is disabled by default. Internal queue is used though and all previous guarantees are still there: if Elasticsearch is down, Temporal will continue to work and it won’t affect core services. Visibility won’t be updated (of course) but all requests will be stored and as soon as ES is back again, everything will be flushed.
Currently index is created when server starts using name from $ES_VIS_INDEX env. Makefile is just for development purpose. You can set it to anything but if name doesn’t start with "temporal-visibility- you need to change template also.
Yes. Basically only list is served from Elasticsearch. As soon as we know Workflow Id/Run Id we go to main database to fetch all required info (history, etc).
Default retention for default namespace should be 1 day. What is unknown? Where do you see it? But even without retention completed workflows should be marked as Completed in Elasticsearch. Retention only controls deletion of the records. If you see workflow not being completed, I need more info on it.
The best way to get server version is to run server binary (temporal-server) and it will output it’s version. tctl also prints it’s version and it is currently in sync with server version. Web version can be different.
