We have deployed temporal self hosted in our production environment using Helm charts, server is v1.19.0, elasticsearch is 7.17.3 and we are using our own instance of MySQL v5.7. During a K8s cluster maintanence, 2 out of 3, elasticsearch pods has been deleted and the same has been rescheduled in different node. After this action, we are seeing following error in UI
List WorkflowExecutions failed: elastic : Error 404 (Not Found) : no such index [temporal_visibility_v1_dev] [type=index_not_found_exception]
To fix the issue, we have to delete the existing helm chart and redeploy the helm chart fresh. How to fix this, is it possible to assign pvc to the elasticsearch pod?
Also, once its re-deployed, we didn’t see older workflow entries in UI, does it not get populated from mysql to elasticsearch?
Please suggest the best possible ways to resolve this issue.