Workflows not visible on UI after docker restart

Community Support
1

Hi Thiomir/Maxim,

In our testing environment some how docker was stopped and that’s why we have to take restart for it and when we did it we found that most of the workflows were vanished from temporal UI but were present in database. So is there any way to bring them back to temporal UI and along with this we found that querying based on search attribute is also not working that’s why signalling is not happening to respective workflows.

Is there any auto purging policy in temporal like after 3 months or any other value may be and if yes can we can configure it as per our need.

Problem with search attribute.

  1. While workflows are created search attributes are being set on workflows with feed values.

  2. But with this existing search attribute we unable to perform search.

  3. But when we add new search attribute and create new workflow then we are able to perform search based on that new added search attribute.

Could you please help in this because we are blocked with our testing.

Thanks!!

2

@Manish_Baraskar
sounds like an issue in the visibility store.

Are you able to get one of the missing workflow executions with temporal workflow describe -w <workflowId> -r <runId>

Is there any auto purging policy in temporal like after 3 months or any other value may be and if yes can we can configure it as per our need.

Retention Period will remove closed workflows after the set period, but there is nothing like that for open workflows

3

Hi @antonio.perez

We are able to fetch the workflows by the cmd you have shared. Can we restore those workflows.

And about the retention policy when our docker got restarted then at that time we were unable to see the running workflows as well because it should not be purged as you said it is applicable for closed workflows only.

4

Hi Team,

Any updates on above queries??

As because our existing search attributes are not working we have to recreate it and I think it is not the best solution if such things happens in higher environment. Please suggest a solution.

5

@Manish_Baraskar

temporal has two data stores, visibility store and default store.

Queries like listworkflow executions hit the visibility store, while the execution API (start, describe, signal… ) hits default store. What you are describing looks like an issue with the visibility store, it might get corrupted during the restart.

what dbs are you using? configuration etc… ? in your cluster

6

Hi @antonio.perez ,

We are using MySql and it is externalize.

Below is the configuration details.

File name : docker-compose-mysql-external-es.yml

version: “3.5”
services:
elasticsearch:
container_name: temporal-elasticsearch
environment:
- cluster.routing.allocation.disk.threshold_enabled=true
- cluster.routing.allocation.disk.watermark.low=512mb
- cluster.routing.allocation.disk.watermark.high=256mb
- cluster.routing.allocation.disk.watermark.flood_stage=128mb
- discovery.type=single-node
- ES_JAVA_OPTS=-Xms256m -Xmx256m
- xpack.security.enabled=false
image: elasticsearch:${ELASTICSEARCH_VERSION}
networks:
- temporal-network
ports:
- 9200:9200
volumes:
- /var/lib/elasticsearch/data
temporal:
container_name: temporal
depends_on:
- elasticsearch
environment:
- DB=mysql
- MYSQL_USER=XXX
- MYSQL_PWD=XXX
- MYSQL_SEEDS=External Server
- DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development-sql.yaml
- ENABLE_ES=true
- ES_SEEDS=elasticsearch
- ES_VERSION=v7
image: temporalio/auto-setup:${TEMPORAL_VERSION}
networks:
- temporal-network
ports:
- 7233:7233
volumes:
- ./dynamicconfig:/etc/temporal/config/dynamicconfig
labels:
kompose.volume.type: configMap
temporal-admin-tools:
container_name: temporal-admin-tools
depends_on:
- temporal
environment:
- TEMPORAL_CLI_ADDRESS=temporal:7233
image: temporalio/admin-tools:${TEMPORAL_VERSION}
networks:
- temporal-network
stdin_open: true
tty: true
temporal-ui:
container_name: temporal-ui
depends_on:
- temporal
environment:
- TEMPORAL_ADDRESS=temporal:7233
- TEMPORAL_CORS_ORIGINS=http://localhost:3000
image: temporalio/ui:${TEMPORAL_UI_VERSION}
networks:
- temporal-network
ports:
- 8080:8080
networks:
temporal-network:
driver: bridge
name: temporal-network

7

Hi @antonio.perez,

Any updates please…

As again we have faced the same issue as mentioned in first query whenever container for temporal and elastic search gets recreated all of our workflows gets vanished from UI and even searching on search attribute also does not work.

Thanks.