# Temporal io restart problem on Kubernetes node restart **URL:** \ **Category:** Community Support\ **Tags:** helm, kubernetes\ **Created:** [September 3, 2020, 3:04pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566 "2020-09-03T15:04:58Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![shawel\_negussie](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/shawel_negussie/32/231_2.png) [@shawel\_negussie](https://community.temporal.io/u/shawel_negussie)\ **Post date:** [September 3, 2020, 3:04pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/1 "2020-09-03T15:04:58Z")
I used helm chart to deploy temporal IO in Kubernetes. It was working great but i noticed frontend and history server are failing to restart. I noticed the following log “waiting for default keyspace to become ready”. is there anywhere to start looking to debug problems. Should it not recover automatically on node restarts ? ---
**Author:** ![markmark](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/markmark/32/88_2.png) [@markmark](https://community.temporal.io/u/markmark)\ **Post date:** [September 4, 2020, 5:52pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/3 "2020-09-04T17:52:01Z")
Hey Shawel, Thank you for the report! This might happen when db schemas have not been created on the database server (or, perhaps – and this shouldn’t actually be the case, I would expect a different error message, I would need to try to repro to be sure – if the db server is not reachable). Do you think you could share (here or in a direct message) - the `helm install ...` command you ran to install temporal, and - the output of `kubectl get pods` command ? That would help me reproduce and debug the problem. Thank you! Mark. ---
**Author:** ![shawel\_negussie](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/shawel_negussie/32/231_2.png) [@shawel\_negussie](https://community.temporal.io/u/shawel_negussie)\ **Post date:** [September 4, 2020, 6:36pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/4 "2020-09-04T18:36:20Z")
hey @markmark thanks for the reply! here is the helm install i used > helm install --namespace temporalio-dev > –set server.replicaCount=1 > –set cassandra.config.cluster\_size=1 > –set prometheus.enabled=false > –set grafana.enabled=false > –set elasticsearch.enabled=false > –set kafka.enabled=false > temporaltest . --timeout 15m here is the kubectl command > ➜ ~ kubectl get pods -n temporalio-dev > NAME READY STATUS RESTARTS AGE > temporaltest-admintools-7c5b9798ff-7jqrf 1/1 Running 0 179m > temporaltest-cassandra-0 1/1 Running 0 24h > temporaltest-frontend-6bf7969c9b-277p5 0/1 Init:2/4 0 27h > temporaltest-history-5ddfb6d5cf-km5hf 0/1 Init:2/4 0 36h > temporaltest-matching-7d67c4fc5-4dskb 0/1 Init:2/4 0 24h > temporaltest-web-7f5bb66cf-lx9xt 1/1 Running 0 24h > temporaltest-worker-7b689c8d-zvrmb 0/1 Init:2/4 0 15h ![Screen Shot 2020-09-04 at 2.27.56 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/1X/dbe233838b31a239756dcdba832e03282ebb22a1.png) For connectivity issues i can ping the cassandra se ![Screen Shot 2020-09-04 at 2.34.31 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/1X/09bacd2587b9d272cb90254d58c8e58538168052.png) rvice from the front end ---
**Author:** ![shawel\_negussie](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/shawel_negussie/32/231_2.png) [@shawel\_negussie](https://community.temporal.io/u/shawel_negussie)\ **Post date:** [September 15, 2020, 3:10pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/5 "2020-09-15T15:10:02Z")
hi @markmark any more insights on this? best ---
**Author:** ![markmark](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/markmark/32/88_2.png) [@markmark](https://community.temporal.io/u/markmark)\ **Post date:** [September 15, 2020, 5:14pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/6 "2020-09-15T17:14:05Z")
Hi Shawel, Apologies for the slow response on my part! I am having a hard time reproducing this. Are you still seeing this issue? Here are my steps (forgive the bird theme;): 1. Create the namespace: ```auto ~/src/helm-charts $ cat ns.json { "apiVersion": "v1", "kind": "Namespace", "metadata": { "name": "duck", "labels": { "name": "duck" } } } ~/src/helm-charts $ kubectl create -f ns.json namespace/duck created ``` 1. install temporal to that namespace ```auto ~/src/helm-charts $ helm install --namespace duck --set server.replicaCount=1 --set cassandra.config.cluster_size=1 --set prometheus.enabled=false --set grafana.enabled=false --set elasticsearch.enabled=false --set kafka.enabled=false temporaltest . --timeout 15m NAME: temporaltest LAST DEPLOYED: Tue Sep 15 09:55:57 2020 NAMESPACE: duck STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: To verify that Temporal has started, run: kubectl --namespace=duck get pods -l "app.kubernetes.io/instance=temporaltest" ``` 1. check that the pods are running (giving it a couple of mins, and ignore kibana for now): ```auto ~/src/helm-charts $ kubectl get pods --namespace duck NAME READY STATUS RESTARTS AGE temporaltest-admintools-76684c7c59-4scgz 1/1 Running 0 4m9s temporaltest-cassandra-0 1/1 Running 0 4m9s temporaltest-frontend-56c4df6845-qdrwn 1/1 Running 2 4m9s temporaltest-history-9655b4746-hg54s 1/1 Running 3 4m9s temporaltest-kibana-8c9c486d7-d66l9 0/1 Running 0 4m9s temporaltest-matching-85f8c9bcc6-gk6mg 1/1 Running 2 4m9s temporaltest-web-84864fdddc-qjtsm 1/1 Running 0 4m9s temporaltest-worker-9bd8b68cf-x6zxh 1/1 Running 2 4m9s ``` 1. do a basic check (create a new temporal namespace, “chickens”;): ```auto ~/src/helm-charts $ kubectl --namespace duck exec -it services/temporaltest-admintools -- bash -c 'tctl --ns chickens namespace register' Namespace chickens successfully registered. ``` ```auto ~/src/helm-charts $ kubectl --namespace duck exec -it services/temporaltest-admintools -- bash -c 'tctl --ns chickens namespace describe' Name: chickens Id: 7487be56-c238-478a-9b2f-0ffa18e1bb98 Description: OwnerEmail: NamespaceData: map[string]string(nil) State: Registered RetentionInDays: 72h0m0s ActiveClusterName: active Clusters: active HistoryArchivalState: Disabled VisibilityArchivalState: Disabled Bad binaries to reset: +-----------------+----------+------------+--------+ | BINARY CHECKSUM | OPERATOR | START TIME | REASON | +-----------------+----------+------------+--------+ +-----------------+----------+------------+--------+ ``` ---
**Author:** ![shawel\_negussie](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/shawel_negussie/32/231_2.png) [@shawel\_negussie](https://community.temporal.io/u/shawel_negussie)\ **Post date:** [September 15, 2020, 5:19pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/7 "2020-09-15T17:19:19Z")
Hey @markmark no problem. thanks for the reply. The failure actually came when i did a rolling update to the kubernetes nodes. It did work for me the first time i installed the chart, created namespace and everything no issues there. It would still come up on new installs. I expected it to come up on node failures, restarts as well which is the current issue. Best Shawel ---
**Author:** ![markmark](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/markmark/32/88_2.png) [@markmark](https://community.temporal.io/u/markmark)\ **Post date:** [September 15, 2020, 5:30pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/8 "2020-09-15T17:30:00Z")
Ah!!! If you could share more details on the rolling update that you performed (here or in a direct message), that would be super-helpful – we will look into testing that scenario (and fixing bugs that, it sounds like, it might have;). Thank you, Mark. ---
**Author:** ![shawel\_negussie](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/shawel_negussie/32/231_2.png) [@shawel\_negussie](https://community.temporal.io/u/shawel_negussie)\ **Post date:** [September 15, 2020, 5:36pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/9 "2020-09-15T17:36:46Z")
Sure it was a kubernetes version upgrade from v15.0 to 15.1 using kops (kops update cluster --yes). It does recycle the all nodes in the cluster. Temporalio never came up after that. ---
**Author:** ![markmark](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/markmark/32/88_2.png) [@markmark](https://community.temporal.io/u/markmark)\ **Post date:** [September 15, 2020, 5:43pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/10 "2020-09-15T17:43:28Z")
Perfect, this is great information, thank you. ---
**Author:** ![shawel\_negussie](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/shawel_negussie/32/231_2.png) [@shawel\_negussie](https://community.temporal.io/u/shawel_negussie)\ **Post date:** [September 21, 2020, 4:08pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/11 "2020-09-21T16:08:12Z")
It seems like the keyspace information from cassandra is gone on node restart from kubernetes. I don’t know why it happens. But i needed to do this by going to admin tool to get this working: > export CASSANDRA\_HOST=temporalio-cassandra.temporalio-dev.svc > export CASSANDRA\_PORT=9042 > > temporal-cassandra-tool create -k temporal > temporal-cassandra-tool create -k temporal\_visibility --replication-factor 1 > > export CASSANDRA\_KEYSPACE=temporal > temporal-cassandra-tool setup-schema -v 0.0 > temporal-cassandra-tool update-schema -d /etc/temporal/schema/cassandra/temporal/versioned/ > > export CASSANDRA\_KEYSPACE=temporal\_visibility > temporal-cassandra-tool create -k temporal\_visibility --replication-factor 1 > temporal-cassandra-tool setup-schema -v 0.0 > temporal-cassandra-tool update-schema -d /etc/temporal/schema/cassandra/temporal/versioned/ ---
**Author:** ![barbarize](https://avatars.discourse-cdn.com/v4/letter/b/8e8cbc/32.png) [@barbarize](https://community.temporal.io/u/barbarize)\ **Post date:** [February 15, 2021, 5:15pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/12 "2021-02-15T17:15:15Z")
Hi @markmark and @shawel_negussie. I’m having the same problem with my [Temporal.io](http://Temporal.io) installation. Temporal is deployed on a Google Kubernetes Engine in a Production environment. And when nodes restart, the server never up again, and logs are showing the message “waiting for default keyspace to become ready”. The Helm command used to install was: ``` helm install \ --set server.replicaCount=1 \ --set cassandra.config.cluster_size=1 \ --set prometheus.enabled=true \ --set grafana.enabled=true \ --set elasticsearch.enabled=true \ --set kafka.enabled=true \ temporal . --timeout 15m -namespace temporal ``` The workaround proposed by @shawel_negussie works sometimes, but I think it’s not an ideal scenario to do at every restart. I don’t know, I’m wondering if we are having a problem with config or environment vars. Because I noticed that temporal-admintools are pointing to IPs to reach other services. So, maybe these IPs have changed on k8s node restart. What do you think about it? I’ll appreciate your help! ---
**Author:** ![barbarize](https://avatars.discourse-cdn.com/v4/letter/b/8e8cbc/32.png) [@barbarize](https://community.temporal.io/u/barbarize)\ **Post date:** [February 16, 2021, 3:38am UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/13 "2021-02-16T03:38:07Z")
@ryland or @maxim can help me? ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [February 16, 2021, 3:40am UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/14 "2021-02-16T03:40:19Z")
AFAIK we don’t recommend helm for production deployments and especially upgrades. ---
**Author:** ![shawel\_negussie](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/shawel_negussie/32/231_2.png) [@shawel\_negussie](https://community.temporal.io/u/shawel_negussie)\ **Post date:** [February 16, 2021, 12:49pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/15 "2021-02-16T12:49:18Z")
I have switched to my own instance of MySQL since this ticket. It seems more stable! Also node affinity seems to help. ---
**Author:** ![barbarize](https://avatars.discourse-cdn.com/v4/letter/b/8e8cbc/32.png) [@barbarize](https://community.temporal.io/u/barbarize)\ **Post date:** [February 16, 2021, 4:02pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/16 "2021-02-16T16:02:38Z")
What is the recommendation for production deployments? ---
**Author:** ![barbarize](https://avatars.discourse-cdn.com/v4/letter/b/8e8cbc/32.png) [@barbarize](https://community.temporal.io/u/barbarize)\ **Post date:** [February 16, 2021, 4:03pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/17 "2021-02-16T16:03:59Z")
Thanks a lot! I’ll try this too. ---
**Author:** ![cg1972](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/cg1972/32/465_2.png) [@cg1972](https://community.temporal.io/u/cg1972)\ **Post date:** [April 5, 2021, 11:20pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/18 "2021-04-05T23:20:47Z")
I just encountered a similar issue in our test environment. We are running a 3 node cassandra cluster in K8 exclusively for the temporal keyspaces. A k8 worker node hosting one of the cassandra pods started evicting pods. As a result of this the cassandra pod was restarted (on the same node). After this all the temporal keyspaces were gone from cassandra. We also run a separate cassandra cluster on the same worker nodes, these were also restarted without any loss of data so this seems to be specific to the temporal keyspaces. Has there been any further investigation into how this could occur? Is there anything in temporal that would trigger the keyspaces to be removed or refreshed? ---
**Author:** ![barbarize](https://avatars.discourse-cdn.com/v4/letter/b/8e8cbc/32.png) [@barbarize](https://community.temporal.io/u/barbarize)\ **Post date:** [April 6, 2021, 10:13am UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/19 "2021-04-06T10:13:18Z")
Did you deploy it using the the helm chart? I couldn’t resolve this issues with Cassandra on my deployment. ---
**Author:** ![cg1972](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/cg1972/32/465_2.png) [@cg1972](https://community.temporal.io/u/cg1972)\ **Post date:** [April 6, 2021, 9:21pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/20 "2021-04-06T21:21:25Z")
Yes, temporal was deployed using the helm chart and the keyspaces were created using the temporal-cassandra-tool. I did notice in one of the posts that it is recommended that replication be set to 3 for production where our test environment would have defaulted to replication factor of 1. I doubt this would have affected the keyspaces being removed? ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [April 6, 2021, 9:42pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/21 "2021-04-06T21:42:04Z")
We strongly advise against running production Cassandra (or any other DB) deployments on K8s, especially using the provided Helm Chart. They are included for quick start scenarios. You are practically guaranteed to run into data loss issues with the Helm Chart based Cassandra deployment. [Next page](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566.md?page=2)