# Temporal io restart problem on Kubernetes node restart **URL:** \ **Category:** Community Support\ **Tags:** helm, kubernetes\ **Created:** [September 3, 2020, 3:04pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566 "2020-09-03T15:04:58Z")\ **Posts on this page:** 8\ **Page:** 2
**Author:** ![cg1972](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/cg1972/32/465_2.png) [@cg1972](https://community.temporal.io/u/cg1972)\ **Post date:** [April 6, 2021, 10:04pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/22 "2021-04-06T22:04:54Z")
Hi Maxim, Just to clarify, the cassandra DB on K8’s that is used for the temporal keyspaces was not deployed using the temporal helm charts. It’s a separate deployment that temporal connects to. Can you provide details on why the helm chart based cassandra deployment can have data losses? Although we did not use the temporal helm charts for deploying cassandra it would be handy knowing what could cause the losses so I can check our cassandra configuration for similar issues. ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [April 6, 2021, 11:08pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/23 "2021-04-06T23:08:51Z")
I’m not expert in K8s. But AFAIK running production quality persistence on it is very non trivial. And I believe Helm chart is not a right technology to manage databases on K8s. ---
**Author:** ![cg1972](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/cg1972/32/465_2.png) [@cg1972](https://community.temporal.io/u/cg1972)\ **Post date:** [April 6, 2021, 11:10pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/24 "2021-04-06T23:10:11Z")
Thanks Maxim. ---
**Author:** ![Vikram\_Bailur](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/vikram_bailur/32/1834_2.png) [@Vikram\_Bailur](https://community.temporal.io/u/Vikram_Bailur)\ **Post date:** [March 25, 2022, 5:05pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/25 "2022-03-25T17:05:57Z")
Hello @maxim I am running into the same problem. Mine is NOT a production env btw. I have deployed to a GKE autopilot cluster using the helm chart (with small modifications by setting resource limits). The cassandra headless clusterIP & pods are running but the frontend-pod is not coming up due to the "waiting for default keyspace to become ready” message. How do i resolve this? thanks Vikram ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [March 25, 2022, 5:57pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/26 "2022-03-25T17:57:52Z")
Hi @Vikram_Bailur Could you confirm that your db schema is full created before Temporal server comes up? Helm charts do run some [batch jobs](https://github.com/temporalio/helm-charts/blob/master/templates/server-job.yaml#L55) to set up the schema which should complete first. The error you mentioned can happen if cassandra is [up](https://github.com/temporalio/helm-charts/blob/master/templates/server-deployment.yaml#L56) but the keyspace hasnt completed setting up. ---
**Author:** ![Vikram\_Bailur](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/vikram_bailur/32/1834_2.png) [@Vikram\_Bailur](https://community.temporal.io/u/Vikram_Bailur)\ **Post date:** [March 25, 2022, 7:30pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/27 "2022-03-25T19:30:02Z")
> [@tihomir](#): > > Helm charts do run some [batch jobs](https://github.com/temporalio/helm-charts/blob/master/templates/server-job.yaml#L55) to set up the schema which should complete first. Hi @tihomir - thanks for your quick response - I think you might be right - i was trying to figure out what is wrong w cassandra. Since Im using GKE Autopilot, resources take some time to get provisioned and I think the batch job might have tried to run without the cassandra pods being fully up & running. This is the message on the Job pod. `*** Can't find temporaltest-cassandra.default.svc.cluster.local: No answer` The cassandra pod also has the following message `containers with unready status: [temporaltest-cassandra]` UPDATE: it finally started up and I think the job has run as well. I did see in the docs a way to setup the cloudsql proxy in the helm chart(to use GCP Cloud Sql as the db instead of cassandra) but i don’t see specific instructions on how to install the helm chart with that (sorry i am new to helm) - do i just use a bare minimum helm chart and then run the helm install for the cloudsqlproxy ? ---
**Author:** ![derek](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/derek/32/245_2.png) [@derek](https://community.temporal.io/u/derek)\ **Post date:** [March 25, 2022, 8:19pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/28 "2022-03-25T20:19:09Z")
> [@Vikram\_Bailur](#): > > Since Im using GKE Autopilot, resources take some time to get provisioned and I think the batch job might have tried to run without the cassandra pods being fully up & running. that should be fine - everything should wait and fail for a while til it comes up. if the batch job didn’t work and isn’t still trying to run, you can create another one - or just log into the admin tools pod and run the commands the batch job would have run. if you have enough issue getting cassandra to work and you want to try cloudsql with proxy, you can deploy it to the cluster by setting a value for `server.sidecarContainers` - [https://github.com/temporalio/helm-charts/blob/master/values.yaml#L21](https://github.com/temporalio/helm-charts/blob/master/values.yaml#L21) this is just a list of containers that looks like any other kubernetes `containers:` section. anything you put in there will run in the same pod as temporal server containers. ---
**Author:** ![Yuvaraj\_Loganathan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/yuvaraj_loganathan/32/243_2.png) [@Yuvaraj\_Loganathan](https://community.temporal.io/u/Yuvaraj_Loganathan)\ **Post date:** [July 13, 2025, 4:18pm UTC](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566/29 "2025-07-13T16:18:40Z")
The problem is temporal helm charts does not enable persitence by default. so any pod restart will loose the data. > [Previous page](https://community.temporal.io/t/temporal-io-restart-problem-on-kubernetes-node-restart/566.md?page=1)