I try to use grpc-health-probe to do a health check, Instead of the original tcp check. I would like to check three services, include frontend, history and matching, and my ‘exec Liveness probe’ is respectively:
- grpc-health-probe -addr=:7233 -service=temporal.api.workflowservice.v1.WorkflowService
- grpc-health-probe -addr=:7234 -service=temporal.api.workflowservice.v1.HistoryService
- grpc-health-probe -addr=:7235 -service=temporal.api.workflowservice.v1.MatchingService
But the liveness probe failed after I install temporal by helm-chart, however I go into the pod and manually execute the above command, I can get the result “status: SERVING”, which indicates the grpc is normal. So I really don’t know what’s wrong, and how to fix it?
Are you using charts from this repo?
If so, can you show result of
kubectl get svc
Do your frontend, history, matching services have external ip? With Temporal helm charts all these services have ClusterIP type which makes them only accessible within the cluster.
If you are running locally via helm charts you could change services to NodePort type here
and if you dont have external cluster / nodeport ip, can port-forward for example history service:
kubectl port-forward services/temporaltest-history-headless 7234:7234
and can run
grpc-health-probe -addr=localhost:7234 -service=temporal.api.workflowservice.v1.HistoryService outside of cluster, if that helps
My description may be misleading you. I have rebuilded the tempooral-server image and copy the grpc-health-probe into image, so the grpc-health-probe can access the port within the cluster, and I use it as ‘Exec Liveness Probe’.