How to observe the temporal cluster?

[tihomir]

Couple related forum posts that might be helpful to you:

Guidance on creating and interpreting Grafana dashboards

Hey @rfwagner, This is a lengthy topic and we plan to provide more information on operationalizing Temporal clusters in the future. We recently started dashboard repo which has some basic dashboards which gives visibility into Temporal service. This is work in progress so please use this just as a reference at this point. In the future we plan to have fully supported version of grafana dashboards using PromQL. All the metric emitted by server are listed in defs.go . So if you see somethings…

What metrics does temporal expose out of box and how to consume this in prometheus?

Hi, there is not much documentation around metrics, the statd which was removed too seemed very complex. I want to to understand a)what metrics does temporal expose by default b) are the metrics namespace specific? c) can i get queue /task list specific metrics? d) how to consume them in prometheus. e) if i am to develop custom metrics what’s the best way, should those be activities in workflows or interceptors?

Grafana graphs - not all graphs are getting populated

Hi i brought up the temporal with the default helm. With this setting I see the Front end graph/dashboard getting populated, However the Canary and Temporal dashboards are always empty see attached [screencapture-localhost-18080-d-SxRYJXZMz-frontend-2020-08-08-13_55_05] [screencapture-localhost-18080-d-58u-1uWGz-canary-2020-08-08-13_54_39] [screencapture-localhost-18080-d-g0SxnLMMk-temporal-2020-08-08-13_54_14]

For SDK Metrics:
Worker tuning guide: Developer's guide - Worker performance | Temporal Documentation
list of metrics produced:
Go SDK
Java SDK
and SDK metrics docs: Temporal SDK metrics reference | Temporal Documentation

If you want to get started pretty quick take a look at this docker compose repo, it has built in prometheus with scrape points as well as SDK and Server dashboards that you can use out of box to get you started in the right direction. Hope this helps.