# Guidance on creating and interpreting Grafana dashboards

**URL:** <https://community.temporal.io/t/guidance-on-creating-and-interpreting-grafana-dashboards/493>\
**Category:** Community Support\
**Tags:** prometheus, metrics\
**Created:** [August 19, 2020, 7:59pm UTC](https://community.temporal.io/t/guidance-on-creating-and-interpreting-grafana-dashboards/493 "2020-08-19T19:59:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rfwagner](https://avatars.discourse-cdn.com/v4/letter/r/a87d85/32.png) [@rfwagner](https://community.temporal.io/u/rfwagner)\
**Post date:** [August 19, 2020, 7:59pm UTC](https://community.temporal.io/t/guidance-on-creating-and-interpreting-grafana-dashboards/493/1 "2020-08-19T19:59:45Z")

</div>

Can you offer some guidance on using prometheus+grafana to monitor Temporal. Some topics that would be useful:

1. what are the top tier metrics that must be monitored
2. a brief high level description of what each top-tier metric is measuring
3. what are the units being displayed
4. real-world experience on correlating user-visible issues to metrics that can help diagnose those issues

Any help on this topic would be much appreciated!

---

<div class="post-metadata">

**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\
**Post date:** [August 20, 2020, 6:34am UTC](https://community.temporal.io/t/guidance-on-creating-and-interpreting-grafana-dashboards/493/3 "2020-08-20T06:34:26Z")

</div>

Hey @rfwagner,  
This is a lengthy topic and we plan to provide more information on operationalizing Temporal clusters in the future. We recently started [dashboard repo](https://github.com/temporalio/dashboards) which has some basic dashboards which gives visibility into Temporal service. This is work in progress so please use this just as a reference at this point. In the future we plan to have fully supported version of grafana dashboards using PromQL.

All the metric emitted by server are listed in [defs.go](https://github.com/temporalio/temporal/blob/master/common/metrics/defs.go). So if you see somethings are missing in the dashboards then you can use the [defs.go](https://github.com/temporalio/temporal/blob/master/common/metrics/defs.go) as a reference.

Some of the most useful metrics are the following:

1. **Service Metric** :  
For each request by service handler we emit `service_requests`, `service_errors`, and `service_latency` metric with `type`, `operation`, and `namespace` tags. This gives you basic visibility into service usage and allows you to look request rates across services, namespaces or even operations.

2. **Persistence Metric** :  
Temporal emits `persistence_requests`, `persistence_errors` and `persistence_latency` metric for each persistence operation. These metrics are tagged with `operation` tag to allow getting request rates, error rates or latencies per operation. These are super useful to identify issues caused by database problems.

3. **Workflow Stats** :  
Temporal also emits counters on completion of workflows. These are super useful in getting overall stats about workflow completion. Use `workflow_success`, `workflow_failed`, `workflow_timeout`, `workflow_terminate` and `workflow_cancel` counters for each type of workflow completion. They are also tagged with namespace tag.

I would start with these 3 basic category of metrics before digging deeper into metrics which gives insights into other internals of the system. Apart from these metric emitted by the server you also want to monitor infrastructure specific metric like cpu, memory, network for all hosts running Temporal roles.

Hope this gives you some starting point. Feel free to ask if you have questions about a specific metric or some specific use case in mind.

---

<div class="post-metadata">

**Author:** ![Steven\_Sun](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/steven_sun/32/29_2.png) [@Steven\_Sun](https://community.temporal.io/u/Steven_Sun)\
**Post date:** [August 27, 2020, 6:38pm UTC](https://community.temporal.io/t/guidance-on-creating-and-interpreting-grafana-dashboards/493/4 "2020-08-27T18:38:54Z")

</div>

> [@samar](#):
>
> rting point. Feel free to ask if you have questions about a specific metric or some specific use case in mind.

@samar  
I asked this question in here, would you mind answer it here. since i always see poll request very high and Maxim mentioned it should not be the case: [Worker Setup Recommendations - #3 by maxim](https://community.temporal.io/t/worker-setup-recommendations/535/3)

---

<div class="post-metadata">

**Author:** ![Nicolas\_Gaviria](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/nicolas_gaviria/32/5525_2.png) [@Nicolas\_Gaviria](https://community.temporal.io/u/Nicolas_Gaviria)\
**Post date:** [June 28, 2024, 3:47pm UTC](https://community.temporal.io/t/guidance-on-creating-and-interpreting-grafana-dashboards/493/6 "2024-06-28T15:47:43Z")

</div>

thanks @samar for the clarifications and the suggestions,

Just one question:  
would it be possible to filter all this metrics by a workflow\_id? it would be great to see the impact of some improvements over time on a single workflow\_id

thanks in advance.
