# What metrics does temporal expose out of box and how to consume this in prometheus?

**URL:** <https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346>\
**Category:** Community Support\
**Tags:** prometheus, metrics\
**Created:** [August 3, 2020, 11:17am UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346 "2020-08-03T11:17:13Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![madhu](https://avatars.discourse-cdn.com/v4/letter/m/a8b319/32.png) [@madhu](https://community.temporal.io/u/madhu)\
**Post date:** [August 3, 2020, 11:17am UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/1 "2020-08-03T11:17:13Z")

</div>

Hi, there is not much documentation around metrics, the statd which was removed too seemed very complex.

I want to to understand  
a)what metrics does temporal expose by default  
b) are the metrics namespace specific?  
c) can i get queue /task list specific metrics?  
d) how to consume them in prometheus.  
e) if i am to develop custom metrics what’s the best way, should those be activities in workflows or interceptors?

---

<div class="post-metadata">

**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\
**Post date:** [August 4, 2020, 5:59pm UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/3 "2020-08-04T17:59:56Z")

</div>

Hey @madhu,  
Yes this is an area where we lack any public documentation at the moment but this is definitely something which is pretty high up in priority among the list of tasks which we plan to address soon.

Temporal server reports a wide variety of metric to help operators get visibility into cluster and setup alerts. We use [tally](https://github.com/uber-go/tally) for reporting metric from the application and it supports multiple backends like prometheus, statsd, and M3db. We generally recommend to run Temporal with Prometheus backend and plan to provide dashboards using promQL to the community very soon. Here is a [dashboard repo](https://github.com/temporalio/dashboards) which we started recently. This is something we are iterating over pretty heavily at the moment and not ready for production use at the moment, but you can definitely use this as a reference to build your own dashboards.

All the metric emitted by server are listed in [defs.go](https://github.com/temporalio/temporal/blob/master/common/metrics/defs.go). So if you see somethings are missing in the dashboards then you can use the [defs.go](https://github.com/temporalio/temporal/blob/master/common/metrics/defs.go) as a reference.

We have provided a [development config](https://github.com/temporalio/temporal/blob/master/config/development_prometheus.yaml) which shows how to run the server using prometheus as the back end. You can also checkout our [helm chart](https://github.com/temporalio/helm-charts) which also has a section on how to run Temporal with prometheus as the metric backend.

---

<div class="post-metadata">

**Author:** ![madhu](https://avatars.discourse-cdn.com/v4/letter/m/a8b319/32.png) [@madhu](https://community.temporal.io/u/madhu)\
**Post date:** [August 4, 2020, 6:40pm UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/4 "2020-08-04T18:40:20Z")

</div>

Thanks much @samar i will check these links and get back. really appreciate .

---

<div class="post-metadata">

**Author:** ![Sandeep\_Paul](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sandeep_paul/32/145_2.png) [@Sandeep\_Paul](https://community.temporal.io/u/Sandeep_Paul)\
**Post date:** [September 7, 2020, 4:05pm UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/5 "2020-09-07T16:05:47Z")

</div>

Hi @samar ,  
“with the context of .29 helm charts”  
in the config we have  
datasources:  
- name: TemporalMetrics  
type: prometheus  
url: http://{{ .Release.Name }}-prometheus-server  
access: proxy  
isDefault: true  
“url: http://{{ .Release.Name }}-prometheus-server” what does this URL stand for and being resolved.

why am I asking this is, I have installed the temporal helm charts tags .29 in X namespace and we have prometheus operator and grafana in another Y namespace.  
Im not getting the datasources in the grafana UI.

please let me know if Iam missing something or how should I proceed to use existing prometheus operator.

regards  
Sandeep

---

<div class="post-metadata">

**Author:** ![madhu](https://avatars.discourse-cdn.com/v4/letter/m/a8b319/32.png) [@madhu](https://community.temporal.io/u/madhu)\
**Post date:** [September 7, 2020, 6:21pm UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/9 "2020-09-07T18:21:20Z")

</div>

basically your question is how to configure “bring your own prometheus”  
i am not too sure, but each component/Role has a promethus section right, will you not be able to provide the prmoethus endpoint there?

```
  frontend:
    # replicaCount: 1
    service:
      type: ClusterIP
      port: 7233
    metrics:
      annotations:
        enabled: true
      serviceMonitor: {}
       prometheus: {//HERE GOES YOUR STUFF??}
 
      # enabled: false
```

---

<div class="post-metadata">

**Author:** ![derek](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/derek/32/245_2.png) [@derek](https://community.temporal.io/u/derek)\
**Post date:** [September 9, 2020, 2:56am UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/10 "2020-09-09T02:56:08Z")

</div>

Hi @Sandeep_Paul - those configs that you are talking about are for connecting the grafana deployed by our helm chart to the prometheus deployed by our helm chart.

If you want to bring your own prometheus and your own grafana you can make use of our dashboards but you’ll use your existing prometheus as datasource and configure your prometheus to scrape metrics in the namespace you have temporal deployed to.

Depending on how you have that set up, things to check include: prometheus is configured to scrape metrics in your temporal namespace, your RBAC settings allow prometheus to scrape metrics in your temporal namespace, and that any annotations you need on temporal deployments are set.

With prometheus operator there’s a note in our default values file about setting serviceMonitor to enabled which should create the required resources for you.

when using your own prom/grafana, install the helm chart with prometheus.enabled=false and grafana.enabled=false as well.

---

<div class="post-metadata">

**Author:** ![sonyantony](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sonyantony/32/1209_2.png) [@sonyantony](https://community.temporal.io/u/sonyantony)\
**Post date:** [September 29, 2021, 11:20pm UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/11 "2021-09-29T23:20:28Z")

</div>

Is Prometheus scraping points enabled in both docker-compose and helm charts by default on the Temporal server ?  
If yes, what is the URL for scraping ?  
If No, how can we enable it ?

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [September 30, 2021, 3:18pm UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/12 "2021-09-30T15:18:23Z")

</div>

Prometheus is not set up by default in the docker compose files.  
To add:

1. Enable it in temporal-\>environment by adding:

`- PROMETHEUS_ENDPOINT=0.0.0.0:8000`

1. Add port 8000 in temporal-\>ports by adding:

-` 8000:8000`

1. Check the metrics on:

`http://localhost:8000/metrics`

---

<div class="post-metadata">

**Author:** ![sonyantony](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/sonyantony/32/1209_2.png) [@sonyantony](https://community.temporal.io/u/sonyantony)\
**Post date:** [October 3, 2021, 4:48am UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/13 "2021-10-03T04:48:49Z")

</div>

Thank You Tihomir. It worked. !!  
( I had tried this using a port 9090 before I posted teh question here. Somehow - I need to check why - it did not work )

---

<div class="post-metadata">

**Author:** ![whitecrow](https://avatars.discourse-cdn.com/v4/letter/w/50afbb/32.png) [@whitecrow](https://community.temporal.io/u/whitecrow)\
**Post date:** [August 5, 2022, 2:02am UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/14 "2022-08-05T02:02:21Z")

</div>

Hi @samar ,  
Is there any update about the metric documentation?

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [August 5, 2022, 2:58am UTC](https://community.temporal.io/t/what-metrics-does-temporal-expose-out-of-box-and-how-to-consume-this-in-prometheus/346/16 "2022-08-05T02:58:11Z")

</div>

You can see SDK metrics docs [here](https://docs.temporal.io/references/sdk-metrics/). Server metrics docs are afaik still in the works and should be available in near future.