# Prometheus metrics configuration confusion

**URL:** <https://community.temporal.io/t/prometheus-metrics-configuration-confusion/6592>\
**Category:** Community Support\
**Tags:** go-sdk, helm, metrics\
**Created:** [November 29, 2022, 5:28am UTC](https://community.temporal.io/t/prometheus-metrics-configuration-confusion/6592 "2022-11-29T05:28:28Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![haojie](https://avatars.discourse-cdn.com/v4/letter/h/53a042/32.png) [@haojie](https://community.temporal.io/u/haojie)\
**Post date:** [November 29, 2022, 5:28am UTC](https://community.temporal.io/t/prometheus-metrics-configuration-confusion/6592/1 "2022-11-29T05:28:28Z")

</div>

I have a little doubt about the configuration of the prometheus metrics. From this [docs](https://docs.temporal.io/references/configuration/#metrics), I learned the ‘listenAddress’ which is for Prometheus to scrape metrics from. But from the [helm charts](https://github.com/temporalio/helm-charts) or other [discussion](https://community.temporal.io/t/configuring-external-promethrus-and-grafana/2187) , I learned that each service has exposed a metrics port, e.g default is 9090. So what is the difference between ‘listenAddress’ and service exposed port ? and How should I configure it?

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [November 29, 2022, 7:24pm UTC](https://community.temporal.io/t/prometheus-metrics-configuration-confusion/6592/2 "2022-11-29T19:24:08Z")

</div>

Is the “process-wider configuration” text in docs maybe what’s creating the confusion?  
For helm charts each temporal service pod defines its metrics format (prometheus) and the listenAddress defaulting to “0.0.0.0:9090”. Each service deployment also sets up prometheus auto discovery, see:

```auto
prometheus.io/job: {{ $.Chart.Name }}-{{ $service }}
prometheus.io/scrape: 'true'
prometheus.io/port: '9090'

```

in server-deployment.yaml for example.

So each service role can emit its metrics on a different port that can be then scraped by Prometheus. My guess is that the docs maybe assume the use of auto-setup image where all Temporal services are deployed onto same container and same process where you would use a single config.

---

<div class="post-metadata">

**Author:** ![haojie](https://avatars.discourse-cdn.com/v4/letter/h/53a042/32.png) [@haojie](https://community.temporal.io/u/haojie)\
**Post date:** [December 5, 2022, 3:43am UTC](https://community.temporal.io/t/prometheus-metrics-configuration-confusion/6592/3 "2022-12-05T03:43:08Z")

</div>

Hi，@tihomir, I check if my understanding is correct. According your description, I can configure the listen address(prometheus) of each temporal service pod in server-deployment.yaml. However, the server-configmap.yaml is used in same process where use a single config.

```auto
global:
      metrics:
        tags:
          type: frontend
        prometheus:
          timerType: histogram
          listenAddress: "0.0.0.0:9090"

```

But I am still a little confusion, will this global configuration affect the each temporal service deployment? what is the use of this global metrics configuration?

---

<div class="post-metadata">

**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\
**Post date:** [December 5, 2022, 8:44pm UTC](https://community.temporal.io/t/prometheus-metrics-configuration-confusion/6592/4 "2022-12-05T20:44:45Z")

</div>

Currently in helm charts every Temporal service is going to configure prometheus and port 9090 for metrics as you have shown. We should probably update this to be configurable from values.yaml, could you please open issue in helm-charts repo?

> what is the use of this global metrics configuration?

global is “global” really if you run all Temporal services (frontend, matching, history, worker) in the same process and have single config for all (this is default unless you specify SERVICES env var in pods to the specific role you want to start).  
If you deploy each service individually like with helm (or same in docker) each pod/container has own config so yes its not that “global” and you could set different configs for each role if you wanted.

---

<div class="post-metadata">

**Author:** ![madhu](https://avatars.discourse-cdn.com/v4/letter/m/a8b319/32.png) [@madhu](https://community.temporal.io/u/madhu)\
**Post date:** [April 4, 2023, 10:32am UTC](https://community.temporal.io/t/prometheus-metrics-configuration-confusion/6592/5 "2023-04-04T10:32:25Z")

</div>

my ops team to complained this, in fact it appers that after upgrading from 1.16 to 1.17 or beyond possibly, the Prometheus integration is broken, in spite of having

[prometheus.io/job:](http://prometheus.io/job:) {{ $.Chart.Name }}-{{ $service }}  
[prometheus.io/scrape:](http://prometheus.io/scrape:) ‘true’  
[prometheus.io/port:](http://prometheus.io/port:) ‘9090’

for each prossess at pod , i dont see 9090 running

netstat -a in each of the pod d0es not show this port at all.

---

<div class="post-metadata">

**Author:** ![madhu](https://avatars.discourse-cdn.com/v4/letter/m/a8b319/32.png) [@madhu](https://community.temporal.io/u/madhu)\
**Post date:** [April 4, 2023, 11:49am UTC](https://community.temporal.io/t/prometheus-metrics-configuration-confusion/6592/6 "2023-04-04T11:49:22Z")

</div>

> [@tihomir](#):
>
> you could set different configs for each role if you wanted

i think its now deprecated and removed,can you please confirm @tihomir ,

> [@Metrics endpoint not being enabled on upgrade](https://community.temporal.io/t/metrics-endpoint-not-being-enabled-on-upgrade/5247/6):
>
> The per service metrics endpoint is deprecated since v1.11.0 [Release v1.11.0 · temporalio/temporal · GitHub](https://github.com/temporalio/temporal/releases/tag/v1.11.0) And it is removed from code in 1.16.0. Please use the global config for metrics, see more here: [Temporal Cluster configuration reference | Temporal Documentation](https://docs.temporal.io/references/configuration/#metrics)

and

> **[Release v1.11.0 · temporalio/temporal](https://github.com/temporalio/temporal/releases/tag/v1.11.0)**
>
> IMPORTANT
> This release contains a serious issue (#1697). Do not use this release. Use v1.11.2 instead.
> Release Highlights
> Schema Changes
> This release includes minor schema changes for MySQL/Postgre...

But, if thats the case, its certainly a bug in the helms, because helms still seem to use per service metric configuration and has no defination for global values.

---

<div class="post-metadata">

**Author:** ![madhu](https://avatars.discourse-cdn.com/v4/letter/m/a8b319/32.png) [@madhu](https://community.temporal.io/u/madhu)\
**Post date:** [April 12, 2023, 4:41am UTC](https://community.temporal.io/t/prometheus-metrics-configuration-confusion/6592/7 "2023-04-12T04:41:29Z")

</div>

Looks like my helms were not upto date, changing/upgrading the helm charts helped.