# Setting PROMETHEUS\_ENDPOINT in docker-compose results in warning logs

**URL:** <https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321>\
**Category:** Community Support\
**Tags:** prometheus, metrics\
**Created:** [July 30, 2020, 5:50pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321 "2020-07-30T17:50:29Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jocelyn](https://avatars.discourse-cdn.com/v4/letter/j/90db22/32.png) [@Jocelyn](https://community.temporal.io/u/Jocelyn)\
**Post date:** [July 30, 2020, 5:50pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/1 "2020-07-30T17:50:29Z")

</div>

Based on [https://github.com/temporalio/temporal/issues/319](https://github.com/temporalio/temporal/issues/319) , I’ve added `PROMETHEUS_ENDPOINT: '0.0.0.0:9090'` to the Temporal docker-compose file. Launching results in:

```auto
{"level":"warn","ts":"2020-07-30T17:38:51.115Z","msg":"error in prometheus reporter","error":"a previously registered descriptor with the same fully-qualified name as Desc{fqName: \"history_size\", help: \"history_size histogram\", constLabels: {}, variableLabels: [operation stats_type namespace]} has different label names or a different help string","logging-call-at":"metrics.go:135"}
{"level":"warn","ts":"2020-07-30T17:38:51.115Z","msg":"error in prometheus reporter","error":"a previously registered descriptor with the same fully-qualified name as Desc{fqName: \"history_size\", help: \"history_size histogram\", constLabels: {}, variableLabels: [stats_type namespace operation]} has different label names or a different help string","logging-call-at":"metrics.go:135"}

```

Is this expected behavior from Temporal?

Also, the helm example shows additional metrics tags, should those be added by default in docker-compose setups?

Thank you

---

<div class="post-metadata">

**Author:** ![markmark](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/markmark/32/88_2.png) [@markmark](https://community.temporal.io/u/markmark)\
**Post date:** [August 1, 2020, 12:10am UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/3 "2020-08-01T00:10:23Z")

</div>

Hey Jocelyn, thank you for trying this out, and for your question!

We haven’t tested prometheus docker-compose setup yet.

Without having looked into this in depth, I _think_ you should be able to provide a prometheus configuration file, that would tell prometheus which endpoints to scrape (e. g. ‘temporal:9090’). Here is the relevant section in prometheus docs:

[https://prometheus.io/docs/introduction/first\_steps/#configuring-prometheus](https://prometheus.io/docs/introduction/first_steps/#configuring-prometheus)

```auto
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

```

Separate from that, the error message you included makes me wonder if we have a bug around using prometheus from inside our docker-compose, in a setup where all service roles are collapsed into one.

I will create a bug report for us to investigate and, if this is a real problem, track that. (you are welcome to create that, too if you prefer to have it under your name). [https://github.com/temporalio/temporal/issues/new/choose](https://github.com/temporalio/temporal/issues/new/choose)

(If you happen to have a docker-compose file with a repro that you are able share with us, it would help us reproducing the problem).)

Thank you!  
Mark.

---

<div class="post-metadata">

**Author:** ![Jocelyn](https://avatars.discourse-cdn.com/v4/letter/j/90db22/32.png) [@Jocelyn](https://community.temporal.io/u/Jocelyn)\
**Post date:** [August 6, 2020, 9:04pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/4 "2020-08-06T21:04:17Z")

</div>

Hi Mark & thank you for your reply. So I did configure Prometheus with a config similar to your suggestion and also configured Grafana with the Temporal dashboards, that part seems to be working fine and I can see some graph data such as memory allocation or goroutines, I’m just not sure all the metrics are working properly.

I’ve just tested again with the latest release of Temporal, and still getting the same warning logs. Here is the docker-compose file to reproduce:

```auto
version: '3.7'
services:
  postgres:
    image: postgres:12.3
    restart: unless-stopped
    ports:
      - '5432:5432'
    environment:
      POSTGRES_USER: temporal
      POSTGRES_PASSWORD: placeholder
    volumes:
      - postgres-data-volume:/var/lib/postgresql/data
  temporal-server:
    image: temporalio/server:0.28.0
    restart: unless-stopped
    ports:
      - '7233:7233'
    environment:
      AUTO_SETUP: 'true'
      DB: 'postgres'
      DB_PORT: '5432'
      POSTGRES_USER: 'temporal'
      POSTGRES_PWD: 'placeholder'
      POSTGRES_SEEDS: 'postgres'
      DYNAMIC_CONFIG_FILE_PATH: 'config/dynamicconfig/development.yaml'
      PROMETHEUS_ENDPOINT: '0.0.0.0:9090'
    depends_on:
      - postgres
  temporal-web:
    image: temporalio/web:0.28.0
    restart: unless-stopped
    ports:
      - '8088:8088'
    environment:
      TEMPORAL_GRPC_ENDPOINT: temporal-server:7233
    depends_on:
      - temporal-server
volumes:
  postgres-data-volume:

```

Running this results in a few warnings related to Prometheus:

```auto
"level":"warn","ts":"2020-08-06T20:10:45.028Z","msg":"error in prometheus reporter","error":"listen tcp 0.0.0.0:9090: bind: address already in use","logging-call-at":"metrics.go:135"}
"level":"warn","ts":"2020-08-06T20:10:45.087Z","msg":"error in prometheus reporter","error":"listen tcp 0.0.0.0:9090: bind: address already in use","logging-call-at":"metrics.go:135"}
"level":"warn","ts":"2020-08-06T20:10:45.192Z","msg":"error in prometheus reporter","error":"listen tcp 0.0.0.0:9090: bind: address already in use","logging-call-at":"metrics.go:135"}
"level":"warn","ts":"2020-08-06T20:10:45.481Z","msg":"error in prometheus reporter","error":"a previously registered descriptor with the same fully-qualified name as Desc{fqName: \"forwarded\", help: \"forwarded counter\", constLabels: {}, variableLabels: [operation]} has different label names or a different help string","logging-call-at":"metrics.go:135"}
"level":"warn","ts":"2020-08-06T20:10:45.534Z","msg":"error in prometheus reporter","error":"a previously registered descriptor with the same fully-qualified name as Desc{fqName: \"forwarded\", help: \"forwarded counter\", constLabels: {}, variableLabels: [operation]} has different label names or a different help string","logging-call-at":"metrics.go:135"}
"level":"warn","ts":"2020-08-06T20:10:49.293Z","msg":"error in prometheus reporter","error":"a previously registered descriptor with the same fully-qualified name as Desc{fqName: \"history_size\", help: \"history_size histogram\", constLabels: {}, variableLabels: [namespace stats_type operation]} has different label names or a different help string","logging-call-at":"metrics.go:135"}
"level":"warn","ts":"2020-08-06T20:10:49.293Z","msg":"error in prometheus reporter","error":"a previously registered descriptor with the same fully-qualified name as Desc{fqName: \"history_size\", help: \"history_size histogram\", constLabels: {}, variableLabels: [stats_type operation namespace]} has different label names or a different help string","logging-call-at":"metrics.go:135"}

```

Which points to [https://github.com/temporalio/temporal/blob/ca33ba46a6352fc17ff82dda1a0e4eda228a68ec/common/service/config/metrics.go#L135](https://github.com/temporalio/temporal/blob/ca33ba46a6352fc17ff82dda1a0e4eda228a68ec/common/service/config/metrics.go#L135)

Adding just the tags doesn’t seem to fix the issue, I tried it by mounting a volume of a modified `config_template.yaml` with tags types frontend, matching, history and worker:

```auto
version: '3.7'
services:
  postgres:
    image: postgres:12.3
    restart: unless-stopped
    ports:
      - '5432:5432'
    environment:
      POSTGRES_USER: temporal
      POSTGRES_PASSWORD: placeholder
    volumes:
      - postgres-data-volume:/var/lib/postgresql/data
  temporal-server:
    image: temporalio/server:0.28.0
    restart: unless-stopped
    ports:
      - '7233:7233'
    environment:
      AUTO_SETUP: 'true'
      DB: 'postgres'
      DB_PORT: '5432'
      POSTGRES_USER: 'temporal'
      POSTGRES_PWD: 'placeholder'
      POSTGRES_SEEDS: 'postgres'
      DYNAMIC_CONFIG_FILE_PATH: 'config/dynamicconfig/development.yaml'
      PROMETHEUS_ENDPOINT: '0.0.0.0:9090'
    volumes:
      - ./config/config_template.yaml:/etc/temporal/config/config_template.yaml:ro
    depends_on:
      - postgres
  temporal-web:
    image: temporalio/web:0.28.0
    restart: unless-stopped
    ports:
      - '8088:8088'
    environment:
      TEMPORAL_GRPC_ENDPOINT: temporal-server:7233
    depends_on:
      - temporal-server
volumes:
  postgres-data-volume:

```

FYI, I haven’t created a bug report for now. Thank you,  
Jocelyn.

---

<div class="post-metadata">

**Author:** ![Jocelyn](https://avatars.discourse-cdn.com/v4/letter/j/90db22/32.png) [@Jocelyn](https://community.temporal.io/u/Jocelyn)\
**Post date:** [August 12, 2020, 7:02pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/5 "2020-08-12T19:02:56Z")

</div>

Opened github ticket [https://github.com/temporalio/temporal/issues/673](https://github.com/temporalio/temporal/issues/673)

---

<div class="post-metadata">

**Author:** ![Pedro\_Almeida](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/pedro_almeida/32/35_2.png) [@Pedro\_Almeida](https://community.temporal.io/u/Pedro_Almeida)\
**Post date:** [November 11, 2020, 9:44am UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/6 "2020-11-11T09:44:21Z")

</div>

Hello,

Is this issue still present in 1.2.1 release? Could be wrong, but believe the ticket was merged in master but it is not present in 1.2.1 release?

When solved, and by looking into the fix, adding this into docker-compose will be enough to use it?

- “PROMETHEUS\_ENDPOINT=0.0.0.0:9090”

Thanks!

---

<div class="post-metadata">

**Author:** ![alex](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/alex/32/15_2.png) [@alex](https://community.temporal.io/u/alex)\
**Post date:** [November 11, 2020, 11:09pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/7 "2020-11-11T23:09:34Z")

</div>

Hi Pedro.  
I’ve just published new [1.3.0 release](https://github.com/temporalio/temporal/releases/tag/v1.3.0) and this thing is fixed there. Please note, you will need to remove `metrics` section from every service and add one to `global` section in your config file.

---

<div class="post-metadata">

**Author:** ![Pedro\_Almeida](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/pedro_almeida/32/35_2.png) [@Pedro\_Almeida](https://community.temporal.io/u/Pedro_Almeida)\
**Post date:** [November 12, 2020, 4:31pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/8 "2020-11-12T16:31:24Z")

</div>

Thank you Alex.

Using the 1.3.0 image we can scrap the metrics using Prometheus. But we noticed that most of the grafana Dashboards available shows empty info, so we began searching for issues.

One thing we noticed is the warnings in the log :

> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:21.760Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "service\_latency", help: "service\_latency histogram", constLabels: {}, variableLabels: [operation]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}  
> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:21.761Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "service\_latency", help: "service\_latency histogram", constLabels: {}, variableLabels: [operation]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}  
> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:21.761Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "service\_requests", help: "service\_requests counter", constLabels: {}, variableLabels: [operation]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}  
> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:21.761Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "service\_requests", help: "service\_requests counter", constLabels: {}, variableLabels: [operation]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}  
> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:21.802Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "forwarded", help: "forwarded counter", constLabels: {}, variableLabels: [operation]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}  
> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:21.802Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "forwarded", help: "forwarded counter", constLabels: {}, variableLabels: [operation]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}  
> temporal\_1 | {“level”:“info”,“ts”:“2020-11-12T16:02:25.600Z”,“msg”:“Get dynamic config”,“name”:“limit.blobSize.error”,“value”:“2097152”,“default-value”:“2097152”,“logging-call-at”:“config.go:79”}  
> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:25.605Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "service\_requests", help: "service\_requests counter", constLabels: {}, variableLabels: [operation]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}  
> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:25.606Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "service\_latency", help: "service\_latency histogram", constLabels: {}, variableLabels: [operation]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}  
> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:25.614Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "history\_size", help: "history\_size histogram", constLabels: {}, variableLabels: [operation stats\_type]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}  
> temporal\_1 | {“level”:“warn”,“ts”:“2020-11-12T16:02:25.617Z”,“msg”:“error in prometheus reporter”,“error”:“a previously registered descriptor with the same fully-qualified name as Desc{fqName: "service\_errors\_execution\_already\_started", help: "service\_errors\_execution\_already\_started counter", constLabels: {}, variableLabels: [operation namespace]} has different label names or a different help string”,“logging-call-at”:“metrics.go:135”}

Note that we are not using any development\_config.yaml while using docker-compose.

Looking into the grafana Dashboard, we noticed that most of graphs that are showing empty results are looking into metrics not available in Prometheus. Example :

temporal\_activity\_execution\_latency\_bucket  
temporal\_workflow\_completed

Metrics that we do have info :

activity\_end\_to\_end\_latency\_bucket  
persistence\_latency\_bucket

Should the grafana dashboards available here [https://github.com/temporalio/dashboards/tree/master/dashboards](https://github.com/temporalio/dashboards/tree/master/dashboards) be used in this setup?

I can send you a file with all metrics being scraped in the Prometheus endpoint in temporal server ( can’t attach files here i think? ). Let me know if it helps.

docker-compose.yml

> version: ‘3’  
> services:  
> mysql:  
> image: mysql:5.7  
> ports:  
> - “3306:3306”  
> environment:  
> - “MYSQL\_ROOT\_PASSWORD=root”  
> temporal:  
> image: temporalio/auto-setup:${SERVER\_TAG:-1.3.0}  
> ports:  
> - “7233:7233”  
> - “9100:9100”  
> volumes:  
> - ${DYNAMIC\_CONFIG\_DIR:-…/config/dynamicconfig}:/etc/temporal/config/dynamicconfig  
> environment:  
> - “DB=mysql”  
> - “MYSQL\_USER=root”  
> - “MYSQL\_PWD=root”  
> - “MYSQL\_SEEDS=mysql”  
> - “DYNAMIC\_CONFIG\_FILE\_PATH=config/dynamicconfig/development.yaml”  
> - “PROMETHEUS\_ENDPOINT=0.0.0.0:9100”  
> depends\_on:  
> - mysql  
> temporal-admin-tools:  
> image: temporalio/admin-tools:${SERVER\_TAG:-1.3.0}  
> stdin\_open: true  
> tty: true  
> environment:  
> - “TEMPORAL\_CLI\_ADDRESS=temporal:7233”  
> depends\_on:  
> - temporal  
> temporal-web:  
> image: temporalio/web:${WEB\_TAG:-1.1.1}  
> environment:  
> - “TEMPORAL\_GRPC\_ENDPOINT=temporal:7233”  
> - “TEMPORAL\_PERMIT\_WRITE\_API=true”  
> ports:  
> - “8088:8088”  
> depends\_on:  
> - temporal

Thanks again for all the help.

---

<div class="post-metadata">

**Author:** ![alex](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/alex/32/15_2.png) [@alex](https://community.temporal.io/u/alex)\
**Post date:** [November 12, 2020, 5:39pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/9 "2020-11-12T17:39:57Z")

</div>

Yes, we are aware of these errors with different label names and working on it right now. Hopefully it will be fixed in next release.

As for dashboards, @samar should have more context.

---

<div class="post-metadata">

**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\
**Post date:** [November 17, 2020, 11:12pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/10 "2020-11-17T23:12:27Z")

</div>

The two examples you shared for metric with empty results are emitted from sdk. Here is the [definition](https://github.com/temporalio/sdk-go/blob/master/internal/common/metrics/constants.go#L48) for `temporal_activity_execution_latency`. Are you sure your workflow/activity workers are correctly configured for metric scrapping?

Dashboards we shared at [https://github.com/temporalio/dashboards/blob/master/dashboards](https://github.com/temporalio/dashboards/blob/master/dashboards) are still work in progress, so at this point you should use those as examples to build your own dashboards. We are still iterating over the monitoring experience for Temporal and soon will create supported dashboards which just works out of the box for others to import.

[SDK dashboards](https://github.com/temporalio/dashboards/blob/master/dashboards/sdk.json) relies on all the metric emitted from workflow/activity workers. So make sure they are configured correctly otherwise this dashboard won’t have any data to show as it relies on metric emitted from client SDK.

---

<div class="post-metadata">

**Author:** ![Pedro\_Almeida](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/pedro_almeida/32/35_2.png) [@Pedro\_Almeida](https://community.temporal.io/u/Pedro_Almeida)\
**Post date:** [November 18, 2020, 12:03pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/11 "2020-11-18T12:03:15Z")

</div>

Hello @samar,

Thanks for the feedback, although i’m not sure what config for the workers i need to check.

Is it at the start of the workers or the server? Any example / docs to check?

Thanks again for the help

---

<div class="post-metadata">

**Author:** ![samar](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/samar/32/12_2.png) [@samar](https://community.temporal.io/u/samar)\
**Post date:** [November 19, 2020, 6:46pm UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/12 "2020-11-19T18:46:34Z")

</div>

You need to configure `MetricsScope` when initializing the client used by worker and then setup prometheus to scrape metric from your workers. Here is an [example](https://github.com/temporalio/samples-go/blob/master/metrics/worker/main.go#L19) which shows how to configure `MetricsScope` when initializing temporal client.

---

<div class="post-metadata">

**Author:** ![Pedro\_Almeida](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/pedro_almeida/32/35_2.png) [@Pedro\_Almeida](https://community.temporal.io/u/Pedro_Almeida)\
**Post date:** [November 23, 2020, 11:32am UTC](https://community.temporal.io/t/setting-prometheus-endpoint-in-docker-compose-results-in-warning-logs/321/13 "2020-11-23T11:32:43Z")

</div>

Thanks @samar,

I was able to expose the metrics from worker this way.
