# Temporal cluster not emitting certain metrics

**URL:** <https://community.temporal.io/t/temporal-cluster-not-emitting-certain-metrics/10870>\
**Category:** Server Deployment\
**Tags:** prometheus, metrics\
**Created:** [February 1, 2024, 1:08am UTC](https://community.temporal.io/t/temporal-cluster-not-emitting-certain-metrics/10870 "2024-02-01T01:08:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![adithyasai](https://avatars.discourse-cdn.com/v4/letter/a/3be4f8/32.png) [@adithyasai](https://community.temporal.io/u/adithyasai)\
**Post date:** [February 1, 2024, 1:08am UTC](https://community.temporal.io/t/temporal-cluster-not-emitting-certain-metrics/10870/1 "2024-02-01T01:08:51Z")

</div>

I am running a local temporal cluster 1.22 deployed in kubernetes through helm.

Port-forwarded the web ui port and the frontend port to the host.

```auto
kubectl -n temporal port-forward services/temporal-web 8080:8080 & kubectl -n temporal port-forward services/temporal-frontend-headless 7233:7233

```

Created a default namespace

```auto
tctl -ns default n register

```

Then I ran the temporal go sample application for metrics - [samples-go/metrics at main · temporalio/samples-go · GitHub](https://github.com/temporalio/samples-go/tree/main/metrics)

```auto
go run metrics/worker/main.go (running in one terminal window)

go run metrics/starter/main.go (another terminal window)

2024/01/31 13:46:19 INFO No logger configured for temporal client. Created default one.
2024/01/31 13:46:19 Started workflow. WorkflowID metrics_workflowID RunID 31698d47-09bd-4b44-b499-47c5197ad40b
2024/01/31 13:46:21 Check metrics at http://localhost:9090/metrics

```

I can see the worker sdk metrics at [http://localhost:9090/metrics](http://localhost:9090/metrics).

But I am looking for `activity_end_to_end_latency` metric which is emitted in the temporal cluster through the history service. When I port-forwarded the history service’s 9090 port and looked at its metric, I couldn’t find `activity_end_to_end_latency` for the default namespace. In fact I couldn’t find any metrics for the default namespace. All I see are metrics for the temporal-system namespace.

```auto
kubectl -n temporal port-forward services/temporal-history-headless 9091:9090

```

Here is my kube services list

```auto
kubectl get services -n temporal
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
postgresql ClusterIP 10.96.54.24 <none> 5432/TCP 18h
postgresql-hl ClusterIP None <none> 5432/TCP 18h
temporal-admintools ClusterIP 10.96.133.126 <none> 22/TCP 18h
temporal-frontend ClusterIP 10.96.120.135 <none> 7233/TCP 18h
temporal-frontend-headless ClusterIP None <none> 7233/TCP,9090/TCP 18h
temporal-history-headless ClusterIP None <none> 7234/TCP,9090/TCP 18h
temporal-kube-state-metrics ClusterIP 10.96.231.19 <none> 8080/TCP 18h
temporal-matching-headless ClusterIP None <none> 7235/TCP,9090/TCP 18h
temporal-prometheus-alertmanager ClusterIP 10.96.223.245 <none> 80/TCP 18h
temporal-prometheus-pushgateway ClusterIP 10.96.59.153 <none> 9091/TCP 18h
temporal-prometheus-server ClusterIP 10.96.114.139 <none> 80/TCP 18h
temporal-web ClusterIP 10.96.45.211 <none> 8080/TCP 18h
temporal-worker-headless ClusterIP None <none> 7239/TCP,9090/TCP 18h

```

Does anyone know what I am missing?

---

<div class="post-metadata">

**Author:** ![Rob\_Temporal](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/rob_temporal/32/1079_2.png) [@Rob\_Temporal](https://community.temporal.io/u/Rob_Temporal)\
**Post date:** [February 1, 2024, 10:23am UTC](https://community.temporal.io/t/temporal-cluster-not-emitting-certain-metrics/10870/2 "2024-02-01T10:23:48Z")

</div>

`activity_endtoend_latency` is no longer emitted, it’s been replaced: [Activity success e2e latency metrics by cretz · Pull Request #575 · temporalio/sdk-go · GitHub](https://github.com/temporalio/sdk-go/pull/575)

It’s an SDK metric, not a server metric.

---

<div class="post-metadata">

**Author:** ![adithyasai](https://avatars.discourse-cdn.com/v4/letter/a/3be4f8/32.png) [@adithyasai](https://community.temporal.io/u/adithyasai)\
**Post date:** [February 1, 2024, 8:09pm UTC](https://community.temporal.io/t/temporal-cluster-not-emitting-certain-metrics/10870/3 "2024-02-01T20:09:28Z")

</div>

I am able to see the sdk metrics. But there are some server metrics that should be emitted but I don’t see them.

For example, these are some server metrics that are emitted as a part of the history service.

> <https://github.com/temporalio/temporal/blob/893a8f3d51567f889dd6e7f973196f7d94bb1cbe/service/history/api/respondactivitytaskcompleted/api.go#L122-L131>

> <https://github.com/temporalio/temporal/blob/aec2c100b6fae91a53f0f4ea3befe8ea683f0d9c/service/history/workflow/metrics.go#L113-L127>

But I don’t see them when I look at the history service metrics. All I see are the metrics for temporal\_system namespace.