Prometheus metrics only have one namespace: temporal_system inside metrics

Community Support
,
1

Bellow is the configuration part of metrics setting.

...
global:
  membership:
    maxJoinDuration: 30s
    broadcastAddress: "192.168.127.18"
  pprof:
    port: 7936
  metrics:
    prometheus:
       timerType: "histogram"
       listenAddress: "192.168.127.18:8000"

services:
  frontend:
    rpc:
      grpcPort: 7233
      membershipPort: 6933
      bindOnIP: "192.168.127.18"
...

problem is the metrics I query from prometheus, only have temporal_system namespace, like bellow:


temporal_workflow_completed{activity_type="none", client_name="temporal_go", group="nta-temporal-cluster-1", instance="collector-001:8000", job="file-sd", namespace="temporal_system", task_queue="none", type="temporal", worker_type="none", workflow_type="temporal_sys_tq_scanner_workflow"}
2

Can you see what all namespaces you have with

tctl namespace list

3 
➜ tctl namespace list | grep Name
Name: sre
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: fortest
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: temporal-system
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: firepipe
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: samples-namespace
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: te
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: it
NamespaceData: map[string]string(nil)
ActiveClusterName: active

Here is one without grep

Name: it
Id: e23f9bc1-3230-4e91-ad12-8380da5dfa9a
Description: 
OwnerEmail: 
NamespaceData: map[string]string(nil)
State: Registered
Retention: 72h0m0s
ActiveClusterName: active
Clusters: active
HistoryArchivalState: Disabled
VisibilityArchivalState: Disabled
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+
4

Thanks, do you enable metrics for each service or just in “global”. Each service individually should enable its metrics (expose the listenAddress port).

5

ic, only in global. let me add this configuration inside each service.

6

how do i enable “global” for all metrics to be emitted in other namespaces in Kubernetes ? My deployment is using helm chart