Prometheus metrics only have one namespace: temporal_system inside metrics

Bellow is the configuration part of metrics setting.

...
global:
  membership:
    maxJoinDuration: 30s
    broadcastAddress: "192.168.127.18"
  pprof:
    port: 7936
  metrics:
    prometheus:
       timerType: "histogram"
       listenAddress: "192.168.127.18:8000"

services:
  frontend:
    rpc:
      grpcPort: 7233
      membershipPort: 6933
      bindOnIP: "192.168.127.18"
...

problem is the metrics I query from prometheus, only have temporal_system namespace, like bellow:


temporal_workflow_completed{activity_type="none", client_name="temporal_go", group="nta-temporal-cluster-1", instance="collector-001:8000", job="file-sd", namespace="temporal_system", task_queue="none", type="temporal", worker_type="none", workflow_type="temporal_sys_tq_scanner_workflow"}

Can you see what all namespaces you have with

tctl namespace list

➜ tctl namespace list | grep Name
Name: sre
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: fortest
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: temporal-system
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: firepipe
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: samples-namespace
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: te
NamespaceData: map[string]string(nil)
ActiveClusterName: active
Name: it
NamespaceData: map[string]string(nil)
ActiveClusterName: active

Here is one without grep

Name: it
Id: e23f9bc1-3230-4e91-ad12-8380da5dfa9a
Description: 
OwnerEmail: 
NamespaceData: map[string]string(nil)
State: Registered
Retention: 72h0m0s
ActiveClusterName: active
Clusters: active
HistoryArchivalState: Disabled
VisibilityArchivalState: Disabled
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+

Thanks, do you enable metrics for each service or just in “global”. Each service individually should enable its metrics (expose the listenAddress port).

ic, only in global. let me add this configuration inside each service.