How to observe the temporal cluster?

[maki_XIE]

I am trying to solve the observability of our temporal cluster, and I am not sure if there is a best practice to measure and monitor if the server is running well, and if the applications using it are running well.
I tried to define a couple of metrics to describe the circumstances, and am not sure if you guys can kindly offer me some documents for the following questions:

1. how to observe the following metrics, how to build them on one board, and set alerts on them

2. do these metrics make sense or if I need other metrics for observation

• If Tasks are Piling up in Queues
  • How many queues, and related workers/signals/queries to them
  • Piling up unhandled tasks in queues
  • The time of the activity event processing time
• Server-Side GRPC methods metrics
  • Total QPS and Successful QPS
  • Latency
• Failures and Risks in Workflow Executions and Activities
  • Activity Failure: If one activity fails (probably all activities of the same type) with a tolerant retry policy, how to quickly find it
  • Activity Latency: If one activity’s process time is increasing to an abnormal amount, how to observe it
  • have the same failure and latency problems for executions

[tihomir]

Couple related forum posts that might be helpful to you:

Guidance on creating and interpreting Grafana dashboards

Hey @rfwagner, This is a lengthy topic and we plan to provide more information on operationalizing Temporal clusters in the future. We recently started dashboard repo which has some basic dashboards which gives visibility into Temporal service. This is work in progress so please use this just as a reference at this point. In the future we plan to have fully supported version of grafana dashboards using PromQL. All the metric emitted by server are listed in defs.go . So if you see somethings…

What metrics does temporal expose out of box and how to consume this in prometheus?

Hi, there is not much documentation around metrics, the statd which was removed too seemed very complex. I want to to understand a)what metrics does temporal expose by default b) are the metrics namespace specific? c) can i get queue /task list specific metrics? d) how to consume them in prometheus. e) if i am to develop custom metrics what’s the best way, should those be activities in workflows or interceptors?

Grafana graphs - not all graphs are getting populated

Hi i brought up the temporal with the default helm. With this setting I see the Front end graph/dashboard getting populated, However the Canary and Temporal dashboards are always empty see attached [screencapture-localhost-18080-d-SxRYJXZMz-frontend-2020-08-08-13_55_05] [screencapture-localhost-18080-d-58u-1uWGz-canary-2020-08-08-13_54_39] [screencapture-localhost-18080-d-g0SxnLMMk-temporal-2020-08-08-13_54_14]

For SDK Metrics:
Worker tuning guide: Developer's guide - Worker performance | Temporal Documentation
list of metrics produced:
Go SDK
Java SDK
and SDK metrics docs: Temporal SDK metrics reference | Temporal Documentation

If you want to get started pretty quick take a look at this docker compose repo, it has built in prometheus with scrape points as well as SDK and Server dashboards that you can use out of box to get you started in the right direction. Hope this helps.

[tihomir]

To add a couple of server metrics grafana queries that might be useful to you:

1. Service Errors by type (change service type to the service you want):

sum(rate(service_error_with_type{service_name="frontend"}[5m])) by (error_type)

2. Service latencies by operation (again you can change service_name if needed):

histogram_quantile(0.95, sum(rate(service_latency_bucket{service_name="frontend"}[5m])) by (operation, le))

3. Persistence latencies by operation

histogram_quantile(0.95, sum(rate(persistence_latency_bucket{}[1m])) by (operation, le))

4. Sync match rate:

sum(rate(poll_success_sync{}[1m])) / sum(rate(poll_success{}[1m]))

5. Workflow lock contention:

histogram_quantile(0.99, sum(rate(cache_latency_bucket{operation="HistoryCacheGetOrCreate"}[1m])) by (le))

6. Shard Lock contention

histogram_quantile(0.99, sum(rate(lock_latency_bucket{operation="ShardInfo"}[1m])) by (le))

7. Shard movement:

sum(rate(sharditem_created_count{}[1m]))

8. Cluster restarts

sum(restarts)

9. Visibility latencies:

histogram_quantile(0.95, sum(rate(task_latency_bucket{operation=~"VisibilityTask.*", service_name="history"}[1m])) by (operation, le))

10. Resources exhausted due to limits:

sum(rate(service_errors_resource_exhausted{}[1m])) by (resource_exhausted_cause)

[Hilman_Adam]

Hi Tihomir,

3. Persistence latencies by operation
   histogram_quantile(0.95, sum(rate(persistence_latency_bucket{}[1m])) by (operation, le))

What’s the unit for this metrics? is it seconds or milliseconds?

[tihomir]

For Prometheus the bucket sizes are in seconds / fractions of seconds.