Properly plot counters with Prometheus in Grafana

Community Support
,
1

Hi,

I have been trying to figure out how to properly plot some of the following counters to show some breakdown on workflow_type or activity_type:

  • temporal_workflow_completed_total
  • temporal_workflow_failed_total
  • temporal_request_total (for plotting all successful activity)
  • temporal_activity_execution_failed_total

it’s been unfruitful so far. Looks like all these metrics are not emitted as zeros and I am not able to plot them accurately. How do people handle these?

Thanks!

2

For these counter metrics you could do for example:

sum(rate(temporal_workflow_completed_total[5m]))

Yes, SDK does not emit these counters for 0, meaning for example workflow_completed_total will be emitted once a workflow execution completes (counter increased by 1).

3

i tried sum(rate(...)) but it doesn’t really show accurate value, same as increase(...)

4

What are you trying to display? Why is data not accurate?
Rate gives you average rate over time. You can always look at the raw metric value in Prometheus for example.

5

I am trying to display:

  • number of succeed/failed workflows by types
  • number of succeed/failed activity by types

Using rates or increase drops to zeros during low-volume.