Emitting custom counter metrics from activity

Community Support
, ,
1

Hello :wave:

We’d like to emit counter metrics either from activities or from service methods deep in the call stack, but initiated from activities.

The problem we’re facing is that we, naturally, have multiple workers publishing different counter metrics for the same timeseries.

A simple concrete example… we have a timeseries my_counter{} and two workers WorkerA and WorkerB that can execute activity MyActivity, in which my_counter{} is incremented by 1. Then the following sequence of invocations occur:

  1. WorkerA executes MyActivity and reports my_counter{} = 1.
  2. WorkerA executes MyActivity and reports my_counter{} = 2.
  3. WorkerA executes MyActivity and reports my_counter{} = 3.
  4. WorkerB executes MyActivity and reports my_counter{} = 1.

I think this confuses Prometheus, which is expecting a monotonically increasing counter for a given timeseries, and ideally, we’d want WorkerB to report my_counter{} = 4.

How have folks typically solved for this?

Thanks!

Albert

2

My guess is we may need to make some opentelemetry config changes to attach the pod name as a label to the metric timeseries. :thinking: