Merging Prometheus Metrics from Temporal SDK and Local Services in a Python Application

Community Support
,
1

Hi everyone,

I’m currently working on a Python application that uses Temporal with a worker and has its own metrics for some external services and internal functions. Additionally, the application also includes metrics from the Temporal SDK.

I’m looking for a solution to merge these metrics into a single Prometheus registry.

What I am doing now is: each time my Prometheus server scrapes the /metrics endpoint, I consume the endpoint where Prometheus exposes its metrics and merge that data with the data my local Prometheus server has. While this approach works, I don’t like the fact that I’m doing this merge operation every time Prometheus scrapes the metrics.

I have been trying to extract the Prometheus Registry from the Temporal SDK to merge it with my local registry, but I can’t find a way to access it. The TelemetryConfig in the Python SDK doesn’t seem to provide any method for accessing the Prometheus configuration.

Does anyone know if there’s a way to access the Prometheus configuration or registry from the Temporal SDK? Any help or pointers would be greatly appreciated!

Thanks in advance!

2

Since the Prometheus metrics are emitted by Rust, combining by default may be difficult.

You can host them at different ports/endpoints and have your Prometheus server scrape both.

Otherwise, if you must combine, you’ll have to use the advanced MetricBuffer concept. This advanced approach does not have a sample, but basically instead of setting Prometheus config on the runtime telemetry options, you set a temporalio.runtime.MetricBuffer. We then populate this buffer with metrics which you can periodically pull off and put into your metric library of choice (in this case a Python Prometheus library).

1 Like
3

Thanks Chad!

After posting this questions and trying to debug a bit more in depth into temporal I realized my task might be difficult.
I read about prometheus scrapping and yes, it’s very simple to scrape them both, so I’m working on upgrading my helm charts in order to have these pieces in place. Thanks for the quick response!