Tags are appearing in the metric name when using statsd configuration

Support
#1

Hi,

I’m running into an issue setting up a datadog dashboard for temporal due to the way the metric names are formatted, for example this is a metric produced now by the frontend service:

temporal.service_latency.namespace.default.operation.DescribeNamespace.service.temporal.type.frontend.95percentile

It looks like tags are being appended to the metric name with dot delimited notation. I’d expect namespace, operation, service, and type to be tags on this metric with the metric name being “temporal.service_latency”.

The reason this metric naming scheme makes it difficult to create datadog dashboard is that each individual combination above needs to be manually added as a series.

Our server config (for the matching server), is as follows:

global:
    metrics:
        tags:
            service: temporal
            type: matching
        statsd:
            hostPort: localhost:8125
            prefix: temporal

My apologies if this has been covered elsewhere, I couldn’t find anything similar to this post. I’ve seen some discussion around using Prometheus, but it seems like using the statsd config should just work in this case. I’ve also seen some discussion around using a custom reporter which may be what we want here?

Thank you!

1 Like
#2

Tally statsd reporter does add "." as delimeter, see here. are you sure you don’t want to produce prometheus style metrics (iirc datadog does have prometheus integration, could be wrong)? If not, the mentioned custom reporter should be ok imo.

#3

Thanks @tihomir!

are you sure you don’t want to produce prometheus style metrics (iirc datadog does have prometheus integration, could be wrong)?

I can give the datadog prometheus integration a try. It would be surprising to me if the behavior via the integration was different from the statsd backend however.

If not, the mentioned custom reporter should be ok imo.

Do you know of or have any code examples where a custom metric reporter is used when creating a server? I’ve seen this post which outlines the code for a custom metric reporting implementation, but not clear on how to put it all together.