Setting up of monitoring with Datadog

According to this doc here - Configure the Temporal Server | Temporal documentation, statsD, prometheus are supported.

I could specify the endpoints for these metrics via the env variables like this - temporal/config_template.yaml at f55d0bbcafdd94f80e4f22521d3aaee5f6b1694c · temporalio/temporal · GitHub. Example, to get the metrics for prometheus i can do PROMETHEUS_ENDPOINT=0.0.0.0:9091

Once this metrics endpoint is available, how can I forward it to datadog to visualize?

1 Like

So I managed to set it up with datadog agent (dogstatsd) as a sidecar container alongside the temporal server container. I set an env variable for temporal:

STATSD_ENDPOINT=localhost:8125

The statsD agent appears to be receiving calls from temporal, but it seems like temporal server is reporting in a syntax that doesn’t appear to be supported:

2021-06-25 05:28:30 UTC | CORE | ERROR | (pkg/dogstatsd/server.go:568 in errLog) | Dogstatsd: error parsing metric message ‘“temporal.build_information.build_date.2021-06-08-20:42:06.build_platform.linux/amd64.build_version.1.10.2.git_branch.HEAD.git_revision.e354132c.git_tag.v1.10.1.go_version.go1.16.4:1|g”’: could not parse dogstatsd metric values: strconv.ParseFloat: parsing “06.build_platform.linux/amd64.build_version.1.10.2.git_branch.HEAD.git_revision.e354132c.git_tag.v1.10.1.go_version.go1.16.4”: invalid syntax

Any ideas?

Looks like one option is to setup custom metrics reporter like that - Temporal Server options | Temporal documentation