According to this doc here - Configure the Temporal Server | Temporal documentation, statsD, prometheus are supported.
I could specify the endpoints for these metrics via the env variables like this - temporal/config_template.yaml at f55d0bbcafdd94f80e4f22521d3aaee5f6b1694c · temporalio/temporal · GitHub. Example, to get the metrics for prometheus i can do PROMETHEUS_ENDPOINT=0.0.0.0:9091
Once this metrics endpoint is available, how can I forward it to datadog to visualize?
So I managed to set it up with datadog agent (dogstatsd) as a sidecar container alongside the temporal server container. I set an env variable for temporal:
STATSD_ENDPOINT=localhost:8125
The statsD agent appears to be receiving calls from temporal, but it seems like temporal server is reporting in a syntax that doesn’t appear to be supported:
2021-06-25 05:28:30 UTC | CORE | ERROR | (pkg/dogstatsd/server.go:568 in errLog) | Dogstatsd: error parsing metric message ‘“temporal.build_information.build_date.2021-06-08-20:42:06.build_platform.linux/amd64.build_version.1.10.2.git_branch.HEAD.git_revision.e354132c.git_tag.v1.10.1.go_version.go1.16.4:1|g”’: could not parse dogstatsd metric values: strconv.ParseFloat: parsing “06.build_platform.linux/amd64.build_version.1.10.2.git_branch.HEAD.git_revision.e354132c.git_tag.v1.10.1.go_version.go1.16.4”: invalid syntax
Any ideas?
Looks like one option is to setup custom metrics reporter like that - Temporal Server options | Temporal documentation
Has Anyone gotten this working with temporal cloud?
I have the prometheus endpoint up with temporal cloud but datadog agent is not able to read from it.
“message”: “Unsupported content-type provided: text/html”, “traceback”: "Traceback (most recent call last):\n File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 1122, in run\n self.check(instance)
Anyone know if anything else is needed here beyond running a DD Agent and setting the STATSD_ENDPOINT var to point at it?
For Temporal Cloud, you can set up PromQL endpoint for your metrics and then can use this script to ship to Datadog.
I set an env variable
STATSD_ENDPOINT=localhost:8125
and statsd agent is receiving calls but I am getting “error parsing metric message” error. I went through setting up a custom report explained here Temporal Server options | Temporal Documentation but seems like the WithCustomMetricsReporter doesn’t exist anymore.
Any suggestion on how to fix this issue or maybe there is an easier way to send metrics to datadog without a custom report setup?
Hey razmin, WithCustomMetricsReporter has been renamed to WithCustomMetricsHandler that you can find at here
For what its worth I noticed that Datadog expects Openmetrics schema and Temporal defaults to the Tally schema, at least it does on our deployment.