How to detect a cron run timeout and alert?

I followed the example here https://github.com/temporalio/samples-java/blob/master/src/main/java/io/temporal/samples/hello/HelloCron.java
The cron runs every minute. I modified the execution and run timeout to this
.setWorkflowExecutionTimeout(Duration.ofMinutes(5))
.setWorkflowRunTimeout(Duration.ofSeconds(10))
I let it run for 2 minutes and then killed the worker and here is what I got

How can I detect that the run timeout happened and raise some alert? And what if I want to alert only when 2 consecutive run failed/timed out?

Thank you in advance!

We emit a workflow_timeout metric tagged with namespace. Unfortunately this is not hooked up for cron workflows yet. Issue#397 tracks this improvement. Once it is fixed you should be able to use this metric to track cron workflows timing out.

Thank you very much samar
Is there a doc around how to use metric to setup monitoring and alerts?
Or I should just learn one of the systems listed here? https://docs.temporal.io/docs/configure-temporal-server/#metrics

Lots of users are running Temporal with prometheus as metric backend. Please refer to our helm-charts for documentation on setting it up. If you search on forum you will find a lot of information on this topic. Few examples:




We are using Datadog. Any support on that.
Google search shows nothing and search datadog in the forum got nothing.
I’ll ask a separated question