Hello, have 2 questions surrounding Workflow Sinks and their use in interceptors. Currently, I have a sink defined that helps me increment a Datadog Metric when called. We currently use these sinks in our workflow interceptors to have metrics upon workflow start and completion. I’d like to also add some sinks to gain activity level metrics (activity starts, failure rates, etc.).
- Would adding a new interceptor that calls some of these sink methods cause any possible non-determinism issues with older running workflows? My gut instinct is no, since the documents say “A sink call isn’t recorded in the Event History of a Workflow Execution (no timeouts or retries)”, but just wanted to double check.
- Which interceptor would be the best place to call this new metrics sink? I’d like to track how often activities fail along with how often they’re started, so should this be located in the workflow worker interceptor or the activity worker interceptor? Specifically, my concern here is that if I have an activity A running that fails + retries multiple times, I’d like to ensure that I get a metric each time it fails and retries. I’m not sure if the workflow interceptor would ensure that the sink gets run for each retry of the activity. On the flip side, I’d also imagine that I’d want metrics for activities being scheduled or started. The former seems like it’d need to be on the workflow interceptor, while the latter should be on the activity interceptor.