Adopt OpenTelemetry for workflow telemetry instrumentation

OpenTelemetry is a new standard and API for software telemetry instrumentation and reporting. It unifies tracing, metrics and logs reporting under one unified standard API and protocol and is a result of joined effort of former OpenTracing and OpenCensus teams.

I propose that Temporal would adopt OpenTelemetry instrumentation for instrumenting and reporting metrics and tracing info for workflow execution.

The exact manner of instrumentation is something that needs to be discussed in detail, but my initiona guess is that each workflow run would be represented as a separate trace, each activity would produce a child span and it should be possible to add metadata to the trace/span context for workflows and activities.

This would give Temporal workflows very good observability story and allows it to seamlessly integrate into whatever o11y front-end or SaaS tooling.

The relevant poll so far indicated that 3/4 of respondents wanted Micrometer and 1/4 OpenTelemetry.

We will evaluate our options. Ideally, we should be able to support pluggable frameworks using interceptors.

The issue with the relevant poll is that most people respond with what they know and it only shows that Micrometer is a very well established metrics library.

In my experience, until one has actually been exposed to the wider telemetry landscape, the popular belief is that metrics and logs is all you need to be operationally observable. But there are much better tools out there that give you all the same metrics but with a much richer set of debugging tools once you completely embrace full spectrum of telemetry data.

I would strongly recommend checking out Honeycomb.io — check out their playground demos to see what a proper observability toolkit can do with well instrumented application data and you will see what I mean.

1 Like

I see your point.

Note, that using Micrometer doesn’t preclude from using any other tracing library like Jaeger or even OpenTelemetry. It doesn’t require any changes in the core framework and can be achieved through interceptors.