Hi,
I need a way to pass the open telemetry trace context into the workflows/activities, so I could create additional spans manually under the external parent span.
I’ve added the otel interceptor to the client options (go.temporal.io/sdk/contrib/opentelemetry), and I see the automatically created spans for workflows/activities. I also see the property “header.fields._tracer-data.data” added to the workflow. But once inside the workflow/activity code I want to populate a Go context.Context with the trace info and I can’t find a way to do so, since I only have workflow.Context.
If you need to create custom spans for activities, the span is already on the normal context like any other OTel library, so you can just use tracer.Start for a new one or trace.SpanFromContext or whatever.
For workflows, since we use workflow.Context it’s a bit different. You would have to set the SpanContextKey in TracerOptions to whatever key you want then you can use that key to obtain the span via workflow.Context.Value().
Note however that workflow code can get replayed over and over and on different workers. This causes all sorts of issues with OTel which doesn’t support span resumption. For your custom spans you may want to use workflow.IsReplaying() to choose not to create spans during replay. The problem here is what if your replaying start but not the end stage of the span? Then it may replay the start but the end will not be replaying. There is no good solution to this. What many do is just start and stop their span immediately when workflow.IsReplaying() is false instead of leaving a span open to potentially never close or only close when workflow is evicted from cache (because workflow may move workers).
Hi,
Regarding the opening new traces for spans, this is the issue that I started the thread for, it happens since the context is not propagated into the go context inside the workflow.
Using @Chad_Retz suggestion to use my own key in the interceptor did not work for me since I also needed the span type in order to extract the span from the workflow.Context, but that span type was defined internally in the interceptor.
In order to solve it I copied the interceptor code to my project so I could have access to the context span key & type, after which I could extract the span fro mthe workflow.Context using this code:
if val := wfctx.Value(mySpanContextKey); val != nil {
vals := val.(*tracerSpan)
ctx = trace.ContextWithSpan(ctx, vals.Span)
}
This is so you can not perform some of these side-effect behaviors like span creation when the workflow is replaying code it has already replayed.
If they are under a different trace, that likely means you aren’t setting the interceptor on the client. The client-side calls make a span and serialize that into Temporal headers to be deserialized as the span parent when other spans are created on the worker.
Ah, I see. That embeds the opentelemetry span type, so I wonder if you could just:
import "go.opentelemetry.io/otel/trace"
// ...
if span, _ := wfctx.Value(mySpanContextKey).(trace.Span); span != nil {
// Use span
}
Also, make sure that ctx you create is only used for OTel span creation purposes and not for anything else. The async functionality of Go contexts in workflows is non-deterministic.