Scheduled workflow traces combined

Community Support
1

Hello!

I am working on instrumenting our Temporal workflows with OpenTelemetry, and I am running into an issue with scheduled workflows. When the spans come in from a scheduled workflow, they are grouped into one gigantic trace instead of being their own distinct trace. This makes looking those traces up nearly impossible. Is there a way to separate these traces?

Here is a sample of the code I am using to set up the tracing interceptor:

import "go.temporal.io/sdk/client"

func GetClient() (client.Client, error) {
    tracingInterceptor, err := opentelemetry.NewTracingInterceptor(
        opentelemetry.TracerOptions{Tracer: otel.Tracer("temporal")}
    )

    config := client.Options{
	    HostPort:     "<port>",
	    Interceptors: []interceptor.ClientInterceptor{tracingInterceptor},
    }

    return client.Dial(config)
}
2

I managed to prevent the propagation of trace data to the schedule by implementing az interceptor, that explicitly removes the _tracer-data header from the context when creating a schedule, right after the tracing interceptor has placed it there.

Here’s how I did it:

// Implements interceptor.ClientInterceptor
type scheduleTraceRemoverInterceptor struct {
	interceptor.ClientInterceptorBase
}

func NewScheduleTraceRemoverInterceptor() interceptor.ClientInterceptor {
	return &scheduleTraceRemoverInterceptor{}
}

func (c *scheduleTraceRemoverInterceptor) InterceptClient(
	next interceptor.ClientOutboundInterceptor,
) interceptor.ClientOutboundInterceptor {
	i := &scheduleTraceRemoverClientOutboundInterceptor{}
	i.Next = next

	return i
}

// Implements interceptor.ClientOutboundInterceptor
type scheduleTraceRemoverClientOutboundInterceptor struct {
	interceptor.ClientOutboundInterceptorBase
}

func (i *scheduleTraceRemoverClientOutboundInterceptor) CreateSchedule(
	ctx context.Context,
	input *interceptor.ScheduleClientCreateInput,
) (client.ScheduleHandle, error) {
	header := interceptor.Header(ctx)
	delete(header, "_tracer-data")

	return i.Next.CreateSchedule(ctx, input)
}

To use this in your GetClient function you need to just add it to the interceptors after the tracing interceptor:

func GetClient() (client.Client, error) {
    tracingInterceptor, err := opentelemetry.NewTracingInterceptor(
        opentelemetry.TracerOptions{Tracer: otel.Tracer("temporal")}
    )

    scheduleTraceRemoverInterceptor := telemetry.NewScheduleTraceRemoverInterceptor()

    config := client.Options{
	    HostPort:     "<port>",
	    Interceptors: []interceptor.ClientInterceptor{
            tracingInterceptor,
            scheduleTraceRemoverInterceptor,
        },
    }

    return client.Dial(config)
}