Trace Propagation in Java using OpenTelemetry

Community Support
, ,
1

Hi, I’m running into an issue where trace ids aren’t properly being propagated through.
I have set up otel and I know it’s correct because before the workflow starts, I see the otel trace id and when I call another microservice I see that same trace id in the logs for the microservice. However, the context propagation isn’t working correctly within temporal and I’m wondering where I’ve messed up

This is how I’ve set up the intercepter beans in my temporal config:
package my.project.calc.config

@Configuration
public class TemporalConfig {

 @Bean
  @Profile("!test")
  public TemporalOptionsCustomizer<WorkflowClientOptions.Builder>
      customWorkflowClientOptionsBuilder(OpenTelemetry openTelemetry) {
    io.opentracing.Tracer tracer = OpenTracingShim.createTracerShim(openTelemetry);
    OpenTracingOptions options = OpenTracingOptions.newBuilder().setTracer(tracer).build();
    return (optionsBuilder) -> {
      optionsBuilder.setInterceptors(new OpenTracingClientInterceptor(options));
      return optionsBuilder;
    };
  }

  @Bean
  @Profile("!test")
  public TemporalOptionsCustomizer<WorkerFactoryOptions.Builder> customWorkerFactoryOptions(
      OpenTelemetry openTelemetry) {
    io.opentracing.Tracer tracer = OpenTracingShim.createTracerShim(openTelemetry);
    OpenTracingOptions options = OpenTracingOptions.newBuilder().setTracer(tracer).build();
    return (optionsBuilder) -> {
      optionsBuilder.setWorkerInterceptors(new OpenTracingWorkerInterceptor(options));
      return optionsBuilder;
    };
  }
}

I also tried it without the Customizer wrapper:

  @Bean
  public WorkflowClientOptions workflowClientOptions() {
    return WorkflowClientOptions.newBuilder()
        .setInterceptors(new OpenTracingClientInterceptor())
        .build();
  }

  @Bean
  public WorkerFactoryOptions workerFactoryOptions() {
    return WorkerFactoryOptions.newBuilder()
        .setWorkerInterceptors(new OpenTracingWorkerInterceptor())
        .build();
  }

Neither of these worked.
I don’t have a backend collector as I don’t need one
Here are the spring properties:

spring:
  temporal:
    connection:
      target: 127.0.0.1:7233
    namespace: default
    workers-auto-discovery:
      packages: my.project.calc.temporal
otel:
  propagators:
    - tracecontext
    - b3
    - b3multi
  traces:
    exporter: none
  logs:
    exporter: none
  metrics:
    exporter: none

example log. in this example, i would be able to track traceid1 in the given service’s logs until the workflow starts. then it switched to traceid2 (or traceid3,4, etc. its not all one trace id, it seems to switch within the workflow)

Here, i call the microservice from an activity and can see that the traceid1 im expecting exists and is being propagated externally but for some reason the temporal activity itself it not using the traceid1 but rather its on traceid2

[calling micro service] traceparent: 00-traceid1-somespanid
traceid = traceid2

I saw that we can manually set context propagators but that doesn’t seem right for trace ids.

Thanks in advance for any tips!