Temporal not writing to OpenTelemetry Collector

Community Support
1

Hi,
I am unable to get Temporal to write to OpenTelemetry Collector, both running locally.

I’m using the docker-compose version of Temporal. I have edited the dynamicconfig/development-sql.yaml file to look like below:

limit.maxIDLength:
  - value: 255
    constraints: {}
system.forceSearchAttributesCacheRefreshOnRead:
  - value: true # Dev setup only. Please don't turn this on in production.
    constraints: {}
exporters:
  - kind:
      signal: traces
      model: otlp
      protocol: grpc
    spec:
      connection:
        insecure: true
        endpoint: localhost:4317

And I have otel-collector running locally via Docker, as described here: Getting Started | OpenTelemetry. I don’t see the otel-collector dumping anything to screen as far Temporal is concerned. But I know that my workflow is instrumented correctly, because things get logged to console when I enable the ConsoleSpanExporter.

docker run -p 4317:4317 -v ${pwd}/otel-config.yaml:/etc/otelcol-contrib/config.yaml otel/opentelemetry-collector-contrib:0.81.0

Note: To get Temporal to write to otel-collector, I’m using the documentation provided here but with one change: https://github.com/temporalio/temporal/blob/master/develop/docs/tracing.md#configuring. You can see that this doc page suggests creating an “otel” top-level stanza in the YAML. Unfortunately that seems wrong, and leads to this error message:

temporal                | Unable to create dynamic config client. Error: unable to read dynamic config: unable to decode dynamic config: yaml: unmarshal errors:
temporal                |   line 8: cannot unmarshal !!map into []struct { Constraints map[string]interface {}; Value interface {} }
2

Got it working by directly writing to an OpenTelemetry backend from the code, using otlp-trace-exporter-http. Not using the above approach of defining exporters in the Temporal yaml file.