Guidance on Logging Temporal Trace Errors with OpenTelemetry & Datadog

Community Support
1

Hi everyone,

I’m currently integrating Temporal with OpenTelemetry and Datadog for distributed tracing. I’ve implemented a custom sampler in Go that is intended to always sample and log error spans. The sampler checks for error indicators—specifically, an "error" attribute or an "otel.status_code" attribute set to Error—and logs the span details when an error is detected.

My goal is to ensure that all error traces are properly logged and visible in Datadog’s temporal log view. However, despite the sampler correctly sampling these error spans, I’m not seeing the expected error logs in Datadog.

Has anyone encountered this issue or could share insights on best practices for logging error spans in the context of Temporal and OpenTelemetry? Are there specific configurations or recommended approaches—either within Temporal or Datadog’s exporter settings—that might be necessary to capture these error details?

Any guidance or pointers to relevant documentation would be greatly appreciated!

Thanks in advance for your help.

package sampling

import (
	"fmt"
	"log"

	"go.opentelemetry.io/otel/codes"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

type CustomSampler struct {
	defaultSampler sdktrace.Sampler
}

func NewCustomSampler(defaultRatio float64) sdktrace.Sampler {
	errorAwareSampler := &CustomSampler{
		defaultSampler: sdktrace.TraceIDRatioBased(defaultRatio),
	}
	return sdktrace.ParentBased(
		errorAwareSampler,
		sdktrace.WithRemoteParentSampled(errorAwareSampler),
		sdktrace.WithRemoteParentNotSampled(errorAwareSampler),
		sdktrace.WithLocalParentSampled(errorAwareSampler),
		sdktrace.WithLocalParentNotSampled(errorAwareSampler),
	)
}

func (s *CustomSampler) ShouldSample(p sdktrace.SamplingParameters) sdktrace.SamplingResult {
	for _, attr := range p.Attributes {
		if attr.Key == "error" || (attr.Key == "otel.status_code" && attr.Value.AsString() == codes.Error.String()) {
			return sdktrace.SamplingResult{
				Decision:   sdktrace.RecordAndSample,
				Attributes: p.Attributes,
			}
		}
	}
	return s.defaultSampler.ShouldSample(p)
}

func (s *CustomSampler) Description() string {
	return fmt.Sprintf("CustomSampler(errors=100%%, other=%s)", s.defaultSampler.Description())
}