Tracing

Traces are emitted through AWS Distro for OpenTelemetry and exported to Grafana Tempo (grafanacloud-traces).

Service identity

Every service sets identity in OTEL_RESOURCE_ATTRIBUTES:

service.name=drive-api-leads
service.namespace=aws.lambda
deployment.environment=prod
deployment.environment.name=prod
cloud.provider=aws
cloud.platform=aws_lambda
cloud.region=ap-southeast-2

The collector also inserts service.namespace, deployment.environment, and deployment.environment.name through attributes/service_attrs so telemetry from the Lambda Telemetry API has the same tags.

Propagation

OTEL_PROPAGATORS is set to:

tracecontext,baggage,xray

TraceUtils.extractTraceContext reads incoming context from API Gateway events. X-Ray context propagation is enabled with OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION=false.

TraceQL examples

Slow leads inbound:

{ resource.service.name = "drive-api-leads" && name = "leadsInboundQueue.handle" && duration > 1s }

By HTTP route:

{ span.http.route = "/leads/in/queue" } | count() by (resource.service.name)

Errors:

{ status = ERROR && resource.service.namespace = "aws.lambda" }

The collector drops platform.* spans through filter/traces to reduce Lambda platform noise.