Metrics

Metrics come from OpenTelemetry auto-instrumentation, CloudWatch Lambda/API Gateway metrics, and service-specific custom metrics.

Sources

Standard Lambda RED

Leads journey

drive_api_leads_stage_total counts every lead crossing every stage of the leads journey. One counter; each headline metric is a selector over it.

Each stage is recorded by one Lambda. The Leads - Overview dashboard draws each path as one stacked bar chart, a bar per stage labelled with the stage value minus its inbound_/outbound_ prefix, green for ok and red for error; the section heading supplies the path. Each path panel's header links open Explore with Loki filtered to one Lambda's stream, whose service_name label is the function name. Webhook failures by destination come from the WebhookError log line of out-webhook-process, not from the metric.

error_code is a fixed classification, not a cause. It says whose problem the failure is: validation_error and guardrail_error point at the caller, internal_error and infra_error at us, external_error at a dependency, partner_error at a partner endpoint. The specific cause - which field failed, which lookup missed, which upstream status - stays in the logs, correlated by trace_id.

skipped is a deliberate non-delivery, such as a test lead against a destination that has not opted in. It is not an error and not a delivery, so it appears in neither rate.

The four inbound_* gate stages emit on rejection only. A lead clearing all four is counted once at inbound_queue{outcome="ok"}. Every in-queue invocation that is not a warmup therefore records exactly one inbound-stage event: an accepted lead (inbound_queue ok), a blocked lead (a gate stage error) or an enqueue failure (inbound_queue error). Summed across all outcomes and including test leads, the inbound-stage events equal in-queue Lambda invocations minus warmups:

sum(
  last_over_time(drive_api_leads_stage_total{metric_version="2", instance!="", stage=~"inbound_(queue|parse|validate|safety|ratelimit)"}[$__range])
  - (last_over_time(drive_api_leads_stage_total{metric_version="2", instance!="", stage=~"inbound_(queue|parse|validate|safety|ratelimit)"}[24h] offset $__range)
     or 0 * last_over_time(drive_api_leads_stage_total{metric_version="2", instance!="", stage=~"inbound_(queue|parse|validate|safety|ratelimit)"}[$__range]))
)

outbound_enrich fires once per lead; every other outbound_* stage fires once per delivery. So outbound_enrich is the per-lead denominator and the rest are per-delivery numerators.

Dimensions are read only from a payload that has already passed inbound_validate, so the gate stages report unknown for all four. state is resolved during enrich, so every stage before inbound_enrich reports state="unknown".

DLQ depth is not part of this metric - SQS publishes it via the CloudWatch integration as aws_sqs_approximate_number_of_messages_visible.

Counter aggregation

The producer exports cumulative counters and identifies each Lambda execution environment with service.instance.id, sourced from its stable AWS_LAMBDA_LOG_STREAM_NAME. Each execution environment is the single authority for its own series: its counters only move forward, and every collector replica that receives a flush forwards the same total. Accumulating in the collector instead does not work here. drive-otel runs several Alloy replicas behind a round-robin ALB, each with independent delta-to-cumulative state, so one producer's deltas are accumulated separately by each replica and the published series moves backwards, which rate() and increase() read as counter resets.

Do not count with rate() or increase(). Every execution environment is a new series that starts at its first lead, and Prometheus never counts a series' first sample as an increment, so increase() drops one lead per new instance and undercounts by around ten percent a day. The exact count over a window is, per series, the last value inside the window minus the last value before it, with zero for a series born inside the window:

sum by (lead_type) (
  last_over_time(S[$__range]) - (last_over_time(S[24h] offset $__range) or 0 * last_over_time(S[$__range]))
)

where S is the full selector. For a time series use $__interval in place of $__range, multiply by 3600000 / $__interval_ms to express each bucket as a per-hour rate, and let the bucket scale with the range (a 15 minute minimum with maxDataPoints of 100 gives 15 minute buckets up to a day and 2 hour buckets at a week). The lookback before the window only has to exceed the longest gap between two flushes of a live execution environment; a series with no sample inside it is treated as newborn and its whole value is counted. Warmups do not flush, so the gap is bounded by instance lifetime, observed at under two hours; 24 hours leaves ample margin.

Serverless WarmUp invokes selected handlers every five minutes, but each handler returns on source="serverless-plugin-warmup" before it records or flushes a lead metric. Warmups therefore appear in aws_lambda_invocations_sum, not in drive_api_leads_stage_total.

Alarms

Per-service CloudWatch alarms are defined in serverless.common.yml and attached by each service's custom.alerts block.

Prod alarms notify PagerDuty and dev+alerts@drive.com.au; non-prod alarm behavior depends on the service's selected alert tier and stage list.