RED: Rate, Errors, Duration
Three numbers per request-handling service: how many, how many failed, how long they took. RED is the fastest way to make every service in a fleet legible in the same shape — and it goes blind the moment work stops being request-shaped.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Three numbers, one shape, every service
RED is deliberately smaller than the golden signals: it drops saturation and keeps the three that describe work *as the caller experiences it*. Rate is requests per second. Errors is the fraction of those that failed. Duration is the distribution of how long they took — a histogram, not an average (see Histograms: A Distribution You Can Afford to Keep Forever).
The reason to adopt it fleet-wide is consistency, and consistency during an incident is worth more than sophistication. When every service publishes the same three series with the same names, an engineer can pivot from an unfamiliar service to its neighbour without relearning anything, and cross-service comparison becomes possible at a glance. That is the actual product of RED: not better numbers, but a shared vocabulary.
It pairs naturally with USE (see USE: Utilization, Saturation, Errors): RED describes the work from the requester's side, USE describes the resource from the machine's side. RED tells you customers are waiting; USE tells you which resource they are waiting for. Neither is a substitute for the other, and most useful service dashboards contain both.
RATE requests_total{service, route, method} → rate over window
ERRORS requests_total{service, route, status_class="5xx"} → fraction of rate
DURATION request_duration_seconds_bucket{service, route} → histogram → p50/p95/p99
service rate err% p50 p95 p99
────────────────────────────────────────────────────────────
checkout-api 1,205 0.4% 210ms 890ms 1,820ms ← tail problem
catalog-api 4,410 0.1% 38ms 72ms 140ms
search-api 680 2.9% 120ms 310ms 520ms ← error problem
inventory-api 2,180 0.1% 45ms 91ms 180ms
Same three columns for every service is the point. Comparing checkout to
catalog requires no context switch, and two different failure shapes
(tail vs errors) are visible in one read.Where RED misleads if read carelessly
The first trap is duration as a mean. A service serving 99% of requests in 40ms and 1% in 8 seconds has a mean around 120ms, which looks unremarkable and describes no actual request (see The Average Was Fine and Users Were Not). Duration must be a distribution, and the alerting threshold belongs on a percentile tied to what users tolerate (see Percentiles: Which One, and How Many Users Is That?).
The second trap is errors defined as HTTP status. Plenty of real failures return 200: a search that returns an empty result set because the index is down, a checkout that succeeds but silently skips the confirmation email, a GraphQL response carrying an errors array with a 200 status. If the error rate is computed from status codes alone, these are invisible — the service reports perfect health while failing its users.
The third trap is aggregation across routes. A service-wide p99 mixes a fast health check with a slow report generator; the aggregate belongs to neither. Splitting duration by route is what turns RED from a summary into a diagnostic, and route is a low-cardinality label as long as it is the *template* (/orders/{id}) rather than the raw path (see Label Sets That Survive a Year).
1# duration as an average, errors as status-code-only, no route split2avg(rate(request_duration_seconds_sum[5m]))3 / avg(rate(request_duration_seconds_count[5m]))4 5rate(requests_total{status=~"5xx"}[5m]) / rate(requests_total[5m])6 7# Reports: 120ms average, 0.1% errors. Looks healthy.8# Reality: 1% of checkouts take 8s; search returns empty9# results with status 200 because the index is down.1# duration as a distribution, per route2histogram_quantile(0.99,3 sum by (route, le) (rate(request_duration_seconds_bucket[5m])))4 5# errors include semantic failures, split by class6sum by (route, error_class) (rate(requests_failed_total[5m]))7 / sum by (route) (rate(requests_total[5m]))8 9# Reports: /checkout p99 = 1.8s (p50 210ms) → tail problem10# /search error_class="index_unavailable" = 2.9% at status 200Same three concepts, three different implementation decisions — distribution instead of mean, semantic errors instead of status codes, per-route instead of per-service. The queries are barely longer and the second set finds two real problems the first set reports as healthy.
When RED is the wrong frame
RED assumes work arrives as discrete requests with a caller waiting. That assumption breaks in several common architectures, and forcing the frame onto them produces dashboards that look fine while the system fails.
For an async queue consumer, per-message processing duration can be excellent while the backlog grows for hours — the consumer is healthy and the *system* is failing, because nobody is measuring how long messages wait before being picked up (see Depth Is Not an Emergency; Age Is). For a batch job, "rate" is meaningless between runs and the signal that matters is whether the run finished before the downstream deadline. For a streaming consumer, lag is the whole story and RED contributes almost nothing.
There is also a subtler blind spot in request services: RED measures requests the service *received*. If a load balancer is rejecting connections, or clients are timing out before their request arrives, those failures never appear in the service's own RED metrics. The service reports a healthy error rate for the requests it saw, which is true and useless. Client-side or edge measurement is the only way to see them (see Coordinated Omission: When the Load Generator Lies for the same blind spot in load testing).
| Workload | Rate means | Errors mean | Duration means | What RED misses |
|---|---|---|---|---|
| HTTP service | requests/sec by route | failed fraction incl. semantic failures | distribution of request duration | Requests rejected before arrival; resource saturation |
| Queue consumer | messages consumed/sec | failed + dead-lettered fraction | per-message processing time | Wait time before pickup — the number that actually matters |
| Batch job | records/sec within a run | failed records or a failed run | total run duration | Lateness against the downstream deadline |
| Streaming consumer | events/sec | deserialization / handler failures | per-event handling time | Consumer lag — none of the three move as lag grows |
| Agent / LLM step | steps or runs/sec | tool errors, refusals, timeouts | total run duration | Time to first token, step count, token cost (see Where an Agent Run Actually Spends Its Time) |
Key points
- Rate, Errors, Duration — three series per service, named identically fleet-wide so any engineer can read any service.
- Duration must be a distribution split by route; a service-wide mean describes no real request.
- Errors must include semantic failures, not just 5xx — plenty of real failures return 200.
- RED describes work from the caller's side; USE describes the resource. Most good dashboards carry both.
- The frame breaks for queues, batch and streaming, where wait time, lateness and lag are the signals that predict failure.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Change or demand → service: each request begins costing more time or failing more often.
- 2Service → duration histogram: the bucket distribution shifts; the tail buckets fill while the median buckets barely move.
- 3Aggregation → visibility: a service-wide average or an unsplit percentile absorbs the shift, so the dashboard stays calm.
- 4Errors → status codes: semantic failures return 200 and never reach the error series, so the error panel stays flat too.
- 5Team → conclusion: the dashboard says healthy while customers report failures, and trust in the dashboard drops.
- • "Average duration is 120ms, so we are fast." A bimodal distribution has a mean that describes no request in it.
- • "Error rate is 0.1%." If errors are counted from status codes, semantic failures returning 200 are excluded from that number entirely.
- • "The consumer's duration metric is healthy." Per-message processing time says nothing about how long messages waited in the queue first.
- • "Our RED metrics show no errors during the outage." A service cannot count requests that never reached it; edge and client measurement are separate signals.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Rate: `rate(requests_total[5m])` split by route and method, with the same metric name in every service.
- • Errors: failed fraction by error class, where "failed" is defined semantically rather than by status code alone.
- • Duration: a histogram, read at p50/p95/p99 per route — never as a sum-over-count average.
- • For async workloads, add the signal RED omits: oldest-message age, consumer lag, or lateness against schedule.
- • Standardize metric names and label sets across the fleet so RED reads identically everywhere.
- • Emit duration as a histogram with buckets chosen around the SLO threshold, and always split by route template.
- • Define errors semantically in code — a `requests_failed_total` counter incremented by the handler — rather than inferring from status.
- • Add the workload-appropriate fourth signal for async systems: oldest-message age, consumer lag, or schedule lateness.
- • Measure at the edge as well as in the service, so rejected and timed-out requests are visible somewhere.
- • Compare the service's own error rate against edge-measured failures for the same window; a large gap means requests are failing before arrival.
- • Check that p99 per route differs meaningfully from the service-wide p99 — if it does not, the split is not yet capturing route diversity.
- • Trigger a known semantic failure in staging and confirm it appears in the error series.
- • Per-route histograms multiply series count: routes × buckets × status classes adds up quickly (see [[cardinality]]).
- • Semantic error definitions require application code to participate, which means they can be forgotten in new handlers.
- • Fleet-wide naming conventions constrain teams that want service-specific metrics, and enforcing them costs review time.
- • Enforce the metric naming convention in a shared library or service template, so drift cannot happen quietly.
- • Alert on the duration percentile tied to the SLO rather than on the average (see SLOs: A Target, a Window, and a Reason).
- • Add a check that every new route appears in the per-route series within a release of shipping.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe fleet table and query examples are teaching constructs. Metric names and query syntax vary by stack; the three concepts do not.
- WORKLOAD-SPECIFICRED is designed for request-response services. For queues, batch and streaming it must be extended with wait time, lateness or lag, which are not derivable from the three.