Anatomy of a feature: how Uber decides whose request to drop

“Overloaded? Drop the excess.” Load shedding sounds free, and that is exactly what makes it dangerous.

A rejection is not free. The server still accepts the connection, still reads the bytes, still parses the request, still decides, still writes an answer back. Under 3x load a naive shedder spends its CPU refusing work: goodput drops toward zero and the system settles there - busy, stable, useless.

Uber hit that wall and built Cinnamon. This is an anatomy of the feature in three layers: what you can observe from outside, what the mechanism is, and why it is built the way it is - and then how the rest of the industry answers the same question.

The surface layer: what you can observe

A service behind Cinnamon holds 300% overload with p50 latency up about 50%. In Uber’s load test the traffic jumps from about 3,000 to about 6,500 RPS: the rejection rate climbs from ~30% to ~75% and plateaus in 10 seconds, and in steady state it stays inside a spread of about 10 percentage points.

overload sustained    300%, p50 +50%
load jump             ~3,000 to ~6,500 RPS
rejection rate        ~30% to ~75% in 10 s
steady-state spread   ~10 ppt, CoDel ~30 ppt

Uber’s previous shedder, QALM, was built on CoDel - and this class of controller oscillates: a narrow rejection band while the ratio is small, then a spread of ~30 points once it has to reject more than half of the traffic. “Reject everything” and “reject nothing”, taking turns.

And with no shedding at all the ending is familiar: instances fail health checks, restarts concentrate the load on the survivors, autoscaling arrives too late. The death spiral.

The middle layer: what Cinnamon is

6 tiers x 128 cohorts = 768 priorities, and four moving parts.

Tier is business criticality. t0 is critical infrastructure, t1 is online user traffic - booking a ride - down through t5, background map refresh. The tier is assigned at the edge and travels in the request context down the whole call chain.

Cohort is a shard of users, an idea borrowed from WeChat’s overload control. When 5% of tier-1 traffic has to go, it is the same 5% of users being dropped on every service in the chain. 128 cohorts also set the resolution: shedding moves in steps of one cohort, roughly 0.8% of a tier.

The rejector compares a request’s priority with an atomic threshold. Cost: about one microsecond per request.

A PID controller re-computes the shed ratio every ~500 ms over a bounded ~30 second history.

The deep layer: why it is built this way

Why cohorts. Let five hops each shed their own independent 5%. A user’s chance to fail somewhere along the chain is 1 - 0.95^5, about 23% - not 5%. Uncoordinated shedding multiplies down the chain. Cohorts make the decision deterministic: rejected at the first hop means rejected at the fifth, and every other user passes the chain end to end.

Why PID. CoDel sees only the current state of the queue, so it swings between extremes. A PID controller carries history in its integral term and converges on a stable ratio - that is where the 10 second plateau comes from. The control theory is a century old, which Uber’s own title happily admits.

Why one microsecond beats a smarter algorithm. Shedding must cost less than the work it saves. Otherwise you get a congestive failure of your own making: throughput with the shedder on drops below what it was with the shedder off.

The same feature, across the industry

WeChat came first. DAGOR, the system Cinnamon borrowed cohorts from, runs admission control on two priorities: a business priority fixed by action type, and a user priority - a hash of the user ID, with the hash function rotated every hour. A server declares itself overloaded when the average queuing time crosses 20 ms, and piggybacks its current admission level on every response, so upstreams stop sending doomed requests at all. Fully decentralized, across 3,000+ services. The reasoning is the same as Uber’s: a high-priority user is “honoured all the way through the call graph” instead of failing at hop four of five.

Meta: Defcon. The unit of degradation is the knob - a feature with a name, an owner, an oncall rotation and a criticality level. L3 targets 5% resource savings and handles load spikes like New Year’s Eve; L2 targets 10% for the loss of a full data center region; L1 targets 20% for global emergencies. A human pulls the lever, and the reaction is measured in minutes.

Netflix. Four priority buckets - CRITICAL, DEGRADED, BEST_EFFORT, BULK - modeled on Linux tc-prio, paired with client fallbacks in the earlier iteration of the system. During an infrastructure incident described in their 2024 write-up, Netflix shed more than half of all requests while availability for user-initiated requests held above 99.4%.

Google. Criticality is a first-class field of the RPC stack, propagated automatically, with adaptive throttling on the client: the client learns the rejection rate and stops sending on its own.

Stripe. Four limiters guard the API; two of them are load shedders. The fleet usage shedder always reserves a slice of the infrastructure for critical requests - cross the line and non-critical calls get a 503. The interesting part is how rarely the deep layers fire: the plain rate limiter rejects millions of requests a month, the worker utilization shedder about a hundred. The unit of degradation is the API method.

LinkedIn: Hodor. Holistic Overload Detection and Overload Remediation - detectors living inside the process. On the JVM the primary signal is garbage collection overhead: when GC starts eating the service, Hodor sheds just enough traffic for it to recover, adaptively, with no configuration. It runs on 1,000+ microservices and has prevented hundreds of overloads. The unit here is not the request at all - it is the process’s own health.

Amazon. Walked away from fallbacks altogether: fallback code rots unexercised for years, then makes the outage worse. The bet is on constant work and static stability - do the same amount of work in every state, and a spike has nowhere to come from.

Envoy. Adaptive concurrency: a gradient controller derives the concurrency limit from observed latency - the same family of controllers Netflix open sourced as concurrency-limits, and Alibaba ships client-side as Sentinel. No priorities at all, on purpose: priorities need a queue, and a queue is itself latency.

DoorDash asks a different question. Their argument: local shedders and circuit breakers protect one service, but microservice failures live in the interactions - retry storms, cascades, metastable states - where local mechanisms “are not very effective”. Hence their experiments with Aperture, an open source controller that takes the global view: metrics in, YAML policies evaluated centrally, and coordinated actions - load shedding, distributed rate limiting - pushed to every node. They also offer a cautionary tale for the local approach: one of their own shedders, configured with too high an initial concurrency limit, overloaded its service at startup.

The layer beneath the algorithm

The shedding algorithm - PID, CoDel, gradient - is the top layer and the most replaceable one. The decision that actually shapes the system sits below: what is the unit of degradation.

Uber and WeChat chose the user. Meta chose the feature. Netflix and Google chose the request. Stripe chose the API method. LinkedIn chose the process. Amazon chose constancy of work instead of degradation. Envoy chose not to choose. And DoorDash asks where the choice should live at all - inside the service or above it.

That choice decides who pulls the lever, whether the reaction takes seconds or minutes, and what the person on the other end of the request actually sees.

Reading

• Cinnamon: Using Century Old Tech to Build a Mean Load Shedder and the PID controller follow-up - Uber Engineering
• Overload Control for Scaling WeChat Microservices - Zhou et al., SoCC’18, summarized by The Morning Paper
• Defcon: Preventing Overload with Graceful Feature Degradation - Meza et al., OSDI’23
• Enhancing Netflix Reliability with Service-Level Prioritized Load Shedding - Netflix TechBlog, 2024
• Scaling your API with rate limiters - Stripe Engineering
• Hodor: Detecting and addressing overload in LinkedIn microservices - LinkedIn Engineering
• Handling Overload - Google SRE Book
• Avoiding Fallback in Distributed Systems - Amazon Builders’ Library
• Failure Mitigation for Microservices: An Intro to Aperture - DoorDash Engineering