Ship on Friday: feature flags, and the flag that cost $440 million

It is Friday afternoon, the branch is green, and the team is having the argument again. Ship it now, or leave it until Monday?

The argument is usually framed as courage versus caution. It is neither. It is a question about which of two entirely different events you are about to perform, and most teams have quietly fused them into one. Charity Majors drew the line years ago: a deploy is “the process of building, testing, and rolling out changes to your production software”; a release is “the process of changing user experience in a meaningful way.” Deploys should be boring and frequent. Releases are a product decision, and they deserve their own dial.

A feature flag is that dial. Get it right and Friday stops being a risk category. Get it wrong and you get Knight Capital.

The flag that outlived its feature

On August 1, 2012, Knight Capital Americas deployed new order-routing code to eight production servers. A technician copied it to seven of them.

That alone would have been survivable. What made it fatal was a decision taken earlier and entirely inside the codebase: the new code reused a flag that had, for years, activated an obsolete function called Power Peg. The old function had been disabled but never deleted. On the seven updated servers the flag now meant “run the new logic.” On the eighth, the flag still meant what it had always meant, and Power Peg woke up after a decade of sleep.

In 45 minutes Knight sent millions of erroneous orders into 154 stocks and lost roughly $440 million - a firm-ending amount for a firm that had been worth about that much on Monday. The SEC’s order of October 16, 2013 brought the first enforcement action ever taken under the market access rule and a $12 million penalty.

Read the failure carefully, because the usual moral is the wrong one. The deploy error was the trigger. The bomb was a retired flag that was still wired to live code, and a team that treated flag reuse as an economy. Every practice in the rest of this post exists to keep that particular bomb out of your repository.

What a flag actually buys you

Flags are not new. Flickr described the technique publicly in December 2009 in Flipping Out: one trunk, no long-lived branches, everything integrated on commit, incomplete features hidden behind flippers and shipped anyway. Pete Hodgson gave the practice its canonical taxonomy eight years later in Feature Toggles on Martin Fowler’s site.

The mechanics are simple. The value is in what stops happening:

• Merges stop being events. Nothing lives on a branch long enough to rot; the integration cost is paid a line at a time instead of in one Friday-afternoon lump.
• Rollback stops requiring a deploy. The fastest way to undo a change is a config write that reaches every process in seconds, not a pipeline run that reaches them in minutes.
• The blast radius becomes a parameter. One percent, five percent, one region, one paying customer who agreed to be first.
• Release timing leaves engineering. Marketing can turn the feature on at 9 a.m. Tuesday without a human being on call at 8:55.

That last one is what actually makes Friday boring. If the change is already in production, dark and off, then Friday’s deploy carries no user-visible behavior at all - and the release, whenever it happens, is a dial you can turn back.

Architecture: evaluate locally, fail predictably

The naive implementation calls a flag service on every evaluation. It works beautifully until the flag check sits in a hot path and every request grows a network round trip, or until the flag service has a bad afternoon and takes your application with it.

Serious implementations invert the flow. The SDK pulls the full rule set once, keeps it in memory, and evaluates locally - no network call on the hot path, evaluation in microseconds. Updates arrive by streaming (server-sent events, gRPC, or a websocket), so a change made in the console reaches every process in about a second. That is eventual consistency, deliberately chosen: for a percentage rollout, a second of skew between two servers is harmless, and the alternative costs you a round trip on every request forever.

Two properties matter more than the transport:

The kill switch must not depend on what it protects. If the flag service is unreachable, the SDK must serve the last known good rule set from local cache, and if there is no cache, a fallback value compiled into the deploy. A kill switch that requires a healthy control plane to be pulled is not a kill switch; it is a wish. Test this path deliberately - block the flag service in staging and confirm the application still starts, still serves, and still honors sane defaults.

Bucketing must be deterministic and sticky. Percentage rollouts work by hashing a stable identifier - user id, account id, device id - together with the flag key, then comparing the hash against the threshold. Same user, same flag, same answer, on every server, on every request. Roll a die per request instead and a user in a 5% rollout will flicker between the old and new experience twenty times a session, which will look like a haunting rather than a bug. Hashing the flag key into the bucket also prevents the same 5% of users from being the guinea pigs for every experiment you ever run.

And log what you evaluated. A flag decision that is not in your telemetry turns every “it works for me” into an archaeology project. Emit the flag key, the variant served, and the reason - rule match, percentage bucket, default - alongside the request. It costs a few bytes and pays for itself the first time a support ticket mentions a screen half your users cannot see.

The blast radius is a parameter, and the return path is always open.

Schema changes under a flag

Hiding a button behind a flag is trivial. Hiding a change to the shape of your data is the part that separates teams that use flags from teams that trust them.

The pattern is expand and contract, also known as parallel change: never migrate in place, always run both shapes side by side long enough to change your mind. Flags are what let you move between the phases without a deploy.

1. Expand. Deploy the new schema alongside the old. Nothing reads or writes it yet. This step must be backward compatible on its own - a new nullable column, a new table, a new index - so that a rollback of the application code changes nothing about the database.
2. Dual write. Flip the first flag: the application writes both shapes, still reads the old one. Now the new structure is filling with real production traffic while the old one remains the source of truth. Backfill history in the background.
3. Verify, then read. Before switching reads, compare them. Run the new query alongside the old one, return the old result, and log the differences - the technique GitHub packaged as Scientist. When the mismatch rate is flat at zero, flip the second flag and let a small cohort read from the new shape. If anything looks wrong, flip it back; because both writes are still happening, no data has diverged.
4. Contract. Only after reads have been at 100% long enough to survive a full business cycle - month-end, quarterly job, the annual report nobody remembers - delete the old read path, the old writes, the flags, and finally the old schema.

The order matters in both directions. Every phase is independently reversible, and no phase requires the previous one to be undone in a hurry. That is the whole point: the migration becomes a sequence of small, boring, revertible steps instead of one irreversible Saturday.

Expand and contract, with the flags marked. Every arrow can be reversed without a deploy.

Wire the rollback to the metrics, not to a human

A percentage rollout is a controlled experiment, and controlled experiments deserve a stopping rule written down in advance.

The mature setup connects the flag system to the telemetry pipeline: when a flag is at 5%, the platform compares the exposed cohort against the rest on a small set of guardrail metrics - error rate, latency at the tail, and one or two business numbers that would notice a broken checkout. Cross a threshold and the flag turns itself off, in seconds, without paging anybody.

Three details make the difference between an automatic rollback and an automatic outage:

• Compare cohorts, not time windows. Traffic at 4 p.m. Friday looks nothing like traffic at 4 a.m., and a threshold set against yesterday will fire on the daily curve. Comparing the flagged cohort against the unflagged one at the same moment removes most of that noise for free.
• Give the rule enough traffic to be right. A 5% cohort of a low-volume endpoint may see a handful of requests a minute; a single retry storm crosses any threshold you like. Require a minimum sample before the rule is allowed to fire, or you have built a random number generator with production access.
• Make the rollback loud. Automatic means nobody was paged, not nobody was told. The flag flipping itself off is an incident that already resolved: it belongs in the channel, with the metric that tripped it and a link to the exposed cohort, so Monday starts with a diagnosis instead of a mystery.

Four lifetimes, one discipline

Every flag is technical debt that you took on knowingly. The interest is paid in state: each live flag doubles the number of configurations your system can be in, and integration testing stops being meaningful somewhere around the fourth or fifth simultaneous flag.

Hodgson’s taxonomy is still the right one, and the useful part is that each category has a different expiry:

• Release flags live days to weeks. They exist to decouple deploy from release, and they must be deleted the moment the rollout reaches 100%.
• Experiment flags live weeks to months - as long as the experiment needs to reach significance, and not one sprint longer.
• Ops flags are long-lived by design: kill switches, load shedding, rate limiters, the switch that turns off the recommendation engine when it starts timing out. These stay, and they get tested like the safety equipment they are.
• Permission flags live for years, because they are not really flags at all - they are entitlements, the plan tier, the beta cohort, the enterprise feature. Give them their own home in the product model rather than letting them hide in the same table as this week’s rollout.

“Savvy teams view their Feature Toggles as inventory which comes with a carrying cost,” Hodgson wrote, “and seek to keep that inventory as low as possible.” The teams that hold that line put deletion in the Definition of Done, or add an expiry date to the flag record and let a test fail when the date passes.

Uber went further and automated it. Their tool Piranha, open sourced in March 2020, parses the abstract syntax tree, finds code guarded by a stale flag, deletes the condition, deletes the code that becomes unreachable, deletes the now-pointless tests, and opens the pull request. Uber has used it to remove roughly two thousand stale flags across its mobile codebases.

Four categories, four expiries. The first two are debts with a due date; the last two are architecture.

Which brings us back to Knight Capital. The Power Peg flag was a release flag that had been left in the codebase for years after its feature was retired, and then reused because deleting it looked like work with no payoff. Nothing in this section is hygiene. It is the whole safety case.

Never reuse a flag key. A retired flag key should be as dead as a retired database column: the code deleted, the key retained in the registry as tombstoned, and any attempt to reintroduce it rejected in review. Flag keys are cheap. Meaning collisions are not.

Which service, and the standard underneath

The vendor question matters less than it used to. OpenFeature is a vendor-neutral flag API that joined the CNCF in June 2022 and reached incubating status in November 2023: your application codes against one SDK interface, and the provider behind it - a hosted service, a self-hosted open-source server, or a static file for local development - becomes swappable. The related remote evaluation protocol standardizes the wire format between an application and whatever evaluates its flags.

The practical rule for a small team: start with a file or a table plus a cache, adopt the standard interface early so nothing in your code depends on the vendor, and move to a managed service the day someone outside engineering needs to flip a flag safely.

The Friday checklist

Friday is fine when all of these are true:

• the change is behind a flag that defaults to off, and the deploy is dark
• the flag evaluates locally, with a cached rule set and a compiled-in fallback
• bucketing is deterministic, so the exposed cohort stays the same set of people
• guardrail metrics compare cohorts, and the rollback rule can fire without a human
• the rollback path has been tested this quarter, not asserted in a design document
• the flag has an owner, an expiry date, and a deletion ticket that already exists

And Friday is not fine when the release is the deploy, the rollback is a pipeline run, or the flag can only be turned off by the system that is currently on fire. That is not a calendar problem. It is an architecture problem wearing a calendar costume.

Deploy on Friday. Release on Tuesday. Delete the flag on Wednesday.

Reading

• Feature Toggles (aka Feature Flags) - Pete Hodgson, October 2017. The canonical taxonomy and the carrying-cost argument.
• ParallelChange - Danilo Sato. Expand and contract, in one page.
• Deploys Are the Wrong Way to Change User Experience - Charity Majors. The deploy-release split, argued properly.
• Flipping Out - Flickr, December 2009. Where the practice went public.
• Piranha - Uber’s AST-based flag cleanup tool, open sourced March 2020.
• Scientist - GitHub’s library for running the new path beside the old one and diffing the results.
• OpenFeature - the CNCF flag API standard, incubating since November 2023.
• SEC administrative proceeding 34-70694, October 16, 2013 - the Knight Capital order, and the first enforcement under the market access rule.