8,000 satellites, one off switch: how Starlink went down twice in eight weeks

By the summer of 2025, Starlink was the largest machine ever flown: just under 8,000 satellites in orbit, about 6,900 of them operational, laser-linked into a mesh serving more than six million customers across some 140 markets. A decade of marketing framed it as the network that cannot fail the way ground networks fail - thousands of nodes, no cable to cut, no basement to flood.

On July 24, 2025, at 19:13 UTC, it failed everywhere at once. And seven weeks later, it did it again.

Kentik’s aggregate NetFlow shows traffic to Starlink’s network beginning “a precipitous drop” at 19:13. ThousandEyes watched terminals across every served geography fall into the same loop: connect briefly, lose most packets, drop, search again. For roughly two and a half hours the constellation flew on in perfect physical health - and almost nobody on Earth could use it.

A distributed data plane under one brain

A LEO satellite at Starlink’s altitude moves at about 7.5 km/s, which means the network’s topology changes every second of every day. Running full route computation on board - Dijkstra over a moving graph of thousands of nodes - is expensive in exactly the two things a satellite has least of: power and compute. So SpaceX made the classic software-defined-networking trade, at planetary scale. The satellites are a fast, mostly stateless data plane. The intelligence lives on the ground: routes and allocations are computed centrally and pushed up as timestamped snapshots.

Measurement research pinned the cadence: a global scheduler re-allocates the user-to-satellite-to-gateway path every 15 seconds, for every terminal on the planet (Mohan et al., WWW 2024). One scheduler. The whole sky.

The corollary sat quietly in the design for years: a satellite without a fresh snapshot is not a router. It is a very fast piece of metal that no longer knows whom to serve.

150 minutes in July

19:13 UTC   global traffic collapse begins
            terminals loop: connect, lose,
            search again
21:28 UTC   traffic starts returning
21:40 UTC   service mostly restored
~2.5 h      total, per SpaceX

Publicly, SpaceX said this, through Michael Nicolls, VP of Starlink Engineering: the outage was “due to failure of key internal software services that operate the core network.” To partners it said more, as reported at the time: an upgrade procedure was rolling out to the ground-based compute clusters that host the control plane, and the update overloaded the hardware - the network stopped serving packets while the change propagated. No public engineering postmortem followed. The most detailed accounts of Starlink’s worst outage still belong to third-party monitors.

The third-party data is unambiguous about the shape of the failure. BGP barely flinched - Kentik saw a handful of routes withdrawn while “the vast majority were unfazed”: the roads to Starlink stayed announced, and they were empty. Some terminals briefly established paths that leaked a trickle of traffic with heavy loss before dying - a data plane sporadically alive, with nobody directing it. ThousandEyes’ verdict: a centralized control plane failure. Up to 61,000 users reported outages at the peak on Downdetector alone.

The blast radius

Every continent at the same minute - that is what a single brain buys you. In Ukraine, where Starlink carries front-line connectivity, the commander of drone forces Robert Brovdi wrote that “Starlink is down across the entire front”: combat missions flew without video feeds, reconnaissance fell back to strike drones, for what he called “a long 150 minutes in the war.” His conclusion is the whole post in four words: diversify your communications. Maritime links, remote industrial sites, emergency services - the customers who bought Starlink precisely because it was supposed to be the option that does not fail - went dark together.

What broke, in architecture terms

The data plane was redundant eight thousand ways. The control plane was redundant zero ways: one logical system allocating the entire sky in 15-second heartbeats, hosted on one set of ground clusters, updated - that day - in a way that took all of it down at once.

Notice what the failure mode was not. It was not degraded service, higher latency, a continent shedding load. It was binary, planet-wide, and instant, because between “fresh allocations” and “nothing” the design had no intermediate state: no fly-on-last-known-good-for-N-minutes mode, no regional autonomy, no partial schedule. When the scheduler went silent, every terminal on Earth went into searching in the same second.

And the change management inverted the usual hierarchy. The reported trigger was a routine-looking rollout - to the one layer of the system whose failure is global by construction. What the account to partners does not describe is a canary that failed safe, a rollback that fired on error rates, or geography-staged propagation. One hundred and fifty minutes of worldwide binary outage is strong evidence that none of them stood in the path.

The irony

For the satellites themselves, SpaceX wrote the industry’s reference on safe fleet updates. Their SmallSat 2023 paper - Over-the-vacuum Update: Starlink’s Approach for Reliably Upgrading Software - is about exactly two problems: how to safely update a machine you can never touch again, and how to orchestrate thousands of such updates on a rapid cadence without users noticing, borrowing the best of terrestrial practice. Updating a spacecraft is treated as existential risk, engineered accordingly, and it works.

The ground got less respect. The compute clusters that are the constellation’s brain took an update that overloaded them - the exact failure class the orbital pipeline was built to make impossible.

Eight weeks later, an encore

On September 15, at about 04:35 UTC, it happened again - to a constellation that had meanwhile grown by several hundred more satellites. Shorter this time, roughly an hour, and it caught most of the United States asleep: more than 43,000 US reports peaked on Downdetector within minutes, and Ukraine’s drone-force commander posted that Starlink was “once again down across the entire front line.” NBC News noted the outage followed the rollout of T-Mobile’s satellite service on the network.

The official explanation was not similar to July’s. It was identical - the same sentence, from the same VP of Engineering: “failure of key internal software services that operate the core network.” When one sentence explains two global outages eight weeks apart, it has stopped being an explanation. It is a pattern.

And the pattern is older than 2025. On April 8, 2023, Starlink degraded globally for several hours over an expired ground station certificate. Musk called that ground-side single point vulnerability “inexcusable” - his word, two years before the single point on the ground took the network down twice in one summer.

Three lessons

A distributed system is measured by the independence of its failure modes, not the count of its nodes. Eight thousand satellites behind one scheduler are one node with excellent PR. Whatever redundancy you pile into the data plane, a single control plane above it is a global single point of failure. All three of Starlink’s global software outages - 2023, and twice in 2025 - originated on the ground, where the brain lives, not in orbit, where the nodes do.

The control plane deserves stricter deployment discipline than the data plane, not looser. Staged by geography. Canaried against error rates, with automatic rollback. And backed by a degraded mode in which the data plane keeps working on last-known-good state while the brain restarts. Starlink’s terminals had exactly one fallback state, and it was “searching.”

Marketed resilience is more dangerous than an honest SPOF. “We have thousands of nodes” is a narrative, and its effect is to relax exactly the people who should be architecting around you. The pattern is not Starlink’s alone: one CrowdStrike content update took down millions of Windows hosts in 2024; Cloudflare’s November 2023 incident left its data plane serving while the control plane and analytics were down for days; an AWS us-east-1 internal-network event in December 2021 turned half the internet’s control knobs unresponsive. Different logos, same shape: the plane that thinks is smaller, more central, and less carefully deployed than the plane that works.

The constellation never stopped flying. Through all three outages the sky stayed full of perfectly healthy satellites that nobody could talk to - the cleanest demonstration yet that in distributed systems you do not count the nodes. You count the brains.

Reading

• Starlink Outage Analysis: July 24, 2025 - ThousandEyes Internet Research Team
• The Starlink Outage and Its Impact on Community Gateways - Kentik
• Michael Nicolls on the July cause - SpaceX, July 24, 2025
• Starlink outage affects tens of thousands, including Ukrainian military - CNN Business, September 15, 2025
• Starlink outage caused by expired ground station certificates - Data Center Dynamics, April 2023
• Over-the-vacuum Update: Starlink’s Approach for Reliably Upgrading Software - SSC23-II-01, SmallSat 2023
• A Multifaceted Look at Starlink Performance - Mohan et al., WWW 2024