How One Uncaught Rust Exception Took Out Cloudflare

Every so often, the internet likes to remind us that it is not a magical cloud powered by optimism and caffeine. It is, instead, a huge stack of software, configuration files, assumptions, and highly paid adults trying very hard not to press the wrong button before lunch. On November 18, 2025, that reminder arrived with all the subtlety of a fire alarm in a library: Cloudflare suffered a major outage, and a startling number of websites and apps began face-planting in public.

What made the incident so fascinating was not just the scale. It was the path. This was not some movie-script cyberattack with neon maps and dramatic keyboard pounding. It was a chain reaction: a routine database permissions change, a bot-management feature file that quietly grew too large, a limit inside Cloudflare’s newer FL2 proxy, and one uncaught Rust error that turned a bad input into a full-blown panic. From there, the outage spread the way bad news always does on the internet: instantly, globally, and with a thousand people yelling, “Wait, why is everything down?”

If you want the short version, here it is: Rust did not “break the internet.” An unchecked assumption did. The real story is about error handling, blast radius, system design, and what happens when a small internal mistake gets a first-class ticket onto critical infrastructure.

When Cloudflare Sneezed, the Internet Caught a Cold

Cloudflare sits in front of a huge slice of the web, helping sites load faster, block attacks, route traffic, and generally keep digital civilization from turning into soup. When its network has a bad day, plenty of other services have a bad day too. During this outage, users reported problems reaching big-name platforms such as ChatGPT, X, Spotify, Canva, Uber, and even Downdetector, which is a bit like the fire extinguisher catching fire.

That scale is what made the outage feel so eerie. It was not a single app malfunctioning. It was the odd experience of opening several unrelated websites and getting different flavors of “nope.” For regular users, it felt random. For engineers, it felt familiar: if many unrelated services fail at once, the problem is probably hiding in shared infrastructure.

That shared layer, in this case, was Cloudflare’s core proxy path. Once it stumbled, everything sitting on top of it started wobbling too.

What Actually Went Wrong

A routine database change opened the first trapdoor

The outage started with a permissions update to one of Cloudflare’s database systems. On paper, that sounds boring enough to put a golden retriever to sleep. In reality, it changed how a ClickHouse query behaved. A query that had previously returned metadata from one database began returning additional metadata from an underlying schema as well.

That difference mattered because Cloudflare’s Bot Management system used the query output to generate a frequently refreshed feature file. This file helps the system score requests and decide what looks like automated bot traffic versus legitimate human traffic. Since bot behavior changes constantly, the file is updated and pushed quickly across the network. Speed is part of the product. Unfortunately, speed also makes a lovely vehicle for spreading mistakes.

Once the query began returning extra rows, the feature file ballooned. A file that normally carried roughly 60 machine-learning features suddenly shot past 200. And because this file was automatically propagated across Cloudflare’s machines, the bad input did not stay local for long. It went global.

The file got bigger than the code expected

Cloudflare’s proxy modules include limits to prevent runaway memory use and to preallocate resources for performance. That is sensible engineering. Nobody wants a core traffic-handling service freelancing its memory habits in production.

But the Bot Management path had a hard limit on how many runtime features it expected to load. That ceiling was set at 200, comfortably above the normal count. Then the oversized file arrived and said, “Surprise.”

In other words, the system was not merely handed bad data. It was handed bad data that broke one of its built-in assumptions. Once that assumption failed, the question became whether the software would degrade gracefully or throw a chair through the window.

FL2 hit an error and panicked

Cloudflare explained that the newer FL2 proxy path contained Rust code that checked this feature limit. When the bad file crossed the threshold, the code path wound up panicking with the now-famous message that it had called Result::unwrap() on an Err value.

If you do not live inside Rust docs for fun, here is the plain-English version. Rust uses Result to represent operations that can either succeed or fail. Calling unwrap() is basically telling the language, “I am sure this worked, hand me the success value.” If it did not work, Rust panics. That is not a gentle shrug. That is the program equivalent of dropping the tray, flipping the table, and exiting the room.

So yes, one uncaught Rust exception more precisely, one panic-triggering unwrap() on an error path became the headline. But the deeper issue was not “Rust bad.” The deeper issue was that a critical piece of traffic infrastructure was allowed to treat malformed internal input as unrecoverable.

Why This Became a Major Cloudflare Outage Instead of a Small Bug

Internal input was treated as trustworthy

One of the most important lessons from the Cloudflare outage is painfully simple: internally generated data can still be dangerous. Engineers often harden systems against customer input, attacker input, and random internet nonsense, yet give internal pipelines a warm, trusting handshake. That works right up until it absolutely does not.

Cloudflare itself highlighted this in its remediation plan, saying it would harden ingestion of Cloudflare-generated configuration files the same way it treats user-generated input. That is a big admission, and an important one. If a file can crash a service, it does not matter whether that file came from a stranger on the internet or your own automation.

The bad file propagated fast

Bot Management depends on rapid updates because threats change fast. That means Cloudflare designed the system to distribute new feature data aggressively across the network. Great for fighting adaptive bots. Less great when the update is wrong.

This is the classic tradeoff in large-scale systems: the faster your control plane moves, the bigger the blast radius when it moves in the wrong direction. A slow rollout may annoy product teams. A global rollout of a bad file annoys the entire internet.

The symptoms were misleading

Incidents are rarely kind enough to announce themselves with an accurate label. Early in the outage, Cloudflare suspected a huge DDoS attack. That guess was not completely irrational. The network was failing in a strange way, and a coincidental issue with the status page made things look even more suspicious. To the humans in the incident room, it briefly looked like multiple systems might be under attack at once.

That matters because misdiagnosis burns time. While teams investigated elevated traffic and issues in Workers KV, the real trigger was still the bot-management configuration file. This is another reason postmortems are so valuable: they turn the messy, emotional timeline of an outage into a readable chain of cause and effect, rather than a foggy memory of everyone yelling at dashboards.

The Timeline of the Incident

Cloudflare said the database access control change was deployed at 11:05 UTC. Impact on customer traffic began at 11:28 UTC. The first automated test caught issues at 11:31, manual investigation began at 11:32, and the incident call started at 11:35.

From 11:32 to 13:05 UTC, teams chased what initially appeared to be degraded Workers KV behavior and related downstream failures. At 13:05 UTC, Workers KV and Cloudflare Access were partially bypassed to reduce impact. By 13:37, engineers had focused their efforts on rolling the Bot Management configuration file back to a known good version.

At 14:24 UTC, Cloudflare stopped generating and propagating new Bot Management configuration files. At 14:30 UTC, the main impact was largely resolved as a correct file was deployed globally. Full resolution came later, at 17:06 UTC, after downstream systems were restarted and lingering issues were cleared.

That timeline is worth studying because it shows how outages often unfold in layers. The primary issue may be fixed first, but side effects retries, login backlogs, overloaded dashboards, dependent services recovering unevenly can keep the incident alive long after the original bug has been caged.

What Made the Cloudflare Rust Panic So Interesting

Rust prevented some classes of bugs, but not this one

Rust has earned its reputation for memory safety, ownership rules, and making entire categories of low-level bugs much harder to write. But languages are not magical exorcists. They reduce some risks; they do not repeal engineering reality.

In this incident, the problem was not memory corruption or a dangling pointer. It was logic, assumptions, and failure handling. The system encountered an error state, and a panic occurred in a production path that served critical traffic. Rust gave the developers tools to express recoverable errors. The system simply hit a path where those tools were not used defensively enough.

That is why the better lesson is not “never use Rust.” It is “never let critical infrastructure assume a bad internal state is impossible.”

Old and new systems failed differently

Cloudflare noted that both its newer FL2 proxy and older FL proxy were affected, but in different ways. FL2 returned HTTP 5xx errors. The older FL path did not crash in the same way, yet it still behaved badly by generating incorrect bot scores. That meant some customers using bot-score rules could see false positives even without hard failures.

This nuance matters because it reveals a subtle truth: graceful degradation is not the same as correct behavior. A system that stays “up” while making bad decisions can still cause real harm. The old path looked sturdier from one angle, but it was hardly perfect. One version failed loudly; another failed awkwardly.

Cloudflare’s Fixes and Why They Matter

Cloudflare’s remediation list was refreshingly concrete. The company said it would validate internally generated configuration files more aggressively, add more global kill switches, stop core dumps and other error reports from overwhelming system resources, and review failure modes across core proxy modules.

Those are not cosmetic changes. They target the exact weak points the incident exposed:

• Input validation for internal data
• Safer global rollback controls
• Better containment when errors happen
• A broader audit of “what happens if this assumption is wrong?”

That last one is the money question. Every large system has assumptions buried inside it like old wiring in the walls. Most days they hum quietly. On the wrong day, one of them sparks.

What the Rest of Us Should Learn From This

The Cloudflare outage is a terrific case study for anyone who builds, deploys, monitors, or writes about infrastructure. First, internal automation deserves the same suspicion as external input. Second, rapid global propagation needs equally rapid kill switches. Third, error paths in critical software should be treated like first-class product features, not sad little alleyways nobody wants to visit.

And finally, the modern web is still far more centralized than many people realize. When a single infrastructure provider trips, ordinary users do not see “an issue in a bot-scoring feature file.” They see their favorite apps refusing to load, their work tools stalling, and the general sense that the internet has suddenly become haunted.

That is why this story traveled so far. It was technical, yes. But it was also deeply human. One assumption. One panic. One morning where millions of people discovered just how much of the web depends on systems they never think about until those systems break.

Conclusion

So, how did one uncaught Rust exception take out Cloudflare? Not by itself. It took a permissions change, a query behavior shift, duplicate feature rows, an oversized Bot Management file, a hard runtime limit, a panic-triggering unwrap(), and a fast-moving deployment path with enormous reach. The exception was the dramatic part, but the outage was really a systems story.

And that may be the most useful takeaway of all. Big outages are rarely caused by one villain in a trench coat. They are caused by several reasonable choices linking arms at exactly the wrong moment. Cloudflare’s postmortem showed just how quickly that can happen and why resilience is not only about preventing bad inputs, but surviving them with dignity when they slip through anyway.

What Incidents Like This Feel Like in Real Life

If you have ever been on call during a serious production incident, the Cloudflare story probably felt less like a shocking revelation and more like a flashback with better branding. First there is confusion. Then there is denial. Then someone says, “Is it just us?” and five dashboards answer, “Absolutely not.” The hardest part in the opening minutes is that the system is broken, but the story of how it is broken has not arrived yet.

That is why the experience of outages like this tends to be so emotionally weird. The technical details are precise, but the human experience is chaos. Alerts multiply. Screenshots fly into chat. People begin speaking in fragments. One team thinks it is storage. Another thinks it is networking. A third is quietly wondering whether lunch is now canceled by destiny.

Cloudflare’s outage had that unmistakable incident-room energy. At first, the symptoms looked like a possible attack. That happens all the time in high-stress response work: the earliest explanation is often the loudest one, not the right one. A weird traffic pattern plus a failing status page? Sure, that smells like malicious activity. Until it doesn’t. Then everybody has to pivot without losing another precious half hour.

There is also a special kind of dread that comes with discovering the failure is internal. An external attack is scary, but psychologically tidy. You have an adversary. You have a story. An internal failure is messier. It means your own trusted process smuggled the problem into production wearing a visitor badge that said, “Hi, I’m normal.”

And then comes the rollback dance, which is less like flipping a switch and more like trying to land a plane during crosswinds while someone keeps reading you contradictory weather updates. You cannot just slam everything backward and hope. You have dependencies, caches, retries, backlogs, and systems that recover at different speeds. Even after the root cause is boxed up and escorted out of the building, the secondary mess keeps moving.

That is why these incidents leave such a mark on engineers. You remember the graphs, yes, but you also remember the feeling of watching assumptions collapse in real time. You remember how one harmless-looking configuration change turned into a full-scale outage. You remember how every “that should never happen” is really just a sentence that has not met the right Tuesday yet.

There is a strange silver lining, though. Good postmortems turn painful mornings into durable knowledge. They replace blame with anatomy. They help teams build better validation, safer rollouts, sharper monitoring, and humbler software. In that sense, the Cloudflare outage was not just a cautionary tale. It was a master class in the kind of resilience work that usually only gets funded after the internet catches fire a little.