CrowdStrike 2024: how one update crashed 8.5 million computers

Nabayn did not work on this incident. This is our analysis of what is on the public record, written so other organisations can learn from it.

On 19 July 2024, computers around the world began crashing to a blue screen. Airlines, hospitals and shops were among those affected. There was no attacker. A security company had shipped a faulty update, and it spread very quickly. It remains one of the clearest examples of how much depends on software most people have never heard of.

What happened

CrowdStrike makes security software called Falcon. On Friday 19 July 2024, at 04:09 UTC, it released a content configuration update for its Windows sensor as part of regular operations. The update resulted in a Windows system crash.¹

The affected systems were Windows computers running sensor version 7.11 and above that were online and received the update between 04:09 and 05:27 UTC, when the faulty update was reverted. Mac and Linux computers were not affected.¹

So the bad update was live for only 78 minutes. That did not matter much, because a computer that had already received it kept crashing. Microsoft estimated that it affected 8.5 million Windows devices, which was less than one percent of all Windows machines. It noted that although the percentage was small, the broad economic and societal impacts reflected the use of CrowdStrike by enterprises running many critical services.²

Why testing did not catch it

CrowdStrike’s preliminary review separates two kinds of update. Its main sensor software goes through extensive testing, followed by a staged rollout, starting with its own staff and then early adopters. Customers can also choose how quickly they take it.¹

The faulty update was a different kind, called Rapid Response Content, designed to reach customers quickly as new threats appear. Its content is checked by a tool called the Content Validator. On 19 July, because of a bug in that validator, one of two new content instances passed validation despite containing problematic data.¹

The recovery

Recovery was slow because the machines needed attention one by one. CrowdStrike’s public hub reported that greater than 97% of Windows sensors were online by 24 July, and about 99% by 29 July, compared with before the update.³

Microsoft said the incident showed the interconnected nature of the technology ecosystem, and was a reminder to prioritise safe deployment and disaster recovery.²

What a small business can learn

You do not run global infrastructure, but you do depend on suppliers, and you may build or commission software of your own. These are our suggestions.

Roll out changes gradually where you can. If you have several computers, or several sites, do not update them all at the same moment for anything critical. Try one first. Many tools have an option for this.

Keep updating anyway. It would be the wrong lesson to turn updates off. Security updates fix real problems, and being far behind is a much more common risk.

Plan for your main tool being down. Could you take a booking or a payment if your computers would not start this morning? Keep a paper fallback, and a contact list you can reach from a phone.

Keep recovery information somewhere else. Recovery keys and support numbers do not help if they are stored only on the machine that will not start.

Ask how your suppliers release changes. For anything critical, ask whether they test and stage their updates, and how quickly they can undo one.

If you build software, test the tests. The validator itself had a bug here. Automated checks are essential, but they are not proof. Release to a small group first, and make sure you can undo the change quickly. We apply this to our own work, with checks that block a release and a private preview before anything goes live.

For wider thinking on dependence on other people’s software, see our analyses of SolarWinds and Log4Shell.