2024Non-malicious outage
CrowdStrike Falcon update outage
19 July 2024
A defective content configuration update (Channel File 291) for CrowdStrike's Falcon sensor caused Windows systems that received it to crash and fail to restart normally. Many machines needed manual recovery. It is included here because it is a landmark availability incident, not because anyone attacked anything.
- Victim
- Windows hosts running the CrowdStrike Falcon sensor worldwide
- Sector
- Cross-sector (airlines, healthcare, banking, media and others)
- Public attribution
- Not a cyber attack. CrowdStrike said the outage was caused by a defective content update and was not a cyberattack.
Impact
Microsoft estimated that about 8.5 million Windows devices were affected.
Root cause
A content update that was not caught by validation before it was deployed to all customers at once, triggering a fault in software running with kernel privileges (CrowdStrike root cause analysis).
Lessons learned: the controls it teaches
- Stage updates, including security content updates, in rings with canary deployments.
- Business continuity plans must cover non-malicious outages and mass manual recovery, including access to encryption recovery keys.
- Understand the operational risk of software that runs with kernel privileges.