How a software update caused one of the world’s biggest IT blackouts

A fault with an update issued by cybersecurity company CrowdStrike led to a cascade effect among global IT systems Friday, with industries ranging from banking to airlines facing outages.

Banks and health care providers saw their services disrupted and TV broadcasters went offline as businesses worldwide grappled with the ongoing outage. Air travel has been hit hard, too, with planes grounded and services delayed.

At the heart of the issue is Texas-based cybersecurity vendor CrowdStrike. On Friday, the cybersecurity firm experienced a major disruption following an issue with a software update.

So what happened, exactly? CNBC takes a look.

CrowdStrike is a cybersecurity vendor that develops software to help companies detect and block hacks. It is used by many of the world’s Fortune 500 companies, including major global banks, healthcare and energy companies.

CrowdStrike is what’s known as an “endpoint security” firm as it uses cloud technology to apply cyber protections to devices that are connected to the internet.

This differs from alternative approaches used by other cyber firms, which involve applying protection directly to backend server systems.

On Friday, people around the world began encountering an error screen known as the “blue screen of death.”

This issue — a common problem among PCs, for example if a machine overheats — was the result of an update from cybersecurity firm CrowdStrike concerning its Falcon product. 

Falcon is a platform developed by the company that’s designed to stop cyber breaches using cloud technology — it is at the heart of the firm’s focus on endpoints. CrowdStrike said Friday it is in the process of rolling back the update globally.

CrowdStrike’s software requires deep access to a computer’s operating system to scan for threats. In the case of Friday’s outage, machines running Microsoft’s Windows operating system crashed due to a fault in the way a software update issued by CrowdStrike interacted with Windows.

“We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD [blue screen of death]) and get stuck in a restarting state. We approximate impact started around 19:00 UTC on the 18th of July,” Microsoft said in an update at 5:40 a.m. ET.

“We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance,” the company added.

Earlier, Microsoft said its cloud services had been restored after an outage that affected its Azure services and Microsoft 365 suite of apps in the central U.S. region. A company spokesperson said these are two different and non-related issues — one issue relates to Azure, the other is linked to CrowdStrike.

They added that they “anticipate a resolution is forthcoming,” in respect to the CrowdStrike problem.

CrowdStrike is “actively working with customers impacted by a defect found in a single content update for Windows hosts,” CEO George Kurtz said Friday in a update on social media platform X. He added that Mac and Linux hosts are not affected.

“This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed,” Kurtz said.

That fix could be hard to implement, though. Andy Grayland, chief information and security officer at threat intelligence firm Silobreaker, said that in order to implement a fix, engineers would have to go into each individual data center running windows.

They’d then have to log in, navigate to a certain CrowdStrike file, delete it, and then reboot the entire system, he said.

“Where machines are encrypted, complex encryption keys also need to be entered manually. Unless Microsoft and CrowdStrike (if they are involved) pull something miraculous out of the bag, this could be painful to recover from.”