Google issues apology, incident report for hours-long cloud outage

Google issues apology, incident report for hours-long cloud outage


Thomas Kurian, CEO of Google Cloud, speaks at a cloud-computing conference held by the company in 2019.

Michael Short | Bloomberg | Getty Images

Google apologized for a major outage that the company said was caused by multiple layers of flawed recent updates.

The company released an incident report late on Friday that explained hours of downtime on Thursday. More than 70 Google cloud services stopped working properly across the globe, knocking down or disrupting dozens of third-party services, including Cloudflare, OpenAI and Shopify. Gmail, Google Calendar, Google Drive, Google Meet and other first-party products also malfunctioned.

“We deeply apologize for the impact this outage has had,” Google wrote in the incident report. “Google Cloud customers and their users trust their businesses to Google, and we will do better. We apologize for the impact this has had not only on our customers’ businesses and their users but also on the trust of our systems. We are committed to making improvements to help avoid outages like this moving forward.”

Thomas Kurian, CEO of Google’s cloud unit, also posted about the outage in an X post on Thursday, saying “we regret the disruption this caused our customers.”

Google in May added a new feature to its “quota policy checks” for evaluating automated incoming requests, but the new feature wasn’t immediately tested in real-world situations, the company wrote in the incident report. As a result, the company’s systems didn’t know how to properly handle data from the new feature, which included blank entries. Those blank entries were then sent out to all Google Cloud data center regions, which prompted the crashes, the company wrote.

Engineers figured out the issue in 10 minutes, according to the company. However, the entire incident went on for seven hours after that, with the crash leading to an overload in some larger regions.

As it released the feature, Google did not use feature flags, an increasingly common industry practice that allows for slow implementation to minimize impact if problems occur. Feature flags would have caught the issue before the feature became widely available, Google said.

Going forward, Google will change its architecture so if one system fails, it can still operate without crashing, the company said. Google said it will also audit all systems and improve its communications “both automated and human, so our customers get the information they need asap to react to issues.” 

— CNBC’s Jordan Novet contributed to this report.

WATCH: Google buyouts highlight tech’s cost-cutting amid AI CapEx boom

Google buyouts highlight tech's cost-cutting amid AI CapEx boom



Source

Inside a Utah desert facility preparing humans for life on Mars
Technology

Inside a Utah desert facility preparing humans for life on Mars

Hidden among the majestic canyons of the Utah desert, about 7 miles from the nearest town, is a small research facility meant to prepare humans for life on Mars. The Mars Society, a nonprofit organization that runs the Mars Desert Research Station, or MDRS, invited CNBC to shadow one of its analog crews on a […]

Read More
Ethereum is powering Wall Street’s future. The crypto scene at Cannes shows how far it’s come
Technology

Ethereum is powering Wall Street’s future. The crypto scene at Cannes shows how far it’s come

CANNES — Wall Street’s new plumbing is being built on Ethereum and this week its architects took over the same French Riviera villas and red carpet venues that host the Cannes Film Festival in May. The Ethereum Community Conference, or EthCC, took over the beachside town that was swarming with crypto founders, developers, and some […]

Read More
Trump inaugural impersonators scammed donors out of crypto, feds say
Technology

Trump inaugural impersonators scammed donors out of crypto, feds say

Sopa Images | Lightrocket | Getty Images Scammers impersonating the President Donald Trump and Vice President JD Vance Inaugural Committee fraudulently stole hundreds of thousands of dollars in cryptocurrency, according to a new complaint filed by federal prosecutors. The perpetrators used phony email addresses made to look like they belonged to the inaugural committee to […]

Read More