Facebook outage: What we know so far

According to Down Detector, thousands of users reported that they could not use any of Facebook Inc’s services just after 17:00 on October 4. Facebook, Instagram, and WhatsApp users were left high and dry.

Chief technology officer Mike Schroepfer tweeted that the company was experiencing network issues that led to the outage.

*Sincere* apologies to everyone impacted by outages of Facebook powered services right now. We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible

— Mike Schroepfer (@schrep) October 4, 2021

Facebook’s services were done for almost six hours. This morning (October 5), Facebook released a statement with more details of what had happened.

“Our engineering teams have learned that configuration changes on the backbone routers that co-ordinate network traffic between our data centres caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt,” the statement reads.

What made this outage even more catastrophic was the fact that Facebook’s internal tools and systems, used by employees, were down too – making it more difficult for staff to resolve the issue.

As of this morning, all of Facebook’s services appeared to be functioning as normal.

We’re now back and running at 100%.

💚 Thank you to everyone around the world today for your patience while our teams worked diligently to restore WhatsApp. We truly appreciate you and continue to be humbled by how much people and organizations rely on our app every day. 💚

— WhatsApp (@WhatsApp) October 5, 2021

The root cause of the outage appears to be a broken border gateway protocol (BGP) route update that was applied by engineering teams earlier in the day.

Simply put, this means that the servers that run the applications became unreachable when this update was applied.

Cloudflare vice president Dane Knecht took to Twitter to give an update on what was experienced.

. @Facebook DNS and other services are down. It appears their BGP routes have been withdrawn from the internet. @Cloudflare 1.1.1.1 started seeing high failure in last 20mins.

— Dane Knecht (@dok2001) October 4, 2021

For a more in-depth look into BGP, click here. 

“People and businesses around the world rely on us every day to stay connected. We understand the impact outages like these have on people’s lives, and our responsibility to keep people informed about disruptions to our services. We apologise to all those affected, and we’re working to understand more about what happened today so we can continue to make our infrastructure more resilient,” Facebook Inc’s statements reads.

Twitter upsurge

With the three popular social media apps down, many users flocked to Twitter to see what the problem was, thinking the cause was their own connection or WiFi. However, everyone was quick to learn that it was a global problem.

Facebook used Twitter to keep its users updated.

To the huge community of people and businesses around the world who depend on us: we're sorry. We’ve been working hard to restore access to our apps and services and are happy to report they are coming back online now. Thank you for bearing with us.

— Meta (@Meta) October 4, 2021

This shone the spotlight firmly on Twitter.

https://twitter.com/Twitter/status/1445078208190291973?s=20

Billions of dollars lost

Facebook makes billions of dollars each year on ad revenue, and even a minute of outage can be catastrophic for the company.

As a result of the outage, it is estimated that around $6b (R90b) of Mark Zuckerberg’s personal wealth was lost.

@livingwithwill asked how much money was being lost by advertisers who can't run ads. (Opportunity cost.)

According to Nielsen, average return on ad spend is 2.87:1, so that would make it $299,000/minute ($17,950,000/hour).

— 👽 ♫ I'm dopé, I hope you're dopé, too ♫ 👽 (@WhatDope) October 5, 2021

Facebook Inc’s stock fell 4.9% yesterday, as a result of both the outage and damaging claims by a whistleblower. 

The whistleblower and the Facebook Files

Frances Haugen, a former Facebook employee, alleged in an interview the day before the outage (October 3) that the company is unable to properly deal with hate and misinformation on its platform – specifically in the lead up to the 2020 US elections.

Haugen has leaked internal documents, called ‘The Facebook Files’, which detail how the company focuses on growth and its own interests, rather than the public good. Facebook has denied these allegations.

Watch her interview here:

Today (October 5), Haugen is set to testify at a hearing convened by the Senate Commerce consumer protection panel.