The prolonged global shutdown of Facebook’s apps prompted the company’s engineers to seek to fix the problem at one of its data centers in California.
The outage disrupted all Facebook apps, including WhatsApp, Instagram and Messenger, globally.
This affected billions of users and millions of advertisers. The outage also shut down nearly all of the company’s internal systems that employees use to communicate and work.
Several employees explained that they resorted to speaking through Outlook email accounts provided by work because Facebook is essentially working on an internal version of the social network that is not currently accessible.
And while employees can send emails to each other. But they cannot send or receive email messages from external addresses.
And since the company requires employees to sign in with their work accounts to access tools like Google Docs and Zoom, these services also don’t work. This prompted some employees to use alternative services such as FaceTime and Discord.
The company’s engineers were sent to a major US data center in California for restoration, which means that the repair cannot be done remotely.
Complicating matters further, the outage temporarily disrupted some employees’ ability to access company buildings and conference rooms. According to for newspaper The New York Times, which first reported that engineers were sent to the data center.
Read also: Facebook services are back after a six-hour outage
Facebook engineers had to visit the California data center in person
In an email to employees sent shortly after the service was restored, Chief Technology Officer Mike Schroepfer said the issue “affects our networking backbone that connects all our data centers together.
He cautioned in the note: If you are not actively working on the restore, please be patient and do not rush to reload everything to prevent slowing down the network restore.
The company did not provide a detailed explanation of the outage. This is despite the fact that outside experts say it was due to a problem with BGP networking technology, or Border Gate Protocol.
Santosh Janardan, Vice President of Infrastructure at Facebook, posted: post It says the outage was the result of a configuration error change. He added that the company had no evidence that user data had been compromised as a result of this downtime.
“Our engineering teams learned that configuration changes for the core routers that coordinate network traffic between our data centers caused issues that led to this outage,” Janardan wrote. This disruption to network traffic has had a ripple effect on the way our data centers communicate, bringing our services to a halt.
Read also: WhatsApp, Facebook and Instagram are down globally for hours