Last Thursday CenturyLink, the nation’s third largest telecommunications company, announced that it was experiencing phone and internet services technical difficulties, and expected that the disruptions would be fixed in four hours.
However, it wasn’t until Friday morning that most outages were resolved and the FCC, prompted by Commissioner Jessica Rosenworcel, has launched an investigation of CenturyLink to identify the problem that had hundreds of thousands of landline customers unable to reach 911 emergency services in Texas, California, Oregon, Colorado, Washington, New Mexico, Idaho, Montana, Missouri, Arizona and Massachusetts, and possibly other states.
FCC Chairman Ajit Pai called the outage “completely unacceptable” because people who needed help couldn’t use the emergency number.
“Its breadth and duration are particularly troubling,” he said.
In addition to disrupting 911 services, the outage also caused outages of Verizon network services in at least two states, New Mexico and Montana
CenturyLink hasn’t provided a public statement regarding the cause of the outages, but investigative journalist Brian Krebs provided in a Tweet an event summary from one of CenturyLink’s core customers.
“Root Cause: A CenturyLink network management card in Denver, CO was propagating invalid frame packets across devices.”
“Fix Action: To restore services the card in Denver was removed from the equipment, secondary communication channel tunnels between specific devices were removed across the network, and a polling filter was applied to adjust the way the packets were received in the equipment. As repair actions were underway, it because apparent that additional restoration steps were required for certain nodes, which included either line card resets of Field Operations dispatches for local equipment login. Once completed, all services restored,” CenturyLink wrote in a trouble ticket.
Due to a similar six-hour outage blamed on CenturyLink in April 2014, more than 10 million people in Washington, Minnesota and North Carolina were unable to make calls to 911, according to the FCC, which ordered the company to pay $16 million to resolve the matter.
The FCC said that CenturyLink was required to timely notify all affected Public Safety Answering Points (PSAP) of the April 2014 outage, but the company failed to do so.
It is not know if CenturyLink contacted PSAPs after learning of the outages beginning Thursday. A number of state regulators said they too will be investigating CenturyLink’s outages.
The FCC fined AT&T $5.25 million for two nationwide outages in March and May 2017 that lasted a total of six hours and resulted in the failure of 15,200 failed 911 calls.