For more than three hours on the night of Nov. 1, Verizon users in the greater Burlington area were without service. The Burlington fire department lost a line of communication. Police had some digital devices knocked offline. And anyone trying to reach 911 couldn’t get through.
Verizon customers lost service because 26 of the company’s cell towers in the area went down. State officials say that was caused, in part, by a stretch of FirstLight’s optical fiber being cut when a bridge — along with a pipe protecting utility lines — was washed out by floodwaters in Muddy Brook, between Williston and South Burlington.
But one cut fiber isn’t supposed to stop service. Verizon, like any modern telecommunications provider, would ideally have “resilient rings” throughout its network, so it can continue to provide service in the case of fiber cuts — often caused by falling trees or utility poles — by redirecting digital traffic in the other direction.
“The importance of maintaining resilient communications during and after a disaster cannot be overstated,” an expert panel wrote in a report to the Federal Communications Commission just last week. “Sometimes, the ability to communicate during and after a disaster is a matter of life and death.”
The Halloween storm, which caused some $6 million to $10 million in damage in the state, wasn’t a “disaster” by FCC standards, but it does offer a test for how Vermont’s telecommunications networks might hold up if things were worse.
So what happened to Verizon’s network in its most populated service area?
Verizon confirmed that “fiber belonging to a third party wireline provider” was cut — apparently referring to the Muddy Brook washout — and said service was rerouted to a secondary route.
“While in the process of repairing the damaged fiber, the other side of the ring went down and disrupted service to customers for over 3 hours in the Burlington, Williston and Colchester area,” David Weissmann, a spokesperson for Verizon, said in an email.
What caused the other side of the ring to go down?
Weissmann did not respond to follow-up questions. FirstLight, which provides fiber “backhaul” connecting Verizon’s towers to its main network, did not respond to an email seeking comment.
Whatever happened to FirstLight’s network knocked out more than just Verizon towers in the Burlington area.
John Quinn, secretary of the Agency of Digital Services, said state offices in Middlebury also experienced outages because of the FirstLight fiber break. The state has a $3.5 million contract with FirstLight to provide connectivity to its data centers, and FirstLight’s network will be used for the state’s E911 system for the next five years.
Quinn pointed to the Muddy Brook fiber cut as the cause of the outage. He did not respond to an email asking if he knew what else happened that caused FirstLight to lose redundancy.
Clay Purvis, director of telecommunications at Vermont’s Department of Public Service, said he suspected the second event was also a problem with FirstLight’s equipment.
FirstLight sent an outage report to DPS on Nov. 6 saying the E911 system was not effected by the outage. It said the primary event was the cables being washed out at 9:11 a.m. on Nov. 2. Then, about six hours later, it says: “Secondary event at 15:00 hours on existing fiber not cut once washout severed remaining fibers in duct.”
Burlington police sent out an email at 4:46 p.m. Nov. 1 saying Verizon service had been down for more than an hour.
Purvis said he wasn’t sure what to make of FirstLight’s explanation, but that Verizon officials said they are looking to gather more information about it.
He added that he was somewhat surprised that Verizon relied on only one service provider to keep its towers connected. “I’m questioning why this particular fiber cut took out 26 towers,” he said. “It’s not a bad idea to purchase backhaul from more than one provider.”
But “outages happen,” he added. Purvis said he was more concerned about Verizon failing to notify relevant state agencies about the outage, which left users unable to place emergency calls in the midst of a major storm.
“My understanding is nobody from the company reached out to the 911 board despite the fact it was a widespread event,” he said. Barb Neal, executive director of the E911 Board, said early last week that it had not received any reports on outages during the Halloween storm.
Vermont’s currently expects wireline telephone operators (such as Consolidated Communications) to report incidents that prevent their users from making emergency calls. However, the state has never implemented rules that require reporting from landline providers, cable companies (like Comcast) or wireless operators (AT&T, Verizon, T-Mobile) who provide telephone and voice services.
Neal said the board is drafting rules that would apply to cable companies, who have taken much of the residential and business telephone market from traditional landline providers in recent years, mostly by bundling VOIP with television and internet.
However, those rules would not apply to Verizon, or other wireless providers, which are largely exempt from state regulation. In the case of major outages, they are required to send reports to the FCC, which then forwards them to the impacted state.
That means that wireless companies can decide how forthcoming they want to be with state regulators, and their customers, especially when outages don’t reach the FCC reporting threshold.
Weissmann of Verizon said the company “is refining our system to provide notice of outages to Vermont state, local and public safety agencies” after the post-Halloween incident.
Correction: A previous version of this story said the Verizon outage occurred on Halloween night. It occurred on Nov. 1 due to a storm that began on Halloween night. And although the state expects landline providers to submit outage reports, there are no rules requiring them to do so.
