AURORA, Ill. – The urgent email to members of the Central Station Alarm Association could hardly begin to explain the dire straits that had befallen Alarm Detection Systems.
In the correspondence sent Oct. 19, CSAA Executive Director Jay Hauhn wrote that ADS had an immediate need for “thousands” of AES-IntelliNet radio transceivers due to a failure of one of the multiple mesh networks operated by the regional security company. If they had any of the modules to spare, members were asked to overnight them to ADS‘ Aurora, Ill., headquarters. Stat.
ADS had already been contending with the failure of its network for 10 days when the rally cry went out. All told, roughly 10,500 customers were affected, with no means to monitor the commercial and residential subscriber accounts throughout the greater Chicagoland area and Kenosha County in Wisconsin.
“After about one week of the network being down, I looked around at all the employees. Everyone is stressed and exhausted. Nobody is getting enough sleep, including the owners, who are fully immersed in the seriousness of having more than 10,000 fire and burglar alarms failing to report,” says Nick Bonifas, corporate counsel for ADS. “And I realized that for all of our company efforts, we had only restored 4% of the network.”
Corrupt Data Packet Found to Be Cause of Network Failure
The first indication that something was awry struck on Oct. 9. The network did not fail all at once. Instead, Bonifas explains, “it took days for the issues to progress.” AES Corp. engineers were quickly dispatched to Aurora to help ADS troubleshoot the reason for the alarm signal disruption.
It was determined the network was being engulfed by a tsunami of data brought there by a corrupt data packet. The rogue packet closely resembled a normal, transmittable data packet, but with part of the code edited in such a way that restricted alarm signal transmission.
Bonifas says the genesis of the bad packet has yet to be determined and remains under investigation. (SSI continues to report the story and will provide updates when further information becomes available.)
“That packet created an echo. It is not being received or processed by the IP link to pass off on to alarm automation,” says AES Director of Technical Support Lee Dubé, who continues to work closely with ADS. “Alarm signals do make it through. However, we see some latency in that processing simply because of the amount of traffic occurring in the RF network due to that corruption.”
Inside the IP link, itself an aggregator of the RF signals, is the smallish transceiver, which demodulates the information, processes it and converts into IP traffic. To make each affected subscriber whole again will require a truck roll to swap in a new transceiver or a newly reprogrammed used unit.
Customers impacted by the outage will be receiving a credit for the signaling interruption, according to ADS. And no customer will be paying for monitoring services that they did not receive. The company declined to provide an estimated dollar amount in cost to its bottom line.
“Right now, we still have about 700 customers still affected by the outage. So, we are still in a crisis. Down the road, we’ll work the calculators to see what happened financially,” Bonifas says.
ADS has been heavily supported by AES throughout the event. The company has sent many of its employees to the Aurora facility, shipping around 6,300 transceivers from overseas, among other support. AES Corp. CEO & President Bill Kieckhafer says the company has installed more than 600,000 of the same radio units worldwide in the past 20 years. And it has never seen this problem before.
“This is truly one of a kind, but ADS is in total control. They are a very professional organization,” Kieckhafer says. “They have a very specific plan because that is what we are supporting. We have made tremendous progress.”
Bonifas says from the onset of the event the company has remained singularly focused on its customers’ well-being and expediting the return of full monitoring services as quickly as possible. The ordeal is considered to be the most difficult challenge ADS has encountered in its 48 years in business. Not only has the company contended with the logistics and execution of a plan to restore 10,500 customers to service, Bonifas says, but there is an emotional drain that is hard to describe.
“Somehow our company dug deep to find the resolve to restore the network,” he says. “Once ADS gets service restored, we’ll be able to better share our experiences from this situation. I think the industry will be very interested to hear what we learned.”
UPDATE: AES Tells How ADS’ Mesh Network Failed
