Slashdot Mirror
Software Bug Behind Biggest Telephony Outage In US History (bleepingcomputer.com)
An anonymous reader writes: A software bug in a telecom provider's phone number blacklisting system caused the largest telephony outage in US history, according to a report released by the US Federal Communications Commission (FCC) at the start of the month. The telco is Level 3, now part of CenturyLink, and the outage took place on October 4, 2016.
According to the FCC's investigation, the outage began after a Level 3 employee entered phone numbers suspected of malicious activity in the company's network management software. The employee wanted to block incoming phone calls from these numbers and had entered each number in fields provided by the software's GUI. The problem arose when the Level 3 technician left a field empty, without entering a number. Unbeknownst to the employee, the buggy software didn't ignore the empty field, like most software does, but instead viewed the empty space as a "wildcard" character. As soon as the technician submitted his input, Level 3's network began blocking all incoming and outgoing telephone calls — over 111 million in total.
According to the FCC's investigation, the outage began after a Level 3 employee entered phone numbers suspected of malicious activity in the company's network management software. The employee wanted to block incoming phone calls from these numbers and had entered each number in fields provided by the software's GUI. The problem arose when the Level 3 technician left a field empty, without entering a number. Unbeknownst to the employee, the buggy software didn't ignore the empty field, like most software does, but instead viewed the empty space as a "wildcard" character. As soon as the technician submitted his input, Level 3's network began blocking all incoming and outgoing telephone calls — over 111 million in total.
If you want to route all 212 area code numbers to a specific carrier you can just enter '212' and it will route them. If you want go do a NPA-NXX, just enter '212555'. Since it's longest match it will also work for a 'thousands block' (ie, 2125551) and even down to the individual number (2125551212). US numbers don't mean a whole lot, but in other countries they specify specific geographic regions, carriers or number types. The backend database takes longest match for the most flexibility and the EMS UI is nothing more than a glorified frontend directly to the DB. There's little business logic actually protecting you.
In a lot of cases, you want a wildcard match. I route a number of prefixes to different carriers with longer matches but I have a blank entry to default fall back directly to Level3 if I don't have any other carriers to handle calls.
Everyone who uses Sonus knows this is how it works. It sounds like they gave a task to someone and only trained them on one piece of data entry. The fact that 800 people had access to this highly specialized software without higher level tooling that adds in the required business logic is the terrifying piece.
I may well be missing something, but I'm still not seeing why this scheme provides any benefit over one where you explicitly ask for a wildcard if that's what you want
Phone numbers are hierarchical and variable length (as opposed to e.g. IP addresses which are fixed length). a) The most common mistake to make is to route only a particular number or set of numbers as opposed to the hierarchy - using the shortest match by default avoids this mistake b) the routing algorithm used normally also works on a hierarchy, so a wildcard match apart from the end of the number can be very costly and unwise c) that's the way it's "always" been done and so doing something different would be conusing.