Slashdot Mirror
Software Bug Behind Biggest Telephony Outage In US History (bleepingcomputer.com)
An anonymous reader writes: A software bug in a telecom provider's phone number blacklisting system caused the largest telephony outage in US history, according to a report released by the US Federal Communications Commission (FCC) at the start of the month. The telco is Level 3, now part of CenturyLink, and the outage took place on October 4, 2016.
According to the FCC's investigation, the outage began after a Level 3 employee entered phone numbers suspected of malicious activity in the company's network management software. The employee wanted to block incoming phone calls from these numbers and had entered each number in fields provided by the software's GUI. The problem arose when the Level 3 technician left a field empty, without entering a number. Unbeknownst to the employee, the buggy software didn't ignore the empty field, like most software does, but instead viewed the empty space as a "wildcard" character. As soon as the technician submitted his input, Level 3's network began blocking all incoming and outgoing telephone calls — over 111 million in total.
According to the FCC's investigation, the outage began after a Level 3 employee entered phone numbers suspected of malicious activity in the company's network management software. The employee wanted to block incoming phone calls from these numbers and had entered each number in fields provided by the software's GUI. The problem arose when the Level 3 technician left a field empty, without entering a number. Unbeknownst to the employee, the buggy software didn't ignore the empty field, like most software does, but instead viewed the empty space as a "wildcard" character. As soon as the technician submitted his input, Level 3's network began blocking all incoming and outgoing telephone calls — over 111 million in total.
Check the spec - perhaps it was by design or not called out to ignore empty entries?
A null/blank input taken as a wildcard is certainly not a feature.
Even labeling that as a mere bug is putting it mildly. More like gargantuan fuck-up.
If you want to route all 212 area code numbers to a specific carrier you can just enter '212' and it will route them. If you want go do a NPA-NXX, just enter '212555'. Since it's longest match it will also work for a 'thousands block' (ie, 2125551) and even down to the individual number (2125551212). US numbers don't mean a whole lot, but in other countries they specify specific geographic regions, carriers or number types. The backend database takes longest match for the most flexibility and the EMS UI is nothing more than a glorified frontend directly to the DB. There's little business logic actually protecting you.
In a lot of cases, you want a wildcard match. I route a number of prefixes to different carriers with longer matches but I have a blank entry to default fall back directly to Level3 if I don't have any other carriers to handle calls.
Everyone who uses Sonus knows this is how it works. It sounds like they gave a task to someone and only trained them on one piece of data entry. The fact that 800 people had access to this highly specialized software without higher level tooling that adds in the required business logic is the terrifying piece.
Ha ha. No, this is not just a bug. The fuckup goes much deeper than that. "An empty field acts as a wildcard" is the least of your problems. It may or may not be expected behaviour for a GUI. "Not finding it during testing" is par for the course for GUIs for this sort of thing. You're not supposed to give wrong input, even accidentally!
The real problem is thinking a GUI is appropriate to feed lists of boring numbers through. By hand, no less. It's way too easy to accidentally leave a field empty or --if it's a micro-managing form like windows IP address entry type things-- copy part of a numer in the wrong line, shift it a sub-field, or something else similarly silly.
What we have here is a mismatch between user interface and purpose, cooked up without thinking. This is the same mode that makes users stupid, but now it was the designer who wasn't thinking. The focus was on "getting some input fields done", not on "how will this be used and what might the consequences be?" The deeper problem is TFIing such lists. GUIs are entirely stupid for this.
Compare "here, have a GUI" with this sequence: Check the list then feed it to the system as a textfile. It gets queued. Then check the list as it appears in the system against your original list. The system probably should make explicit just what it will do with each entry, like "block one number" or "block a range of numers". Possibly have someone else look over the proposed actions. THEN activate it.
So the problem is that the workflow is entirely too stupid to live. And it was shaped into that form by a GUI.
If it says "filter by" and you enter nothing you're saying filter by nothing, i.e. don't filter, i.e. give me everything.
Plenty of software works like that. Otherwise the user is going to have to enter * in 47 different fields.
Now if there's a minimum number of selections (filter by at least one of foo, bar and froblgobl) that should be enforced somewhere in the software, twice.
It was probably created by one of these full-stack unicorns I keep hearing about.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
In 1987 I had just taken a job at the local Telco and was hitting a steep learning curve. My experience to that point had been PC computers and networks, assembler, CBASIC dBase and the like. This was an IBM System/38 and their billing software used RPG/III, which was a real structured language unlike its spaghetti-GOTO RPG/II cousin, but aspects were still position sensitive and opcodes were silly-simple compared to languages with which I was familiar. It was more like assembler than anything else. Most data flows consisted of running commands that generated a relational input stream sort of like an SQL query, through simple RPG programs.
We had just installed an ITT 1210 switch and ITT had sent over a block of sample RPG code demonstrating how to parse the various fields and flags appearing on call tapes. My boss provided specs for the internal call ticket system they were using and the simple (!) task was to write a shim that generated a batch of call tickets from each tape. Pretty straightforward, tedious without being intricate. But one part of their code slapped me across the face when I examined it.
The tape recorded end time and call duration in whole seconds, call start time would need to be calculated. They had supplied a routine to do this but it didn't make any sense because I could see no modulo 60 arithmetic in it, they were applying the simple RPG subtraction opcode on the zoned fields. I spent the most mystified HOUR of my LIFE searching the language manuals for that surely described RPG's 'magic' ops for manipulating times and dates, which I assumed had to be there because IBM is GREAT and I am STUPID... finding none. Forced to conclude that I was looking at concept code that was dashed off hurriedly in two minutes I confronted my boss with it (and my solution) but it was a hard sell at first, because my boss was incredulous too.
<blink>down the rabbit hole</blink>