Slashdot Mirror
The Inside Story On the San Francisco Network Hijacking
snydeq writes "A source with direct knowledge of San Francisco's IT infrastructure has tipped off Paul Venezia to the real story behind Terry Childs' lockout of San Francisco's network, providing a detailed account of the city's FiberWAN, interdepartmental politics, and Terry Childs himself. Childs pleaded not guilty to charges of tampering yesterday and is being held on $5 million bail. According to the source, Childs' purview was limited to the city's FiberWAN — a network he himself built and, believing no one competent enough to touch the network but himself, guarded religiously, sharing details with no one, including routing configuration and log-in information. Childs was so concerned about the network's security that he refused even to write router and switch configurations to flash. But what may prove difficult for the prosecution in its case against Childs is that his restricted access to the network was widely known and accepted among managers and the city's other network engineers. Venezia, who has been suspicious of the official story from the start, suspects that the Childs case may be that 'of an overprotective admin who believed he was protecting the network — and by extension, the city — from other administrators whom he considered inferior, and perhaps even dangerous.' Further evidence is that fact that the network, from what Venezia understands, has been running smoothly since Childs' arrest."
So instead of letting the air out of the car's tires, a car he loved, he simply wouldn't give the keys to dangerous drivers.
"He's using a quantum encryption scheme! That'll take hours to break!"
You can try and defend him and glorify him all you want... but as a professional system administrator he should have known that his singular access and pathological behavior was more dangerous than helpful.
What if, instead of being fired he was the victim of an accident or crime? What if he had a health problem? What if a serious, life threatening issue came up (say, you know, an earthquake) that caused the system to be unstable and, at the same time, prevented him from getting there to fix things?
He's still a criminal. But, he's not alone in his behaviour. Whoever his managers are sound to be guilty of criminal negligence. This never should have been possible in a city government the size of San Francisco. Especially when it comes to critical infrastructure. If I were a citizen of San Fran I'd be asking why heads aren't rolling at the highest levels. Why was this allowed to happen? In San Francisco, where you think they'd have no problem finding competent replacements.
Absolutely mind boggling.
short version: if you bad to computers, we bad to you!
I find that easy to believe. Even easier to believe that they didn't know this was the case, or knew but did not understand.
so the network is NOT locked up, it's just unrestoreble after "password recovery."
sounds like what they need to do is get some qualified engineers to redesign it, and when it's on paper, pull the plug on everything, and reconfigure from scratch.
because if it isn't saved in flash, it's going away as soon as the power light goes out.
which makes our jailed genius a little less than blazing fast. in fact, about half fast. parts of the system ARE going to go down. it's the nature of the beast. no records, no writes... the first time the janitor plugs in a 18-amp vacuum in a rack, it's gone.
they'll come along and take his Cisco cert away for not saving the configs, if for nothing else.
if this is supposed to be a new economy, how come they still want my old fashioned money?
IANAL, but isn't $5 million US for bail a bit excessive for this?
That's my first reaction to the news. Critical infrastructure should have redundancy everywhere, including the support staff.
To give a stupid but obvious example what if Childs was run over by a car? OK, he wouldn't care but all the rest of SF would.
So they should never have put the network online until the information was in several places (the brains of several people if formal electronic/paper records were too inflexible).
Stll, this sounds like political infighting more than ever. Given the situation why were they trying to fire a critical person like Childs? Sounds like some bureaucrat with an ego as big as Childs would be involved to cause this, rather than Childs "going rogue". And he (the bureaucrat) was more skilled in the political game. Of course this person would be covering his tracks, and not be obvious in any way. So Childs and the whole of SF lost. His firing does not make sense otherwise, given his critical position.
Ah, the fun of weaving conspiracy theories :-)
Every software company I have worked for... if one or two people were hit by a bus... the company would be out-of-business. Management knew this... fellow developers knew it. Its a commonplace thing. Engineers take the work so *personally*. "No one can touch that code but me... " blah... blah. Ånd the stupid management goes along w/ these primadonna's. Of course... if they demanded more money... they'd be gone in a NY minute.
He's certainly guilty of being a bad employee, as well as affirming all of those user-unfriendly IT sterotypes (those are often true, BTW). But criminal?
In America, they have to prove that first. Looking at the statute, it seems it all comes down to the issue of "without permission." The main point the article makes is that he might have had at least understood or standing permission to do most or all of what he did. Just like when you take your parents' car somewhere as a teenager, it isn't theft if it's understood that you are allowed to use it.
The article is one-sided, and his alleged refusal to give up the passwords looks bad (perhaps he is remaining silent until he speaks with counsel), but proving he didn't have permission might be hard. Ergo, no criminal.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
Every time I see a situation like this, I have to wonder what would happen if an "indispensable" person got hit by a bus. It strikes me that Childs was using his absolute control of the network as a way to put the fear of god in others within the department while attaining more prestige and autonomy than he deserved. The fact that Childs locked everyone out of the system after apparently receiving a poor job assessment backs that up. Sooner or later, the IT department had to take action to strip his stranglehold of the network, especially if he was on the verge of burnout or increasingly difficult to deal with.
I suspect that no one had the interpersonal wherewithal to figure out how to approach him in a non-confrontational manner. The best approach would have been to find someone who Childs respected who could share the load and provide backup and support while the organization attempted to deal with an overly possessive employee who is behaving irrationally.
People who fiddle with government machines get let of and win people elections! Those that STOP people fiddling with Machines get locked up on $5 mill bail....:D:D
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
I know someone who worked on the cisco side with this guy. This had been going on for a while. The dude was threatening co-workers doing all kinds of odd stuff. The idea that he was somehow just a little protective is an off the charts miss-representation.
Never worked for the government, have you? ;)
Management is where people who are too incompetent for technical work go. No one gets fired, they get moved to different departments. As a last resort, they get assigned to 'special projects' for about a year in the hopes that everyone will forget what an imbecile they are, and will be safe to move back into the management structure.
funny I find it VERY easy to believe. Right now only 3 people in my own district now the running of the network, and only 1 by extension of that the complete configuration of the OS X server running the mac portion of the district. I have a emergency recovery manual I wrote myself, but it is under lock and key by me to keep all but 2 people from knowing it because I KNOW the other techs and administrators are incompetent political appointees who will royally screw things up and cause much more damage than they solve if they try to implement it without know what is going on.
"Slashdot, where telling the truth is overrated but lying is insightful."
It's nice to believe that but, to abuse an oft-quoted phrase, quis sacko ipsos pointyhaires?
Before you can fire someone for being a complete idiot, you have to not be totally out to lunch yourself. More importantly you have to possess evidence to back up your decision which is at least strong enough to outweigh the political costs of making it.
If you think this all sounds like a load of crap, then consider yourself lucky that you have never been in the middle of it.
It seems pretty idiotic to me. I still think they should throw this guy in the clink, but at the same time, I think some of his superiors should be told to collect their belongings and then have security escort them through the front door, because there was a colossal breakdown of management here if a single guy was permitted to basically hold the entire network's architecture in his head.
The world's burning. Moped Jesus spotted on I50. Details at 11.
Or you write them a glowing recommendation and help them get promoted out.
New! Device Legs: These legs will help your poor OEM installed product escape any hamfistedness it may encounter. Ava
If the others were so stupid as to not do anything about this waaaaayyyyy before, then maybe, just maaayyyybe he was right. They are too stupid to be let loose on the network. :-D
-- I ignore anonymous replies to my comments and postings.
I get a little tired with the "hit by a bus" example. My coworkers use it all the time as an excuse to make me document everything to the Nth degree.
Maybe they could suggest "crushed in an orgy" or "broke lightspeed and turned to photons". Getting hit by a bus is such a boring way to go.
This analogy is spot on, and whoever modded it off-topic obviously is incapable of understanding the topic and shouldn't have had the keys to the mod-car in the first place.
Infuriate left and right
Open source does not equal open data.
.sig withheld by request
In my experience, it's a rare company indeed whose managers can fathom the implications of a situation like this. In general, I'm unable to get management to even understand Rule Zero of system administration. Which is: Do everything you need to do to be drop dead certain that you always have a reasonable backup of your important systems. This doesn't sound too difficult, but in practice it's difficult to convince managers that an event that could happen with probability == 0.01 could ever happen...
"Not an actor, but he plays one on TV."
.
modded +3, Informative.
but this attitude sets off alarms.
exposing a geek who despises his supervisors and is used to thinking of the server rooms as his personal playground.
haha..probably right on..I came from the outside, now I work in county govt going on 7 years. I have more knowledge in my little finger than 99 net-sys admins / network engineers picked at random who are working inside the county govt. CLUELESS! No excuse for getting yourself arrested thou... maybe not too late for "I forgot the password" as a defense. Any ideas how to clean up these laggards? Its our freekin tax money down the drain!!
I post AC because of my position, which is basically a guy who was hired as the second network tech to help manage the network for a sizeable city (large enough that we host several professional sports teams). I had no real qualifications other than knowing how to google my way out of most basic computer situations. My supervisor managed all City-owned Cisco equipment and it has only been 2 of us for 2 years. We manage over 300 Cisco devices at over 100 sites and I can honestly say that after reading a few more details on this story, I can easily understand how this can happen in a local government. I believe that the problem is in management. We have similar problems in our City regarding the lack of passing of knowledge and lack of staffing, but we have a good security team that knows more about Cisco networks than the 2 of us that regularly work on the Cisco equipment in our City. They are not normally watching our backs (that we know of) but they would certainly do so if they got a bad vibe about us. We have to share passwords with them and they have as much access to our equipment as we do. It is simply a requirement in a publicly owned system that knowledge is shared. Taxpayers have payed for the equipment and expect that there are not single points of failure. There are many reasons that more people than work on one thing on a regular basis have knowledge of and access to the most basic systems. If there was no redundancy, then it is a fundamental failure of management.....I'm not saying the guy should have set one password and not passed it on.....but I understand.
I find the situation startling familiar. It's downright creepy to read this scenario. Back in the late 90s I was the sysadmin of a moderately sized ISP. When we started out I was one of three network engineers hired to build the ISP; eventually I ended up in 'charge' of the system. Like the article I also was very protective of my network, and as paranoid as this individual is made out to be. Granted I was in my 20s and suitably arrogant to boot, more on this in a moment. As time went on first one, then the other guy quit after working 80 hours a week without the possibility of time off...things only got worse as people quit. When it was down to me I made sure the owners knew the passwords to everything, but they lacked any knowledge of how to do anything. This came back to haunt me later as you'll see. Eventually I too got fed up and went to work for another company that wasn't a direct competitor. Before I left I advised management on changing all passwords for both of our sakes. I tried to explain everything but nobody understood the technical aspects. Two months later I got a visit from the FBI. 8 grueling hours of interrogation later from armed men I found out that the entire network had crashed, and I was under suspicion as having remotely logged in and crashing their system. It wasn't until later I found out they never hired a replacement, and my system simply collapsed due to lack of maintenance. It's easy to be painted out as the bad guy when you intimately know the network while being managed by a bunch of clueless twits. I don't know if that's the case in this guy's case, but I can see it working either way.
Consider mentoring. The God complex management style rarely works out well in the end.