Slashdot Mirror
The Inside Story On the San Francisco Network Hijacking
snydeq writes "A source with direct knowledge of San Francisco's IT infrastructure has tipped off Paul Venezia to the real story behind Terry Childs' lockout of San Francisco's network, providing a detailed account of the city's FiberWAN, interdepartmental politics, and Terry Childs himself. Childs pleaded not guilty to charges of tampering yesterday and is being held on $5 million bail. According to the source, Childs' purview was limited to the city's FiberWAN — a network he himself built and, believing no one competent enough to touch the network but himself, guarded religiously, sharing details with no one, including routing configuration and log-in information. Childs was so concerned about the network's security that he refused even to write router and switch configurations to flash. But what may prove difficult for the prosecution in its case against Childs is that his restricted access to the network was widely known and accepted among managers and the city's other network engineers. Venezia, who has been suspicious of the official story from the start, suspects that the Childs case may be that 'of an overprotective admin who believed he was protecting the network — and by extension, the city — from other administrators whom he considered inferior, and perhaps even dangerous.' Further evidence is that fact that the network, from what Venezia understands, has been running smoothly since Childs' arrest."
So instead of letting the air out of the car's tires, a car he loved, he simply wouldn't give the keys to dangerous drivers.
"He's using a quantum encryption scheme! That'll take hours to break!"
It's hard to believe that management didn't care that a single employee was the only one who knew anything about critical infrastructure, no matter whether the employee arranged things this way because he thought no-one else was good enough or because this was his was of becoming entrenched.
You can try and defend him and glorify him all you want... but as a professional system administrator he should have known that his singular access and pathological behavior was more dangerous than helpful.
What if, instead of being fired he was the victim of an accident or crime? What if he had a health problem? What if a serious, life threatening issue came up (say, you know, an earthquake) that caused the system to be unstable and, at the same time, prevented him from getting there to fix things?
He's still a criminal. But, he's not alone in his behaviour. Whoever his managers are sound to be guilty of criminal negligence. This never should have been possible in a city government the size of San Francisco. Especially when it comes to critical infrastructure. If I were a citizen of San Fran I'd be asking why heads aren't rolling at the highest levels. Why was this allowed to happen? In San Francisco, where you think they'd have no problem finding competent replacements.
Absolutely mind boggling.
short version: if you bad to computers, we bad to you!
Simon Travaglia? Is that you?
How can I believe you when you tell me what I don't want to hear?
so the network is NOT locked up, it's just unrestoreble after "password recovery."
sounds like what they need to do is get some qualified engineers to redesign it, and when it's on paper, pull the plug on everything, and reconfigure from scratch.
because if it isn't saved in flash, it's going away as soon as the power light goes out.
which makes our jailed genius a little less than blazing fast. in fact, about half fast. parts of the system ARE going to go down. it's the nature of the beast. no records, no writes... the first time the janitor plugs in a 18-amp vacuum in a rack, it's gone.
they'll come along and take his Cisco cert away for not saving the configs, if for nothing else.
if this is supposed to be a new economy, how come they still want my old fashioned money?
IANAL, but isn't $5 million US for bail a bit excessive for this?
That's my first reaction to the news. Critical infrastructure should have redundancy everywhere, including the support staff.
To give a stupid but obvious example what if Childs was run over by a car? OK, he wouldn't care but all the rest of SF would.
So they should never have put the network online until the information was in several places (the brains of several people if formal electronic/paper records were too inflexible).
Stll, this sounds like political infighting more than ever. Given the situation why were they trying to fire a critical person like Childs? Sounds like some bureaucrat with an ego as big as Childs would be involved to cause this, rather than Childs "going rogue". And he (the bureaucrat) was more skilled in the political game. Of course this person would be covering his tracks, and not be obvious in any way. So Childs and the whole of SF lost. His firing does not make sense otherwise, given his critical position.
Ah, the fun of weaving conspiracy theories :-)
Every software company I have worked for... if one or two people were hit by a bus... the company would be out-of-business. Management knew this... fellow developers knew it. Its a commonplace thing. Engineers take the work so *personally*. "No one can touch that code but me... " blah... blah. Ånd the stupid management goes along w/ these primadonna's. Of course... if they demanded more money... they'd be gone in a NY minute.
He's certainly guilty of being a bad employee, as well as affirming all of those user-unfriendly IT sterotypes (those are often true, BTW). But criminal?
In America, they have to prove that first. Looking at the statute, it seems it all comes down to the issue of "without permission." The main point the article makes is that he might have had at least understood or standing permission to do most or all of what he did. Just like when you take your parents' car somewhere as a teenager, it isn't theft if it's understood that you are allowed to use it.
The article is one-sided, and his alleged refusal to give up the passwords looks bad (perhaps he is remaining silent until he speaks with counsel), but proving he didn't have permission might be hard. Ergo, no criminal.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
Every time I see a situation like this, I have to wonder what would happen if an "indispensable" person got hit by a bus. It strikes me that Childs was using his absolute control of the network as a way to put the fear of god in others within the department while attaining more prestige and autonomy than he deserved. The fact that Childs locked everyone out of the system after apparently receiving a poor job assessment backs that up. Sooner or later, the IT department had to take action to strip his stranglehold of the network, especially if he was on the verge of burnout or increasingly difficult to deal with.
I suspect that no one had the interpersonal wherewithal to figure out how to approach him in a non-confrontational manner. The best approach would have been to find someone who Childs respected who could share the load and provide backup and support while the organization attempted to deal with an overly possessive employee who is behaving irrationally.
People who fiddle with government machines get let of and win people elections! Those that STOP people fiddling with Machines get locked up on $5 mill bail....:D:D
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
I know someone who worked on the cisco side with this guy. This had been going on for a while. The dude was threatening co-workers doing all kinds of odd stuff. The idea that he was somehow just a little protective is an off the charts miss-representation.
1. Terry: you selfish bastard, if your network cannot be maintained without you, you have failed as an admin 2. The city of SF: common sense - try it out some time 3. The tax payers: what did you do to deserve this?
I get a little tired with the "hit by a bus" example. My coworkers use it all the time as an excuse to make me document everything to the Nth degree.
Maybe they could suggest "crushed in an orgy" or "broke lightspeed and turned to photons". Getting hit by a bus is such a boring way to go.
i'm not sure if i am being trolled
are you lampooning how a paranoid schizophrenic thinks or are you actually also a paranoid schizophrenic?
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
This analogy is spot on, and whoever modded it off-topic obviously is incapable of understanding the topic and shouldn't have had the keys to the mod-car in the first place.
Infuriate left and right
Power cycle the network equipment. If it comes back up, pay him for the rest of the year as severance and let him go his own way. If it doesn't come back up, put him away for 10-15 years for public endangerment, and fine him whatever the cost is to the city to recreate the network and for any loss of productivity in the meantime. Either way he is a terrible admin - no one single person should be a single point of failure. What if he got hit by Muni at lunch one day?
Open source does not equal open data.
.sig withheld by request
In my experience, it's a rare company indeed whose managers can fathom the implications of a situation like this. In general, I'm unable to get management to even understand Rule Zero of system administration. Which is: Do everything you need to do to be drop dead certain that you always have a reasonable backup of your important systems. This doesn't sound too difficult, but in practice it's difficult to convince managers that an event that could happen with probability == 0.01 could ever happen...
"Not an actor, but he plays one on TV."
.
modded +3, Informative.
but this attitude sets off alarms.
exposing a geek who despises his supervisors and is used to thinking of the server rooms as his personal playground.
all networks once configured properly, run smoothly until they don't.
when they don't, there's one man who can fix it.
I can fully understand setting up a complex system and getting it working perfectly and then some other admin or consultant coming in and fucking it up.
when they fuck it up, you have to fix it. And you don't get bonus pay for that.
not only that, but network/system administrators have to worry a lot about whether management wants to can them simply because things are running so smoothly that they have nothing to do. Which is bullshit because half of the job is keeping up with current tech trends, learning new technologies, and protecting your network on a daily basis. I don't blame the man for guarding his creation jealously. When you start handing over the keys, you are no longer necessary. You get paid too much and this kid who just quit his job of six months from bimblebomble.com seems to know how to do what you do. And we can pay him a lot less and potentially cut out benefits.
They're using their grammar skills there.
He wouldn't write configs to flash?
It means they can't power cycle or reboot anything, or the network is screwed.
No device stays up forever.
It also means they just have to power cycle a switch to gain access to it, and then do what they can to figure out how it was configured.
IOW: They have to break it to fix it.
Let's hope his mailbox doesn't flow over and crash the servers with congrat mails for sysadmin appreciation day next week.
The city has a huge issue here.
This guy will have a hard time getting a job in the future, and a guy with his credentials commands a lot more than he is making right now.
If it turns out that the facts of this case are far from the original story, and nobody from the city is stepping in to correct it, then SF is in the same situation as the US when Ashcroft pointed the finger at the Anthrax guy (who recently won a big chunk of change for the false accusation).
Something tells me that the wheels of government turn slowly enough that even if they wanted to correct themselves at this point, they won't until well after the publicity is over.
That law is there to make it possible for administrators to do their work. If you are working with emails, and you happen to see a few, you don't go to jail for it.
But monitoring his bosses' email so you can tell what they are saying specifically about him is highly unlikely to be in his job description, and thus he is not protected when he does that. Nor should he be.
http://lkml.org/lkml/2005/8/20/95
I find the situation startling familiar. It's downright creepy to read this scenario. Back in the late 90s I was the sysadmin of a moderately sized ISP. When we started out I was one of three network engineers hired to build the ISP; eventually I ended up in 'charge' of the system. Like the article I also was very protective of my network, and as paranoid as this individual is made out to be. Granted I was in my 20s and suitably arrogant to boot, more on this in a moment. As time went on first one, then the other guy quit after working 80 hours a week without the possibility of time off...things only got worse as people quit. When it was down to me I made sure the owners knew the passwords to everything, but they lacked any knowledge of how to do anything. This came back to haunt me later as you'll see. Eventually I too got fed up and went to work for another company that wasn't a direct competitor. Before I left I advised management on changing all passwords for both of our sakes. I tried to explain everything but nobody understood the technical aspects. Two months later I got a visit from the FBI. 8 grueling hours of interrogation later from armed men I found out that the entire network had crashed, and I was under suspicion as having remotely logged in and crashing their system. It wasn't until later I found out they never hired a replacement, and my system simply collapsed due to lack of maintenance. It's easy to be painted out as the bad guy when you intimately know the network while being managed by a bunch of clueless twits. I don't know if that's the case in this guy's case, but I can see it working either way.
It seems that both his co-workers and his management recognized his considerable expertise. So, if he thought that he was working with incompetent freeloaders, then he probably was right. This has nothing to do with primadonna status. The problem here is workload and responsibility.
I support several hundred servers and two of my backups are telecommuting from the west coast. Their Unix expertise is limited to creating user accounts, pushing power buttons, and sending me emails with stupid questions. I don't go so far as to lock them out of "my" servers, but I do my best to keep these two characters busy in their sandbox.
This has nothing to do with my (admittedly very considerable) ego, but has everything to do with me being able to enjoy weekends without being called to fix various problems. When something breaks, the ops calls me and not the two clowns in California. Any problem - big or small - they will find me, wake me up, drag me into a telecon, where I would have to fix the problem while simultaneously explaining to them how I did it and answering "are we there yet" questions from various random managers who couldn't sleep at night.
I would love to have a colleague whom I can trust to do upgrades and architectural changes, so I can spend more time fishing. The way things are right now, I am forced to keep other sysadmins at an arm's length just so I don't have to work even more hours (for which I am not being paid) to clean up their messes.
"Sure, the odds are 1000:1 against that I'll be hit by a bus, but there are a lot of ways disaster can strike, and they add up. You willing to ignore 5:1 odds? How about 10:1, or 15:1?"
This is why technical people need to strive to learn to have relationships with supervisors of a non-technical bent. From reading the article, it seems that Childs' demeanor meant that he could easily be dismissed as the brilliant-but-whacked-out-network-curmudgeon. Fair or not, that means that all of his concerns could be waved off as paranoia (for instance, him trying to get an information security policy in place). Unfortunately, the wisdom of our caution only becomes evident when a disaster occurs or is narrowly averted (e.g. "Thank God we backed that data up!").
On the other hand, non-technical managers should learn to not instantly dismiss the concerns of technical people as unlikely or unrealistic.
"I know someone who worked on the cisco side with this guy .. The dude was threatening co-workers"
What was the name of this someone, who did Terry Childs threaten, what was the nature of these threats?
davecb5620@gmail.com
Consider mentoring. The God complex management style rarely works out well in the end.