Slashdot Mirror
The Inside Story On the San Francisco Network Hijacking
snydeq writes "A source with direct knowledge of San Francisco's IT infrastructure has tipped off Paul Venezia to the real story behind Terry Childs' lockout of San Francisco's network, providing a detailed account of the city's FiberWAN, interdepartmental politics, and Terry Childs himself. Childs pleaded not guilty to charges of tampering yesterday and is being held on $5 million bail. According to the source, Childs' purview was limited to the city's FiberWAN — a network he himself built and, believing no one competent enough to touch the network but himself, guarded religiously, sharing details with no one, including routing configuration and log-in information. Childs was so concerned about the network's security that he refused even to write router and switch configurations to flash. But what may prove difficult for the prosecution in its case against Childs is that his restricted access to the network was widely known and accepted among managers and the city's other network engineers. Venezia, who has been suspicious of the official story from the start, suspects that the Childs case may be that 'of an overprotective admin who believed he was protecting the network — and by extension, the city — from other administrators whom he considered inferior, and perhaps even dangerous.' Further evidence is that fact that the network, from what Venezia understands, has been running smoothly since Childs' arrest."
The giant flash was just some solar burst.. it wasnt anubis' ship
There's no Freedom like UFP-dom
You're wrong. Your comparison with Diebold does not even merit cursory contemplation.
This is a sig. It is like every other sig in the world, except that it is mine, and it is different.
It's hard to believe that management didn't care that a single employee was the only one who knew anything about critical infrastructure, no matter whether the employee arranged things this way because he thought no-one else was good enough or because this was his was of becoming entrenched.
You can try and defend him and glorify him all you want... but as a professional system administrator he should have known that his singular access and pathological behavior was more dangerous than helpful.
What if, instead of being fired he was the victim of an accident or crime? What if he had a health problem? What if a serious, life threatening issue came up (say, you know, an earthquake) that caused the system to be unstable and, at the same time, prevented him from getting there to fix things?
He's still a criminal. But, he's not alone in his behaviour. Whoever his managers are sound to be guilty of criminal negligence. This never should have been possible in a city government the size of San Francisco. Especially when it comes to critical infrastructure. If I were a citizen of San Fran I'd be asking why heads aren't rolling at the highest levels. Why was this allowed to happen? In San Francisco, where you think they'd have no problem finding competent replacements.
Absolutely mind boggling.
short version: if you bad to computers, we bad to you!
Simon Travaglia? Is that you?
How can I believe you when you tell me what I don't want to hear?
so the network is NOT locked up, it's just unrestoreble after "password recovery."
sounds like what they need to do is get some qualified engineers to redesign it, and when it's on paper, pull the plug on everything, and reconfigure from scratch.
because if it isn't saved in flash, it's going away as soon as the power light goes out.
which makes our jailed genius a little less than blazing fast. in fact, about half fast. parts of the system ARE going to go down. it's the nature of the beast. no records, no writes... the first time the janitor plugs in a 18-amp vacuum in a rack, it's gone.
they'll come along and take his Cisco cert away for not saving the configs, if for nothing else.
if this is supposed to be a new economy, how come they still want my old fashioned money?
IANAL, but isn't $5 million US for bail a bit excessive for this?
Honestly, I am surprised the FBI or some other government branch hasn't stepped in on the matter and taken over. If the fiber/wan deals with E911 and other critical functions of the city, I think the city government needs to allow the higher government branches to intervene.
Either use the higher government interaction or just take him out back and start breaking each finger and toe until he talks.
That's my first reaction to the news. Critical infrastructure should have redundancy everywhere, including the support staff.
To give a stupid but obvious example what if Childs was run over by a car? OK, he wouldn't care but all the rest of SF would.
So they should never have put the network online until the information was in several places (the brains of several people if formal electronic/paper records were too inflexible).
Stll, this sounds like political infighting more than ever. Given the situation why were they trying to fire a critical person like Childs? Sounds like some bureaucrat with an ego as big as Childs would be involved to cause this, rather than Childs "going rogue". And he (the bureaucrat) was more skilled in the political game. Of course this person would be covering his tracks, and not be obvious in any way. So Childs and the whole of SF lost. His firing does not make sense otherwise, given his critical position.
Ah, the fun of weaving conspiracy theories :-)
Every software company I have worked for... if one or two people were hit by a bus... the company would be out-of-business. Management knew this... fellow developers knew it. Its a commonplace thing. Engineers take the work so *personally*. "No one can touch that code but me... " blah... blah. Ånd the stupid management goes along w/ these primadonna's. Of course... if they demanded more money... they'd be gone in a NY minute.
He's certainly guilty of being a bad employee, as well as affirming all of those user-unfriendly IT sterotypes (those are often true, BTW). But criminal?
In America, they have to prove that first. Looking at the statute, it seems it all comes down to the issue of "without permission." The main point the article makes is that he might have had at least understood or standing permission to do most or all of what he did. Just like when you take your parents' car somewhere as a teenager, it isn't theft if it's understood that you are allowed to use it.
The article is one-sided, and his alleged refusal to give up the passwords looks bad (perhaps he is remaining silent until he speaks with counsel), but proving he didn't have permission might be hard. Ergo, no criminal.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
Every time I see a situation like this, I have to wonder what would happen if an "indispensable" person got hit by a bus. It strikes me that Childs was using his absolute control of the network as a way to put the fear of god in others within the department while attaining more prestige and autonomy than he deserved. The fact that Childs locked everyone out of the system after apparently receiving a poor job assessment backs that up. Sooner or later, the IT department had to take action to strip his stranglehold of the network, especially if he was on the verge of burnout or increasingly difficult to deal with.
I suspect that no one had the interpersonal wherewithal to figure out how to approach him in a non-confrontational manner. The best approach would have been to find someone who Childs respected who could share the load and provide backup and support while the organization attempted to deal with an overly possessive employee who is behaving irrationally.
**WAY of Topic** Except when a McCain Ex-Advisor came out to say the Diebold CEO went to 2 Democrat area to "Patch" the Machines in the '02 elections...(those 2 area turned Conservative in that election)... **BACK on topic** But sounds like Childs was a great Admin! The worse thing that can happen to a network is other Admins! You can't have them sticking routers on your network and let them think they know more that you! :D
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
This makes no sense. A properly secure network should be in complete control of those creating it, simply through password and other authentication. Sure, good documentation is helpful in a worst case scenario, but you really need a hit-by-a-bus contingency team.
The ______ Agenda
People who fiddle with government machines get let of and win people elections! Those that STOP people fiddling with Machines get locked up on $5 mill bail....:D:D
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
I know someone who worked on the cisco side with this guy. This had been going on for a while. The dude was threatening co-workers doing all kinds of odd stuff. The idea that he was somehow just a little protective is an off the charts miss-representation.
1. Terry: you selfish bastard, if your network cannot be maintained without you, you have failed as an admin 2. The city of SF: common sense - try it out some time 3. The tax payers: what did you do to deserve this?
I get a little tired with the "hit by a bus" example. My coworkers use it all the time as an excuse to make me document everything to the Nth degree.
Maybe they could suggest "crushed in an orgy" or "broke lightspeed and turned to photons". Getting hit by a bus is such a boring way to go.
Let's leave out the legal ramifications here, and let's not go to the hysteria of "he's being thrown to the wolves to protect management" or "he's an evil hacker who shut down the city government networks."
When it comes down to it, one has to ask what Childs' job was. He was supposed to manage the network for the San Francisco city government.
As a result, he was supposed to implement policy as communicated to him by his bosses... but he also had the latitude to take actions to support the spirit of those policies where the right action was unclear. And yes, this is a Pollyanna-esque (is that a word?) view of the situation, but it leaves out the concept of malice as the driving force for either side - because it didn't start out as a plan to shut down the city.
Somehow it morphed into him becoming the sole support for the network routers, be it through arrogance ("I can't believe anyone else would do this right!") or being the only one available ("There's nobody else who works here who even understands the need!"), and at that point this became an incident waiting to happen.
So, either he refused to do his job (at which point he would have deserved to be fired), or his job was such that he was prevented from doing it (at which point professional ethics would have suggested his resignation - or at least, that's what engineering associations would have recommended in similar scenarios).
Instead, he stayed on and we have the current state of affairs.
Strike while the irony is hot! -- The Freethinker
Don't make ad hominem attacks please. I called the article one-sided, and merely presented a legal analysis of his case. I did not "rationalize" or "glorify" him. Truth be told, I actually tend to dislike IT geeks. They tend to be rude and have no personality and think they are smarter than everyone (which is usually not the case) and believe they are God's gift to an organization. Such attitudes should not be tolerated, regardless of how skilled an IT guy is.
With that said, government organizations tend to take a lowest common denominator attitude with IT departments. They don't pay shit, so the cheapest guy gets hired, often resembling a DMV employee. So I can see how a guy could get possessive about his network. He must know what the average city employee is like: Under-trained, bad attitude, and can't be fired due to unions.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
i'm not sure if i am being trolled
are you lampooning how a paranoid schizophrenic thinks or are you actually also a paranoid schizophrenic?
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
This analogy is spot on, and whoever modded it off-topic obviously is incapable of understanding the topic and shouldn't have had the keys to the mod-car in the first place.
Infuriate left and right
Power cycle the network equipment. If it comes back up, pay him for the rest of the year as severance and let him go his own way. If it doesn't come back up, put him away for 10-15 years for public endangerment, and fine him whatever the cost is to the city to recreate the network and for any loss of productivity in the meantime. Either way he is a terrible admin - no one single person should be a single point of failure. What if he got hit by Muni at lunch one day?
Open source does not equal open data.
.sig withheld by request
The magical pixie dust created everytime an OSS program runs.
In my experience, it's a rare company indeed whose managers can fathom the implications of a situation like this. In general, I'm unable to get management to even understand Rule Zero of system administration. Which is: Do everything you need to do to be drop dead certain that you always have a reasonable backup of your important systems. This doesn't sound too difficult, but in practice it's difficult to convince managers that an event that could happen with probability == 0.01 could ever happen...
"Not an actor, but he plays one on TV."
.
modded +3, Informative.
but this attitude sets off alarms.
exposing a geek who despises his supervisors and is used to thinking of the server rooms as his personal playground.
If I can be forgiven for porting my response here:
The InfoWorld article linked to is remarkable and revealing, in particular, to me, because I have seen this exact scenario in multiple work settings. The people with actual networking knowledge and talent control access so that the employees with "just enough knowledge to be dangerous" don't BREAK THE SYSTEM.
That's not ego or theory. I've seen it happen so many times I couldn't count: technicians who think they know what they're doing but don't adequately research their ideas (or study enough in general) are prone to wreaking all sorts of havoc on the network. This Childs fellow may well be controlling or even arrogant. But what if -- just humor the notion -- in his work environment he was actually right? That had he shared access with the less competent admins with which he may have been surrounded, the San Fran government would have had a far less stable, secure network.
I don't know, but given what I've seen, it's quite plausible. Not his call to make, I'd agree. But then, it seems that for some time, his direct superior didn't insist otherwise. Bad call, of course -- but not Childs' fault.
I'm starting to suspect his arrest and being charged were ridiculously hasty and unnecessary. Conceivably the outcome of his immediate superior(s) running an exaggerated "renegade" story up the chain of command, as much out of interpersonal distaste for Childs as actual concern over his reluctance to give up a password on demand.
Perhaps the new gap-filler for managerial incompetence: employee prosecution.
all networks once configured properly, run smoothly until they don't.
when they don't, there's one man who can fix it.
I can fully understand setting up a complex system and getting it working perfectly and then some other admin or consultant coming in and fucking it up.
when they fuck it up, you have to fix it. And you don't get bonus pay for that.
not only that, but network/system administrators have to worry a lot about whether management wants to can them simply because things are running so smoothly that they have nothing to do. Which is bullshit because half of the job is keeping up with current tech trends, learning new technologies, and protecting your network on a daily basis. I don't blame the man for guarding his creation jealously. When you start handing over the keys, you are no longer necessary. You get paid too much and this kid who just quit his job of six months from bimblebomble.com seems to know how to do what you do. And we can pay him a lot less and potentially cut out benefits.
They're using their grammar skills there.
Post a challenge on alt.2600 and The Pirate Bay to all comers: award the first cracker to get back root with a lifetime supply of Red Bull and a 10 bill shopping spree at the Frys in Palo Alto. They'll have the guy's password sliced and diced by Wednesday.
No mod points, no meta-moderating/Firehose/all the other free work Slashdot wants me to do.
He wouldn't write configs to flash?
It means they can't power cycle or reboot anything, or the network is screwed.
No device stays up forever.
It also means they just have to power cycle a switch to gain access to it, and then do what they can to figure out how it was configured.
IOW: They have to break it to fix it.
Let's hope his mailbox doesn't flow over and crash the servers with congrat mails for sysadmin appreciation day next week.
This is total speculation on my part, but maybe they never asked him nicely for the PW. I could totally see the new security coordinator wanting to use him as an example to others, and threatening him with termination right off the bat.
(I hate slashdot playlets, but just this once, let us imagine the following exchange.)
Security Coordinator: I want all the passwords to the routing equipment.
Admin: Why do you need them?
S.C.: I'm not here to explain myself to you. Give me the PWs or you're fired!
Admin: OK
S.C.: So you're going to give them to me?
Admin: You haven't told me why you want them.
S.C.: That's it! You're fired!
Admin: OK, I'll go clear out my desk.
S.C.: Wait, aren't you going to give me the passwords now? Come back here! Don't walk away when I'm talking to you!
Admin: I'm sorry, but you're no longer my boss.
The End (?)
OK, a quick quiz! Did you spot the criminal act? Trick question! In this scenario, there wasn't one!
Or, maybe when the owner said they wanted something they own, he should have handed it over.
The city has a huge issue here.
This guy will have a hard time getting a job in the future, and a guy with his credentials commands a lot more than he is making right now.
If it turns out that the facts of this case are far from the original story, and nobody from the city is stepping in to correct it, then SF is in the same situation as the US when Ashcroft pointed the finger at the Anthrax guy (who recently won a big chunk of change for the false accusation).
Something tells me that the wheels of government turn slowly enough that even if they wanted to correct themselves at this point, they won't until well after the publicity is over.
That law is there to make it possible for administrators to do their work. If you are working with emails, and you happen to see a few, you don't go to jail for it.
But monitoring his bosses' email so you can tell what they are saying specifically about him is highly unlikely to be in his job description, and thus he is not protected when he does that. Nor should he be.
http://lkml.org/lkml/2005/8/20/95
So based on your statement, passwords, ACLs, social security numbers and other extremely sensitive data should be visible to the public. Could you please post all that information about your own system(s)? Otherwise, STFU.
Posting semi anonymously for obvious reasons.
The university I went to issued default passwords of the last several digits of the owners social security number.
The school is a public school in a state that had an open records law.
The open records of accounts payable and receivable of the school included the vendor/individuals TIN in the left hand column for accurate identification of vendors and individual with similar names.
The second day of the semester one could stop by the administration building and pick up a handy password list ^h^h public finance statement.
A random test at the end of the semester, from a borrowed account, indicated about one third of the student body never changed their passwords. (same for first year faculty.)
One semester I was the only person to pick up a copy even though they had four printed out just in case somebody asked for it.
The damage was mostly limited to a couple of scripts that tested that yes there was almost no security and at least half a dozen people that could trivially compromise a large percentage of the accounts on campus.
I find the situation startling familiar. It's downright creepy to read this scenario. Back in the late 90s I was the sysadmin of a moderately sized ISP. When we started out I was one of three network engineers hired to build the ISP; eventually I ended up in 'charge' of the system. Like the article I also was very protective of my network, and as paranoid as this individual is made out to be. Granted I was in my 20s and suitably arrogant to boot, more on this in a moment. As time went on first one, then the other guy quit after working 80 hours a week without the possibility of time off...things only got worse as people quit. When it was down to me I made sure the owners knew the passwords to everything, but they lacked any knowledge of how to do anything. This came back to haunt me later as you'll see. Eventually I too got fed up and went to work for another company that wasn't a direct competitor. Before I left I advised management on changing all passwords for both of our sakes. I tried to explain everything but nobody understood the technical aspects. Two months later I got a visit from the FBI. 8 grueling hours of interrogation later from armed men I found out that the entire network had crashed, and I was under suspicion as having remotely logged in and crashing their system. It wasn't until later I found out they never hired a replacement, and my system simply collapsed due to lack of maintenance. It's easy to be painted out as the bad guy when you intimately know the network while being managed by a bunch of clueless twits. I don't know if that's the case in this guy's case, but I can see it working either way.
"Sure, the odds are 1000:1 against that I'll be hit by a bus, but there are a lot of ways disaster can strike, and they add up. You willing to ignore 5:1 odds? How about 10:1, or 15:1?"
This is why technical people need to strive to learn to have relationships with supervisors of a non-technical bent. From reading the article, it seems that Childs' demeanor meant that he could easily be dismissed as the brilliant-but-whacked-out-network-curmudgeon. Fair or not, that means that all of his concerns could be waved off as paranoia (for instance, him trying to get an information security policy in place). Unfortunately, the wisdom of our caution only becomes evident when a disaster occurs or is narrowly averted (e.g. "Thank God we backed that data up!").
On the other hand, non-technical managers should learn to not instantly dismiss the concerns of technical people as unlikely or unrealistic.
"I know someone who worked on the cisco side with this guy .. The dude was threatening co-workers"
What was the name of this someone, who did Terry Childs threaten, what was the nature of these threats?
davecb5620@gmail.com
http://www.csoonline.com.au/index.php/id;1895501252;fp;2;fpid;1
Yet the city gave hime full admin access to a critical, and sensitive system. The city also didn't bother to insure that the system was safe from being locked out in that manner.
IMO: if Childs goes to prison, the city's IT managers should go with him.