Slashdot Mirror
The Inside Story On the San Francisco Network Hijacking
snydeq writes "A source with direct knowledge of San Francisco's IT infrastructure has tipped off Paul Venezia to the real story behind Terry Childs' lockout of San Francisco's network, providing a detailed account of the city's FiberWAN, interdepartmental politics, and Terry Childs himself. Childs pleaded not guilty to charges of tampering yesterday and is being held on $5 million bail. According to the source, Childs' purview was limited to the city's FiberWAN — a network he himself built and, believing no one competent enough to touch the network but himself, guarded religiously, sharing details with no one, including routing configuration and log-in information. Childs was so concerned about the network's security that he refused even to write router and switch configurations to flash. But what may prove difficult for the prosecution in its case against Childs is that his restricted access to the network was widely known and accepted among managers and the city's other network engineers. Venezia, who has been suspicious of the official story from the start, suspects that the Childs case may be that 'of an overprotective admin who believed he was protecting the network — and by extension, the city — from other administrators whom he considered inferior, and perhaps even dangerous.' Further evidence is that fact that the network, from what Venezia understands, has been running smoothly since Childs' arrest."
The giant flash was just some solar burst.. it wasnt anubis' ship
There's no Freedom like UFP-dom
You're wrong. Your comparison with Diebold does not even merit cursory contemplation.
This is a sig. It is like every other sig in the world, except that it is mine, and it is different.
You can try and defend him and glorify him all you want... but as a professional system administrator he should have known that his singular access and pathological behavior was more dangerous than helpful.
What if, instead of being fired he was the victim of an accident or crime? What if he had a health problem? What if a serious, life threatening issue came up (say, you know, an earthquake) that caused the system to be unstable and, at the same time, prevented him from getting there to fix things?
He's still a criminal. But, he's not alone in his behaviour. Whoever his managers are sound to be guilty of criminal negligence. This never should have been possible in a city government the size of San Francisco. Especially when it comes to critical infrastructure. If I were a citizen of San Fran I'd be asking why heads aren't rolling at the highest levels. Why was this allowed to happen? In San Francisco, where you think they'd have no problem finding competent replacements.
Absolutely mind boggling.
so the network is NOT locked up, it's just unrestoreble after "password recovery."
sounds like what they need to do is get some qualified engineers to redesign it, and when it's on paper, pull the plug on everything, and reconfigure from scratch.
because if it isn't saved in flash, it's going away as soon as the power light goes out.
which makes our jailed genius a little less than blazing fast. in fact, about half fast. parts of the system ARE going to go down. it's the nature of the beast. no records, no writes... the first time the janitor plugs in a 18-amp vacuum in a rack, it's gone.
they'll come along and take his Cisco cert away for not saving the configs, if for nothing else.
if this is supposed to be a new economy, how come they still want my old fashioned money?
Every software company I have worked for... if one or two people were hit by a bus... the company would be out-of-business. Management knew this... fellow developers knew it. Its a commonplace thing. Engineers take the work so *personally*. "No one can touch that code but me... " blah... blah. Ånd the stupid management goes along w/ these primadonna's. Of course... if they demanded more money... they'd be gone in a NY minute.
He's certainly guilty of being a bad employee, as well as affirming all of those user-unfriendly IT sterotypes (those are often true, BTW). But criminal?
In America, they have to prove that first. Looking at the statute, it seems it all comes down to the issue of "without permission." The main point the article makes is that he might have had at least understood or standing permission to do most or all of what he did. Just like when you take your parents' car somewhere as a teenager, it isn't theft if it's understood that you are allowed to use it.
The article is one-sided, and his alleged refusal to give up the passwords looks bad (perhaps he is remaining silent until he speaks with counsel), but proving he didn't have permission might be hard. Ergo, no criminal.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
Every time I see a situation like this, I have to wonder what would happen if an "indispensable" person got hit by a bus. It strikes me that Childs was using his absolute control of the network as a way to put the fear of god in others within the department while attaining more prestige and autonomy than he deserved. The fact that Childs locked everyone out of the system after apparently receiving a poor job assessment backs that up. Sooner or later, the IT department had to take action to strip his stranglehold of the network, especially if he was on the verge of burnout or increasingly difficult to deal with.
I suspect that no one had the interpersonal wherewithal to figure out how to approach him in a non-confrontational manner. The best approach would have been to find someone who Childs respected who could share the load and provide backup and support while the organization attempted to deal with an overly possessive employee who is behaving irrationally.
**WAY of Topic** Except when a McCain Ex-Advisor came out to say the Diebold CEO went to 2 Democrat area to "Patch" the Machines in the '02 elections...(those 2 area turned Conservative in that election)... **BACK on topic** But sounds like Childs was a great Admin! The worse thing that can happen to a network is other Admins! You can't have them sticking routers on your network and let them think they know more that you! :D
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
I know someone who worked on the cisco side with this guy. This had been going on for a while. The dude was threatening co-workers doing all kinds of odd stuff. The idea that he was somehow just a little protective is an off the charts miss-representation.
funny I find it VERY easy to believe. Right now only 3 people in my own district now the running of the network, and only 1 by extension of that the complete configuration of the OS X server running the mac portion of the district. I have a emergency recovery manual I wrote myself, but it is under lock and key by me to keep all but 2 people from knowing it because I KNOW the other techs and administrators are incompetent political appointees who will royally screw things up and cause much more damage than they solve if they try to implement it without know what is going on.
"Slashdot, where telling the truth is overrated but lying is insightful."
More details here
Childs, who works in the city Department of Technology, allegedly created a password that gave him exclusive access to the city's new FiberWAN (wide area network), authorities told the newspaper. He has refused to divulge the password, leaving other system administrators locked out.
Undoing Childs' alleged tampering could cost millions of dollars, city officials said. In the meantime, the system is operating, even though administrators have limited or no access.
Childs, who has worked for the city for about five years, had been disciplined in recent months for poor job performance, and supervisors had tried to fire him, the newspaper reported.
"They weren't able to do it -- this was kind of his insurance policy," an official who spoke on the condition of anonymity told the newspaper. Childs allegedly began tampering with the computer system June 20, building a tracing system to monitor what other administrators were saying or doing about his personnel case.
More details here
The Chronicle also reported on Wednesday that Childs has a 25-year-old felony criminal record in Kansas, where he was convicted of aggravated robbery and aggravated burglary stemming from charges filed in 1982. Childs was on probation or parole until 1987, according to records uncovered by the newspaper. Childs had disclosed the felony conviction when he applied for the San Francisco job five years ago.
Childs had been highly regarded in the technology department until he became a "rogue employee that got a bit maniacal," Newsom said.
"He was very good at what he did, and sometimes that goes to people's heads," the mayor said. "And we think that's what this is about."
Childs' problems with the department got serious June 20 when he started taking photographs of the agency's new head of security after she began an audit of who had password access to the system, the newspaper said. Childs' frightening behavior prompted the woman to lock herself in an office
His supervisors' concerns grew when they discovered he had given himself exclusive access to the system and had developed a way to spy on his bosses' e-mails related to his conduct. Childs was ordered to leave work July 9 for alleged insubordination.
It's not actually the case of people not knowing the passwords or such, from what I've read in the news sources, it looks like he's locked out the other people who should have access. Think of him as a an admin who either changed the passwords of all the other admin users or deleted their accounts.
If I can be forgiven for porting my response here:
The InfoWorld article linked to is remarkable and revealing, in particular, to me, because I have seen this exact scenario in multiple work settings. The people with actual networking knowledge and talent control access so that the employees with "just enough knowledge to be dangerous" don't BREAK THE SYSTEM.
That's not ego or theory. I've seen it happen so many times I couldn't count: technicians who think they know what they're doing but don't adequately research their ideas (or study enough in general) are prone to wreaking all sorts of havoc on the network. This Childs fellow may well be controlling or even arrogant. But what if -- just humor the notion -- in his work environment he was actually right? That had he shared access with the less competent admins with which he may have been surrounded, the San Fran government would have had a far less stable, secure network.
I don't know, but given what I've seen, it's quite plausible. Not his call to make, I'd agree. But then, it seems that for some time, his direct superior didn't insist otherwise. Bad call, of course -- but not Childs' fault.
I'm starting to suspect his arrest and being charged were ridiculously hasty and unnecessary. Conceivably the outcome of his immediate superior(s) running an exaggerated "renegade" story up the chain of command, as much out of interpersonal distaste for Childs as actual concern over his reluctance to give up a password on demand.
Perhaps the new gap-filler for managerial incompetence: employee prosecution.
haha..probably right on..I came from the outside, now I work in county govt going on 7 years. I have more knowledge in my little finger than 99 net-sys admins / network engineers picked at random who are working inside the county govt. CLUELESS! No excuse for getting yourself arrested thou... maybe not too late for "I forgot the password" as a defense. Any ideas how to clean up these laggards? Its our freekin tax money down the drain!!
I post AC because of my position, which is basically a guy who was hired as the second network tech to help manage the network for a sizeable city (large enough that we host several professional sports teams). I had no real qualifications other than knowing how to google my way out of most basic computer situations. My supervisor managed all City-owned Cisco equipment and it has only been 2 of us for 2 years. We manage over 300 Cisco devices at over 100 sites and I can honestly say that after reading a few more details on this story, I can easily understand how this can happen in a local government. I believe that the problem is in management. We have similar problems in our City regarding the lack of passing of knowledge and lack of staffing, but we have a good security team that knows more about Cisco networks than the 2 of us that regularly work on the Cisco equipment in our City. They are not normally watching our backs (that we know of) but they would certainly do so if they got a bad vibe about us. We have to share passwords with them and they have as much access to our equipment as we do. It is simply a requirement in a publicly owned system that knowledge is shared. Taxpayers have payed for the equipment and expect that there are not single points of failure. There are many reasons that more people than work on one thing on a regular basis have knowledge of and access to the most basic systems. If there was no redundancy, then it is a fundamental failure of management.....I'm not saying the guy should have set one password and not passed it on.....but I understand.
So based on your statement, passwords, ACLs, social security numbers and other extremely sensitive data should be visible to the public. Could you please post all that information about your own system(s)? Otherwise, STFU.
Posting semi anonymously for obvious reasons.
The university I went to issued default passwords of the last several digits of the owners social security number.
The school is a public school in a state that had an open records law.
The open records of accounts payable and receivable of the school included the vendor/individuals TIN in the left hand column for accurate identification of vendors and individual with similar names.
The second day of the semester one could stop by the administration building and pick up a handy password list ^h^h public finance statement.
A random test at the end of the semester, from a borrowed account, indicated about one third of the student body never changed their passwords. (same for first year faculty.)
One semester I was the only person to pick up a copy even though they had four printed out just in case somebody asked for it.
The damage was mostly limited to a couple of scripts that tested that yes there was almost no security and at least half a dozen people that could trivially compromise a large percentage of the accounts on campus.
Yeah. You'd think there should be some sort of law against excessive bail.
I find the situation startling familiar. It's downright creepy to read this scenario. Back in the late 90s I was the sysadmin of a moderately sized ISP. When we started out I was one of three network engineers hired to build the ISP; eventually I ended up in 'charge' of the system. Like the article I also was very protective of my network, and as paranoid as this individual is made out to be. Granted I was in my 20s and suitably arrogant to boot, more on this in a moment. As time went on first one, then the other guy quit after working 80 hours a week without the possibility of time off...things only got worse as people quit. When it was down to me I made sure the owners knew the passwords to everything, but they lacked any knowledge of how to do anything. This came back to haunt me later as you'll see. Eventually I too got fed up and went to work for another company that wasn't a direct competitor. Before I left I advised management on changing all passwords for both of our sakes. I tried to explain everything but nobody understood the technical aspects. Two months later I got a visit from the FBI. 8 grueling hours of interrogation later from armed men I found out that the entire network had crashed, and I was under suspicion as having remotely logged in and crashing their system. It wasn't until later I found out they never hired a replacement, and my system simply collapsed due to lack of maintenance. It's easy to be painted out as the bad guy when you intimately know the network while being managed by a bunch of clueless twits. I don't know if that's the case in this guy's case, but I can see it working either way.
But it didn't work.
This, someone has suggested, could be because he'd nuked the root account and "root's" name was not root (so you have to crack the password AND the name).
On being sacked, they nuked his account.
Which was the one which worked to administrate the network.
Which doesn't exist because they nuked his account.
http://www.csoonline.com.au/index.php/id;1895501252;fp;2;fpid;1
Yet the city gave hime full admin access to a critical, and sensitive system. The city also didn't bother to insure that the system was safe from being locked out in that manner.
IMO: if Childs goes to prison, the city's IT managers should go with him.