Slashdot Mirror
The Inside Story On the San Francisco Network Hijacking
snydeq writes "A source with direct knowledge of San Francisco's IT infrastructure has tipped off Paul Venezia to the real story behind Terry Childs' lockout of San Francisco's network, providing a detailed account of the city's FiberWAN, interdepartmental politics, and Terry Childs himself. Childs pleaded not guilty to charges of tampering yesterday and is being held on $5 million bail. According to the source, Childs' purview was limited to the city's FiberWAN — a network he himself built and, believing no one competent enough to touch the network but himself, guarded religiously, sharing details with no one, including routing configuration and log-in information. Childs was so concerned about the network's security that he refused even to write router and switch configurations to flash. But what may prove difficult for the prosecution in its case against Childs is that his restricted access to the network was widely known and accepted among managers and the city's other network engineers. Venezia, who has been suspicious of the official story from the start, suspects that the Childs case may be that 'of an overprotective admin who believed he was protecting the network — and by extension, the city — from other administrators whom he considered inferior, and perhaps even dangerous.' Further evidence is that fact that the network, from what Venezia understands, has been running smoothly since Childs' arrest."
The giant flash was just some solar burst.. it wasnt anubis' ship
There's no Freedom like UFP-dom
If no one can get it, no one can mess it up, which might prove he was a capable admin.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
You're wrong. Your comparison with Diebold does not even merit cursory contemplation.
This is a sig. It is like every other sig in the world, except that it is mine, and it is different.
It's hard to believe that management didn't care that a single employee was the only one who knew anything about critical infrastructure, no matter whether the employee arranged things this way because he thought no-one else was good enough or because this was his was of becoming entrenched.
You can try and defend him and glorify him all you want... but as a professional system administrator he should have known that his singular access and pathological behavior was more dangerous than helpful.
What if, instead of being fired he was the victim of an accident or crime? What if he had a health problem? What if a serious, life threatening issue came up (say, you know, an earthquake) that caused the system to be unstable and, at the same time, prevented him from getting there to fix things?
He's still a criminal. But, he's not alone in his behaviour. Whoever his managers are sound to be guilty of criminal negligence. This never should have been possible in a city government the size of San Francisco. Especially when it comes to critical infrastructure. If I were a citizen of San Fran I'd be asking why heads aren't rolling at the highest levels. Why was this allowed to happen? In San Francisco, where you think they'd have no problem finding competent replacements.
Absolutely mind boggling.
short version: if you bad to computers, we bad to you!
Simon Travaglia? Is that you?
How can I believe you when you tell me what I don't want to hear?
so the network is NOT locked up, it's just unrestoreble after "password recovery."
sounds like what they need to do is get some qualified engineers to redesign it, and when it's on paper, pull the plug on everything, and reconfigure from scratch.
because if it isn't saved in flash, it's going away as soon as the power light goes out.
which makes our jailed genius a little less than blazing fast. in fact, about half fast. parts of the system ARE going to go down. it's the nature of the beast. no records, no writes... the first time the janitor plugs in a 18-amp vacuum in a rack, it's gone.
they'll come along and take his Cisco cert away for not saving the configs, if for nothing else.
if this is supposed to be a new economy, how come they still want my old fashioned money?
IANAL, but isn't $5 million US for bail a bit excessive for this?
Honestly, I am surprised the FBI or some other government branch hasn't stepped in on the matter and taken over. If the fiber/wan deals with E911 and other critical functions of the city, I think the city government needs to allow the higher government branches to intervene.
Either use the higher government interaction or just take him out back and start breaking each finger and toe until he talks.
That's my first reaction to the news. Critical infrastructure should have redundancy everywhere, including the support staff.
To give a stupid but obvious example what if Childs was run over by a car? OK, he wouldn't care but all the rest of SF would.
So they should never have put the network online until the information was in several places (the brains of several people if formal electronic/paper records were too inflexible).
Stll, this sounds like political infighting more than ever. Given the situation why were they trying to fire a critical person like Childs? Sounds like some bureaucrat with an ego as big as Childs would be involved to cause this, rather than Childs "going rogue". And he (the bureaucrat) was more skilled in the political game. Of course this person would be covering his tracks, and not be obvious in any way. So Childs and the whole of SF lost. His firing does not make sense otherwise, given his critical position.
Ah, the fun of weaving conspiracy theories :-)
I would have thought the government would have deemed Teddy a TERRORIST EXTREMEST PLOTTING TO BLOW UP THE CITY and shipped him off to Guantanamo Bay by now...
Every software company I have worked for... if one or two people were hit by a bus... the company would be out-of-business. Management knew this... fellow developers knew it. Its a commonplace thing. Engineers take the work so *personally*. "No one can touch that code but me... " blah... blah. Ånd the stupid management goes along w/ these primadonna's. Of course... if they demanded more money... they'd be gone in a NY minute.
He's certainly guilty of being a bad employee, as well as affirming all of those user-unfriendly IT sterotypes (those are often true, BTW). But criminal?
In America, they have to prove that first. Looking at the statute, it seems it all comes down to the issue of "without permission." The main point the article makes is that he might have had at least understood or standing permission to do most or all of what he did. Just like when you take your parents' car somewhere as a teenager, it isn't theft if it's understood that you are allowed to use it.
The article is one-sided, and his alleged refusal to give up the passwords looks bad (perhaps he is remaining silent until he speaks with counsel), but proving he didn't have permission might be hard. Ergo, no criminal.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
then there is sitting in a holding cell, still protecting your turf... from the guys you are supposed to be protecting it for
the guy is over the deep end, he is criminally culpable for denying access to the people he built the network for
at best, he can probably use an insanity defense, like paranoid schizophrenia, because his actions are on the extreme paranoid end if this latest revelation about his motives and actions ring true
he's certainly mentally fragile. he shouldn't have that much exclusive control over such an important government network, that's for sure
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Every time I see a situation like this, I have to wonder what would happen if an "indispensable" person got hit by a bus. It strikes me that Childs was using his absolute control of the network as a way to put the fear of god in others within the department while attaining more prestige and autonomy than he deserved. The fact that Childs locked everyone out of the system after apparently receiving a poor job assessment backs that up. Sooner or later, the IT department had to take action to strip his stranglehold of the network, especially if he was on the verge of burnout or increasingly difficult to deal with.
I suspect that no one had the interpersonal wherewithal to figure out how to approach him in a non-confrontational manner. The best approach would have been to find someone who Childs respected who could share the load and provide backup and support while the organization attempted to deal with an overly possessive employee who is behaving irrationally.
If he's so smart he must have had a death contingency in place. The city might just have to use it, without killing him of course.
**WAY of Topic** Except when a McCain Ex-Advisor came out to say the Diebold CEO went to 2 Democrat area to "Patch" the Machines in the '02 elections...(those 2 area turned Conservative in that election)... **BACK on topic** But sounds like Childs was a great Admin! The worse thing that can happen to a network is other Admins! You can't have them sticking routers on your network and let them think they know more that you! :D
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
This makes no sense. A properly secure network should be in complete control of those creating it, simply through password and other authentication. Sure, good documentation is helpful in a worst case scenario, but you really need a hit-by-a-bus contingency team.
The ______ Agenda
People who fiddle with government machines get let of and win people elections! Those that STOP people fiddling with Machines get locked up on $5 mill bail....:D:D
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
I know someone who worked on the cisco side with this guy. This had been going on for a while. The dude was threatening co-workers doing all kinds of odd stuff. The idea that he was somehow just a little protective is an off the charts miss-representation.
How does open source prevent this from happening?
Only the State obtains its revenue by coercion. - Murray Rothbard
This incident effective highlights the organizational and management failure of the whole IT system at San Francisco public department. Its insanely incompetent and gross neglect of duties on everyone's part who made this guy the Lord of the network. There needs to be effective administration, documentation and oversight of things like these. I hope the people of SF wake up and demand some structured organization and competent people to manage these systems.
No Sig for you.!
1. Terry: you selfish bastard, if your network cannot be maintained without you, you have failed as an admin 2. The city of SF: common sense - try it out some time 3. The tax payers: what did you do to deserve this?
whenever you read stuff like this, about how hard it was for childs to setup the fiberwan network, and the complexity of the router details etc etc...
all you can, poorly designed equipment/software
in this day and age, why on earth can't you just plug the routers into the wall and they configure ?
the obvious answer is that the people who buy them [like childs] have a vested interest - they would loose their well paid jobs if it became simple, so they never buy simple stuff...
If Childs really was so damned concerned about the lack of skills within his own team, he should have been going out of his way to document his work, train the other staff and lift the standards. A person of that level of ability has a responsibility to raise the bar and his management should have known better.
Its obvious that his superiors are the ones largely to blame for letting this go one as long as possible but really, a person of Childs' skills/caliber could have done so much to turn the situation around its not funny.
Stories like this are a tragedy on multiple levels. Sad fact is though, this happens all the time in IT....
I get a little tired with the "hit by a bus" example. My coworkers use it all the time as an excuse to make me document everything to the Nth degree.
Maybe they could suggest "crushed in an orgy" or "broke lightspeed and turned to photons". Getting hit by a bus is such a boring way to go.
<3
I'm wondering whether, in the days since the guy was arrested, any of the hardware is having trouble? Yes, they're reporting the network is running smoothly, but is that because nothing has broken or because there's enough redundancy in the system to keep things going? I'd think, in a setup as large as SF must need, SOMETHING would have malfed in the last few days.
I'm curious because it'd be interesting to know if the guy's network-fu is as good as everyone's been saying.
It only requires physical access to the router and a few minutes. Thousands of dollars of time for all the routers, but not more. What am I missing here?
But, if Terry Childs really wanted to avoid this, he could have just put the password(s) in a safe. Something happens to him, the safe gets drilled open, and everyone is happy. If Childs is simply refusing to give the passwords, then bill him for drilling the safe, and fire him. If I was that distrusting, it's what I would have done, because it would save me from jail and make them prove just how badly they needed the passwords. I'm not that distrusting. I would have a safe, yes, but only give out the combo to my supervisor. Then both of us would have that warm, fuzzy feeling.
Let's leave out the legal ramifications here, and let's not go to the hysteria of "he's being thrown to the wolves to protect management" or "he's an evil hacker who shut down the city government networks."
When it comes down to it, one has to ask what Childs' job was. He was supposed to manage the network for the San Francisco city government.
As a result, he was supposed to implement policy as communicated to him by his bosses... but he also had the latitude to take actions to support the spirit of those policies where the right action was unclear. And yes, this is a Pollyanna-esque (is that a word?) view of the situation, but it leaves out the concept of malice as the driving force for either side - because it didn't start out as a plan to shut down the city.
Somehow it morphed into him becoming the sole support for the network routers, be it through arrogance ("I can't believe anyone else would do this right!") or being the only one available ("There's nobody else who works here who even understands the need!"), and at that point this became an incident waiting to happen.
So, either he refused to do his job (at which point he would have deserved to be fired), or his job was such that he was prevented from doing it (at which point professional ethics would have suggested his resignation - or at least, that's what engineering associations would have recommended in similar scenarios).
Instead, he stayed on and we have the current state of affairs.
Strike while the irony is hot! -- The Freethinker
Don't make ad hominem attacks please. I called the article one-sided, and merely presented a legal analysis of his case. I did not "rationalize" or "glorify" him. Truth be told, I actually tend to dislike IT geeks. They tend to be rude and have no personality and think they are smarter than everyone (which is usually not the case) and believe they are God's gift to an organization. Such attitudes should not be tolerated, regardless of how skilled an IT guy is.
With that said, government organizations tend to take a lowest common denominator attitude with IT departments. They don't pay shit, so the cheapest guy gets hired, often resembling a DMV employee. So I can see how a guy could get possessive about his network. He must know what the average city employee is like: Under-trained, bad attitude, and can't be fired due to unions.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
1. Know anyone that was hit by a bus?
2. Know anyone that was hit by a train?
3. Know anyone that was hit by a car?
4. Know anyone that was hit by lightning?
The odds are greater that he is six degrees of separation from Saddam Hussein than any of the above.
That still does not justify what he has done. Granted he is very dedicated and detail oriented, like most of us.
However, from the standpoint of personal responsibility -and integrity- he should have provided a means to allow some trusted individual the means to access these systems -or to provide the means- in the event that he might have been vaporised in a NEO asteroid impact.
Obviously, there is some other dynamic that controls what is occurring in this instance. More than likely there is one or more PHBs that have absolutely no clue as to what this guy does everyday and have elected to *choose a method* that would eliminate or reduce his position. What should be done is to eliminate the PHBs who have no clue and move this guy into management with a team that he can direct.
I've got your sig, right here.
i'm not sure if i am being trolled
are you lampooning how a paranoid schizophrenic thinks or are you actually also a paranoid schizophrenic?
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
I just love America's current paranoic political correctness. Com'n folks, just spit it out. Never mind the "what if her got hit by a bus" crap -- what you mean is "what if he was dead?" And, why mention poor oversight or poor management -- how about " everybody responsible for overseeing what this guy did in his work completely fucked up and should beheld equally responsible for whatever it's going to cost the city to fix the problem".
Go to Blockbuster and rent 'Brazil'. It will provide a very good answer for you. Torrent all your porn I don't care, but rent this. Terry Gilliam deserves whatever royalties he gets. And look for the scenes with Robert De Niro in them, his character is crazy and funny as hell... I never even figured out he was in it till about the 4th time I saw it.
-- I ignore anonymous replies to my comments and postings.
This analogy is spot on, and whoever modded it off-topic obviously is incapable of understanding the topic and shouldn't have had the keys to the mod-car in the first place.
Infuriate left and right
This is about power.
Reading the story, I get the feeling that this guy didn't want to protect "his" network. Instead he wanted to avoid getting obsolete or being replaceable. His main concern was staying in power and have the last word against his superiours he couldn't get along with.
It's that kind of guy who makes things overcomplicated and puts his hands on everything redundantly just to make others dependent on him. Remember that sentence about not writing configs to flash? That's exactly what he needed: Nothing works without him. (And I'm sure he was willingly risking that his oh-so-well-protected network could fail because he is not in place)
So this is not the type of guy I would want to administer my network. Neither is it what I would call an "expert sysadmin". It's just someone with lots of sysadmin knowledge. But he obviously isn't able to act like a professional.
Never worked for the government, have you? ;)
Believe me--it's every bit as bad in the private sector...
"Not an actor, but he plays one on TV."
Power cycle the network equipment. If it comes back up, pay him for the rest of the year as severance and let him go his own way. If it doesn't come back up, put him away for 10-15 years for public endangerment, and fine him whatever the cost is to the city to recreate the network and for any loss of productivity in the meantime. Either way he is a terrible admin - no one single person should be a single point of failure. What if he got hit by Muni at lunch one day?
Open source does not equal open data.
.sig withheld by request
In my experience, it's a rare company indeed whose managers can fathom the implications of a situation like this. In general, I'm unable to get management to even understand Rule Zero of system administration. Which is: Do everything you need to do to be drop dead certain that you always have a reasonable backup of your important systems. This doesn't sound too difficult, but in practice it's difficult to convince managers that an event that could happen with probability == 0.01 could ever happen...
"Not an actor, but he plays one on TV."
Let's say that Childs did indeed build an excellent network. Let's say that he was indeed the only one competent to run it. Let's say that his SF city bosses did indeed let him run everything and keep it's operating and access details secret from them. The second his bosses, who own the network say he has to give them access, he has to give them access.
Childs doesn't get to decide the policy controlling that network. Even if the city managers and/or their other sysadmins are going to screw it up, it's theirs to screw up.
It doesn't matter that there's more to this story. Unless the back-back story is that Childs is secretly the mayor of San Francisco, he's got to give up the password.
--
make install -not war
The only person on staff who understands the network is unavailable (forget why for a moment)... now the city must find someone else who also can understand the network before any changes can be made. This is a bad thing??
By making access more difficult (but NOT impossible, which is a very important point) this admin has forced the city to employ fairly knowledgeable people to maintain or change a network that, from the article at least, seems to warrant the skill of someone knowledgeable.
We don't know all the details, but we do know that #1 no outage has yet occured and #2 the equipment is still in place and can be reconfigured from scratch in the normal ways. This situation simply requires a skilled admin's touch to prevent any down time or inconvenience. Find one, and you don't have a problem.
Sure, this guy has overstepped his role and probably is not someone you'd want on your team, but he has effectively forced the city to utilize skilled people in a situation that seems to benefit greatly from using skilled people but a situation where skilled people all too often are not used.
It's not all bad, but there are hopefully better ways to accomplish the same goal. Why the hell is this guy in jail tho?
Here is a link to a story about convicted terrorist Inderjit Singh Reyat who will have to raise $500,000 to get bail in Canada http://www.canada.com/vancouversun/story.html?id=04aa2643-7845-40e6-8f59-e283bae49176
Much much lower than the 5million this tech guy will have to produce.
It's a car analogy fer crying out loud...
"The past was erased, the erasure was forgotten, the lie became truth." ~1984 George Orwell
The internet at my favorite Starbucks went out.. Now I actually gotta drink this shit. I say death sentence. Death sentence for everyone.
.
modded +3, Informative.
but this attitude sets off alarms.
exposing a geek who despises his supervisors and is used to thinking of the server rooms as his personal playground.
If I can be forgiven for porting my response here:
The InfoWorld article linked to is remarkable and revealing, in particular, to me, because I have seen this exact scenario in multiple work settings. The people with actual networking knowledge and talent control access so that the employees with "just enough knowledge to be dangerous" don't BREAK THE SYSTEM.
That's not ego or theory. I've seen it happen so many times I couldn't count: technicians who think they know what they're doing but don't adequately research their ideas (or study enough in general) are prone to wreaking all sorts of havoc on the network. This Childs fellow may well be controlling or even arrogant. But what if -- just humor the notion -- in his work environment he was actually right? That had he shared access with the less competent admins with which he may have been surrounded, the San Fran government would have had a far less stable, secure network.
I don't know, but given what I've seen, it's quite plausible. Not his call to make, I'd agree. But then, it seems that for some time, his direct superior didn't insist otherwise. Bad call, of course -- but not Childs' fault.
I'm starting to suspect his arrest and being charged were ridiculously hasty and unnecessary. Conceivably the outcome of his immediate superior(s) running an exaggerated "renegade" story up the chain of command, as much out of interpersonal distaste for Childs as actual concern over his reluctance to give up a password on demand.
Perhaps the new gap-filler for managerial incompetence: employee prosecution.
Sooo pathetic.
He was the network admin - given permission to design and setup the system. If the management doesn't have the passwords then who's to blame here?
You get fired when you don't do your job... not arrested. Until today.... apparently.
I said no... but I missed and it came out yes.
is this the one that involves the men in black suits or the space aliens?
i seem to have hit a nerve with the paranoid schizophrenic establishment
i apologize for offending you with my need for plausability
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Even if that's the whole story, it makes him guilty of incompetence (for designing a network with an administrator as a single point of failure) or insubordination (for refusing to hand over the passwords). I don't see a criminal act here.
Heck, if he was fired *first*, then refusing to hand over the passwords couldn't even be called insubordination.
You are all missing the point. While everyone is locked out of the system he created , it is running his code and all the missing fractional pennies created by rounding errors are being deposited into his secret offshore account, he only needs to hold out four more days and then he's a gazillionaire...
I am not that much interested if he is a criminal or not. Most probably he is. But his crime is definetely not worth $5M bail. This amount should be reserved for his boss(es).
all networks once configured properly, run smoothly until they don't.
when they don't, there's one man who can fix it.
I can fully understand setting up a complex system and getting it working perfectly and then some other admin or consultant coming in and fucking it up.
when they fuck it up, you have to fix it. And you don't get bonus pay for that.
not only that, but network/system administrators have to worry a lot about whether management wants to can them simply because things are running so smoothly that they have nothing to do. Which is bullshit because half of the job is keeping up with current tech trends, learning new technologies, and protecting your network on a daily basis. I don't blame the man for guarding his creation jealously. When you start handing over the keys, you are no longer necessary. You get paid too much and this kid who just quit his job of six months from bimblebomble.com seems to know how to do what you do. And we can pay him a lot less and potentially cut out benefits.
They're using their grammar skills there.
Post a challenge on alt.2600 and The Pirate Bay to all comers: award the first cracker to get back root with a lifetime supply of Red Bull and a 10 bill shopping spree at the Frys in Palo Alto. They'll have the guy's password sliced and diced by Wednesday.
No mod points, no meta-moderating/Firehose/all the other free work Slashdot wants me to do.
I haven't seen a clear sequence of events here. Exactly what preceded his being arrested? He doesn't seem to have been accused of any positive harmful acts. He was just removed (somehow) from a controlling position, leaving the system locked.
Once arrested, he has the right to insist on not being interrogated without his attorney present, and because of that problem with the public defender's office, his first attorney had to be replaced. So there's no way anyone can legitimately ask him for the password until he gets a proper lawyer.
He may have been insubordinate, but that's not a criminal offense.
I suspect that it's going to come out that the "security coordinator" the city hired asked for the passwords, and he refused. This is the person who says they were intimidated by his photographing them, which, one could argue, was a reasonable security precaution against a "social engineering" attack.
Hasn't anyone noticed that there's a money crunch going on. 10 to 1 his bosses just wanted to get access so they could start stripping it to get their bonuses for saving money. Childs just wouldn't let the incompetent fools screw up the city IT that's been working and still is working just to save their asses.
Was his job to cater to his personal interests, or those of his employer?
Because for the life of me I can't see how his method of operation is good for his employer (well, half-truth, really. It's a risk his employer knowingly and willingly chose to wear, and low as the odds for something like this may be, it blew up in their face).
Nevertheless, he put his personal interests above those of his employer, so he's at fault, just as much as they are for allowing him to practice.
-
Isn't there a basic right to remain silent. He shouldn't have to tell anyone the password.
There are ways that he could lock himself out. If he disabled password recovery and setup command authorization on the enable command, that could do the trick. From the way this article reads, I find this plausible. Imagine that he setup the AAA server in a very common way. He ties it to AD or whatever external directory and adds himself into the group that has access. He is fired and his account is disabled or deleted. Now they can't get in because they think they are authenticating to the router. Now he doesn't just need to give them the password but explain how the whole system works. To the people that put him in jail after firing him. That is my 2c wild speculation. I also think it's unfair to blame him for the lack of knowledge transfer. It sounds like he was the only one who had any chance of being able to understand how this network worked. Designing complex MPLS networks is not easy and I don't blame him for not spoon feeding configs to junior admins. If the city wants redundant brain power, they can pay for it. Maybe they should have thought about having more than one network engineer of his caliber. That is all.
At one point he was concerned about the security of the FiberWAN routers in remote offices, so he had them set up without saving the config to flash. 'If they go down, I'll get alerted, and connect up to them and reload the config. [...] He eventually conceded and (ahem) decided that disabling password recovery was sufficient security.â
I'm sure with the right court orders, the city can force him to surrender the passwords, and all his copies of configs.
Failing that.. for the right price, the device manufacturer might be able to do something to bypass the authentication requirement for local access, or "re-enable" password recovery methods.
(Using various undocumented backdoor techniques, that require manipulating the hardware)
Some of you are acting surprised. Of COURSE Childs is up to his neck in Ben Gay. Here, let me help you.
The majority of (not all) IT managers detest the fact that they're less than half as bright as their most competent employees. Such managers make themselves feel better by coming up with myriad ways -- some subtle and behind-the-back and others overt power pushing -- to irritate and vex and frustrate and otherwise stress such employees.
Many high-skill employees with such managers love what they do, and the high compensation, so they stay put. But they become increasingly worn and exasperated and fragile from the maltreatment and lack of appreciation.
The semi-politically savvy ones do what I do: keep your mouth shut in meetings, no matter how potentially valuable your thoughts; praise every notion your manager mentions, whether brilliant or impossibly moronic; be excessively self-deprecating and humble (an act); when it's a coworker's incompetence that causes it, let things BREAK even if you can prevent it (it's too threatening to reveal you know better); and FOLLOW POLICY AND YOUR BOSSES DIRECTIVES, even if they're bad, counterproductive, pointless, exhausting, spirit-robbing policies or directives.
The less politically savvy ones, even the ones who start out with decent dispositions, often turn into hostile, defensive and otherwise emotionally reactive shells that burn out and burn their bridges. And maybe do foolish things like Childs did.
If you're in IT, don't look for "meaning" in your work. Don't take your work personally. You'll end up crashing and burning, because IT middle management is among the worst management in all of business. (Rest assured, I speak from experience.) Collect the kick-butt checks and BE NONTHREATENING TO YOUR BOSS AND MAKE HIM HAPPY, even if that makes it impossible to produce anywhere near the quality of results you could.
Or, demote yourself back to rack-and-stack or a help desk. Because let me tell you, your boss could be a disaster for the company and you will still never beat him. All you have are intelligence and talent. Your boss has HIS boss's ear, and his boss will assume that whatever he says about you is spot-on. HE WINS. (You might even get arrested.)
Just trying to help you avoid becoming a Childs. Have watched them come and go. Typically it's without a legal prosecution, but having your mind and dignity and reputation toyed with, and in many cases being walked out the door, are still bad enough.
"Sure, the odds are 1000:1 against that I'll be hit by a bus, but there are a lot of ways disaster can strike, and they add up. You willing to ignore 5:1 odds? How about 10:1, or 15:1?"
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
This is total speculation on my part, but maybe they never asked him nicely for the PW. I could totally see the new security coordinator wanting to use him as an example to others, and threatening him with termination right off the bat.
(I hate slashdot playlets, but just this once, let us imagine the following exchange.)
Security Coordinator: I want all the passwords to the routing equipment.
Admin: Why do you need them?
S.C.: I'm not here to explain myself to you. Give me the PWs or you're fired!
Admin: OK
S.C.: So you're going to give them to me?
Admin: You haven't told me why you want them.
S.C.: That's it! You're fired!
Admin: OK, I'll go clear out my desk.
S.C.: Wait, aren't you going to give me the passwords now? Come back here! Don't walk away when I'm talking to you!
Admin: I'm sorry, but you're no longer my boss.
The End (?)
OK, a quick quiz! Did you spot the criminal act? Trick question! In this scenario, there wasn't one!
The city has a huge issue here.
This guy will have a hard time getting a job in the future, and a guy with his credentials commands a lot more than he is making right now.
If it turns out that the facts of this case are far from the original story, and nobody from the city is stepping in to correct it, then SF is in the same situation as the US when Ashcroft pointed the finger at the Anthrax guy (who recently won a big chunk of change for the false accusation).
Something tells me that the wheels of government turn slowly enough that even if they wanted to correct themselves at this point, they won't until well after the publicity is over.
That law is there to make it possible for administrators to do their work. If you are working with emails, and you happen to see a few, you don't go to jail for it.
But monitoring his bosses' email so you can tell what they are saying specifically about him is highly unlikely to be in his job description, and thus he is not protected when he does that. Nor should he be.
http://lkml.org/lkml/2005/8/20/95
So based on your statement, passwords, ACLs, social security numbers and other extremely sensitive data should be visible to the public. Could you please post all that information about your own system(s)? Otherwise, STFU.
Posting semi anonymously for obvious reasons.
The university I went to issued default passwords of the last several digits of the owners social security number.
The school is a public school in a state that had an open records law.
The open records of accounts payable and receivable of the school included the vendor/individuals TIN in the left hand column for accurate identification of vendors and individual with similar names.
The second day of the semester one could stop by the administration building and pick up a handy password list ^h^h public finance statement.
A random test at the end of the semester, from a borrowed account, indicated about one third of the student body never changed their passwords. (same for first year faculty.)
One semester I was the only person to pick up a copy even though they had four printed out just in case somebody asked for it.
The damage was mostly limited to a couple of scripts that tested that yes there was almost no security and at least half a dozen people that could trivially compromise a large percentage of the accounts on campus.
Doctor Strangelove.
Turn it whichever way you want, there's at least a couple crackheads involved. The d00d included.
Being a professional means following certain rules of conduct even if you know of dysfunctions in an organization:
a) Both the network admin and the managers should have required documentation to be delivered and maintained. If the network admin couldn't trust the city to be responsible with the documentation than the project should never have been started or he should have avoided becoming involved with was apparently a doomed system. Likewise, any management that lets a new network be deployed for any reasonable amount of time w/o receiving and verifying documentation is incompetent and should be replaced. Rather than blaming either party, it seems both brought this problem about and they should just admit it. Blaming and withholding knowledge is what junior engineers do, not heads of departments or senior IT people.
b) When the owner of a system demands password access, you give it to them....end of story. No matter what. Just like any property owner can destroy what they own...it's not the technicians responsibility to stop the owner from shooting themselves in the foot. Of course, this assumes the employee warned management and argued substantially against improper acts. I mean, he could have just said....."OK, I completely disagree with letting individual A have administrative access to the network. I believe it would cause a total disaster and I've spent the last X months and Y dollars building the network, all of which may be lost by this action. In fact, I feel so strongly against this act, that if you request me to give you root passwords...I will comply but immediately quit and not be available w/o a substantial fee to fix any mess the individual creates". That would normally send appropriate warning signals to management and may have ended the matter right there.
c) Lastly, I can't make head or tails of why the admin didn't write the router/switch config files to flash. Any equipment anywhere can fail at any time. Unlike servers, routers in certain situations can not depend on having network access to retrieve configuration files if they fail...and even downtime in seconds can be horribly painful to an organization....so, I think any good cisco engineer writes config files to flash, copies the config files regularly to a backup flash, and then makes a third backup to a remote file server. If this guy wasn't doing this, than I am finding it hard to believe that he was as much an expert as is portrayed.
I think that about covers the professional side of the matter.
I find the situation startling familiar. It's downright creepy to read this scenario. Back in the late 90s I was the sysadmin of a moderately sized ISP. When we started out I was one of three network engineers hired to build the ISP; eventually I ended up in 'charge' of the system. Like the article I also was very protective of my network, and as paranoid as this individual is made out to be. Granted I was in my 20s and suitably arrogant to boot, more on this in a moment. As time went on first one, then the other guy quit after working 80 hours a week without the possibility of time off...things only got worse as people quit. When it was down to me I made sure the owners knew the passwords to everything, but they lacked any knowledge of how to do anything. This came back to haunt me later as you'll see. Eventually I too got fed up and went to work for another company that wasn't a direct competitor. Before I left I advised management on changing all passwords for both of our sakes. I tried to explain everything but nobody understood the technical aspects. Two months later I got a visit from the FBI. 8 grueling hours of interrogation later from armed men I found out that the entire network had crashed, and I was under suspicion as having remotely logged in and crashing their system. It wasn't until later I found out they never hired a replacement, and my system simply collapsed due to lack of maintenance. It's easy to be painted out as the bad guy when you intimately know the network while being managed by a bunch of clueless twits. I don't know if that's the case in this guy's case, but I can see it working either way.
What's with the hate against busses? The number of cars in the US significantly outweighs the number of busses, and the number of drunk drivers, speeders, reckless drivers etc are significantly higher for cars than busses.
Just say "hit by a car". The speed at which it's lethal for a but, it's probably lethal for a car as well.
We do not live in the 21st century. We live in the 20 second century.
But it didn't work.
This, someone has suggested, could be because he'd nuked the root account and "root's" name was not root (so you have to crack the password AND the name).
On being sacked, they nuked his account.
Which was the one which worked to administrate the network.
Which doesn't exist because they nuked his account.
Well, one monday morning I rolled in a little late with a hangover and was real puzzled that all the programmers were playing cards, reading newspapers and doing anything except programming. When I asked I was told - "But Andy, there's *no* software".
"Oh come on, you're joking right?" "No software. No libraries, No source". (scratches head, curses, signs in with the CTO's password (heh!) and sure enough - No software. All the disk volumes except the OS wiped clean. "OK, we have a backup right?" "Sure, Alexis has it" (... in the meantime Mike, the CTO arrives). Andy to Mike - "We have a little problem...." One hour later Alexis walks in, sits down at his desk and reads his newspaper. Two minutes later, various clearly audible (from 100km away) discussions about the feeping backup.
It turns out that Alexis had gotten cross when the backup did a full rather than incremental backup so in a fit of anger he scrubbed the system clean. That folks is how Singular Computer Applications almost died c.a. 1989.
(I won't even describe the chaos the same person did to a running Novell server - things get interesting when logged in users are deleted (especially when they're still building programs)).
But you're right about the "run over by bus" thing. At Singular, if the CTO had been hosed in the early days the company would have died - he'd written most if not all of the core libraries.
Andy
I'm from San Francisco, and I can tell you that
For those of you that remember the Paranoia game that was set in a futuristic post-apocoliptic San Fransisco this is what happens when you annoy a High Programmer. They should have sent in a troubleshooting team (with lasers and tac nukes) to deal with the problem. Or maybe they still will. ;-)
I have to say this "inside story" just sounds like somebody is laying out the groundwork for a defense. Other comments have made a parallel with the Reiser trial. Rightly so, I think.
"Sure, the odds are 1000:1 against that I'll be hit by a bus, but there are a lot of ways disaster can strike, and they add up. You willing to ignore 5:1 odds? How about 10:1, or 15:1?"
This is why technical people need to strive to learn to have relationships with supervisors of a non-technical bent. From reading the article, it seems that Childs' demeanor meant that he could easily be dismissed as the brilliant-but-whacked-out-network-curmudgeon. Fair or not, that means that all of his concerns could be waved off as paranoia (for instance, him trying to get an information security policy in place). Unfortunately, the wisdom of our caution only becomes evident when a disaster occurs or is narrowly averted (e.g. "Thank God we backed that data up!").
On the other hand, non-technical managers should learn to not instantly dismiss the concerns of technical people as unlikely or unrealistic.
Technical people would do well to plan some showy but innocuous sabotage to drive the point home.
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
The master password is his preciiiouuss!
There's nothing and nobody missing this time round. A control freak who takes the BOFH stories too seriously is way more likely than someone setting up a time bomb in the network.
"I know someone who worked on the cisco side with this guy .. The dude was threatening co-workers"
What was the name of this someone, who did Terry Childs threaten, what was the nature of these threats?
davecb5620@gmail.com
Most don't understand that risk management is not just 'what is the likelihood of a bad thing happening'. It is 'what is the likelihood of a bad thing happening, AND what are consequences if they do happen'.
An exaggerated example might be: suppose I am a contractor with journeymen carpenters who never ever miss when driving in a nail, no matter the tool. Normally I have them use a hammer to drive in a nail, but want them to start using a 10 tonne block of steel because they only have to hit the nail once to drive it in. I will be able to save a lot on wages since the job will be done faster, I can sell my homes for cheaper, and drive my competition into the dust... but one problem, even though the likelihood of them missing the nail is microscopically low, the reality is that if they miss the nail with the hammer, they hurt their thumbs... if they miss the nail with the 10 tonne weight, they die. I need to find a better way to cut costs... the likelihood is small, but the risk is still too great.
-- I ignore anonymous replies to my comments and postings.
I can't help but ask if there is some ghastly mind altering substance in the water or food in Northern California?
OK in jest, but only half in jest.
Quite clearly this is a medical situation.
This poor fellow is a victim of his own enthusiasm, 24*7*365 is a big enough number to drive anybody clean off their rocker.
Perhaps it's time to form a trade or professional guild that's got a few teeth?
....than being the sole developer of a mission critical piece of software - I have intimate knowledge of this scenario, trust me.
The responsible, professional, and correct thing to do in this case (obviously the primary functional requirements come first, which in this case would entail a high degree of security) to is make the system as easy to understand as possible. No amount of documentation is ever going to eliminate the need for someone else to come along and figure out the system.
The next step is to ensure that management understand how precarious the house of cards is, when one person is responsible for something so critical.
Unless you do this, you are simply not professional.
Or maybe it's because many of us have seen manage management-types that similarly demonstrate pathologically dishonest or even criminal behavior, especially when it comes time for somebody to take the blame for a given situation?
http://www.csoonline.com.au/index.php/id;1895501252;fp;2;fpid;1
Yet the city gave hime full admin access to a critical, and sensitive system. The city also didn't bother to insure that the system was safe from being locked out in that manner.
IMO: if Childs goes to prison, the city's IT managers should go with him.
One of my professors called this the "truck factor" - the number of people to be struck dead without the project being discontinued. Higher numbers are better ..
something clever to make me stand out!
and to managers it's all your fault when that .01% happens because you talked about it so much so must have caused it!
There's a key point in no one mentioned yet. What is the City of San Francisco policy for access to passwords? Is his manager allowed to have access to the Terry's passwords. I also work for a government organization. My manager doesn't have access to the key passwords. If he were to ask for an account or root, that request would have to be signed by his manager and be submitted to to security officer's access team. He came up through my team and still has more in depth application knowledge, but by the security policy, he's not allowed root access.
Bottom line, Terry Childs may be simply be sticking the security policies in place as he understands them. You don't just do what your manager wants if that would violate the rules of the game.
If Terry's network that's a strong argument for his innocence. His best option is probably going to be to insist on a trial and handle any disclosures with an attorney making sure he isn't blamed for not ignoring the city's policies.
Look, you go to work, collect a paycheck, and go home. This assmunch doesn't OWN a fucking thing. Regardless of whether he built it or not, it ISN'T HIS. He gets PAID (six figures) to do what he is told. That includes give access to whomever the management orders him to.
He is a total jackass and deserves to rot in jail. He is not helping anyone with his current stance.
For sure Admins who dont want to share power and are egocentric socipaths are common. Some of them may be mentally deranged, have paranoia, and some may de plain assholes. Some of them may really see trouble in giving power to people whom you don't want to mess around with your config. Some of these guys do very good technical adminitration. But this is no good work. But what is bad is that his practice has been observed for years by his superiors. If I operate something whis costs millions to build, and where my operations are based upon, i should always ask myself: "If the admin get's hit by a truck, who can help me to even power the network on?". If the answer is "nobody" you are doing something wrong. In the same way, as a good admin makes preparations for him leaving, or at least designs the infrascructure with a certain orthogonality in resoponsibilities. E.g. i preferred in university that handling the backups in none of my business, Even if i do something completely wrong, even if i leave, they would be able to restore the data on a completely different system without even asking me. And when you go for no reason what, honestly giving the information for access is just professional conduct. To my experience even if you give the passwords, they dont know how to handle things.
You're doing it wrong.
No tags here.
Back in the IBM-PC days, I was the senior software engineer on the project that ported Easywriter to the IBM-PC from the Apple ][. Because the programmers on the project all smoked dope constantly, I kept backups of the work in progress every day. On the day the product was released, IBM asked John Draper where the source code was. He told them I had stolen it. The police arrived at my home and arrested me. The demanded all backup disks from me on penalty that they would issue search warrants for all my friends homes in Berkeley if I did not comply. When they led me into the lab in handcuffs, I walked over to John Draper's desk, moved one piece of paper, and there were the official release source code disks. They made me sign a piece of paper that said I hadn't been arrested, but administratively detained. I was really pissed off for years. It was absolutely my policy to keep off site backups because no one else in the company could be trusted to assure on a daily basis that we had the best code saved. This was before source code control systems. Of course I never refused to give the information to management when they asked. But they came after me for having it, when it was my job to have it. In my opinion, the clueless tell management that a crime has been committed, and management call the cops. I should have sued everyone involved in this travesty of justice. But quitting and leaving town was the best revenge, and not having to work with these people further was a benefit.
If anyone cares about the real story I recommend you read this article http://www.infoworld.com/archives/emailPrint.jsp?R=printThis&A=/article/08/07/18/30FE-sf-network-lockout_1.html . Or read the comments by Dana Horn on this wired article. http://blog.wired.com/27bstroke6/2008/07/former-san-fran.html Just because Hans Reiser was guilty doesn't mean every geek that is accused is guilty.
Panel F, Relay #70
a fascinating story. so basically the guy locked down the routers so they used passwords on the box rather than radius, and didn't save the configs to ram.
In my mind the guy will have been doing this to protect his job more than anything else, to make him indispensible.
As for recovering the network... i guess you put a snoop on each interface of a router, see what routing protocols are running, manually pick through the routes that are getting advertised, and therefore reverse engineer the routers config, router by router.
An potential enormus undertaking, depending on how many devices we're talking about. Test routers could be built up and swapped 1 by one.
If he was super paranoid though, maybe he's encrypted the routing protocols (easy to do on a cisco), in which I wouldn't know where to start (though maybe cisco would).
Good luck to the people from cisco that's all I'll say... I'd be trying to get a cheeky bill of $5000 per router for work like that.
Wow. On standby all day, all week for 365 weeks. Or months, even? No wonder he burned out.