Slashdot Mirror
The Inside Story On the San Francisco Network Hijacking
snydeq writes "A source with direct knowledge of San Francisco's IT infrastructure has tipped off Paul Venezia to the real story behind Terry Childs' lockout of San Francisco's network, providing a detailed account of the city's FiberWAN, interdepartmental politics, and Terry Childs himself. Childs pleaded not guilty to charges of tampering yesterday and is being held on $5 million bail. According to the source, Childs' purview was limited to the city's FiberWAN — a network he himself built and, believing no one competent enough to touch the network but himself, guarded religiously, sharing details with no one, including routing configuration and log-in information. Childs was so concerned about the network's security that he refused even to write router and switch configurations to flash. But what may prove difficult for the prosecution in its case against Childs is that his restricted access to the network was widely known and accepted among managers and the city's other network engineers. Venezia, who has been suspicious of the official story from the start, suspects that the Childs case may be that 'of an overprotective admin who believed he was protecting the network — and by extension, the city — from other administrators whom he considered inferior, and perhaps even dangerous.' Further evidence is that fact that the network, from what Venezia understands, has been running smoothly since Childs' arrest."
>In San Francisco, where you think they'd have no
>problem finding competent replacements.
I guess then that you've never been to San Francisco? San Francisco can't balance their budget and had a hiring freeze since 2007 and laid off a lot of people, and only had a skeleton crew running things like IT departments. So things like a network freeze were just bound to happen sooner or later.
George W. Bush isn't the only political leader in the USA who can't balance a budget and is also incompetent and has an incompetent staff. Just look at many state and local governments in places like New York and California. They all want Federal hand-outs to help balance their budgets.
Remember, Slashdot does not have a -1 disagree moderation, and no, troll, flamebait, and overrated are not substitutes.
If you post the standard 10% for release, he could possibly come up with the $500,000. By mortgaging any property he owns, he just might be able to get that.
The bigger deal is that I guess they think he's a flight risk.
-Rob
Biblical fiscal responsibility
Not to ruin a +5 interesting with facts but the article said he ended up saving the configs and disabling recovery.
So basically the devices are fine, if they reboot they will come back online.
However the only way to regain access is to factory reset which would wipe the configs.
My expectation is that Cisco or someone else is just going to use a hardware device to read the configs out of nvram bypassing wipe config recovery.
Baring that solution Cisco and a partner will likely just write a set of new configs and replace the devices one by one with new units leaving the original devices intact.
Lots of options when you have the amount of experts and cash that Cisco does and you can bet Cisco is making sure to take care of this customer so people continue to buy the products
Power cycle the network equipment. If it comes back up, pay him for the rest of the year as severance and let him go his own way. If it doesn't come back up, put him away for 10-15 years for public endangerment, and fine him whatever the cost is to the city to recreate the network and for any loss of productivity in the meantime. Either way he is a terrible admin - no one single person should be a single point of failure. What if he got hit by Muni at lunch one day?
He wouldn't write configs to flash?
It means they can't power cycle or reboot anything, or the network is screwed.
No device stays up forever.
It also means they just have to power cycle a switch to gain access to it, and then do what they can to figure out how it was configured.
IOW: They have to break it to fix it.
If you post the standard 10% for release, he could possibly come up with the $500,000. By mortgaging any property he owns, he just might be able to get that.
You understand how bail bondsmen work, right?
That $500,00 doesn't remain his, it becomes the bondsman's cut for getting him out of jail.
Even if he can mortgage his property, it's still bullshit that he has to lose half a million dollars just to get bail.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
You've obviously never worked for local government.
If the masses can keep you down, you're not the Ubermensch.
Putting a network card into promiscuous mode is not the same as using root access to copy your boss's email store, nor is it running the password file through jack the ripper.
This sort of logic is where nerd myopia falls right on its four-eyed face. If he was reading in on personnel-related email, it really doesn't matter what measures he used, he still fucked up. Especially so if he acted on them.
The argument that his bosses were l00s3rz because they were conducting normal business through email without any special encryption doesn't fly anywhere sorry. Professional job, professional rules, l335ness does not apply.
Business. Numbers. Money. People. Computer World.
"There is a big difference between "in the performance of their duty" and "because are able to do so, they felt like doing so and so they went ahead and did so.""
The thing is you have to prove it that an admin did it for BOFH style "shits and giggles" or some other motivation other than official use - beyond a reasonable doubt. That's a pretty big hurdle for a prosecution. Some would call it an impossible hurdle.
That's for when the email is in-flight. Once it hits storage, an admin basically has free reign. As email gets older and older, it gets less protected. Beyond 180 days it's unprotected - the gubmint can even do a search without a warrant.
Email isn't as protected as paper documents, as the last time this came up before the 6'th circuit, it was refused review on procedural grounds.
Don't ask me, go read the law yourself. ECPA of 1986.
If you think that the legal privacy of email is pretty weak because of the ECPA, this was an *improvement* on privacy back in 1986 because prior to that, email was basically equivalent to shouting out the window (and sometimes still is). Once the ECPA passed, BBS operators like myself became paranoid so we decided to put up disclaimers announcing that users should not expect privacy. Such disclaimers during login and registration notified the users and thus shielded the admin from privacy lawsuits and such. Some people think that this gets rid of plausible deniability, because once you say your users have no privacy, the guys in the FBI PartyVan parked in your driveway might suspect that you know what your users are doing, or so the theory goes. But a section of the CDA of 1996 supposedly shields the admins from the actions of a service's users. It gets really complicated if you research even a little bit of this stuff.
--
BMO
I think you're full of brown smelly stuff.
The ECPA only seems to apply to common carriers and public information services. I don't see any evidence it provides any liability for the sysadmins of internal networks.
If you're not IANAL, here it is:
http://www.usdoj.gov/criminal/cybercrime/wiretap2510_2522.htm
And even if so, you're being really retarded if you think that reading his bosses' email falls under the "system monitoring" provision of the law.
Business. Numbers. Money. People. Computer World.
"The burden of proof is on you to back up your bullshit, and I'm a calling you on it. Quote some laws here, if you can."
I'll do you one better:
I'll point you at a book on the matter:
http://www.amazon.com/Netlaw-Your-Rights-Online-World/dp/0078820774
And I'll quote from here:
http://www.rbs2.com/email.htm
The executive summary of what I've been talking about and what you've been talking out your ass about:
"Reading e-mail that is stored on a computer is not an "interception" under 18 U.S.C. 2510, et seq., because an interception must be contemporaneous with the transmission of the message between different locations. Steve Jackson Games v. U.S. Secret Service, 816 F.Supp. 432, 442 (W.D.Tex. 1993), aff'd, 36 F.3d 457, 460 (5thCir. 1994). This holding has been accepted in several subsequent cases, including Wesley College v. Pitts, 974 F.Supp. 375, 384-390 (D.Del. 1997); U.S. v. Moriarty, 962 F.Supp. 217, 221 (D.Mass. 1997); Bohach v. City of Reno, 932 F.Supp. 1232, 1235-36 (D.Nev. 1996)."
--
BMO - Not a lawyer, but dammit I can read for myself.
My best guess from my understanding of SFGOV is that his boss answers directly to the mayor.
Most of San Francisco government answers directly to the mayor. San Francisco is a city and county so it has no city council, or city manager only county supervisors, a controller and a mayor, along with many other oddities that are only in San Francisco.
Fortunately/unfortunately there is a civil service commission and fairly strong employees unions that cover all but the political appointees, somewhat muting the mayors power. (oh and the board of supervisors and the voters can override the mayor when ever they feel like it, but the mayor still has ultimate control, unless he has been over ridden. )
Work bio at MMWD
I have been following this story in the San Francisco Chronicle.
According to their reporting, he was asked by management for the passwords. He said no.
Then he was asked by police for the passwords. He still said no.
Then they had him arrested.
The reporting by various news organizations has been marred by confusion
on the part of reporters and average people between controlling network hardware and controlling
data on various servers. They often seem to describe it as data being
stored on the network he controlled.
It has been my experience that non-technical people do not really know what a server is.
These days, most people have an idea what a network is (like the Internet). So they
think either their data is stored on their desktop/laptop or it is stored on the network.
If it is true that he sometimes did not write the router configs to flash, that sounds to
me like a "deadman switch." If he got hit by a bus, how would they service the UPS that
backs up the router? The batteries eventually need to be replaced. He may have built it
so that he had to be around for years to keep it running. It is running fine for now,
but they can't power anything off without potentially losing its configuration.
One city official essentially said something like "Worst case, we hire someone to reconfigure
or replace the entire network."
Since he plead not guilty, he will get a trial to determine if what he did was criminal.
"We can't solve problems by using the same kind of thinking we used when we created them." -- Albert Einstein
This all depends on who accepts it when. If when Childs started his lockdown, he was under the watch of a manager that either didn't care, or agreed with him, and so he did as he pleased.
A lot of IT organizations have a single person that is the core, the one person that has comprehensive knowledge of all systems and fully understands how they interact. These are the people that are brought in on any major problem or decision, and whose input counts more than double. (and often simply hold "veto power") Now I'm not saying this is a good thing, I'm just saying it happens from time to time and you have to accept that. Some systems just evolve in this direction, and once they get past a certain point, it becomes very hard to change them.
There IS one easy way to solve these problems, but it involves the managers taking a walk out on a shaky limb and take some heat. One example is a week of paid leave. On Monday Joe's manager announces "Joe is on paid leave effective immediately. (no warning to Joe OR the staff in advance of this) Go home Joe, see you in a week and enjoy your paid time off, courtesy of the company." Then, "OK for the next week you are on your own. NO ONE is to call, page, IM, email, or otherwise contact Joe for ANY REASON. Joe got hit by a bus this morning on the way to work, that's how you will behave. You are to keep written track of every problem you run into this week that you would normally rely on Joe to help with. Do not simply shelve problems for next week - treat them like Joe is never coming back." Any critical questions you bring to ME, and I will call Joe if it's really necessary, but I will not be happy about it, and be prepared to justify to me that you've already tried everything else possible. If I find someone is hiding problems for next week there will be serious disciplinary action taken.
Needless to say, when Joe gets back on Monday, the next 2-4 weeks will probably be planned out, documenting things and teaching people how to do stuff. You could also make this a two week leave depending on your situation. If you're a big organization, the longer the better, but at the worst three weeks will shake out most of the bugs. This also gives the managers a very clear picture of how well distributed knowledge is within the department. You've probably heard someone say "but what if you got hit by a bus tomorrow?" when discussing something you are the only one that knows how to do. Now you get answers. We call this the "hit by a bus test". Any decently sized IT department with one central person should conduct this test periodically, say every two years. The first one should be a gimme. If on the second test, things have not improved over the first, time to take disciplinary action. Letting one of your staff continue to hold the keys to the kingdom is unacceptable and is everybody's fault to some degree.
I work for the Department of Redundancy Department.