Slashdot Mirror


San Fran Hunts For Mystery Device On City Network

alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."

45 of 821 comments (clear)

  1. The story keeps changing. by khasim · · Score: 5, Insightful

    From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.

    Given that they cannot hunt down a single device on the network, I'd have to agree with that assessment.

    MAC address ... switch port ... it should be easy.

    1. Re:The story keeps changing. by DogDude · · Score: 4, Insightful

      1. Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.

      2. Assuming that they have wireless on their network, there's no way to find wireless devices, since they can be put inside of locked buildings. Unless your name is "Superman", there's no real way to find exactly where wireless devices are, as far as I know.

      --
      I don't respond to AC's.
    2. Re:The story keeps changing. by goose-incarnated · · Score: 4, Insightful

      ... Unless your name is "Superman", there's no real way to find exactly where wireless devices are, as far as I know.

      And exactly how would superman find it? Xray vision? How would he then know he found it?

      --
      I'm a minority race. Save your vitriol for white people.
    3. Re:The story keeps changing. by moderatorrater · · Score: 4, Insightful

      Agreed. If they're still having problems at this point, they're incompetent jackasses. However, that's not an excuse for the employee to be a jackass too.

    4. Re:The story keeps changing. by the_B0fh · · Score: 3, Insightful

      2) It's a freaking terminal server. How many wireless terminal servers have you seen?

    5. Re:The story keeps changing. by Crudely_Indecent · · Score: 5, Insightful

      If Superman had any IT skills, he'd perform a traceroute to determine the devices gateway. Once the gateway was determined, block the mac address from accessing the network. If the admin of that device is worth his salt, he'll change the mac address and continue. They could then specifically enable allowed devices and forbid all others.

      Forget finding it, make the network inaccessible.

      City of SF Admins, if this proves to be your resolution, you owe me $150 for 1 hour of my time. Sorry, I do not bill in lower increments.

      --


      "Lame" - Galaxar
    6. Re:The story keeps changing. by mollymoo · · Score: 3, Insightful

      From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.

      That and setting up the routers so they lose their configuration on reset. Even if your boss is an idiot, you get your concerns on the record and a direct instruction on the record and then do what you're fucking well told.

      --
      Chernobyl 'not a wildlife haven' - BBC News
  2. Siding with the network guy by John+Jamieson · · Score: 5, Insightful

    Man, the more I read about this story, the more inclined I am to believe the network admin.

    He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.

    1. Re:Siding with the network guy by sgtrock · · Score: 3, Insightful

      Then you've never worked for the kind of clueless idiots this guy was working for. Supervisors do NOT need access. Any competent manager knows that's the case. What's needed is more than one competent individual to have access, with backup keys kept in sealed envelopes that are kept in a safe with only logged access to it in case both are hit by a bus on the same day.

      BTW, did you miss the part of the case where for _years_ the admin in question begged, _BEGGED_ for someone else who was competent to be hired so he wasn't a single point of failure? That he continually pointed out that there was no DR plan whatsoever?

      Nope, this guy made a serious error in judgment in not making sure that the mayor's office had the access information ahead of time. His supervisors are clearly incapable of administering that network and shouldn't be let anywhere near a console.

    2. Re:Siding with the network guy by geminidomino · · Score: 3, Insightful

      His actions were extremely stupid, but I fail to see why this idiot's relatively non-disruptive actions rise to the level of criminal prosecution.

      Thou shalt not expose the government's incompetence.

  3. The City of SF is undermining its case! by StandardCell · · Score: 3, Insightful

    If the city can't even complete one of the most basic network administration tasks of finding a physical device on a network, I think they have absolutely no right to accuse anyone of "hijacking" their network. I hope the defense attorney for Terry Childs brings this up.

  4. not necessarily wrong... by damn_registrars · · Score: 5, Insightful

    your employer's passwords are NOT yours, no matter how stupid you think your boss is.

    Refusing to give out passwords to higher-ups is not always the wrong thing to do. If you are the network admin, and your job is to maintain security of the network, wouldn't it be reasonable to refuse to hand out passwords to people outside of the network administration roles?

    Although I can say that an admin can make that choice at his or her own peril. After all, the higher-ups can always opt to fire the admin and replace him or her with someone who is willing to seek security of their job over security of the network they are paid to administer.

    --
    Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
    1. Re:not necessarily wrong... by Lonewolf666 · · Score: 4, Insightful

      Agreed.

      If a boss I don't entirely trust demanded my password, I'd offer to upgrade his account to the same privileges at mine, but he'd NOT get MY password.

      The reason is that if he does something stupid that will show up in logfiles, he can damn well do it on his account and get logged doing so ;-)

      --
      C - the footgun of programming languages
    2. Re:not necessarily wrong... by Vancorps · · Score: 3, Insightful

      I'm confused, does any admin ever give up his own account password?

      In my company we have a blanket policy, never give out passwords, ever... as admin I don't need someone else's password to get into their mailbox and retrieve information that's needed by another employee while the content owner is out of contact. Of course I always notify the mailbox owner that I had to go in as I have to have a specific reason.

      Are there environments out there where you would be expected to give up your password? I can understand keeping a password database for service accounts which all admins should be able to access if they manage it but I can't imagine a scenario when I'd need someone else's password. Even if the thing is encrypted, I have the recovery key so again I don't need their password.

  5. Re:to quote bash.org... by FireStormZ · · Score: 4, Insightful

    The admin might not be stupid he might be an ass

    1) He placed a rouge device (his personal property) on the SF network
    2) He set all the network devices on the network to lose all info on a reboot
    3) He will hand over the passwords (after jail) to all the devices except the rogue

    You can make equipment hard to find ( mac masquerading comes to mind )... I'm only adequate in terms of networking but I am pretty sure someone who is really good can play a mean game of hide and seek. Who knows *what* he was doing with that device? and were I the network admin I would have to *on principle alone* rebuild everything after this guy left..

    --
    "Ahh! Arrogance and stupidity in the same package, how efficient of you!" --Londo Molari
  6. Mod Parent Up by mpapet · · Score: 5, Insightful

    I'd like to add that while the way he handled being surrounded by idiots was wrong, he was clearly surrounded by idiots.

    No documentation?
    No change control?
    No diagrams?

    What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.

    I worked one place where a guy with a great deal of responsibility died. (here today dead tomorrow kind of thing) His peers blamed *everything* on him simply because they could. This sounds like the same thing.

    --
    http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
    1. Re:Mod Parent Up by AioKits · · Score: 4, Insightful

      What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.

      Well, every Stalin needs his Trotsky!

      --
      "Quote me as saying I was mis-quoted." -Groucho Marx
    2. Re:Mod Parent Up by Sobrique · · Score: 5, Insightful

      Wait, you mean blame it all on the guy who left (be it through death or a cushy new job) isn't standard practice everywhere?

    3. Re:Mod Parent Up by rickb928 · · Score: 5, Insightful

      I took a gig recovering documentation and re-establishing procedures for a great admin who died as well. He really did great docs, but no one had ever used them, and they couldn't figure out the 'copy file piopoiop.dfj to the \asic\wer\2344\sdf.msdfn folder' sort of directions.

      And the crew there immediately set to removing, replacing, and destroying all of his systems. He was a Novell hardliner (so was I), and when he was gone, his boss succumbed and the Windows bigots prevailed. Much taxpayer money was spent replacing perfectly functional systems. Mind you their clients were still running Novell, so there was some disconnect when they would get a request for support and start saying 'you have to upgrade (ha!) to Windows'. Their clients, for reasons best left undisclosed, could not upgrade. Both physically impossible and logistically impractical. Start with being 60-1600 meters below the ocean surface, and it only gets more difficult from there.

      I'm a little surprised that SF hasn't worked this out. There are plenty of outfits eager to do what is necessary, for a fee of course.

      And yes, finding a device is not impossible. Finding the connection to the network is the obvious first step. After that, well, kill it.

      Unless it's hiding. That would be unfortunate.

      ps- This guy, by many accounts, was brilliant. And a little off the wall. Goes together.

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    4. Re:Mod Parent Up by mrjohnson · · Score: 3, Insightful

      I don't get it. The thing's gotta have a mac address that can be found on a switch somewhere. That'll give you a port number and a patch cable to follow until it's found.

      Nah, it's way more fun to blame the guy in prison.

    5. Re:Mod Parent Up by ElizabethGreene · · Score: 3, Insightful

      There were network diagrams, they indicated they found several copies at his house. No, I don't consider that unusual at all. I carried one in my purse at my prior employer, and a electronic versions on my pda and laptop. When the pager goes off you want to fix it NOW, not drive in to get a circuit ID off of a piece of paper on a wall. They indicated there was some documentation as well, and there was some on the (It didn't have a password on it until Paul Venezia ran an article about it.. thanks Paul) Disaster Recovery sharepoint site. Change management is still kind of fuzzy. They indicated they found some "Configuration files" on his PC, but didn't happen to mention if they were date stamped. -e

    6. Re:Mod Parent Up by _Sprocket_ · · Score: 3, Insightful

      I'm a little surprised that SF hasn't worked this out. There are plenty of outfits eager to do what is necessary, for a fee of course.

      From the article...

      After a dramatic jailhouse meeting with San Francisco's mayor one week after his arrest, Childs handed over the data, but DTIS Chief Administrative Officer Ron Vinson said Wednesday that the city now expects to spend more than $1 million to clean up the mess. To date, DTIS has paid out $182,000 to Cisco contractors and $15,000 in overtime costs, he said in an e-mail interview.

      The city has also set aside a further $800,000 to address the problem. Vinson did not specify what the additional money was expected to cover, but if the city has to hire network consultants to remap, reconfigure and lock down its network, this would not be an unreasonable estimate. The city has also retained a security consulting firm called Secure DNA to conduct a vulnerability assessment of its network.

      And there you have it folks, a million-dollar employee; over-worked and under-appreciated by a management too incompetent to understand the issues the guy dealt with much less manage him and his work effectively. Sadly, it's not a very uncommon story.

      One of the fun bug-a-boos that show up in these stories is the cost of damage an intruder (or in this case, rogue employee) "causes" the target. I've been on the inside of a number of US Government incidents and seen the cost estimate damages. To someone on the outside, they seem pretty insane. The question that the public often asks is something like "how can changing one password cause so much damage?" But the numbers I've seen are pretty much on target (plus or minus some variance for estimates) - they represent real expenses associated with work to properly ensure the systems are truely owned by their rightful owners again. And they cover resources (i.e. hard drives) lost to criminal investigative bodies / evidence lockers. But the real gotcha to these things is that these expenses either should have been spent as part of the normal management cycle without an attached incident or, even better, could have been a fraction of the eventual cost if the resources were spent to improve the environment or hire proper talent in the first place.

    7. Re:Mod Parent Up by OnlineAlias · · Score: 4, Insightful

      Could be an IBM 3174 like device too, running SNA. Fact is, the article and and court filings aren't clarifying any of this and leave the door open for mass amounts of conjecture and sensationalizing, both in the media and on Slashdot. Which, of course, is exactly what everyone is doing...

  7. Re:MAC search by Archangel+Michael · · Score: 5, Insightful

    I learned early on, that most people don't see the difference between a $12 hour high school geek and a $75 hr network administrator. All most people see is that both do roughly the same job and there is $63 hour difference.

    Most of the time, the $12 hr guy is doing most of the same work as the $75 hour guy. The big difference is when crap like this comes up, the $12 hour guy can spend years trying to figure out what the $75 hr guy can figure out in 5 minutes.

    Even when the $12 hr guy screws up, the response is "But he was cheaper". It is cheaper to keep a $12 hr guy trying to keep crapware off a computer, rather than a $75 hour guy who doesn't allow crapware in the first place.

    The point I'm making, is that a $75 hr guy is worth it, but only to people where time has real value. People who place no value on TIME, don't care about anything other than $ per HR

    --
    Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
  8. Re:You're an 1D10T by denis-The-menace · · Score: 4, Insightful

    I wish I had mod point for you.

    Chances are that internal policies prevent the use of "hacker" tools to secure the network.

    Again, the PHBs are idiots!

    --
    Obama's legacy: (N)othing (S)ecure (A)nywhere and (T)error (S)imulation (A)dministration
  9. Simple co-dependency by Anonymous Coward · · Score: 3, Insightful

    If you find that you are "holding the place together", IT-wise, you are likely part of the co-dependency and are part of the problem.

    IT and the other management have both agreed to ignore each other, literally or otherwise, allowing each (and the individual personalities) to do things "their way"; damn the best practices, good management, logical, financial, or even legal issues.

    Except when things go wrong.

    Like a breakup, they can get ugly. And, as the IT guy, you will always lose for it is not your Business, but theirs. You are simply hired help.

  10. Re:to quote bash.org... by russotto · · Score: 4, Insightful

    2) He set all the network devices on the network to lose all info on a reboot

    I wonder if this one is just a complete misunderstanding. One article says that they were set to lose configuration files on "reset". That's pretty typical -- if you have some device you don't have the password to, you can do a full factory reset and get it back to the default password, but that also wipes the configuration files. He might have told his incompetent bosses that, and they thought he meant they'd lose the files on a reboot instead.

    Anyway, if this guy is what they're making him out to be, they need to completely wipe and reconfigure the network anyway; it's the only way to be sure he didn't leave a few presents for them.

  11. Admin code of ethics. by khasim · · Score: 4, Insightful

    What would you think of a doctor who, because some exec somewhere decided he should, pushed the WRONG medication / procedure to you?

    Where does your ethical responsibility end and the boss's desires begin?

    To me there isn't even a question. Fire me. Go ahead. I will get another job.

  12. Re:You're an 1D10T by Archangel+Michael · · Score: 4, Insightful

    Yes, both of those are true (Mac, Ping). Even NMAP responses can be spoofed. However the likelihood of all three being done is not likely. However NMAP will reveal a used IP, and a mac table somewhere will identify what port it is hanging on. Packets have to be routed to it somehow.

    And I agree with your last point. I'm a Libertarian. ;)

    --
    Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
  13. Re:I've Changed my mind. by Medievalist · · Score: 4, Insightful

    Oh yeah, let's give him a break. Oops, he's been by hit by a bus. Where's his disaster recovery plan? That's right, there isn't one.

    My bet is, it's sitting right in the middle of his old desk blotter, in a fat manila folder marked "Disaster Recovery and Service Continuity Plans". These clowns would never find it there in a million years. The infamous missing passwords are probably in a letter-size envelope in the top left desk drawer, too.

  14. Don't mod that "funny". by khasim · · Score: 5, Insightful

    It appears that the idiot "boss" is attempting to generate support for the claim that this guy is a "problem" by paying unreasonable amounts to "repair" the "damage" he did.

    It's difficult to "prove" that a guy did millions of dollars of "damage" ... without a bill for millions of dollars of "repairs".

    Any competent network admin could map out the network and document it for FAR less than the hundreds of thousands of dollars that is being thrown about.

  15. Not always by Weaselmancer · · Score: 3, Insightful

    When users ask for Admin privilages, they should be told to go fsck themselves. No matter who they are.

    I'm a software developer. For the first few weeks working here IT wouldn't give me admin rights on my own box. I couldn't install software.

    So I sat here and did nothing. Not because that's what I wanted. But because that's all I could do, until they gave me permissions on my machine.

    Generally speaking, you're right. Most people in a business should be locked down. But not everyone. Depends on the person - depends on the work they're doing.

    --
    Weaselmancer
    rediculous.
  16. Re:Malice and stupidity. by funwithBSD · · Score: 3, Insightful

    Big assumption.

    They probably deleted all those "useless files" on the fileserver when they fired him.

    And the "terminal server" is probably his iPhone...

    --
    Never answer an anonymous letter. - Yogi Berra
  17. Re:Simple: by cecille · · Score: 4, Insightful

    ...you tell us, Mr. "anonymous".

    --
    ...no two people are not on fire.
  18. Re:Simple: by funaho · · Score: 3, Insightful

    Oh man, that is so hilarious. I love this part especially:

    I cannot find any information in my MCSE bootcamp journal on how to handle this

    Just more proof that MCSE certification is completely useless other than for getting a job. :)

  19. Re:Malice and stupidity. by moxley · · Score: 3, Insightful

    I disagree.

    It isn't that simple; it seems that there is waaaaay more to the story that some ego tripping sysadmin.

    Everytime another piece of the story or fact about what happened comes out it seems to vindicate Mr. Childs to some degree (not that his judgement was flawless in how this was handled, but still).

    Is he still locked up? If so it's a travesty.

    It seems like those who are trying to have him tarred and feathered constantly want to make it look like he's some super-e-terrorist who was holding the entire city for ransom and has dealt an economic blow from which the city will never recover.

    I am not saying everything he did was right, or that he committed no wrongs here; but I think it's pretty obvious that this was viewed as a pissing match by those in the city who wanted him to hand over that information and they have gone to great lengths to make it look like something much more malicious than it was in the press.

    He may have had very good reason to protect it; (I mean aside from the fact that it appears as though those who wanted him to hand it over were incompetent) - because I don't think anyone would put their own ass on the line for jailtime and the loss of their job unless there was something else going on. I am not saying I know this to be true, just that that is how it appears to me based on the available information.

    At this point I view anything coming from the anti-Child's side of this issue with a healthy does of skepticism and try to read through the sensationalization. Something has always stunk about this situation.

  20. Re:Simple: by CrossChris · · Score: 5, Insightful

    MCSE:

    Must Consult Someone Experienced

    Minesweeper Consultant and Solitaire Expert

  21. Re:Simple: by interiot · · Score: 3, Insightful

    It would be a lot harder to track down that way.

    Not really. A network admin should be able to track down the thing, but it will take a lot of work to scan network logs. From the network standpoint, it doesn't matter if the gateway is running on a PC, or running on a VM inside a PC... the network traffic looks the same.

  22. The new WarLords by DeanFox · · Score: 4, Insightful


    I'm reminded of a conversation I had some 25 years ago with a co-worker IBM mainframe technician. IBM management was incensed that uneducated morons turning screwdrivers could make 70k a year. Back then as much as what they were paying top MBA stuff shirt types. They were on a mission to get salary levels down to "reality" paying these screwdriver wielding monkeys what they were (in their minds) really worth.

    Attitudes have changed but not a lot. 93% of companies that loose their data center for 10 days or more due to a disaster filed for bankruptcy within one year. 50% filed bankruptcy immediately (National Archives & Records Administration in Washington). One can't say the same thing about those over paid MBAs.

    It may be awhile before IT matures into a "profession" like doctor or lawyer however I personally believe we're holding the keys. The world can't function now without us.

    -[d]-

  23. Re:Simple: by ajrs · · Score: 5, Insightful

    and your not sniffing the traffic to these boxes why?

  24. Re:You're an 1D10T by gad_zuki! · · Score: 5, Insightful

    >But everyone who supports more government ought to take a look at the incompetence here.

    Im one of those crazies who doesnt support more or less government. Just better government.

  25. Re:Simple: by kimvette · · Score: 4, Insightful

    Nobody knows what the passwds are to get into them so I can't log in and find out what they do.

    1. Boot from floppy, optical media, network, etc.
    2. mount [/dev/sda1|/dev/hda1] /mnt -o rw
    3. chroot /mnt
    4. passwd root [password]
    5. ??????
    6. PROFIT!

    No yanking to do. A reboot and 5 minutes of down time. Bang. Dead. Done.

    --
    The Christian Right is Neither (Christian nor right). See: Matthew 23, Matthew 25, Ezekiel 16:48-50
  26. Re:Simple: by Firehed · · Score: 5, Insightful

    Poison gas ? You think that's all an evil supercomputer will do ? NO ! It will spontaneously develop godlike powers, take over the universe and unravel the very fabric of reality around you !

    It may also mock you with nonexistent cake.

    --
    How are sites slashdotted when nobody reads TFAs?
  27. Re:Simple: by blind+biker · · Score: 4, Insightful

    I have a huge admiration for your honesty. You are an exceptional person.

    --
    "The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
  28. Re:Simple: by Firehed · · Score: 3, Insightful

    Holy crap, +5 insightful? I like my karma as much as anyone else, so no complaints, but... huh?

    --
    How are sites slashdotted when nobody reads TFAs?