San Fran Hunts For Mystery Device On City Network

alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."

Comments

Simple:

[SilentBob0727]

Power cycle it with a city-wide EMP.

Re:Simple:

[Ethanol-fueled]

All they have to do is look for the small black box with a lone, onerous blinking red LED.

Re:Simple:

[bratwiz]

All they have to do is look for the small black box with a lone, onerous blinking red LED.

Don't forget the obligatory RED and BLUE wires. Every small black box with lone onerous blinking red LED MUST have red and blue wires. Its a rule.

Re:Simple:

[74nova]

I beg to differ. "Loose" devices are exactly what they're looking for

Re:Simple:

[elrous0]

As someone who watches a lot of movies, I think I can help them find it. I suggest you look for the ominous looking computer with a single red eye. You'll know you're close when it activates some devious self-defense system (probably involving poisonous gas). Pay careful attention to the background music, as it will provide valuable cues on when to run.

Re:Simple:

[iced_tea]

Could it be possible that the device is actually virtual? Like a Virtual Machine running under VMware or Virtual PC somewhere, with the software obfuscated or hidden? It would be a lot harder to track down that way.

Re:Simple:

[DittoBox]

"Onerous?"

You mean like...I don't know, dictionaries?

Re:Simple:

[clone53421]

An EMP disrupts electronics by inducing massive currents in the thin circuitry of the circuit boards and integrated chips. They're permanently burned. They won't power-cycle, they'll just fry.

Naw... if you really want to power-cycle it, just disrupt the electrical service to the entire city. You'd probably have to leave it off for a fair length of time, though, in case the device was on UPS.

Re:Simple:

[mcgrew]

It could be both onerous and ominous.

Re:Simple:

[RetiredMidn]

More likely multiple LEDs, like this.

Re:Simple:

[UnknowingFool]

No, no, no! You have to obliterate the planet from orbit. It's the only way to be sure.

Re:Simple:

[Provocateur]

No, cool would be having the phone ring and the voice on the other end turns out to be Dennis Hopper:

Pop quiz, hotshot: your network's all screwed up! What do you do? What do you do?

Re:Simple:

[MyLongNickName]

Could it be related to this firehose entry?

Re:Simple:

[JamesP]

Reminds me of a guy I knew who used piezoelectric fire lighters (it's the one used in stoves) to test the watchdogs on circuits he built.

He fired it over the processor and the interference would be enough to disturb it (electrically isolated of couse, the spark would not go to the device, only the EM interference).

Re:Simple:

[Indras]

No, no, no! You have to obliterate the planet from orbit. It's the only way to be sure.

Didn't work in Star Wars: Knights of the Old Republic.

Re:Simple:

[cecille]

...you tell us, Mr. "anonymous".

Re:Simple:

[NormalVisual]

"Whoa!"

Re:Simple:

[Anonymous Brave Guy]

Pay careful attention to the background music, as it will provide valuable cues on when to run.

Oh, please. Movies like that are soooooo 1990s!

In the new century we handle this sort of scenario with game techniques. You just save the current state of the world every few seconds, while sending your guy out into the field. There won't be any change in the music until it's too late for him, but then you just reload, activate all his power-ups, and go kick the red-eye'd mystery device back to where it came from.

Just be careful if the red eye is moving from side to side and you catch a glint of silver. Those guys from the sci-fi shows are trying to muscle in on our turf.

Re:Simple:

[funaho]

Oh man, that is so hilarious. I love this part especially:

I cannot find any information in my MCSE bootcamp journal on how to handle this

Just more proof that MCSE certification is completely useless other than for getting a job. :)

Re:Simple:

[Windows_NT]

Ive heard stories that relate to this. And its not that someone outside hooked this piece of equipment up, its something they have forgot about.
I read about a server that was in a room, and the room had some modifications done to it, and they ended up drywalling the server inside the wall (i dont know know how they did it). It ended up being like 5 years later they had no idea where this PDC signal was coming from and they had to physically follow the network cable to the computer and found it.
I found the story, kind of:
Server 54

Re:Simple:

[fataugie]

Or the guy defusing it is color blind
(that's why he's wearing orange pants and a lime green shirt).

Re:Simple:

[CrossChris]

MCSE:

Must Consult Someone Experienced

Minesweeper Consultant and Solitaire Expert

Re:Simple:

[gsgriffin]

I'm putting my money that its a Mac server that everyone passes by and says, "Oh, that's Mac, it couldn't possibly be that. Why bother checking. It must be from the Evil Empire. We're looking for black, not white."

Re:Simple:

[interiot]

It would be a lot harder to track down that way.

Not really. A network admin should be able to track down the thing, but it will take a lot of work to scan network logs. From the network standpoint, it doesn't matter if the gateway is running on a PC, or running on a VM inside a PC... the network traffic looks the same.

Re:Simple:

[TheoMurpse]

I'm sorry, San Francisco, I'm afraid I can't let you do that.

Re:Simple:

[gardyloo]

Oh, good point. Now the onus is on him to explain what he meant.

Re:Simple:

[Intron]

I think we're unanimous.

Re:Simple:

[clone53421]

The real question, though, is this: If your alternate personality made the bomb, does your present consciousness have the subliminal knowledge of which wire defuses it?

Re:Simple:

[MPAB]

And because of Murphy's law the drywalled server never overheats or has downtime, unlike its well-cared-for counterparts.

Re:Simple:

[ShadowBlasko]

The real question, though, is this: If your alternate personality made the bomb, does your present consciousness have the subliminal knowledge of which wire defuses it?

Depends on when it was I guess.

Back in 2001 I did some emergency wiring work that had to be done in 72 hours at our shop.

Now, we are only there 10 weeks a year, so after the end of the 10 weeks it was forgotten about.

I was very sleep deprived and manic when I finished the job, and to this day I have NO idea how I did some of the connections I did. I just hope and pray it all keeps working. Some day some part of it will fail, and I'll have to re-do the entire building.

Note to self:

When sleep deprived, always work from the list, and write down what you did. One thing at a time, and document everything.

Re:Simple:

[Lord Apathy]

Not at all uncommon. I've got 3 fucking servers in my system room that nobody knows what they hell they are for. The are all running 2.4 kenels so they are as old as the fucking hills. Nobody knows what the passwds are to get into them so I can't log in and find out what they do. And naturally the previous systems administrator that installed them didn't document shit.

The only thing that is known about them is they used to do something important just nobody remembers what it was. Management is to afraid that they might still be doing something important and won't let me yank them out to find out what they do. So while management sits there with their collective heads up their collective asses these three servers sit there taking up space in my racks on my network.

When these thing do finally fall over I hope they are doing something important.

Re:Simple:

[ajrs]

and your not sniffing the traffic to these boxes why?

Re:Simple:

[rah1420]

they ended up drywalling the server inside the wall

For the love of God, Montressor!

Re:Simple:

[Lord Apathy]

Because I'm a fucking dumbass and didn't think about it....

Re:Simple:

[Windows_NT]

Sorry the link doesnt work? here: Server 54 Story

Re:Simple:

[clone53421]

Maybe he knew you'd know, and he spent all day thinking about the wrong wire.

Re:Simple:

[clone53421]

With a username like "Lord Apathy", I'm guessing he isn't being paid enough to care that much.

Re:Simple:

[KillerBob]

If you're really lazy, you could also unplug their network cables and see what breaks... :P

Re:Simple:

[macdaddy]

Because I'm a fucking dumbass and didn't think about it....

Are you waiting for someone to disagree with you? ;-)

Re:Simple:

[interstellar_donkey]

Modern rouge networked devices don't have red and blue wires. They vibrate. Usually it's someone's electric razor connected to the network, but ever once in a while, it's a dildo with an IP address.

Re:Simple:

[MarkGriz]

Because I'm a fucking dumbass and didn't think about it....

You should apply for a job at the San Francisco IT department. I hear they are short an incompetent network administrator.

Re:Simple:

[Lord Apathy]

If you're really lazy, you could also unplug their network cables and see what breaks... :P

I figured that once I yanked them out of the racks we would see who bitched first then we would know.

Re:Simple:

[kimvette]

Nobody knows what the passwds are to get into them so I can't log in and find out what they do.

1. Boot from floppy, optical media, network, etc.
2. mount [/dev/sda1|/dev/hda1] /mnt -o rw
3. chroot /mnt
4. passwd root [password]
5. ??????
6. PROFIT!

No yanking to do. A reboot and 5 minutes of down time. Bang. Dead. Done.

Re:Simple:

[Lord Apathy]

With a username like "Lord Apathy", I'm guessing he isn't being paid enough to care that much

And you would be correct.

Re:Simple:

[clone53421]

No, it sounds like they have plenty of those. They got rid of the competent guy, remember...

Re:Simple:

[ShadowBlasko]

always work from the list, and write down what you did. One thing at a time, and document everything.

This seems sensible under all conditions. Being tired is no excuse for being sloppy.

I have a sleep disorder.

There are times when, for no real discernible reason, my brain decides that I will not be sleeping for a few days. Sometimes upwards of 100 hours.

When you have been awake for 4 days, (at least in my case) you get a serious case of "While I'm at it" syndrome.

Tasks that can not be completed in 10 minutes (or without getting up) are nigh impossible. I can still work, but I am extremely easily distracted and will often forget why I am in the room I was in.

Example: I went to the fridge to get some water, and decided that I should clean it while I was there, then decide to do the dishes since I threw stuff out of the fridge, then decide to do the laundry since I had no clean towels, and while I was in the basement doing the laundry I noticed that I needed to organize the basement and throw out old computer parts. Meanwhile, upstairs, my glass of water has long since evaporated, and the task I was doing before that is long forgotten.

Thus, when I get like that, I work from a list, and only what is on the list gets done, in the order it went on the list.

Re:Simple:

[AmberBlackCat]

In either case, you definitely want wi-fi and definitely don't want the Sony battery.

Re:Simple:

[Firehed]

Poison gas ? You think that's all an evil supercomputer will do ? NO ! It will spontaneously develop godlike powers, take over the universe and unravel the very fabric of reality around you !

It may also mock you with nonexistent cake.

Re:Simple:

[goodmanj]

That's enough verbal onanism for one day.

Re:Simple:

[blind biker]

I have a huge admiration for your honesty. You are an exceptional person.

Re:Simple:

[isorox]

Because I'm a fucking dumbass and didn't think about it....

Finally, proof that slashdot helps you at work! I'll redouble my efforts at spending time on this site during work hours

Re:Simple:

[afidel]

Nah, a terminal server in this context generally means a router with a multiport serial cable (hydra or octal cables are common names) attached. They allow you to dial into one device and connect to everything else. We used to even assign IP's to the async serial port so you could simply telnet to an IP and get into the connected devices console, worked well when you used adjacent subnets =)

Re:Simple:

[Firehed]

Holy crap, +5 insightful? I like my karma as much as anyone else, so no complaints, but... huh?

Re:Simple:

[ignavus]

So you have writing Slashdot posts on your list then?

Re:Simple:

[macdaddy]

Not really. A network admin should be able to track down the thing, but it will take a lot of work to scan network logs. From the network standpoint, it doesn't matter if the gateway is running on a PC, or running on a VM inside a PC... the network traffic looks the same.

It shouldn't take any competent netadm more than 5 minutes to track down any device to a specific port on a switch. There are no logs to look at. What do you think is logged that you'd want to look at to track it? Seriously, it's incredibly simple to do. The thing has an IP and for that IP to be useable on the network it must be in the RIB (read: route table). With less than a minute's work a netadm should be able to track down that route to the router that's originating it. I don't care how big your network it. It should take less than a minute. Once you've found the router originating the route you've almost certainly found the router with an L3 interface in the same broadcast domain as the target device (the router could also be redistributing a static route in which case the static route would point you to the device in possession of said prefix, or a trail of bread crumbs of multiple static routes that will eventually lead you to the device using that prefix). If the router is part of that broadcast domain then it will have an entry in the ARP table for the target IP and will give you the device's MAC. From that router's config the netadm can determine where all that broadcast domain is accessible. Ie, what L2 switches downstream of the router have that VLAN on it. The netadm can examine the CAM table (SAT in Cabletron-speak, bridge forward table in generic terms) to figure out which interface the target's MAC is associated with. That will point him to the correct downstream switch. The netadm will do the same thing on that switch to track the target device their the broadcast domain until he find the one access interface that the target device is connected to. Once he finds that interface he visits that wiring closet and tracks the cable down manually to the target device.

Really, it's much easier than it sounds. Once you've done this once or twice it will become second nature. This should not take a competent netadm more than 5 minutes. I don't care how big the network is. This isn't rocket science. The City of San Francisco is just trying to make their case sound worse than it really is. It would take a truly incompetent IT department to not be able to find that device. I would say that it was impossible to be that incompetent but I'm sure someone would try to prove me wrong.

The story keeps changing.

[khasim]

From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.

Given that they cannot hunt down a single device on the network, I'd have to agree with that assessment.

MAC address ... switch port ... it should be easy.

Re:The story keeps changing.

[DogDude]

1. Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.

2. Assuming that they have wireless on their network, there's no way to find wireless devices, since they can be put inside of locked buildings. Unless your name is "Superman", there's no real way to find exactly where wireless devices are, as far as I know.

Re:The story keeps changing.

[Fx.Dr]

...his boss whom he considered an idiot...I'd have to agree with that assessment

Second that motion. I'd say these guys are like the Marx Brothers of network administration, except they don't know the Secret Woid, so it looks like they're a couple notches down.

Re:The story keeps changing.

[goose-incarnated]

... Unless your name is "Superman", there's no real way to find exactly where wireless devices are, as far as I know.

And exactly how would superman find it? Xray vision? How would he then know he found it?

Re:The story keeps changing.

[autocracy]

Seriously? http://en.wikipedia.org/wiki/Radio_direction_finding

Re:The story keeps changing.

[moderatorrater]

Agreed. If they're still having problems at this point, they're incompetent jackasses. However, that's not an excuse for the employee to be a jackass too.

Re:The story keeps changing.

[the_B0fh]

2) It's a freaking terminal server. How many wireless terminal servers have you seen?

Re:The story keeps changing.

[Lumpy]

I CAN find a wireless device It's called Radio direction finding, with the right gear you can do it, and I have located 802.11g devices with it. It's not hard.

so you may start calling me SUPERMAN.

Re:The story keeps changing.

[chill]

The other end of that wireless device plugs into a wire, which has a MAC and then runs to a switch port.

Re:The story keeps changing.

[bratwiz]

... Unless your name is "Superman", there's no real way to find exactly where wireless devices are, as far as I know.

And exactly how would superman find it? Xray vision? How would he then know he found it?

Um, that's actually an easy one-- he'd zap it with his heat ray vision and then if it stopped, he found it... if not, well-- Ooops!

Re:The story keeps changing.

[Crudely_Indecent]

If Superman had any IT skills, he'd perform a traceroute to determine the devices gateway. Once the gateway was determined, block the mac address from accessing the network. If the admin of that device is worth his salt, he'll change the mac address and continue. They could then specifically enable allowed devices and forbid all others.

Forget finding it, make the network inaccessible.

City of SF Admins, if this proves to be your resolution, you owe me $150 for 1 hour of my time. Sorry, I do not bill in lower increments.

Re:The story keeps changing.

[IntlHarvester]

City of SF Admins, if this proves to be your resolution, you owe me $150 for 1 hour of my time. Sorry, I do not bill in lower increments.

I know nobody RTFAs, but the city is spending $1 million on consultants to rebuild the network, so sorry a guy like is just too cheap for this project.

Re:The story keeps changing.

[LizardKing]

Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.

By the time his boss thought to ask for the password(s), he had already been fired. Any obligation he had to his boss had disappeared. The same goes for documentation and written procedures - I'm not going to document anything after I've been sacked. In this case the guy had been arguing for written procedures to be put in place, but no one in authority would sign them off as any failures would then be their ultimate responsibility. It should be the managers that are taking flack for this, as so often with IT cock ups.

Re:The story keeps changing.

[Sobrique]

I'd qualify the 'your boss is your boss' thing. I think a Sysadmin _does_ have a grounds for professionalism and ethics - just because your boss demands that you go raid the email server to see where that cute secretary he fancies hangs out, doesn't mean you should comply.

Now, as regards passwords and what not, I would be inclined to agree - you've got no right as a professional to lock out the owner of the kit, from their stuff. However I'd also say escalating it higher because there's 'serious ethical implications' in some situations isn't unreasonable. Not that this necessarily relates to this particular case - I don't know the details, so I won't comment - I just wanted to point out that there are good and valid reasons not to comply with a demand like this from your direct 'boss'.

Re:The story keeps changing.

[Shakrai]

Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.

My obligation to my employer (in this case the city of San Francisco) trumps my obligation to my PHB. If I think my PHB is a moron and is going to cause a shitload of damage to my employer then I think I could make a good case for refusing to give him the passwords.

Of course that's not where it would end.... I would have to explain to his boss what the problem was -- or go even further up the chain of command if he was also a moron.

Assuming that they have wireless on their network, there's no way to find wireless devices

Wireless devices still have MAC addresses. By tracing the MAC address you'd get a switch port. If that switch port has an AP plugged into it then you know it's a wireless device and probably know it's general location (the AP doesn't have limitless range).

there's no real way to find exactly where wireless devices are, as far as I know

Oh, there's a way.... it's just out of the reach of most of us.

Re:The story keeps changing.

[FooAtWFU]

and I do development on some software that will use RF data from your existing wireless access points to triangulate and display the physical location of every user and device on your network!

So you can call me, uh, Jerry Siegel, I guess? :| that's not as impressive...

Re:The story keeps changing.

[mollymoo]

From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.

That and setting up the routers so they lose their configuration on reset. Even if your boss is an idiot, you get your concerns on the record and a direct instruction on the record and then do what you're fucking well told.

Re:The story keeps changing.

[Qzukk]

Unless your name is "Superman", there's no real way to find exactly where wireless devices are, as far as I know.

So does the blue and red spandex underwear come with the radio signal triangulation gear, or do you have to pay extra?

Re:The story keeps changing.

[Jah-Wren Ryel]

No joke. You'll pay $300+/hr for a top guy from a place like IBM Global Services or HP's technical consulting group.

Re:The story keeps changing.

[dougmc]

What makes you assume it's a wireless device?

The article doesn't say anything like that.

It's probably wired into the network, stuffed in a closet or a ceiling somewhere. Perhaps it does have another interface, a wireless one -- but it could be a backdoor without that too. Or perhaps it's not a backdoor at all.

In any event, that they are trying to find it via legal means rather than network means does indeed suggest that they're incompetent. And even if they can't physically find it, they should be able to disable it easily enough.

Re:The story keeps changing.

[L4t3r4lu5]

It's written into my contract that I do not document the domain admin password, and that I do not share it with anybody outside of the technical IT team without written confirmation from the Network Admin. The IT Manager and Head both agreed to this, and it won't change while I work here.

When users ask for Admin privilages, they should be told to go fsck themselves. No matter who they are.

Re:The story keeps changing.

[oodaloop]

I'd mod you up, but you have to say 'swordfish' first.

Re:The story keeps changing.

[Gojira Shipi-Taro]

regarding point 2:

Trivialy easy to determine if it's a wireless device. TURN OFF THE WIRELESS NETWORK. If the device goes away, it's wireless. Then simply change the security configuration on the network, and problem solved. The offending device is no longer on the network, and its physical location is irrelevant.

Elsewise, if it doesn't go away, it's a wired device, and normal network investigation should work just fine.

Re:The story keeps changing.

[pacman on prozac]

They could always do something crazy like track the MAC to a port and go trace the cable to find the device, I guess that wouldn't make such a good story though.

If they're using Cisco switches and it's linked via copper then they could probably work out where it is without leaving their seats, use the inbuilt tdr to find out how long the cable is, then use the location of the switch and a bit of common sense to work out where the device is likely to be.

If it's a terminal server then it's not likely to be hanging off a 3km long fibre somewhere in a duct under the city. It'll be within serial cable distance of all the other kit, more than likely in their main computer room with some bloody great octal cables hanging out the back. I suspect it'd take someone clued up approx 5 minutes to identify it as it will look rather different to any of their other routers purely due to the cabling run to/from it.

The more I read about this "ebil admin" story the less I believe any of it.

Re:The story keeps changing.

[element-o.p.]

I was thinking the same thing. Couldn't you traceroute and show arp tables to find where it is?!?!

Disclaimer: I am a sys admin, but not for the municipality of San Francisco, so my ignorance of their network architecture might be masking something that makes this procedure non-trivial. For the life of me, I can't image what, however.

Re:The story keeps changing.

[clone53421]

Routers will usually lose the configuration when you do a reset (as opposed to power-cycling the device), and I'm not surprised that some incompetent superior didn't know the difference between a reset and a reboot.

MAC search

[jeffy210]

Um, do what any network admin does with a rouge device. Search out what port its MAC address is connected to and then start tracing the cable?

I'm fairly certain most all current managed switches allow for this. Even with unmanaged ones you can hunt down which unmanaged switch it is connected to and snoop from there.

Re:MAC search

[Lumpy]

Exactly, hell I can sit down with my laptop and tell you what switch it's connected to in 20 minutes. Bet you $50.00 the community strings on all their network gear is still set to public and private :)

Are the IT people they hire completely dysfunctional? Or do they do what most cities do and not actually hire IT people or networking admins because they command a real salary instead of the $12.00 an hour that someone handy with computers get's...

Re:MAC search

[Yvan256]

I'd think that a red device would be easy to spot in a server room.

Re:MAC search

[Soruk]

It's probably hidden in a wall cavity somewhere, a bit like that Netware server in the news a few years back.

Re:MAC search

[the_B0fh]

Apparently this was why he refused to give out the admin passwords - he thought, and so far, it appears that he is correct, that they are all morons.

Re:MAC search

[Archangel Michael]

I learned early on, that most people don't see the difference between a $12 hour high school geek and a $75 hr network administrator. All most people see is that both do roughly the same job and there is $63 hour difference.

Most of the time, the $12 hr guy is doing most of the same work as the $75 hour guy. The big difference is when crap like this comes up, the $12 hour guy can spend years trying to figure out what the $75 hr guy can figure out in 5 minutes.

Even when the $12 hr guy screws up, the response is "But he was cheaper". It is cheaper to keep a $12 hr guy trying to keep crapware off a computer, rather than a $75 hour guy who doesn't allow crapware in the first place.

The point I'm making, is that a $75 hr guy is worth it, but only to people where time has real value. People who place no value on TIME, don't care about anything other than $ per HR

Re:MAC search

[d_ron_218]

I worked for a company where they cheaped out on the switch infrastructure and bought low-end Dell switches for the entire network. The kind that don't let you see the MAC address table.

Some guy decided to bring in his Linksys router from home so he could use his laptop and his desktop at the same time (instead of, you know, asking IT to add a second port at his desk). Problem was he left DHCP running on the thing, which obviously led to some confusion. Took forever to find it.

Then again it sounds like the city of 'cisco bough nothing but Cisco gear, so who knows what's really going on here...

Re:MAC search

[Baricom]

How, then, can they use the management functions of the equipment if they can't get to it?

Terry Childs provided the passwords to the mayor on July 22. The city "...[was] able to regain complete control of the network," according to the deputy director of the Department of Technology Information Services.

Re:MAC search

[myz24]

Not at all. I dealt with this very issue twice for the same organization. They bought wireless routers and wanted to use them like access points. They put port 1 on the network and placed a computer on port 2, never using the WAN port. This is better setup than using the WAN port because you can't as easily access the computers behind the WAN port. The problem was they wouldn't disable DHCP causing all sorts of issues. Twice I went in and explained that they MUST disable DHCP if they want to use the router in this fashion and last I heard they reset the routers again and were having the same issues. Of course, my name gets dragged in the mud because they think I'm the idiot.

Re:MAC search

[idontgno]

I'm sure the rocket scientist in GP post was just using the switched ports (4, on the typical Linksys home router product) on the LAN side, not realizing that the router's firmware was still active even if it was just being used as a dumb switch.

Hell, if you've gone from wired to wireless at your house, and you happen to have this old router laying around now, and you need a few more 10/100ports ... whadday gonna do, run out and buy a new switch, or grab the one you've already got? And who's gonna think of disabling the DHCP service at first, if it was configured running months ago when you decommisioned it?

A perfectly forseeable screw-up, and another good example why lusers aren't allowed to plug their crap into the network.

In the immortal words of some faux-German sage: "Das Network is nicht fur gefingerpoken und mittengrabben."

Re:MAC search

[Migraineman]

At my previous job (optical network equipment manufacturer, now defunct,) we ran a nifty TSR web server with a single fake news release webpage. Surprisingly, the TSR app would continue running even after logging out. We left it running on a lab computer for about a week before making the internal URL available. When we announced it, it took about 20 minutes before a team of IT guys barged through the lab doors. They chased wall plate tags, right up to the machine in question.
ITguy1: This is it.
ITguy2: It's not logged in.
ITguy3: Double check the wall plate number.
ITguy1: Yep, that's it. But nobody's logged in?
ITguy2: Pull the cable. [*yank*]
ITguy1: (on phone) That's it? Okay, good.
ITguy3: That's it? Fuck it, take the whole machine.

With that, they grabbed the whole machine and took off. Took them a day or two to figure out what was happening. The couldn't pin it on anyone, as there had been numerous log-ins on the lab machine. However, the gave us a knowing nod of the head and a shake of the finger. I do believe that they appreciated the prank, but couldn't officially say so.

to quote bash.org...

[SomeGuyFromCA]

<erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.

Re:to quote bash.org...

[FireStormZ]

The admin might not be stupid he might be an ass

1) He placed a rouge device (his personal property) on the SF network
2) He set all the network devices on the network to lose all info on a reboot
3) He will hand over the passwords (after jail) to all the devices except the rogue

You can make equipment hard to find ( mac masquerading comes to mind )... I'm only adequate in terms of networking but I am pretty sure someone who is really good can play a mean game of hide and seek. Who knows *what* he was doing with that device? and were I the network admin I would have to *on principle alone* rebuild everything after this guy left..

Re:to quote bash.org...

[alnya]

He placed a rouge device (his personal property) on the SF network

My guess is it'll be next to his guyliner

Re:to quote bash.org...

[Chris Mattern]

What is this fascination with red devices? Should I start painting my network gear red?

Re:to quote bash.org...

[Yvan256]

I still don't understand why everyone keeps saying the rogue device is red.

Re:to quote bash.org...

[russotto]

2) He set all the network devices on the network to lose all info on a reboot

I wonder if this one is just a complete misunderstanding. One article says that they were set to lose configuration files on "reset". That's pretty typical -- if you have some device you don't have the password to, you can do a full factory reset and get it back to the default password, but that also wipes the configuration files. He might have told his incompetent bosses that, and they thought he meant they'd lose the files on a reboot instead.

Anyway, if this guy is what they're making him out to be, they need to completely wipe and reconfigure the network anyway; it's the only way to be sure he didn't leave a few presents for them.

Re:to quote bash.org...

[mcgrew]

Will you people please learn how to spell rogue correctly?

1. Yuo must be new here!
2. How do you know he wasn't referring to a device for applying women's makeup?
3. Transposition of two letters in a word is a common typographical error and should not be considered idiocy unless the same error is made multiple times in the same post
4. Logged in users don't have to preview before posting.
5. If you're going to be a pedant, well, the word "rogue" in your sentence should have quotes around it =P

Re:to quote bash.org...

[gEvil (beta)]

What is this fascination with red devices? Should I start painting my network gear red?

Of course you should. It makes it operate at faster speeds. I thought everyone knew this.

Re:to quote bash.org...

[_Sprocket_]

Will you people please learn how to spell rogue correctly?

That's like lipstick on a pig.

This is a job for nmap

[Jeremiah Cornelius]

Hey! Fyodor! They need your number!

Fyodor spent much of this summer scanning tens of millions of IPs on the Internet (plus collecting data contributed by some enterprises) to determine the most commonly open ports. Nmap now uses that empirical data to scan more effectively.
Zenmap Topology and Aggregation features were added, as discussed in the next news item.
Hundreds of OS detection signatures were added, bringing the total to 1,503.
Seven new Nmap Scripting Engine (NSE) scripts were added. These automate routing AS number lookups, "Kaminsky" DNS bug vulnerability checking, brute force POP3 authentication cracking, SNMP querying and brute forcing, and whois lookups against target IP space. Many valuable libraries were added as well.
Many performance improvements and bug fixes were implemented. In particular, Nmap now works again on Windows 2000.

With just nmap, my old buddies at Farm9 could have sussed this out in a few hours. I think they are still around - as Red Siren / Getronics.

Ahh. I miss running netcat at 3 AM!

Siding with the network guy

[John Jamieson]

Man, the more I read about this story, the more inclined I am to believe the network admin.

He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.

Re:Siding with the network guy

[evilviper]

I don't know what part of this you think he's technically right on, other than that he worked for incompetents, which seems to be true.

Well, the fact that they're contracting outside Cisco experts now suggests nobody else there was technically competent enough to manage the network.

The fact that the network stayed up and running without a hitch, while he was in jail and nobody else had access, suggests he did know what he was doing, and refusing to allow anyone to access the routers to make changes seems to work quite well to keep the system working.

The fact that his supervisors are moronic and useless is no small thing, either.

That said, his actions are still beyond reprehensible.

His actions were extremely stupid, but I fail to see why this idiot's relatively non-disruptive actions rise to the level of criminal prosecution.

Re:Siding with the network guy

[John Jamieson]

The part that he seem right on is that his management was so inept, that they could not be trusted to touch anything.

I have not seen anything to indicate he "built a house of cards".
It looks more like he built a palace and put a secure wall around it so the local hoodlums (his management) could not vandalize it.

I think he got a little too attached to his work and cannot let go. (obsessive compulsive?)
Though his actions were extremely dumb, I am not sure I would agree with reprehensible... I might leave that word for describing his (non)management team.

Re:Siding with the network guy

[HauntedCrown]

but he may have been technically right.

The best kind of right!

Re:Siding with the network guy

[SecurityGuy]

Mea culpa. I just RTed the FA again, and his supervisors DID ask for access. He should have been fired immediately. IANAL, so I can't speak to the validity of the criminal charges, but this guy is clearly WAY in the wrong.

Re:Siding with the network guy

[mcgrew]

He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.

I'm guessing he has a four digit slashdot UID!

Re:Siding with the network guy

[sgtrock]

Then you've never worked for the kind of clueless idiots this guy was working for. Supervisors do NOT need access. Any competent manager knows that's the case. What's needed is more than one competent individual to have access, with backup keys kept in sealed envelopes that are kept in a safe with only logged access to it in case both are hit by a bus on the same day.

BTW, did you miss the part of the case where for _years_ the admin in question begged, _BEGGED_ for someone else who was competent to be hired so he wasn't a single point of failure? That he continually pointed out that there was no DR plan whatsoever?

Nope, this guy made a serious error in judgment in not making sure that the mayor's office had the access information ahead of time. His supervisors are clearly incapable of administering that network and shouldn't be let anywhere near a console.

Re:Siding with the network guy

[geminidomino]

His actions were extremely stupid, but I fail to see why this idiot's relatively non-disruptive actions rise to the level of criminal prosecution.

Thou shalt not expose the government's incompetence.

Re:Siding with the network guy

[sgtrock]

The problem that I think you're overlooking is that this guy was stuck in a siege mentality. For years he'd been asking for help and his management chose to ignore him and apparently, in some cases, actively blocked his requests from going further up the chain to someone who'd listen.

Now consider the fact that he was working 50-60 hour weeks and was on call 24/7/365. I've been there, done that, hated the T shirt. Believe me, after a few years of that (and he had apparently been suffering with those kinds of working conditions for 5 or 6), you'd be a little nuts, too.

Did he make an error in judgment? Yes. Was it egregious and irresponsible? Not under the circumstances. In my view, quite the opposite. Was it criminal? Maybe. But since when is the law necessarily about common sense? :(

The scene when they find the server

[UnknowingFool]

I'm sure the scene will be like this:

As Indy deciphered the symbols, he found the correct sequence of tiles to push. The huge stone door slowly opened. Indy grabbed a torch and headed inside. At the end of the long room, there it was on the throne: A massive server. It was archaic, and it appeared to be attached to a punch card reader. Along the sides of the room, there were two rows statutes of archers pointed at the center. Indy made his way slowly to the monitor and keyboard of the server. He brushed away the dust and hit the spacebar. The screen turned on slowly and it displayed:

SCO Server 1.0

Your license has expired. You owe use $699.
>_

Suddenly the archers rotated positions and were aimed at Indy.

"Oh boy."

Re:The scene when they find the server

[vmxeo]

Suddenly the archers rotated positions and were aimed at Indy.
"Oh boy."

You forgot to mention the part where Indy flashes white and is replaced by Dr. Sam Beckett.

I've Changed my mind.

[misterjava66]

When I first heard what the rogue-SF-admin had done, I was very negative on his actions.

Now, that once again, and now at least for the third time, I hear of absolute stupidity and ineptness on the group at sf, I am certain the so called rogue was right on the ball from the beginning.

Re:I've Changed my mind.

[Anonymous Coward]

Oh yeah, let's give him a break. Oops, he's been by hit by a bus. Where's his disaster recovery plan? That's right, there isn't one. He fscked his employer with his trumped up little admin attitude. Like most admins, he's on a power trip because he has root access on a network. The shit should have been fired, then sued him into oblivion for illegally locking up infrastructure that doesn't belong to him. Give him a few months jail time to top it off, he'll never get a decent job again.

Re:I've Changed my mind.

[Medievalist]

Oh yeah, let's give him a break. Oops, he's been by hit by a bus. Where's his disaster recovery plan? That's right, there isn't one.

My bet is, it's sitting right in the middle of his old desk blotter, in a fat manila folder marked "Disaster Recovery and Service Continuity Plans". These clowns would never find it there in a million years. The infamous missing passwords are probably in a letter-size envelope in the top left desk drawer, too.

Re:I've Changed my mind.

[geminidomino]

Who modded this insightful? Part of the reason he was getting canned was because he was PUSHING for the sort of documentation and recovery plans you're snarling about. None of the PHBs wanted to put their names on it because if they came up short, it would be their asses on it.

Sparcstation In The Wall

[gentimjs]

I recall hearing a story about a Sun Sparcstation 2 at my old college that had accidentilly got sealed inside a wall by construction folks when re-working the building the CS lab was in to eliminate a few closets for structural support reasons.. nobody could find it (shock!), but kept using it as a DNS server for another six years. It was found about 2 years after it stopped responding to ping when some component (nvram?) let out, and it started beeping after a power flicker.

Re:Sparcstation In The Wall

[GregMcD]

Your might be thinking of the Novell NetWare server story. University of North Carolina in 2001. It was physically MIA for 4 years yet kept doing the Energizer Bunny routine. I was a Novell Reseller at the time and the story made a great sales pitch. http://www.techweb.com/wire/story/TWB20010409S0012

Re:Sparcstation In The Wall

[scaryjohn]

I bet when the city finds this box, wherever it is, all it will have is one toggle switch labled "Magic / More Magic".

Re:Please - It's San Francisco or simply "The City

[Anonymous Coward]

Hey, at least they didn't say "Frisco".

FoxHunt

[ka9dgx]

1> Yes.. people could be hurt because the network in question is used to save lives, so it's OK not to hand the keys to an idiot.

2> It's easy to find wireless devices... I've personally been doing it since the 1980's.. it's called a fox hunt here in the Chicago area. We used to get 1 minute of transmission every 5... with WiFi you can just ping the dang thing... how easy is that?

--Mike--

Re:FoxHunt

[pilgrim23]

There is an old, probably apocryphal tale from the days of Novel Netware and IPX of the forgotten server. A loan machine runs headless with a quiet fan and no lights in a corner of a room. New remodeling puts the server behind sheet rock and there it sits walled up and running for years. One day a power spike causes a head crash and suddenly a national billing system dies. It takes a tech tracing a cat5 cable into a wall to find it.

Re:FoxHunt

[leuk_he]

it was covered on /.

http://www.techweb.com/wire/story/TWB20010409S0012

Re:FoxHunt

[guruevi]

I've heard the same story, only with coax cable (yes, I've been around that long...).

If their network is CAT-5 based on smart switches (HP, Cisco or any somewhat decent network vendor, heck even Linksys and Netgear have smart switches) they can find it quite simply by running a simple command on the switches, I even have a perl script that will do it for me.

Of course, it wouldn't surprise me if they still use 'dumb' switches or if the consultants simply said among them: let's get these stupid , just sit around and rack up the charges.

Just remember.

[AltGrendel]

These are the guys that the "rogue" admin said were too stupid to run the thing in the first place.

You think they've learned anything about the gear since then? No wonder they're having problems.

Malice and stupidity.

[twitter]

Why is Slashdot linking to stories that paint the network administrator as a bad guy when he's so obviously surrounded by morons? These are the same people who published all of their user names and passwords. That puts the cost of this "hijacking" into perspective. The cost of trusting their employee with the powers required to do the job was zero.

Re:Malice and stupidity.

[bratwiz]

Why can't he be a bad guy AND be surrounded by morons-- you know, the old "bad guy surrounded by morons" routine...???

Re:Malice and stupidity.

[erroneus]

You mean like the VP of the United States? That has been done before.

Re:Malice and stupidity.

[Misch]

Why can't he be a bad guy AND be surrounded by morons-- you know, the old "bad guy surrounded by morons" routine...???

Dark Helmet: Who is he?
Colonel Sandurz: He's an asshole sir.
Dark Helmet: I know that! What's his name?
Colonel Sandurz: That is his name sir. Asshole, Major Asshole!
Dark Helmet: And his cousin?
Colonel Sandurz: He's an asshole too sir. Gunner's mate First Class Philip Asshole!
Dark Helmet: How many asholes do we have on this ship, anyway?
[Entire bridge crew stands up and raises a hand]
Entire Bridge Crew: Yo!
Dark Helmet: I knew it. I'm surrounded by assholes!
[Dark Helmet pulls his face shield down]
Dark Helmet: Keep firing, assholes!

Re:Malice and stupidity.

[Anonymous Psychopath]

There do appear to be a lot of morons involved in this scenario, and Childs was one of them. Basically what he said was "I am smarter than all of you, so I will do things my way, and trust me, you'll be better off."

Except they weren't, because he doesn't appear to be anywhere near as smart as he thinks he is. Even if he was smarter than the gaggle of incompetents he worked for.

Re:Malice and stupidity.

[BlackSnake112]

There do appear to be a lot of morons involved in this scenario, and Childs was one of them. Basically what he said was "I am smarter than all of you, so I will do things my way, and trust me, you'll be better off."

Either I have bad luck or I keep on finding people who think exactly that way. We have even had meetings where all agreed on a specific solution to the problem. Right after my boss say well we are going to do it this other way, we know better. Even if the other way was a better solution.

Some people have egos that are way too big fir their own good. I am not saying I am perfect. I use solution that I know work. If there are better ones please show me. I have no issue changing my way of doing things for a better one. I know a lot of people who will not change. Even when a better way is show to them.

Re:Malice and stupidity.

[funwithBSD]

Big assumption.

They probably deleted all those "useless files" on the fileserver when they fired him.

And the "terminal server" is probably his iPhone...

Re:Malice and stupidity.

[moxley]

I disagree.

It isn't that simple; it seems that there is waaaaay more to the story that some ego tripping sysadmin.

Everytime another piece of the story or fact about what happened comes out it seems to vindicate Mr. Childs to some degree (not that his judgement was flawless in how this was handled, but still).

Is he still locked up? If so it's a travesty.

It seems like those who are trying to have him tarred and feathered constantly want to make it look like he's some super-e-terrorist who was holding the entire city for ransom and has dealt an economic blow from which the city will never recover.

I am not saying everything he did was right, or that he committed no wrongs here; but I think it's pretty obvious that this was viewed as a pissing match by those in the city who wanted him to hand over that information and they have gone to great lengths to make it look like something much more malicious than it was in the press.

He may have had very good reason to protect it; (I mean aside from the fact that it appears as though those who wanted him to hand it over were incompetent) - because I don't think anyone would put their own ass on the line for jailtime and the loss of their job unless there was something else going on. I am not saying I know this to be true, just that that is how it appears to me based on the available information.

At this point I view anything coming from the anti-Child's side of this issue with a healthy does of skepticism and try to read through the sensationalization. Something has always stunk about this situation.

The City of SF is undermining its case!

[StandardCell]

If the city can't even complete one of the most basic network administration tasks of finding a physical device on a network, I think they have absolutely no right to accuse anyone of "hijacking" their network. I hope the defense attorney for Terry Childs brings this up.

You're an 1D10T

[Archangel Michael]

1) They were firing the guy, so he was no longer in the employ of the city, so his boss, was no longer his boss.

2) You don't know what you're talking about. Every IP address on the network should be known. Either through DHCP or static IP address map. A ping sweep should reveal any IP address in use, that shouldn't be. From the ping sweep, one can arp the unknown IPs to get a MAC address, and do a lookup on the Manufacturer code to know what KIND of device the MAC could be. one could use NMAP to try to discover type of device as well. Then you start going to every port on every switch with rogue IPs hanging off it, and manually looking at what is attached at the other end.

As for wireless access points, if you don't have control over them, you pull the freakin plug. Unsecured Access points and open access points should be VLANed off from administrative networked, including not allowing VPN tunnels from unsecured and open wireless access point.

If the boss allows crap like that on the network, he is an idiot, and shouldn't have the Passwords and access codes to anything.

Re:You're an 1D10T

[larry bagina]

Ping replies can be disabled. MACs can be faked. But everyone who supports more government ought to take a look at the incompetence here.

Re:You're an 1D10T

[denis-The-menace]

I wish I had mod point for you.

Chances are that internal policies prevent the use of "hacker" tools to secure the network.

Again, the PHBs are idiots!

Re:You're an 1D10T

[Archangel Michael]

Yes, both of those are true (Mac, Ping). Even NMAP responses can be spoofed. However the likelihood of all three being done is not likely. However NMAP will reveal a used IP, and a mac table somewhere will identify what port it is hanging on. Packets have to be routed to it somehow.

And I agree with your last point. I'm a Libertarian. ;)

Re:You're an 1D10T

[ACMENEWSLLC]

If you have SNMP and a Winders PC;

log.txt (list of your switches / routers)
192.168.1.1
192.168.1.2

file1.bat
for /f "tokens=1,2,3,4,5,6,7,8,9,10" %%i in (log.txt) do call distcmd.bat %%i %%j %%k %%l %%m %%n %%o %%p %%q %%r

distcmd.bat (change public to be your snmp community snmputil is from net-snmp-5.4.1-3.win32.exe / free)
snmputil walk %1 public .1.3.6.1.2.1.17.4.3.1.1 >> %1.MAC.log
snmputil walk %1 public .1.3.6.1.2.1.17.4.3.1.2 >> %1.PORT.log

Now you have a list of every port and ever MAC that runs through that port (don't forget about uplink ports having all MACs listed to them.)

In the file you'll have to do some snmp decimal to hex conversions;
MAC is 00:17:a4:d7:31:99
SNMP walking gives a result of which is
the same MAC, but converted to SNMP notation this becomes .17.4.3.1.1.0.23.164.215.49.153
Part of this is the MAC address in decimal .0.23.164.215.49.153 Put those number in calc and convert each to a hex and you get
the mac address in hex.

So in the output of these batch files
Variable = .17.4.3.1.2.0.64.140.109.101.123
Value = Integer32 24

Means that on port 24 there is MAC address 00:40:8c:6d:65:7b which is the .0.64.140.109.101.123 converted to Hex through Calc.

Anyway, ping the device, arp -a and locate the MAC address, dump the above against all your routers/switches, convert your MAC address you are searching for to SNMP decimal, and search the .txt files for a switch with a port where that is the only MAC on the port and you've found the device (or a hub between the device.)

Re:You're an 1D10T

[mr_mischief]

Packets might have nothing to do with a terminal server. He could be dialed into it using a modem and controlling the console port of an otherwise legitimate Unixish server.

Re:You're an 1D10T

[gad_zuki!]

>But everyone who supports more government ought to take a look at the incompetence here.

Im one of those crazies who doesnt support more or less government. Just better government.

not necessarily wrong...

[damn_registrars]

your employer's passwords are NOT yours, no matter how stupid you think your boss is.

Refusing to give out passwords to higher-ups is not always the wrong thing to do. If you are the network admin, and your job is to maintain security of the network, wouldn't it be reasonable to refuse to hand out passwords to people outside of the network administration roles?

Although I can say that an admin can make that choice at his or her own peril. After all, the higher-ups can always opt to fire the admin and replace him or her with someone who is willing to seek security of their job over security of the network they are paid to administer.

Re:not necessarily wrong...

[Lonewolf666]

Agreed.

If a boss I don't entirely trust demanded my password, I'd offer to upgrade his account to the same privileges at mine, but he'd NOT get MY password.

The reason is that if he does something stupid that will show up in logfiles, he can damn well do it on his account and get logged doing so ;-)

Re:not necessarily wrong...

[Vancorps]

I'm confused, does any admin ever give up his own account password?

In my company we have a blanket policy, never give out passwords, ever... as admin I don't need someone else's password to get into their mailbox and retrieve information that's needed by another employee while the content owner is out of contact. Of course I always notify the mailbox owner that I had to go in as I have to have a specific reason.

Are there environments out there where you would be expected to give up your password? I can understand keeping a password database for service accounts which all admins should be able to access if they manage it but I can't imagine a scenario when I'd need someone else's password. Even if the thing is encrypted, I have the recovery key so again I don't need their password.

Re:not necessarily wrong...

[autocracy]

Often times an account such as Unix root or Windows Administrator will have a randomly generated password that's sealed in an envelope. Envelope is locked in a box, with some kind of anti-tamper on the envelope... all this is usually under multiple control. Nobody uses the account unless shit + fan. Admins then have their own equivalent access level accounts.

Re:Please - It's San Francisco or simply "The City

[Sobrique]

No no. "The City" is quite clearly "The City of London". And no where near San Francisco. (I wonder if they use Cisco hardware though, which might make the San Fran - Cisco more apt)

Where to look...

[s0litaire]

Did they try the Rouge Admin's office. It's probably that beige box under his desk... Either that or he made up the device and it does not exist, he's laughing at them ripping the place apart trying to find it :D

Re:Where to look...

[caluml]

Slightly OT - if Americans spell analogue as analog, why isn't rogue spelled rog?

Mod Parent Up

[mpapet]

I'd like to add that while the way he handled being surrounded by idiots was wrong, he was clearly surrounded by idiots.

No documentation?
No change control?
No diagrams?

What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.

I worked one place where a guy with a great deal of responsibility died. (here today dead tomorrow kind of thing) His peers blamed *everything* on him simply because they could. This sounds like the same thing.

Re:Mod Parent Up

[AioKits]

What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.

Well, every Stalin needs his Trotsky!

Re:Mod Parent Up

[Sobrique]

Wait, you mean blame it all on the guy who left (be it through death or a cushy new job) isn't standard practice everywhere?

Re:Mod Parent Up

[rickb928]

I took a gig recovering documentation and re-establishing procedures for a great admin who died as well. He really did great docs, but no one had ever used them, and they couldn't figure out the 'copy file piopoiop.dfj to the \asic\wer\2344\sdf.msdfn folder' sort of directions.

And the crew there immediately set to removing, replacing, and destroying all of his systems. He was a Novell hardliner (so was I), and when he was gone, his boss succumbed and the Windows bigots prevailed. Much taxpayer money was spent replacing perfectly functional systems. Mind you their clients were still running Novell, so there was some disconnect when they would get a request for support and start saying 'you have to upgrade (ha!) to Windows'. Their clients, for reasons best left undisclosed, could not upgrade. Both physically impossible and logistically impractical. Start with being 60-1600 meters below the ocean surface, and it only gets more difficult from there.

I'm a little surprised that SF hasn't worked this out. There are plenty of outfits eager to do what is necessary, for a fee of course.

And yes, finding a device is not impossible. Finding the connection to the network is the obvious first step. After that, well, kill it.

Unless it's hiding. That would be unfortunate.

ps- This guy, by many accounts, was brilliant. And a little off the wall. Goes together.

Re:Mod Parent Up

[BrokenHalo]

I have seen a number of posts to the effet that that this particular sysad is being an asshat, but there are two points to be learned here regardless:

1. If you employ someone to look after your security, you don't put obstructions in his way when he does it.

2. If you employ someone to look after your security, be very very nice to him. ;-) (on the principle of "do not meddle with dragons, for you are crunchy and taste good with ketchup.")

Re:Mod Parent Up

[moderatorrater]

Their clients, for reasons best left undisclosed, could not upgrade...Start with being 60-1600 meters below the ocean surface...

Good job, tightlips ;)

Re:Mod Parent Up

[mrjohnson]

I don't get it. The thing's gotta have a mac address that can be found on a switch somewhere. That'll give you a port number and a patch cable to follow until it's found.

Nah, it's way more fun to blame the guy in prison.

Re:Mod Parent Up

[BlackSnake112]

Their clients, for reasons best left undisclosed, could not upgrade...Start with being 60-1600 meters below the ocean surface...

Good job, tightlips ;)

I knew Atlantis was somewhere.

Re:Mod Parent Up

[Psmylie]

It's actually in our documentation to blame the guy on the way out. It works well.

When I leave, though, I'm planning on EARNING the blame I'm sure to get :)

Re:Mod Parent Up

[ElizabethGreene]

There were network diagrams, they indicated they found several copies at his house. No, I don't consider that unusual at all. I carried one in my purse at my prior employer, and a electronic versions on my pda and laptop. When the pager goes off you want to fix it NOW, not drive in to get a circuit ID off of a piece of paper on a wall. They indicated there was some documentation as well, and there was some on the (It didn't have a password on it until Paul Venezia ran an article about it.. thanks Paul) Disaster Recovery sharepoint site. Change management is still kind of fuzzy. They indicated they found some "Configuration files" on his PC, but didn't happen to mention if they were date stamped. -e

Re:Mod Parent Up

[_Sprocket_]

I'm a little surprised that SF hasn't worked this out. There are plenty of outfits eager to do what is necessary, for a fee of course.

From the article...

After a dramatic jailhouse meeting with San Francisco's mayor one week after his arrest, Childs handed over the data, but DTIS Chief Administrative Officer Ron Vinson said Wednesday that the city now expects to spend more than $1 million to clean up the mess. To date, DTIS has paid out $182,000 to Cisco contractors and $15,000 in overtime costs, he said in an e-mail interview.

The city has also set aside a further $800,000 to address the problem. Vinson did not specify what the additional money was expected to cover, but if the city has to hire network consultants to remap, reconfigure and lock down its network, this would not be an unreasonable estimate. The city has also retained a security consulting firm called Secure DNA to conduct a vulnerability assessment of its network.

And there you have it folks, a million-dollar employee; over-worked and under-appreciated by a management too incompetent to understand the issues the guy dealt with much less manage him and his work effectively. Sadly, it's not a very uncommon story.

One of the fun bug-a-boos that show up in these stories is the cost of damage an intruder (or in this case, rogue employee) "causes" the target. I've been on the inside of a number of US Government incidents and seen the cost estimate damages. To someone on the outside, they seem pretty insane. The question that the public often asks is something like "how can changing one password cause so much damage?" But the numbers I've seen are pretty much on target (plus or minus some variance for estimates) - they represent real expenses associated with work to properly ensure the systems are truely owned by their rightful owners again. And they cover resources (i.e. hard drives) lost to criminal investigative bodies / evidence lockers. But the real gotcha to these things is that these expenses either should have been spent as part of the normal management cycle without an attached incident or, even better, could have been a fraction of the eventual cost if the resources were spent to improve the environment or hire proper talent in the first place.

Re:Mod Parent Up

[Dmala]

Seriously, when I finally leave my current job I'm going to be very disappointed if my name isn't cursed out on a weekly basis for at least a year.

Re:Mod Parent Up

[cez]

Nope...it doesn't have to have a MAC address on the network. The point of a "terminal Server" is to provide OOB (Out of Bandwidth) Management. While, technically it can have an ethernet connection to the network, the "terminal" part of the terminal server provides a console connection to a router (the device itself is not a router either, but a black-box with multiple console outs)hence...its not physically on the network, but terminally connected to a device that is.

What it would have (if it is similar to how I use them, and yes I am a WAN specialist) is a phone-line for dial in access in case of emergencies.

See MRV's InReach product line for more information.

...though it could have a MAC address on the network, just saying it doesn't have too, and if it is "mysterious" and / or put there maliciously, in all liklihood will not, or it will be spoofed to prevent detection.

Re:Mod Parent Up

[OnlineAlias]

Could be an IBM 3174 like device too, running SNA. Fact is, the article and and court filings aren't clarifying any of this and leave the door open for mass amounts of conjecture and sensationalizing, both in the media and on Slashdot. Which, of course, is exactly what everyone is doing...

Re:Mod Parent Up

[daveywest]

Wait, you mean blame it all on the guy who left (be it through death or a cushy new job) isn't standard practice everywhere?

I had to actually threaten legal action against a former employer who repeatedly claimed all the failures after I left were sabotage. Maybe its my fault for not grooming a successor, but there was some truth when I suggested my knowledge deserved higher pay.

Re:Mod Parent Up

[sootman]

Old joke, many variants:

The new _____ finds a note from his predecessor: "There are two envelopes in the upper drawer. When you are in trouble for the first time, open the first envelope. When you are in a big trouble for the second time, open the second envelope." In a couple of years he got into trouble, opened the first envelope he got from his predecessor and read: "Blame everything on me." He did so and got out of trouble. A couple years later he got into a big trouble again and opened the second envelope. It said: "Prepare two envelopes..."

Re:Mod Parent Up

[afidel]

Not really, a terminal server could easily have a modem on one end and a bunch of serial cables on the other, not at all an uncommon setup.

Re:Mod Parent Up

[rickb928]

?

NetWare was doing it before Microsoft knew what it was. 'It' being file sharing.

My NetWare servers were going 100+ days uptime when I had to reboot the NT servers weekly.

NetWare was offering usable directory services when Active Directory was still moist and unbaked. Of course StreetTalk predated them all, and was awesome, as in it worked. How quickly we forget...

GroupWise still, IMHO is still a preeminent mail and groupware system. iFolder was cool, ZenWorks kicks butt.

NetWare got it done. Not flawless, but damned good. Windows Server '03 finally delivered on the promise first given with NT. NetWare worked before NT was in production.

Woops, I might be a little too parochial. Sue me.

Re:Please - It's San Francisco or simply "The City

[Registered Coward v2]

No no. "The City" is quite clearly "The City of London". And no where near San Francisco. (I wonder if they use Cisco hardware though, which might make the San Fran - Cisco more apt)

Huh? London is only about 142 miles SE from San Francisco and with a population of about 2000 people barely qualifies as a city, let alone "The City" moniker.

Simple co-dependency

[Anonymous Coward]

If you find that you are "holding the place together", IT-wise, you are likely part of the co-dependency and are part of the problem.

IT and the other management have both agreed to ignore each other, literally or otherwise, allowing each (and the individual personalities) to do things "their way"; damn the best practices, good management, logical, financial, or even legal issues.

Except when things go wrong.

Like a breakup, they can get ugly. And, as the IT guy, you will always lose for it is not your Business, but theirs. You are simply hired help.

Re:Simple co-dependency

[Sobrique]

It's grossly unprofessional to 'ransom' you're employer in such a way. The job of a _good_ sysadmin, is essentially to make himself redundant. I consider it professionally acceptable to leave it 'almost' redundant, in the 'can read slashdot all day, and provide a good service' kind of sense.

When you're fighting fires, you're failing.

Re:Simple co-dependency

[the_B0fh]

You may want to stop reading what the city says, and find out what really happened.

http://it.slashdot.org/comments.pl?sid=960957&cid=24963255

Re:Please - It's San Francisco or simply "The City

[Sobrique]

Your London may be inferior. Ours definitely warrants a 'City' moniker. Especially when The City of London is distinct from the conurbation that is known as London. And the City of London is actually fairly small - almost exactly a square mile - but ... well, you know what they say. It's not the size, it's how you use it.

cisco command "show cdp neighbors"

[jamcc]

Will track down where any MAC address is connected. If they have the IP, they can get the MAC. If they have the MAC, they can get what port it's plugged into. Find the switch, find the cable, and air-gap it. I know this, and I'm not even a network guy.

No power outage in the Terry Childs case?

[Joe The Dragon]

http://weblog.infoworld.com/venezia/archives/018376.html

An insider claims that the power outage that Terry Childs was accused of using to sabotage the San Francisco network was not a planned outage.

TAGS: Problems, San Francisco's FiberWAN, Terry Childs

If you've been following the Terry Childs case to any degree, you probably know that one of the key allegations keeping him in prison on $5 million bail is that he had willfully planned to cause the network to fail during a planned power outage at the DTIS One Market Plaza Datacenter on July 19th. According to credible information I've recently received, that power outage was only going to affect the cubes and offices in that building, but not the datacenter itself.

Thus, there never was a plan to power down the network core. Thus, there's no way that Childs could have tried to engineer the failure of the network during this planned power outage, since the network core would not have lost power.

[ Follow the Terry Childs saga with InfoWorld special report: Terry Childs: Admin gone rogue. ]

The evidence supporting this claim comes from someone certainly in a position to know: Ramon Pabros, the DTIS Datacenter Supervisor himself. Pabros has been employed by San Francisco's DTIS for a surprising 41 years. He's been the Datacenter Supervisor since 1984. He's been running datacenters for the City of San Francisco since Ronald Reagan's first term, the introduction of the Macintosh, and the second season of The A-Team. It's probably safe to say that he knows what he's doing.

According to my source, he will testify to the fact that he discussed the power outage with Childs several weeks before the outage, and at least 10 days before Childs' arrest. He will also state that Childs specifically asked for confirmation that the datacenter itself would not be affected, and was reassured that it would not lose power.

With this statement, the City's allegations that Childs planned to cause the failure of the FiberWAN basically collapse.

Now, I'm admittedly a stranger to San Francisco politics, and am certainly not a lawyer, but if the DA was going to make these accusations against Childs, shouldn't they have talked to Pabros? If the OMP Datacenter was not going to lose power on that date, then this charge against Childs is essentially the same as charging someone with planning to burgle a store that doesn't exist.

But then again, this is the same DA's office that placed valid group usernames and passwords into the public record, and an IT department that ran public, unprotected websites containing internal emails, core network details, as well as usernames and passwords.

I suppose I really shouldn't be surprised at all.

UPDATE: It appears that Pabros has just announced he will be retiring, effective next Wednesday. I can't help but wonder if one event has anything to do with the other. I do know that there have been a number of odd layoffs from San Francisco's DTIS in the past two weeks.

Posted by Paul Venezia on September 8, 2008 08:48 AM

Re:No power outage in the Terry Childs case?

[JoelisHere]

Paul Venezia has some of the best reporting and editorial comments about this whole case. His post in regards to the 'hidden' device: http://weblog.infoworld.com/venezia/archives/018408.html

Re:No power outage in the Terry Childs case?

[celle]

Either the employees are seeing the writing on the wall or the city is trying to get rid of any childs' loyalists(purge). Most likely both is happening. Heaven forbid they get rid of anyone else who knows how to maintain the network.

Road trip

[Oriumpor]

There are now dozens of cars packed full of cheetos cheap laptops and foul smelling individuals travelling near, or perhaps at the speed limit, towards san francisco. They're full of people thinking the same thing, "Shit if they can't find a wired device, they sure as hell can't find a wireless one!"

Onerous

[Anonymous Coward]

All they have to do is look for the small black box with a lone, onerous blinking red LED.

I find it difficult to understand how a blinking red LED would constitute a heavy burden.

Re:Onerous

[Helix666]

it's a very big LED.

Admin code of ethics.

[khasim]

What would you think of a doctor who, because some exec somewhere decided he should, pushed the WRONG medication / procedure to you?

Where does your ethical responsibility end and the boss's desires begin?

To me there isn't even a question. Fire me. Go ahead. I will get another job.

Re:Admin code of ethics.

[damn_registrars]

You're preaching to the choir. I firmly believed that when I was a LAN administrator my responsibility was first to the integrity of the network and second to anything else.

Unfortunately IT professionals aren't in as much of a seller's market now as they were before. Getting another job isn't always as easy and beneficial as it used to be - and when you add in the new kids coming out of school looking for work, available IT positions can quickly become races to the bottom in terms of salary.

So as much as an admin would prefer to take the moral high ground, they also have to look out for number one. Everything is a trade-off nowadays, unfortunately.

Re:Admin code of ethics.

[anyGould]

From TOFA: "Childs, being held in a jail cell on $5 million bond, also happens to be a former felon convicted of aggravated robbery and burglary stemming from charges over two decades ago, which the city knew when it hired him as a city computer engineer."

Which, considering the rest of the FUD around this case, doesn't surprise me.

if they dont find it, I'll be a hacker holy grail

[Coraon]

I can see it now, the mythical nethack terminus of San Fran, with it is the power to control the settings for the city...if you can find it.

Hey, you're smarter than J. Michael Cook!

[Medievalist]

In fact, you just proved you are smarter than all of these guys.

Oh, sorry, that wasn't much of a compliment, was it?

So that's a good point ..

[bratwiz]

I would be inclined to agree - you've got no right as a professional to lock out the owner of the kit, from their stuff.

Who is actually the OWNER of the system? The boss? Isn't he employed by the same company as the sysadmin? Don't they both have an obligation to safeguard the OWNER'S property and interests? If the sysadmin refuses to hand over the password to sensitive equipment & systems to a (perceived) inept superior-- as long as that guy DOESN'T own the company-- isn't he actually performing his responsibility to the real owner? Which in this case would be the city, and the personification of the city would be the mayor-- and that's exactly who he DID give the passwords to. So it seems to me like he did precisely what he was supposed to do in terms of safeguarding the network and sensitive equipment. Of course he should probably be then fired for failing to keep backups, conops, continuity planning, etc. But that's a different matter.

The admin thought of this ...

[puddles]

and changed the MAC address to C0:FF:EE:C0:FF:EE

or

FE:ED:C0:ED:BA:BE ...

Just saying

Reminds me of a high school prank

[aclarke]

I went to a boarding school in Kenya for high school. The system of bells ran across the campus of several hundred acres and many buildings in a closed loop, with all the bells in series. The system ran through the main office, with the Super Secure Bell System locked in a cabinet there so nobody could access it. Penalty for messing with the system of bells was said to be expulsion.

The problem was, that all you had to do to get all the bells on campus to ring was to wire the loop back into the mains.

We took a clock from the darkroom in the photo lab, and ran two wires through the face plate. We then ran another strip of wire along the minute hand, so whenever the minute hand swept by a certain point on the clock every hour, it would complete the circuit for about 30 seconds and ring every bell on campus.

We then hid this contraption under a pile of wood in the attic of the wood shop. Right after convocation when I could no longer be expelled, I ran into the building and turned it on.

Apparently the bells rang off and on mysteriously for most of the next month of holiday until they managed to follow the loop and find the device. Good times.

Honeypot ?

[Anonymous Coward]

It could even be a Honeypot...

/oblig bash.org quote

[Shoone]

hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.

Don't mod that "funny".

[khasim]

It appears that the idiot "boss" is attempting to generate support for the claim that this guy is a "problem" by paying unreasonable amounts to "repair" the "damage" he did.

It's difficult to "prove" that a guy did millions of dollars of "damage" ... without a bill for millions of dollars of "repairs".

Any competent network admin could map out the network and document it for FAR less than the hundreds of thousands of dollars that is being thrown about.

Re:What is this ``terminal server'' thing anyway?

[lightning01]

I personally don't follow the confusion over what this box is. They indicate it has "router like" login - if they use Cisco's, it's most likely an old cisco terminal server plugged in somewhere. If they can reach it on the network, I'm having a hard time understanding why they can't narrow down where it is. I'm guessing they don't physically label their hardware? What?! I mean, if you can traceroute to it, you can get a MAC address which will give you the device mfg. From there it's a matter of following the cables form the last hop surely to likely boxes. What the hell am I missing here?

Perhaps the article is overly simplistic in its description. Perhaps they've done all this and still can't find it. The MAC address has been changed or tracing 900 cables is taking them a while or something. But I still wouldn't be talking to the press admitting my own departments incompetence. I mean sheesh!

Re:Please - It's San Francisco or simply "The City

[Thansal]

Hissssss

Not always

[Weaselmancer]

When users ask for Admin privilages, they should be told to go fsck themselves. No matter who they are.

I'm a software developer. For the first few weeks working here IT wouldn't give me admin rights on my own box. I couldn't install software.

So I sat here and did nothing. Not because that's what I wanted. But because that's all I could do, until they gave me permissions on my machine.

Generally speaking, you're right. Most people in a business should be locked down. But not everyone. Depends on the person - depends on the work they're doing.

I should have R TFA...

[BrokenHalo]

Sorry to commit the solesism of replying to myself, but I (gasp!) just read TFA.

Childs, who has worked for the city for five years but faced firing for alleged poor performance... ...being held in a jail cell on $5 million bond, also happens to be a former felon convicted of aggravated robbery and burglary stemming from charges over two decades ago, which the city knew when it hired him as a city computer engineer.

Illuminating, but mostly in that it shows all parties in a very dim kind of light. Under the circumstances, I would have hesitated to employ the guy in this capacity anyway...

Re:Simple: Local Incompetence in Play?

[davidsyes]

"All they have to do is look for the small black box with a lone, onerous blinking red LED."

Not to be a grammar/word-choice "Nazi", but I think you meant "ominous".

But, after all this time, one might expect that the NSA would have been on top of this. Anytime a city government fails to locate rogue devices that could compromise local/state/federal/international investigations, the criminals and the undercover agents/officers, and witnesses, as well as payroll and other HR information, the FBI, NSA, and other agencies should take over that aspect where the locals prove incompetent.

It could happen-harder to find that it seems

[mschuyler]

In a big network I could see this happening. I know--computer rooms are supposed to be pristine with every wire perfectly aligned and in place with everything perfectly labeled and mapped--NOT! Most computer rooms I've been in, including my own, are somewhat less than ideal. They kind of grew with no plan. Need more space? Run a jumper. One of the Field Engineers who worked on one of our minis just laughed and said we weren't really that bad--you should see banks--they're the worst. In other words, poor housekeeping is widespread and tolerated. A typical terminal server could be 1RU or even a blade, or a box sitting loose on top of the rack where you can't see it. If I were really devious I would put a small terminal server in a bigger box. If this were intentionally hidden it could be in the ceiling hooked to a 128 port hub in the rafters itself and you'd never even know it. It's a bird's nest of Cat5 around a hub, all looking the same. I'll just bet it's a Class B network, so you've got a tremendous number of possibilities. And if you used virtual networks on Cisco hubs or did some bizarre subnets that simply confounds matters. I feel very confident that I could hide a box in my building that even the pros would have a hard time finding. Of course you could start turning off power until the device disappeared to try to pin down its location, but my guess is no one wants to do that just because someone lost a box. Too funny.

Managed or Unmanaged

[multimediavt]

Well, if it's on a managed network then the IP needs to be mapped to a MAC address (and port on a switch) and the port turned off. Once that's done, tracing the cable to a physical port should take no more than a day.

If it's on an unmanaged switch things get a little more annoying, but you should still be able to track an IP and MAC address to the switch using any open source network tool like WireShark. Find the switch. Pull the cable out of the port, or if you're feeling really adventurous you could bring a replacement switch in and start playing "Is that it?" until you find the bugger.

Must be a slow day for something this ho-hum to make it on /.

The new WarLords

[DeanFox]

I'm reminded of a conversation I had some 25 years ago with a co-worker IBM mainframe technician. IBM management was incensed that uneducated morons turning screwdrivers could make 70k a year. Back then as much as what they were paying top MBA stuff shirt types. They were on a mission to get salary levels down to "reality" paying these screwdriver wielding monkeys what they were (in their minds) really worth.

Attitudes have changed but not a lot. 93% of companies that loose their data center for 10 days or more due to a disaster filed for bankruptcy within one year. 50% filed bankruptcy immediately (National Archives & Records Administration in Washington). One can't say the same thing about those over paid MBAs.

It may be awhile before IT matures into a "profession" like doctor or lawyer however I personally believe we're holding the keys. The world can't function now without us.

-[d]-

What's the problem?

[PPH]

It shouldn't be that difficult to find a piece of h/w on a network.

Interrogate the switches to find the IP/MAC address corresponding to the device you are trying to log on to. In the event that this Childs guy is deviously smart (i.e. patched the switch software to conceal a particular device) one can still use a stand-alone sniffer to trace packets through a system.

Its possible that the 'terminal server' might be virtual, just an app. running on some other piece of hardware that doesn't necessarily have "ACME Terminal Server" and a wining LED on the front. But tracing the network to that particular box isn't difficult (maybe time consuming).

If these people are really that dumb, I can understand why Childs kept them off the system. Reading some of the stories about him, it wouldn't surprise me if he left a bunch of 'dead ends', like phony terminal servers that nobody could find. Or wireless access points not plugged into anything but plastered inside a wall to drive security auditors nuts.

of course this may seem

[nimbius]

malicious, but im on the side of the ex employee. if the device is his, i hope he uses it. it seems like theres alot of incompetence and coverup going on at the sanfran city network. this story has stunk since they decided to imprison him for not giving the password. "unknown wireless device" just further confirms theres a good chance nobody know what the hell theyre doing, and this guy could have been right.

Re:Anyone taking bets?

[blair1q]

$30 it's the boss's desktop machine.

Marx Brothers of network admin: whats the passwd

[pbhj]

Cheeky: What's the password
Dildo: Yeah, that's what I said, what's the password
Bobo: Why are you asking me, I'm asking you ...
Cheeky: You're asking who?
Dildo: Hu doesn't know ...

Hilarity ensues??

More technical info on the device

[snydeq]

Paul Venezia digs a little deeper into this so-called "terminal server" today in his blog:

"From what I can see, it's a device running Cisco IOS that was accessed via telnet. I could generate an identical screenshot to the one entered into evidence in about five minutes using an elderly Cisco 2924-XL Ethernet switch -- a device that's certainly not a terminal server. It's completely unclear to me how they could have possibly come to the conclusion that this is a "terminal server" -- the evidence presented to the court certainly does not support that theory."

Venezia also uncovers additional technical errors in the prosecution's case, which appears to be unraveling with the recent news that the DTIS Datacenter Supervisor Ramon Pabros will testify on Childs' behalf. Since coming forward, Pabros has announced he will be retiring from the DTIS, effective Sept. 17. Coincidence?

Uh? Not that easy.

[phorm]

Sometimes you inherit the fires. Oftimes they may be created by other people, and frankly, without enough co-operation by management (either dealing with consistent firestarters or by hiring supporting staff), you cannot make yourself redundant.

There's only so much time in the day for a given person to do a given set of tasks.

VMware

[nurb432]

Someone loaded vmware server on their desktop that has an extra network card.

Here ya go

[wiredog]

UNC in 2001

How about this approach?

[gmezero]

I used this once to track down which server room a system was located in and while it's not perfect for all occasions, it might help.

Ok, first if you can get an IP for the device, perform a traceroute from 3 or 4 separate sites. Identify it's Gateway if possible, also if find see if you can determine from the traceroutes if it has a common parent node that it's traffic is going through.

Once you've found the most common system talking to it, go to that system and perform ping tests to other systems where you know their physical location in proximity to the system your at, and are only 1 hop away (if possible). The key here is to make sure that all of your samples share as much of the same route as possible to minimize signal noise in your data set you're going to build.

See if you can develop a correlation between ping times and amount of network cable to your sample set. Compare that data to the ping times on your mystery device and you *potentially* have a physical range now in hand to perform your search.

I'll be the first to admit that this approach has limited success based on how your infrastructure is built, but it might help.

Re:How about this approach?

[msaulters]

OR, one could do a traceroute to the IP and check the ARP tables of that gateway.

The problem I suspect is that like most governments, they're still using a mix of very old technology. This thing might not even be running IP. Of course, one then presumes to ask "How did they know it's there in the first place."

Traceroute ?

[billcopc]

I must be missing some key information here, but if the thing has an IP address, they should be able to track it down to the nearest router/switch and follow the cabling, no ? It's not like the thing is sitting in some guy's closet.

What's in a name

[clarkn0va]

Because I'm a...dumbass and didn't think about it....

Or could it be that you just didn't care, Lord Apathy?

db