San Fran Hunts For Mystery Device On City Network

alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."

Comments

The story keeps changing.

[khasim]

From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.

Given that they cannot hunt down a single device on the network, I'd have to agree with that assessment.

MAC address ... switch port ... it should be easy.

Re:The story keeps changing.

[Crudely_Indecent]

If Superman had any IT skills, he'd perform a traceroute to determine the devices gateway. Once the gateway was determined, block the mac address from accessing the network. If the admin of that device is worth his salt, he'll change the mac address and continue. They could then specifically enable allowed devices and forbid all others.

Forget finding it, make the network inaccessible.

City of SF Admins, if this proves to be your resolution, you owe me $150 for 1 hour of my time. Sorry, I do not bill in lower increments.

Siding with the network guy

[John Jamieson]

Man, the more I read about this story, the more inclined I am to believe the network admin.

He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.

not necessarily wrong...

[damn_registrars]

your employer's passwords are NOT yours, no matter how stupid you think your boss is.

Refusing to give out passwords to higher-ups is not always the wrong thing to do. If you are the network admin, and your job is to maintain security of the network, wouldn't it be reasonable to refuse to hand out passwords to people outside of the network administration roles?

Although I can say that an admin can make that choice at his or her own peril. After all, the higher-ups can always opt to fire the admin and replace him or her with someone who is willing to seek security of their job over security of the network they are paid to administer.

Mod Parent Up

[mpapet]

I'd like to add that while the way he handled being surrounded by idiots was wrong, he was clearly surrounded by idiots.

No documentation?
No change control?
No diagrams?

What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.

I worked one place where a guy with a great deal of responsibility died. (here today dead tomorrow kind of thing) His peers blamed *everything* on him simply because they could. This sounds like the same thing.

Re:Mod Parent Up

[Sobrique]

Wait, you mean blame it all on the guy who left (be it through death or a cushy new job) isn't standard practice everywhere?

Re:Mod Parent Up

[rickb928]

I took a gig recovering documentation and re-establishing procedures for a great admin who died as well. He really did great docs, but no one had ever used them, and they couldn't figure out the 'copy file piopoiop.dfj to the \asic\wer\2344\sdf.msdfn folder' sort of directions.

And the crew there immediately set to removing, replacing, and destroying all of his systems. He was a Novell hardliner (so was I), and when he was gone, his boss succumbed and the Windows bigots prevailed. Much taxpayer money was spent replacing perfectly functional systems. Mind you their clients were still running Novell, so there was some disconnect when they would get a request for support and start saying 'you have to upgrade (ha!) to Windows'. Their clients, for reasons best left undisclosed, could not upgrade. Both physically impossible and logistically impractical. Start with being 60-1600 meters below the ocean surface, and it only gets more difficult from there.

I'm a little surprised that SF hasn't worked this out. There are plenty of outfits eager to do what is necessary, for a fee of course.

And yes, finding a device is not impossible. Finding the connection to the network is the obvious first step. After that, well, kill it.

Unless it's hiding. That would be unfortunate.

ps- This guy, by many accounts, was brilliant. And a little off the wall. Goes together.

Re:MAC search

[Archangel Michael]

I learned early on, that most people don't see the difference between a $12 hour high school geek and a $75 hr network administrator. All most people see is that both do roughly the same job and there is $63 hour difference.

Most of the time, the $12 hr guy is doing most of the same work as the $75 hour guy. The big difference is when crap like this comes up, the $12 hour guy can spend years trying to figure out what the $75 hr guy can figure out in 5 minutes.

Even when the $12 hr guy screws up, the response is "But he was cheaper". It is cheaper to keep a $12 hr guy trying to keep crapware off a computer, rather than a $75 hour guy who doesn't allow crapware in the first place.

The point I'm making, is that a $75 hr guy is worth it, but only to people where time has real value. People who place no value on TIME, don't care about anything other than $ per HR

Don't mod that "funny".

[khasim]

It appears that the idiot "boss" is attempting to generate support for the claim that this guy is a "problem" by paying unreasonable amounts to "repair" the "damage" he did.

It's difficult to "prove" that a guy did millions of dollars of "damage" ... without a bill for millions of dollars of "repairs".

Any competent network admin could map out the network and document it for FAR less than the hundreds of thousands of dollars that is being thrown about.

Re:Simple:

[CrossChris]

MCSE:

Must Consult Someone Experienced

Minesweeper Consultant and Solitaire Expert

Re:Simple:

[ajrs]

and your not sniffing the traffic to these boxes why?

Re:You're an 1D10T

[gad_zuki!]

>But everyone who supports more government ought to take a look at the incompetence here.

Im one of those crazies who doesnt support more or less government. Just better government.

Re:Simple:

[Firehed]

Poison gas ? You think that's all an evil supercomputer will do ? NO ! It will spontaneously develop godlike powers, take over the universe and unravel the very fabric of reality around you !

It may also mock you with nonexistent cake.