Slashdot Mirror
San Fran Hunts For Mystery Device On City Network
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.
Given that they cannot hunt down a single device on the network, I'd have to agree with that assessment.
MAC address ... switch port ... it should be easy.
Man, the more I read about this story, the more inclined I am to believe the network admin.
He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.
When I first heard what the rogue-SF-admin had done, I was very negative on his actions.
Now, that once again, and now at least for the third time, I hear of absolute stupidity and ineptness on the group at sf, I am certain the so called rogue was right on the ball from the beginning.
If the city can't even complete one of the most basic network administration tasks of finding a physical device on a network, I think they have absolutely no right to accuse anyone of "hijacking" their network. I hope the defense attorney for Terry Childs brings this up.
your employer's passwords are NOT yours, no matter how stupid you think your boss is.
Refusing to give out passwords to higher-ups is not always the wrong thing to do. If you are the network admin, and your job is to maintain security of the network, wouldn't it be reasonable to refuse to hand out passwords to people outside of the network administration roles?
Although I can say that an admin can make that choice at his or her own peril. After all, the higher-ups can always opt to fire the admin and replace him or her with someone who is willing to seek security of their job over security of the network they are paid to administer.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
The admin might not be stupid he might be an ass
1) He placed a rouge device (his personal property) on the SF network
2) He set all the network devices on the network to lose all info on a reboot
3) He will hand over the passwords (after jail) to all the devices except the rogue
You can make equipment hard to find ( mac masquerading comes to mind )... I'm only adequate in terms of networking but I am pretty sure someone who is really good can play a mean game of hide and seek. Who knows *what* he was doing with that device? and were I the network admin I would have to *on principle alone* rebuild everything after this guy left..
"Ahh! Arrogance and stupidity in the same package, how efficient of you!" --Londo Molari
I'd like to add that while the way he handled being surrounded by idiots was wrong, he was clearly surrounded by idiots.
No documentation?
No change control?
No diagrams?
What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.
I worked one place where a guy with a great deal of responsibility died. (here today dead tomorrow kind of thing) His peers blamed *everything* on him simply because they could. This sounds like the same thing.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
I learned early on, that most people don't see the difference between a $12 hour high school geek and a $75 hr network administrator. All most people see is that both do roughly the same job and there is $63 hour difference.
Most of the time, the $12 hr guy is doing most of the same work as the $75 hour guy. The big difference is when crap like this comes up, the $12 hour guy can spend years trying to figure out what the $75 hr guy can figure out in 5 minutes.
Even when the $12 hr guy screws up, the response is "But he was cheaper". It is cheaper to keep a $12 hr guy trying to keep crapware off a computer, rather than a $75 hour guy who doesn't allow crapware in the first place.
The point I'm making, is that a $75 hr guy is worth it, but only to people where time has real value. People who place no value on TIME, don't care about anything other than $ per HR
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
I wish I had mod point for you.
Chances are that internal policies prevent the use of "hacker" tools to secure the network.
Again, the PHBs are idiots!
Obama's legacy: (N)othing (S)ecure (A)nywhere and (T)error (S)imulation (A)dministration
If you find that you are "holding the place together", IT-wise, you are likely part of the co-dependency and are part of the problem.
IT and the other management have both agreed to ignore each other, literally or otherwise, allowing each (and the individual personalities) to do things "their way"; damn the best practices, good management, logical, financial, or even legal issues.
Except when things go wrong.
Like a breakup, they can get ugly. And, as the IT guy, you will always lose for it is not your Business, but theirs. You are simply hired help.
I wonder if this one is just a complete misunderstanding. One article says that they were set to lose configuration files on "reset". That's pretty typical -- if you have some device you don't have the password to, you can do a full factory reset and get it back to the default password, but that also wipes the configuration files. He might have told his incompetent bosses that, and they thought he meant they'd lose the files on a reboot instead.
Anyway, if this guy is what they're making him out to be, they need to completely wipe and reconfigure the network anyway; it's the only way to be sure he didn't leave a few presents for them.
I think you guys mean "ominous", not "onerous".
What would you think of a doctor who, because some exec somewhere decided he should, pushed the WRONG medication / procedure to you?
Where does your ethical responsibility end and the boss's desires begin?
To me there isn't even a question. Fire me. Go ahead. I will get another job.
Yes, both of those are true (Mac, Ping). Even NMAP responses can be spoofed. However the likelihood of all three being done is not likely. However NMAP will reveal a used IP, and a mac table somewhere will identify what port it is hanging on. Packets have to be routed to it somehow.
And I agree with your last point. I'm a Libertarian. ;)
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
It appears that the idiot "boss" is attempting to generate support for the claim that this guy is a "problem" by paying unreasonable amounts to "repair" the "damage" he did.
It's difficult to "prove" that a guy did millions of dollars of "damage" ... without a bill for millions of dollars of "repairs".
Any competent network admin could map out the network and document it for FAR less than the hundreds of thousands of dollars that is being thrown about.
There do appear to be a lot of morons involved in this scenario, and Childs was one of them. Basically what he said was "I am smarter than all of you, so I will do things my way, and trust me, you'll be better off."
Except they weren't, because he doesn't appear to be anywhere near as smart as he thinks he is. Even if he was smarter than the gaggle of incompetents he worked for.
Eagles may soar, but weasels don't get sucked into jet engines.
I personally don't follow the confusion over what this box is. They indicate it has "router like" login - if they use Cisco's, it's most likely an old cisco terminal server plugged in somewhere. If they can reach it on the network, I'm having a hard time understanding why they can't narrow down where it is. I'm guessing they don't physically label their hardware? What?! I mean, if you can traceroute to it, you can get a MAC address which will give you the device mfg. From there it's a matter of following the cables form the last hop surely to likely boxes. What the hell am I missing here?
Perhaps the article is overly simplistic in its description. Perhaps they've done all this and still can't find it. The MAC address has been changed or tracing 900 cables is taking them a while or something. But I still wouldn't be talking to the press admitting my own departments incompetence. I mean sheesh!
When users ask for Admin privilages, they should be told to go fsck themselves. No matter who they are.
I'm a software developer. For the first few weeks working here IT wouldn't give me admin rights on my own box. I couldn't install software.
So I sat here and did nothing. Not because that's what I wanted. But because that's all I could do, until they gave me permissions on my machine.
Generally speaking, you're right. Most people in a business should be locked down. But not everyone. Depends on the person - depends on the work they're doing.
Weaselmancer
rediculous.
Big assumption.
They probably deleted all those "useless files" on the fileserver when they fired him.
And the "terminal server" is probably his iPhone...
Never answer an anonymous letter. - Yogi Berra
...you tell us, Mr. "anonymous".
...no two people are not on fire.
Oh man, that is so hilarious. I love this part especially:
I cannot find any information in my MCSE bootcamp journal on how to handle this
Just more proof that MCSE certification is completely useless other than for getting a job. :)
I disagree.
It isn't that simple; it seems that there is waaaaay more to the story that some ego tripping sysadmin.
Everytime another piece of the story or fact about what happened comes out it seems to vindicate Mr. Childs to some degree (not that his judgement was flawless in how this was handled, but still).
Is he still locked up? If so it's a travesty.
It seems like those who are trying to have him tarred and feathered constantly want to make it look like he's some super-e-terrorist who was holding the entire city for ransom and has dealt an economic blow from which the city will never recover.
I am not saying everything he did was right, or that he committed no wrongs here; but I think it's pretty obvious that this was viewed as a pissing match by those in the city who wanted him to hand over that information and they have gone to great lengths to make it look like something much more malicious than it was in the press.
He may have had very good reason to protect it; (I mean aside from the fact that it appears as though those who wanted him to hand it over were incompetent) - because I don't think anyone would put their own ass on the line for jailtime and the loss of their job unless there was something else going on. I am not saying I know this to be true, just that that is how it appears to me based on the available information.
At this point I view anything coming from the anti-Child's side of this issue with a healthy does of skepticism and try to read through the sensationalization. Something has always stunk about this situation.
If you have SNMP and a Winders PC;
log.txt (list of your switches / routers)
192.168.1.1
192.168.1.2
file1.bat /f "tokens=1,2,3,4,5,6,7,8,9,10" %%i in (log.txt) do call distcmd.bat %%i %%j %%k %%l %%m %%n %%o %%p %%q %%r
for
distcmd.bat (change public to be your snmp community snmputil is from net-snmp-5.4.1-3.win32.exe / free) .1.3.6.1.2.1.17.4.3.1.1 >> %1.MAC.log .1.3.6.1.2.1.17.4.3.1.2 >> %1.PORT.log
snmputil walk %1 public
snmputil walk %1 public
Now you have a list of every port and ever MAC that runs through that port (don't forget about uplink ports having all MACs listed to them.)
In the file you'll have to do some snmp decimal to hex conversions; .17.4.3.1.1.0.23.164.215.49.153 .0.23.164.215.49.153 Put those number in calc and convert each to a hex and you get
MAC is 00:17:a4:d7:31:99
SNMP walking gives a result of which is
the same MAC, but converted to SNMP notation this becomes
Part of this is the MAC address in decimal
the mac address in hex.
So in the output of these batch files .17.4.3.1.2.0.64.140.109.101.123
Variable =
Value = Integer32 24
Means that on port 24 there is MAC address 00:40:8c:6d:65:7b which is the .0.64.140.109.101.123 converted to Hex through Calc.
Anyway, ping the device, arp -a and locate the MAC address, dump the above against all your routers/switches, convert your MAC address you are searching for to SNMP decimal, and search the .txt files for a switch with a port where that is the only MAC on the port and you've found the device (or a hub between the device.)
MCSE:
Must Consult Someone Experienced
Minesweeper Consultant and Solitaire Expert
Not really. A network admin should be able to track down the thing, but it will take a lot of work to scan network logs. From the network standpoint, it doesn't matter if the gateway is running on a PC, or running on a VM inside a PC... the network traffic looks the same.
I'm reminded of a conversation I had some 25 years ago with a co-worker IBM mainframe technician. IBM management was incensed that uneducated morons turning screwdrivers could make 70k a year. Back then as much as what they were paying top MBA stuff shirt types. They were on a mission to get salary levels down to "reality" paying these screwdriver wielding monkeys what they were (in their minds) really worth.
Attitudes have changed but not a lot. 93% of companies that loose their data center for 10 days or more due to a disaster filed for bankruptcy within one year. 50% filed bankruptcy immediately (National Archives & Records Administration in Washington). One can't say the same thing about those over paid MBAs.
It may be awhile before IT matures into a "profession" like doctor or lawyer however I personally believe we're holding the keys. The world can't function now without us.
-[d]-
Either the employees are seeing the writing on the wall or the city is trying to get rid of any childs' loyalists(purge). Most likely both is happening. Heaven forbid they get rid of anyone else who knows how to maintain the network.
and your not sniffing the traffic to these boxes why?
>But everyone who supports more government ought to take a look at the incompetence here.
Im one of those crazies who doesnt support more or less government. Just better government.
Sometimes you inherit the fires. Oftimes they may be created by other people, and frankly, without enough co-operation by management (either dealing with consistent firestarters or by hiring supporting staff), you cannot make yourself redundant.
There's only so much time in the day for a given person to do a given set of tasks.
If you're really lazy, you could also unplug their network cables and see what breaks... :P
If you believe everything you read, you'd better not read. - Japanese proverb
1. Boot from floppy, optical media, network, etc. /mnt -o rw /mnt
2. mount [/dev/sda1|/dev/hda1]
3. chroot
4. passwd root [password]
5. ??????
6. PROFIT!
No yanking to do. A reboot and 5 minutes of down time. Bang. Dead. Done.
The Christian Right is Neither (Christian nor right). See: Matthew 23, Matthew 25, Ezekiel 16:48-50
It may also mock you with nonexistent cake.
How are sites slashdotted when nobody reads TFAs?
I have a huge admiration for your honesty. You are an exceptional person.
"The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
Holy crap, +5 insightful? I like my karma as much as anyone else, so no complaints, but... huh?
How are sites slashdotted when nobody reads TFAs?