Slashdot Mirror
San Fran Hunts For Mystery Device On City Network
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
Exactly, hell I can sit down with my laptop and tell you what switch it's connected to in 20 minutes. Bet you $50.00 the community strings on all their network gear is still set to public and private :)
Are the IT people they hire completely dysfunctional? Or do they do what most cities do and not actually hire IT people or networking admins because they command a real salary instead of the $12.00 an hour that someone handy with computers get's...
Do not look at laser with remaining good eye.
Hey! Fyodor! They need your number!
Fyodor spent much of this summer scanning tens of millions of IPs on the Internet (plus collecting data contributed by some enterprises) to determine the most commonly open ports. Nmap now uses that empirical data to scan more effectively.
Zenmap Topology and Aggregation features were added, as discussed in the next news item.
Hundreds of OS detection signatures were added, bringing the total to 1,503.
Seven new Nmap Scripting Engine (NSE) scripts were added. These automate routing AS number lookups, "Kaminsky" DNS bug vulnerability checking, brute force POP3 authentication cracking, SNMP querying and brute forcing, and whois lookups against target IP space. Many valuable libraries were added as well.
Many performance improvements and bug fixes were implemented. In particular, Nmap now works again on Windows 2000.
With just nmap, my old buddies at Farm9 could have sussed this out in a few hours. I think they are still around - as Red Siren / Getronics.
Ahh. I miss running netcat at 3 AM!
"Flyin' in just a sweet place,
Never been known to fail..."
You think they've learned anything about the gear since then? No wonder they're having problems.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
Why is Slashdot linking to stories that paint the network administrator as a bad guy when he's so obviously surrounded by morons? These are the same people who published all of their user names and passwords. That puts the cost of this "hijacking" into perspective. The cost of trusting their employee with the powers required to do the job was zero.
Friends don't help friends install M$ junk.
I worked for a company where they cheaped out on the switch infrastructure and bought low-end Dell switches for the entire network. The kind that don't let you see the MAC address table.
Some guy decided to bring in his Linksys router from home so he could use his laptop and his desktop at the same time (instead of, you know, asking IT to add a second port at his desk). Problem was he left DHCP running on the thing, which obviously led to some confusion. Took forever to find it.
Then again it sounds like the city of 'cisco bough nothing but Cisco gear, so who knows what's really going on here...
Your London may be inferior. Ours definitely warrants a 'City' moniker. Especially when The City of London is distinct from the conurbation that is known as London. And the City of London is actually fairly small - almost exactly a square mile - but ... well, you know what they say. It's not the size, it's how you use it.
Could it be possible that the device is actually virtual? Like a Virtual Machine running under VMware or Virtual PC somewhere, with the software obfuscated or hidden? It would be a lot harder to track down that way.
Now, as regards passwords and what not, I would be inclined to agree - you've got no right as a professional to lock out the owner of the kit, from their stuff. However I'd also say escalating it higher because there's 'serious ethical implications' in some situations isn't unreasonable. Not that this necessarily relates to this particular case - I don't know the details, so I won't comment - I just wanted to point out that there are good and valid reasons not to comply with a demand like this from your direct 'boss'.
Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.
My obligation to my employer (in this case the city of San Francisco) trumps my obligation to my PHB. If I think my PHB is a moron and is going to cause a shitload of damage to my employer then I think I could make a good case for refusing to give him the passwords.
Of course that's not where it would end.... I would have to explain to his boss what the problem was -- or go even further up the chain of command if he was also a moron.
Assuming that they have wireless on their network, there's no way to find wireless devices
Wireless devices still have MAC addresses. By tracing the MAC address you'd get a switch port. If that switch port has an AP plugged into it then you know it's a wireless device and probably know it's general location (the AP doesn't have limitless range).
there's no real way to find exactly where wireless devices are, as far as I know
Oh, there's a way.... it's just out of the reach of most of us.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Your might be thinking of the Novell NetWare server story. University of North Carolina in 2001. It was physically MIA for 4 years yet kept doing the Energizer Bunny routine. I was a Novell Reseller at the time and the story made a great sales pitch. http://www.techweb.com/wire/story/TWB20010409S0012
Well, the fact that they're contracting outside Cisco experts now suggests nobody else there was technically competent enough to manage the network.
The fact that the network stayed up and running without a hitch, while he was in jail and nobody else had access, suggests he did know what he was doing, and refusing to allow anyone to access the routers to make changes seems to work quite well to keep the system working.
The fact that his supervisors are moronic and useless is no small thing, either.
His actions were extremely stupid, but I fail to see why this idiot's relatively non-disruptive actions rise to the level of criminal prosecution.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
In fact, you just proved you are smarter than all of these guys.
Oh, sorry, that wasn't much of a compliment, was it?
Who is actually the OWNER of the system? The boss? Isn't he employed by the same company as the sysadmin? Don't they both have an obligation to safeguard the OWNER'S property and interests? If the sysadmin refuses to hand over the password to sensitive equipment & systems to a (perceived) inept superior-- as long as that guy DOESN'T own the company-- isn't he actually performing his responsibility to the real owner? Which in this case would be the city, and the personification of the city would be the mayor-- and that's exactly who he DID give the passwords to. So it seems to me like he did precisely what he was supposed to do in terms of safeguarding the network and sensitive equipment. Of course he should probably be then fired for failing to keep backups, conops, continuity planning, etc. But that's a different matter.
There is an old, probably apocryphal tale from the days of Novel Netware and IPX of the forgotten server. A loan machine runs headless with a quiet fan and no lights in a corner of a room. New remodeling puts the server behind sheet rock and there it sits walled up and running for years. One day a power spike causes a head crash and suddenly a national billing system dies. It takes a tech tracing a cat5 cable into a wall to find it.
- Minutus cantorum, minutus balorum, minutus carborata descendum pantorum.
You're preaching to the choir. I firmly believed that when I was a LAN administrator my responsibility was first to the integrity of the network and second to anything else.
Unfortunately IT professionals aren't in as much of a seller's market now as they were before. Getting another job isn't always as easy and beneficial as it used to be - and when you add in the new kids coming out of school looking for work, available IT positions can quickly become races to the bottom in terms of salary.
So as much as an admin would prefer to take the moral high ground, they also have to look out for number one. Everything is a trade-off nowadays, unfortunately.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
It's written into my contract that I do not document the domain admin password, and that I do not share it with anybody outside of the technical IT team without written confirmation from the Network Admin. The IT Manager and Head both agreed to this, and it won't change while I work here.
When users ask for Admin privilages, they should be told to go fsck themselves. No matter who they are.
Finally had enough. Come see us over at https://soylentnews.org/
It could even be a Honeypot...
Not at all. I dealt with this very issue twice for the same organization. They bought wireless routers and wanted to use them like access points. They put port 1 on the network and placed a computer on port 2, never using the WAN port. This is better setup than using the WAN port because you can't as easily access the computers behind the WAN port. The problem was they wouldn't disable DHCP causing all sorts of issues. Twice I went in and explained that they MUST disable DHCP if they want to use the router in this fashion and last I heard they reset the routers again and were having the same issues. Of course, my name gets dragged in the mud because they think I'm the idiot.
They could always do something crazy like track the MAC to a port and go trace the cable to find the device, I guess that wouldn't make such a good story though.
If they're using Cisco switches and it's linked via copper then they could probably work out where it is without leaving their seats, use the inbuilt tdr to find out how long the cable is, then use the location of the switch and a bit of common sense to work out where the device is likely to be.
If it's a terminal server then it's not likely to be hanging off a 3km long fibre somewhere in a duct under the city. It'll be within serial cable distance of all the other kit, more than likely in their main computer room with some bloody great octal cables hanging out the back. I suspect it'd take someone clued up approx 5 minutes to identify it as it will look rather different to any of their other routers purely due to the cabling run to/from it.
The more I read about this "ebil admin" story the less I believe any of it.
Reminds me of a guy I knew who used piezoelectric fire lighters (it's the one used in stoves) to test the watchdogs on circuits he built.
He fired it over the processor and the interference would be enough to disturb it (electrically isolated of couse, the spark would not go to the device, only the EM interference).
how long until
"All they have to do is look for the small black box with a lone, onerous blinking red LED."
Not to be a grammar/word-choice "Nazi", but I think you meant "ominous".
But, after all this time, one might expect that the NSA would have been on top of this. Anytime a city government fails to locate rogue devices that could compromise local/state/federal/international investigations, the criminals and the undercover agents/officers, and witnesses, as well as payroll and other HR information, the FBI, NSA, and other agencies should take over that aspect where the locals prove incompetent.
Previously: "Linux... Toward the Sunrise..." Now: "Linux... Toward the-- No, now, part of Every Sunrise"
I was thinking the same thing. Couldn't you traceroute and show arp tables to find where it is?!?!
Disclaimer: I am a sys admin, but not for the municipality of San Francisco, so my ignorance of their network architecture might be masking something that makes this procedure non-trivial. For the life of me, I can't image what, however.
MCSE? No, sir...I don't do Windows. Yes, I am an idealist. What's your point?
In a big network I could see this happening. I know--computer rooms are supposed to be pristine with every wire perfectly aligned and in place with everything perfectly labeled and mapped--NOT! Most computer rooms I've been in, including my own, are somewhat less than ideal. They kind of grew with no plan. Need more space? Run a jumper. One of the Field Engineers who worked on one of our minis just laughed and said we weren't really that bad--you should see banks--they're the worst. In other words, poor housekeeping is widespread and tolerated. A typical terminal server could be 1RU or even a blade, or a box sitting loose on top of the rack where you can't see it. If I were really devious I would put a small terminal server in a bigger box. If this were intentionally hidden it could be in the ceiling hooked to a 128 port hub in the rafters itself and you'd never even know it. It's a bird's nest of Cat5 around a hub, all looking the same. I'll just bet it's a Class B network, so you've got a tremendous number of possibilities. And if you used virtual networks on Cisco hubs or did some bizarre subnets that simply confounds matters. I feel very confident that I could hide a box in my building that even the pros would have a hard time finding. Of course you could start turning off power until the device disappeared to try to pin down its location, but my guess is no one wants to do that just because someone lost a box. Too funny.
How about a moderation of -1 pedantic.
I'm putting my money that its a Mac server that everyone passes by and says, "Oh, that's Mac, it couldn't possibly be that. Why bother checking. It must be from the Evil Empire. We're looking for black, not white."
jsut athnoer menagiensls ltitle psrhae for you to dcoede. Why do we wtsae our tmie dnoig tihs?
What it would have (if it is similar to how I use them, and yes I am a WAN specialist) is a phone-line for dial in access in case of emergencies.
See MRV's InReach product line for more information.
...though it could have a MAC address on the network, just saying it doesn't have too, and if it is "mysterious" and / or put there maliciously, in all liklihood will not, or it will be spoofed to prevent detection.
Walk with Music;
It shouldn't be that difficult to find a piece of h/w on a network.
Interrogate the switches to find the IP/MAC address corresponding to the device you are trying to log on to. In the event that this Childs guy is deviously smart (i.e. patched the switch software to conceal a particular device) one can still use a stand-alone sniffer to trace packets through a system.
Its possible that the 'terminal server' might be virtual, just an app. running on some other piece of hardware that doesn't necessarily have "ACME Terminal Server" and a wining LED on the front. But tracing the network to that particular box isn't difficult (maybe time consuming).
If these people are really that dumb, I can understand why Childs kept them off the system. Reading some of the stories about him, it wouldn't surprise me if he left a bunch of 'dead ends', like phony terminal servers that nobody could find. Or wireless access points not plugged into anything but plastered inside a wall to drive security auditors nuts.
Have gnu, will travel.
malicious, but im on the side of the ex employee. if the device is his, i hope he uses it. it seems like theres alot of incompetence and coverup going on at the sanfran city network. this story has stunk since they decided to imprison him for not giving the password. "unknown wireless device" just further confirms theres a good chance nobody know what the hell theyre doing, and this guy could have been right.
Good people go to bed earlier.
You may want to stop reading what the city says, and find out what really happened.
http://it.slashdot.org/comments.pl?sid=960957&cid=24963255
The real question, though, is this: If your alternate personality made the bomb, does your present consciousness have the subliminal knowledge of which wire defuses it?
Depends on when it was I guess.
Back in 2001 I did some emergency wiring work that had to be done in 72 hours at our shop.
Now, we are only there 10 weeks a year, so after the end of the 10 weeks it was forgotten about.
I was very sleep deprived and manic when I finished the job, and to this day I have NO idea how I did some of the connections I did. I just hope and pray it all keeps working. Some day some part of it will fail, and I'll have to re-do the entire building.
Note to self:
When sleep deprived, always work from the list, and write down what you did. One thing at a time, and document everything.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order- Ed Howdershelt Via Tass
Not at all uncommon. I've got 3 fucking servers in my system room that nobody knows what they hell they are for. The are all running 2.4 kenels so they are as old as the fucking hills. Nobody knows what the passwds are to get into them so I can't log in and find out what they do. And naturally the previous systems administrator that installed them didn't document shit.
The only thing that is known about them is they used to do something important just nobody remembers what it was. Management is to afraid that they might still be doing something important and won't let me yank them out to find out what they do. So while management sits there with their collective heads up their collective asses these three servers sit there taking up space in my racks on my network.
When these thing do finally fall over I hope they are doing something important.
Supporting World Peace Through Nuclear Pacification
Wait, you mean blame it all on the guy who left (be it through death or a cushy new job) isn't standard practice everywhere?
I had to actually threaten legal action against a former employer who repeatedly claimed all the failures after I left were sabotage. Maybe its my fault for not grooming a successor, but there was some truth when I suggested my knowledge deserved higher pay.
Paul Venezia digs a little deeper into this so-called "terminal server" today in his blog:
"From what I can see, it's a device running Cisco IOS that was accessed via telnet. I could generate an identical screenshot to the one entered into evidence in about five minutes using an elderly Cisco 2924-XL Ethernet switch -- a device that's certainly not a terminal server. It's completely unclear to me how they could have possibly come to the conclusion that this is a "terminal server" -- the evidence presented to the court certainly does not support that theory."
Venezia also uncovers additional technical errors in the prosecution's case, which appears to be unraveling with the recent news that the DTIS Datacenter Supervisor Ramon Pabros will testify on Childs' behalf. Since coming forward, Pabros has announced he will be retiring from the DTIS, effective Sept. 17. Coincidence?
Someone loaded vmware server on their desktop that has an extra network card.
---- Booth was a patriot ----
I was going to say nmap it, find an old ssh/telnet/ftp exploit, nail it, then use a root escalation exploit to get root access so you can change the passwords :D
Need to make sure ftp or whatever isn't the critical remote service though...
always work from the list, and write down what you did. One thing at a time, and document everything.
This seems sensible under all conditions. Being tired is no excuse for being sloppy.
I have a sleep disorder.
There are times when, for no real discernible reason, my brain decides that I will not be sleeping for a few days. Sometimes upwards of 100 hours.
When you have been awake for 4 days, (at least in my case) you get a serious case of "While I'm at it" syndrome.
Tasks that can not be completed in 10 minutes (or without getting up) are nigh impossible. I can still work, but I am extremely easily distracted and will often forget why I am in the room I was in.
Example: I went to the fridge to get some water, and decided that I should clean it while I was there, then decide to do the dishes since I threw stuff out of the fridge, then decide to do the laundry since I had no clean towels, and while I was in the basement doing the laundry I noticed that I needed to organize the basement and throw out old computer parts. Meanwhile, upstairs, my glass of water has long since evaporated, and the task I was doing before that is long forgotten.
Thus, when I get like that, I work from a list, and only what is on the list gets done, in the order it went on the list.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order- Ed Howdershelt Via Tass
Nah, a terminal server in this context generally means a router with a multiport serial cable (hydra or octal cables are common names) attached. They allow you to dial into one device and connect to everything else. We used to even assign IP's to the async serial port so you could simply telnet to an IP and get into the connected devices console, worked well when you used adjacent subnets =)
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.