Slashdot Mirror
San Fran Hunts For Mystery Device On City Network
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
Um, do what any network admin does with a rouge device. Search out what port its MAC address is connected to and then start tracing the cable?
I'm fairly certain most all current managed switches allow for this. Even with unmanaged ones you can hunt down which unmanaged switch it is connected to and snoop from there.
------
"And may your days be long upon the earth."
Tourists...
I'm a consultant - I convert gibberish into cash-flow.
Seriously? http://en.wikipedia.org/wiki/Radio_direction_finding
SIG: HUP
2> It's easy to find wireless devices... I've personally been doing it since the 1980's.. it's called a fox hunt here in the Chicago area. We used to get 1 minute of transmission every 5... with WiFi you can just ping the dang thing... how easy is that?
--Mike--
1) They were firing the guy, so he was no longer in the employ of the city, so his boss, was no longer his boss.
2) You don't know what you're talking about. Every IP address on the network should be known. Either through DHCP or static IP address map. A ping sweep should reveal any IP address in use, that shouldn't be. From the ping sweep, one can arp the unknown IPs to get a MAC address, and do a lookup on the Manufacturer code to know what KIND of device the MAC could be. one could use NMAP to try to discover type of device as well. Then you start going to every port on every switch with rogue IPs hanging off it, and manually looking at what is attached at the other end.
As for wireless access points, if you don't have control over them, you pull the freakin plug. Unsecured Access points and open access points should be VLANed off from administrative networked, including not allowing VPN tunnels from unsecured and open wireless access point.
If the boss allows crap like that on the network, he is an idiot, and shouldn't have the Passwords and access codes to anything.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
No no. "The City" is quite clearly "The City of London". And no where near San Francisco. (I wonder if they use Cisco hardware though, which might make the San Fran - Cisco more apt)
Will track down where any MAC address is connected. If they have the IP, they can get the MAC. If they have the MAC, they can get what port it's plugged into. Find the switch, find the cable, and air-gap it. I know this, and I'm not even a network guy.
http://weblog.infoworld.com/venezia/archives/018376.html
An insider claims that the power outage that Terry Childs was accused of using to sabotage the San Francisco network was not a planned outage.
TAGS: Problems, San Francisco's FiberWAN, Terry Childs
If you've been following the Terry Childs case to any degree, you probably know that one of the key allegations keeping him in prison on $5 million bail is that he had willfully planned to cause the network to fail during a planned power outage at the DTIS One Market Plaza Datacenter on July 19th. According to credible information I've recently received, that power outage was only going to affect the cubes and offices in that building, but not the datacenter itself.
Thus, there never was a plan to power down the network core. Thus, there's no way that Childs could have tried to engineer the failure of the network during this planned power outage, since the network core would not have lost power.
[ Follow the Terry Childs saga with InfoWorld special report: Terry Childs: Admin gone rogue. ]
The evidence supporting this claim comes from someone certainly in a position to know: Ramon Pabros, the DTIS Datacenter Supervisor himself. Pabros has been employed by San Francisco's DTIS for a surprising 41 years. He's been the Datacenter Supervisor since 1984. He's been running datacenters for the City of San Francisco since Ronald Reagan's first term, the introduction of the Macintosh, and the second season of The A-Team. It's probably safe to say that he knows what he's doing.
According to my source, he will testify to the fact that he discussed the power outage with Childs several weeks before the outage, and at least 10 days before Childs' arrest. He will also state that Childs specifically asked for confirmation that the datacenter itself would not be affected, and was reassured that it would not lose power.
With this statement, the City's allegations that Childs planned to cause the failure of the FiberWAN basically collapse.
Now, I'm admittedly a stranger to San Francisco politics, and am certainly not a lawyer, but if the DA was going to make these accusations against Childs, shouldn't they have talked to Pabros? If the OMP Datacenter was not going to lose power on that date, then this charge against Childs is essentially the same as charging someone with planning to burgle a store that doesn't exist.
But then again, this is the same DA's office that placed valid group usernames and passwords into the public record, and an IT department that ran public, unprotected websites containing internal emails, core network details, as well as usernames and passwords.
I suppose I really shouldn't be surprised at all.
UPDATE: It appears that Pabros has just announced he will be retiring, effective next Wednesday. I can't help but wonder if one event has anything to do with the other. I do know that there have been a number of odd layoffs from San Francisco's DTIS in the past two weeks.
Posted by Paul Venezia on September 8, 2008 08:48 AM
Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.
By the time his boss thought to ask for the password(s), he had already been fired. Any obligation he had to his boss had disappeared. The same goes for documentation and written procedures - I'm not going to document anything after I've been sacked. In this case the guy had been arguing for written procedures to be put in place, but no one in authority would sign them off as any failures would then be their ultimate responsibility. It should be the managers that are taking flack for this, as so often with IT cock ups.
and I do development on some software that will use RF data from your existing wireless access points to triangulate and display the physical location of every user and device on your network!
So you can call me, uh, Jerry Siegel, I guess? :| that's not as impressive...
The World Wide Web is dying. Soon, we shall have only the Internet.
No joke. You'll pay $300+/hr for a top guy from a place like IBM Global Services or HP's technical consulting group.
When information is power, privacy is freedom.
Because it's rouge!
An EMP disrupts electronics by inducing massive currents in the thin circuitry of the circuit boards and integrated chips. They're permanently burned. They won't power-cycle, they'll just fry.
Naw... if you really want to power-cycle it, just disrupt the electrical service to the entire city. You'd probably have to leave it off for a fair length of time, though, in case the device was on UPS.
Alexander Peter Kristopeit bought his basement from his mommy for one dollar.
regarding point 2:
Trivialy easy to determine if it's a wireless device. TURN OFF THE WIRELESS NETWORK. If the device goes away, it's wireless. Then simply change the security configuration on the network, and problem solved. The offending device is no longer on the network, and its physical location is irrelevant.
Elsewise, if it doesn't go away, it's a wired device, and normal network investigation should work just fine.
"Oh my God. This is terrible. This is the end of my Presidency. I'm fucked."; ~ Donald J. Trump
Sorry to commit the solesism of replying to myself, but I (gasp!) just read TFA.
...being held in a jail cell on $5 million bond, also happens to be a former felon convicted of aggravated robbery and burglary stemming from charges over two decades ago, which the city knew when it hired him as a city computer engineer.
Childs, who has worked for the city for five years but faced firing for alleged poor performance...
Illuminating, but mostly in that it shows all parties in a very dim kind of light. Under the circumstances, I would have hesitated to employ the guy in this capacity anyway...
Well, if it's on a managed network then the IP needs to be mapped to a MAC address (and port on a switch) and the port turned off. Once that's done, tracing the cable to a physical port should take no more than a day.
If it's on an unmanaged switch things get a little more annoying, but you should still be able to track an IP and MAC address to the switch using any open source network tool like WireShark. Find the switch. Pull the cable out of the port, or if you're feeling really adventurous you could bring a replacement switch in and start playing "Is that it?" until you find the bugger.
Must be a slow day for something this ho-hum to make it on /.
Often times an account such as Unix root or Windows Administrator will have a randomly generated password that's sealed in an envelope. Envelope is locked in a box, with some kind of anti-tamper on the envelope... all this is usually under multiple control. Nobody uses the account unless shit + fan. Admins then have their own equivalent access level accounts.
SIG: HUP
Who modded this insightful? Part of the reason he was getting canned was because he was PUSHING for the sort of documentation and recovery plans you're snarling about. None of the PHBs wanted to put their names on it because if they came up short, it would be their asses on it.
No, city property belongs to the city and not to the admin. Failing to allow the city access to or control of its own property is a crime known as "conversion." This guy should be thinking about his personality disorders between prison showers.
Sorry the link doesnt work? here: Server 54 Story
Go go Gadget Nailgun!
UNC in 2001
Best Slashdot Co
I used this once to track down which server room a system was located in and while it's not perfect for all occasions, it might help.
Ok, first if you can get an IP for the device, perform a traceroute from 3 or 4 separate sites. Identify it's Gateway if possible, also if find see if you can determine from the traceroutes if it has a common parent node that it's traffic is going through.
Once you've found the most common system talking to it, go to that system and perform ping tests to other systems where you know their physical location in proximity to the system your at, and are only 1 hop away (if possible). The key here is to make sure that all of your samples share as much of the same route as possible to minimize signal noise in your data set you're going to build.
See if you can develop a correlation between ping times and amount of network cable to your sample set. Compare that data to the ping times on your mystery device and you *potentially* have a physical range now in hand to perform your search.
I'll be the first to admit that this approach has limited success based on how your infrastructure is built, but it might help.
Not really, a terminal server could easily have a modem on one end and a bunch of serial cables on the other, not at all an uncommon setup.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Which, considering the rest of the FUD around this case, doesn't surprise me.
It shouldn't take any competent netadm more than 5 minutes to track down any device to a specific port on a switch. There are no logs to look at. What do you think is logged that you'd want to look at to track it? Seriously, it's incredibly simple to do. The thing has an IP and for that IP to be useable on the network it must be in the RIB (read: route table). With less than a minute's work a netadm should be able to track down that route to the router that's originating it. I don't care how big your network it. It should take less than a minute. Once you've found the router originating the route you've almost certainly found the router with an L3 interface in the same broadcast domain as the target device (the router could also be redistributing a static route in which case the static route would point you to the device in possession of said prefix, or a trail of bread crumbs of multiple static routes that will eventually lead you to the device using that prefix). If the router is part of that broadcast domain then it will have an entry in the ARP table for the target IP and will give you the device's MAC. From that router's config the netadm can determine where all that broadcast domain is accessible. Ie, what L2 switches downstream of the router have that VLAN on it. The netadm can examine the CAM table (SAT in Cabletron-speak, bridge forward table in generic terms) to figure out which interface the target's MAC is associated with. That will point him to the correct downstream switch. The netadm will do the same thing on that switch to track the target device their the broadcast domain until he find the one access interface that the target device is connected to. Once he finds that interface he visits that wiring closet and tracks the cable down manually to the target device.
Really, it's much easier than it sounds. Once you've done this once or twice it will become second nature. This should not take a competent netadm more than 5 minutes. I don't care how big the network is. This isn't rocket science. The City of San Francisco is just trying to make their case sound worse than it really is. It would take a truly incompetent IT department to not be able to find that device. I would say that it was impossible to be that incompetent but I'm sure someone would try to prove me wrong.