Slashdot Mirror
San Fran Hunts For Mystery Device On City Network
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
Power cycle it with a city-wide EMP.
Life would be easier if I had the source code.
From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.
Given that they cannot hunt down a single device on the network, I'd have to agree with that assessment.
MAC address ... switch port ... it should be easy.
Um, do what any network admin does with a rouge device. Search out what port its MAC address is connected to and then start tracing the cable?
I'm fairly certain most all current managed switches allow for this. Even with unmanaged ones you can hunt down which unmanaged switch it is connected to and snoop from there.
------
"And may your days be long upon the earth."
The guy costed the city one million?
How much does it cost for San Fran to have an incredibly stupid IT manager that cannot keep his best talent on the job?
Fuck that: im with the rogue guy.
NO SIG
I'd suggest using traceroute if they know the IP address.
Show me packet captures and log entires, or it never happened.
<erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
if the answer isn't violence, neither is your silence / freedom of expression doesn't make it alright
Hey! Fyodor! They need your number!
Fyodor spent much of this summer scanning tens of millions of IPs on the Internet (plus collecting data contributed by some enterprises) to determine the most commonly open ports. Nmap now uses that empirical data to scan more effectively.
Zenmap Topology and Aggregation features were added, as discussed in the next news item.
Hundreds of OS detection signatures were added, bringing the total to 1,503.
Seven new Nmap Scripting Engine (NSE) scripts were added. These automate routing AS number lookups, "Kaminsky" DNS bug vulnerability checking, brute force POP3 authentication cracking, SNMP querying and brute forcing, and whois lookups against target IP space. Many valuable libraries were added as well.
Many performance improvements and bug fixes were implemented. In particular, Nmap now works again on Windows 2000.
With just nmap, my old buddies at Farm9 could have sussed this out in a few hours. I think they are still around - as Red Siren / Getronics.
Ahh. I miss running netcat at 3 AM!
"Flyin' in just a sweet place,
Never been known to fail..."
check the reservoir, they like to mess up water supplies
Let Cyber punk rule!
Come the revolution, the Bourgeois, Capitalistic, "A PARKING STICKER HOLDERS", will be first against the wall!
Tourists...
I'm a consultant - I convert gibberish into cash-flow.
Man, the more I read about this story, the more inclined I am to believe the network admin.
He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.
As Indy deciphered the symbols, he found the correct sequence of tiles to push. The huge stone door slowly opened. Indy grabbed a torch and headed inside. At the end of the long room, there it was on the throne: A massive server. It was archaic, and it appeared to be attached to a punch card reader. Along the sides of the room, there were two rows statutes of archers pointed at the center. Indy made his way slowly to the monitor and keyboard of the server. He brushed away the dust and hit the spacebar. The screen turned on slowly and it displayed:
SCO Server 1.0
Your license has expired. You owe use $699.
>_
Suddenly the archers rotated positions and were aimed at Indy.
"Oh boy."
Well, there's spam egg sausage and spam, that's not got much spam in it.
When I first heard what the rogue-SF-admin had done, I was very negative on his actions.
Now, that once again, and now at least for the third time, I hear of absolute stupidity and ineptness on the group at sf, I am certain the so called rogue was right on the ball from the beginning.
I recall hearing a story about a Sun Sparcstation 2 at my old college that had accidentilly got sealed inside a wall by construction folks when re-working the building the CS lab was in to eliminate a few closets for structural support reasons.. nobody could find it (shock!), but kept using it as a DNS server for another six years. It was found about 2 years after it stopped responding to ping when some component (nvram?) let out, and it started beeping after a power flicker.
Hey, at least they didn't say "Frisco".
2> It's easy to find wireless devices... I've personally been doing it since the 1980's.. it's called a fox hunt here in the Chicago area. We used to get 1 minute of transmission every 5... with WiFi you can just ping the dang thing... how easy is that?
--Mike--
You think they've learned anything about the gear since then? No wonder they're having problems.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
Why is Slashdot linking to stories that paint the network administrator as a bad guy when he's so obviously surrounded by morons? These are the same people who published all of their user names and passwords. That puts the cost of this "hijacking" into perspective. The cost of trusting their employee with the powers required to do the job was zero.
Friends don't help friends install M$ junk.
If the city can't even complete one of the most basic network administration tasks of finding a physical device on a network, I think they have absolutely no right to accuse anyone of "hijacking" their network. I hope the defense attorney for Terry Childs brings this up.
1) They were firing the guy, so he was no longer in the employ of the city, so his boss, was no longer his boss.
2) You don't know what you're talking about. Every IP address on the network should be known. Either through DHCP or static IP address map. A ping sweep should reveal any IP address in use, that shouldn't be. From the ping sweep, one can arp the unknown IPs to get a MAC address, and do a lookup on the Manufacturer code to know what KIND of device the MAC could be. one could use NMAP to try to discover type of device as well. Then you start going to every port on every switch with rogue IPs hanging off it, and manually looking at what is attached at the other end.
As for wireless access points, if you don't have control over them, you pull the freakin plug. Unsecured Access points and open access points should be VLANed off from administrative networked, including not allowing VPN tunnels from unsecured and open wireless access point.
If the boss allows crap like that on the network, he is an idiot, and shouldn't have the Passwords and access codes to anything.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
your employer's passwords are NOT yours, no matter how stupid you think your boss is.
Refusing to give out passwords to higher-ups is not always the wrong thing to do. If you are the network admin, and your job is to maintain security of the network, wouldn't it be reasonable to refuse to hand out passwords to people outside of the network administration roles?
Although I can say that an admin can make that choice at his or her own peril. After all, the higher-ups can always opt to fire the admin and replace him or her with someone who is willing to seek security of their job over security of the network they are paid to administer.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
There's only one "The City" that matters on 9/11. - Rudy G.
No no. "The City" is quite clearly "The City of London". And no where near San Francisco. (I wonder if they use Cisco hardware though, which might make the San Fran - Cisco more apt)
Did they try the Rouge Admin's office. It's probably that beige box under his desk... Either that or he made up the device and it does not exist, he's laughing at them ripping the place apart trying to find it :D
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
I'd like to add that while the way he handled being surrounded by idiots was wrong, he was clearly surrounded by idiots.
No documentation?
No change control?
No diagrams?
What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.
I worked one place where a guy with a great deal of responsibility died. (here today dead tomorrow kind of thing) His peers blamed *everything* on him simply because they could. This sounds like the same thing.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
As usual, our modern government continues to bungle their day to day operations with complete ineptitude.
The only reason this is getting any attention is because the city of San Francisco chose to make the initial debacle a very public affair, and now people are watching.
I went to eat some animal crackers and the box said, "Do not eat if seal is broken." I opened the box and sure enough..
No no. "The City" is quite clearly "The City of London". And no where near San Francisco. (I wonder if they use Cisco hardware though, which might make the San Fran - Cisco more apt)
Huh? London is only about 142 miles SE from San Francisco and with a population of about 2000 people barely qualifies as a city, let alone "The City" moniker.
I'm a consultant - I convert gibberish into cash-flow.
Excellent Marx Brothers reference. Today is going to be a good day.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
San Fransisco is not "The City" to anyone not living in the area in any other way then the residents of Oklahoma consider Oklahoma City "The City".
If a device pings on a network and no one is there to see it, does it exist? (bring on the tree in the forest metaphors!)
If you find that you are "holding the place together", IT-wise, you are likely part of the co-dependency and are part of the problem.
IT and the other management have both agreed to ignore each other, literally or otherwise, allowing each (and the individual personalities) to do things "their way"; damn the best practices, good management, logical, financial, or even legal issues.
Except when things go wrong.
Like a breakup, they can get ugly. And, as the IT guy, you will always lose for it is not your Business, but theirs. You are simply hired help.
Your London may be inferior. Ours definitely warrants a 'City' moniker. Especially when The City of London is distinct from the conurbation that is known as London. And the City of London is actually fairly small - almost exactly a square mile - but ... well, you know what they say. It's not the size, it's how you use it.
Will track down where any MAC address is connected. If they have the IP, they can get the MAC. If they have the MAC, they can get what port it's plugged into. Find the switch, find the cable, and air-gap it. I know this, and I'm not even a network guy.
This is just more evidence that the Government of San Francisco is full of a bunch of Morons.
http://weblog.infoworld.com/venezia/archives/018376.html
An insider claims that the power outage that Terry Childs was accused of using to sabotage the San Francisco network was not a planned outage.
TAGS: Problems, San Francisco's FiberWAN, Terry Childs
If you've been following the Terry Childs case to any degree, you probably know that one of the key allegations keeping him in prison on $5 million bail is that he had willfully planned to cause the network to fail during a planned power outage at the DTIS One Market Plaza Datacenter on July 19th. According to credible information I've recently received, that power outage was only going to affect the cubes and offices in that building, but not the datacenter itself.
Thus, there never was a plan to power down the network core. Thus, there's no way that Childs could have tried to engineer the failure of the network during this planned power outage, since the network core would not have lost power.
[ Follow the Terry Childs saga with InfoWorld special report: Terry Childs: Admin gone rogue. ]
The evidence supporting this claim comes from someone certainly in a position to know: Ramon Pabros, the DTIS Datacenter Supervisor himself. Pabros has been employed by San Francisco's DTIS for a surprising 41 years. He's been the Datacenter Supervisor since 1984. He's been running datacenters for the City of San Francisco since Ronald Reagan's first term, the introduction of the Macintosh, and the second season of The A-Team. It's probably safe to say that he knows what he's doing.
According to my source, he will testify to the fact that he discussed the power outage with Childs several weeks before the outage, and at least 10 days before Childs' arrest. He will also state that Childs specifically asked for confirmation that the datacenter itself would not be affected, and was reassured that it would not lose power.
With this statement, the City's allegations that Childs planned to cause the failure of the FiberWAN basically collapse.
Now, I'm admittedly a stranger to San Francisco politics, and am certainly not a lawyer, but if the DA was going to make these accusations against Childs, shouldn't they have talked to Pabros? If the OMP Datacenter was not going to lose power on that date, then this charge against Childs is essentially the same as charging someone with planning to burgle a store that doesn't exist.
But then again, this is the same DA's office that placed valid group usernames and passwords into the public record, and an IT department that ran public, unprotected websites containing internal emails, core network details, as well as usernames and passwords.
I suppose I really shouldn't be surprised at all.
UPDATE: It appears that Pabros has just announced he will be retiring, effective next Wednesday. I can't help but wonder if one event has anything to do with the other. I do know that there have been a number of odd layoffs from San Francisco's DTIS in the past two weeks.
Posted by Paul Venezia on September 8, 2008 08:48 AM
There are now dozens of cars packed full of cheetos cheap laptops and foul smelling individuals travelling near, or perhaps at the speed limit, towards san francisco. They're full of people thinking the same thing, "Shit if they can't find a wired device, they sure as hell can't find a wireless one!"
All they have to do is look for the small black box with a lone, onerous blinking red LED.
I find it difficult to understand how a blinking red LED would constitute a heavy burden.
What would you think of a doctor who, because some exec somewhere decided he should, pushed the WRONG medication / procedure to you?
Where does your ethical responsibility end and the boss's desires begin?
To me there isn't even a question. Fire me. Go ahead. I will get another job.
listening out for a ticking clock, sure fire way to find things.
Username: root
Password: admin
Either that, or just go to 192.168.1.1 and do a reset to defaults. dd-wrt is your friend! :-)
This post brought to you by your friendly neighborhood MBA.
RedSeal is at: www.redseal.net
Great scanning/tracking tool for network infrastructure inventory control. Automated management of what lies below layer 3 is always a challenge, though.
http://tinyurl.com/4ny52
City IT department run by clueless morons. City shocked. Film at Eleven.
Didn't everybody within a hundred kilometer radius of The City know this long ago?
I can see it now, the mythical nethack terminus of San Fran, with it is the power to control the settings for the city...if you can find it.
-Ours is the wisdom of Solomon, the magic of Merlyn, the fall of Icaris.
A: When you have engine trouble at 35,000 feet, you start emptying your bank account
Routine operations takes little skill. It is the ability to revolve a crises that distinguishes an excellent employee from a mediocre one.
I wonder if it has a bomb attached.
Putting a bomb in a city is nothing new but with a dead man switch you could get a lot of freedom.
"Hey I have a pretty big bomb in a city. Leave me the fk alone or I'll stop telling it not to go off!" Set a bunch of keys based on some quasi random human algorithm (Girls I slept with encrypted... people I secretly hate by date of birth etc.) and then patch it through random web servers to stop the kill switch and you're home free.
Interesting how little control we have over the internet is all I'm saying.
In fact, you just proved you are smarter than all of these guys.
Oh, sorry, that wasn't much of a compliment, was it?
Man, only in San Francisco. We just... we just do things our own way here. Honestly, it's probably just under some homeless dude who's using it as a bed because it's warm.
There's a limit to that though, and that usually is where time becomes money.
Sure, you could wait days to have the stuck printer fixed, or various other small things.
But when something major comes up that stops the cash flow, that's usually when people start thinking about the importance these things, which is usually too late.
When a sysadmin is good, he's often not noticed. Mainly because there's a lack of screwups to draw attention to him. It's when there's somebody to take blame for IT-disasters that people really go looking for him.
Who is actually the OWNER of the system? The boss? Isn't he employed by the same company as the sysadmin? Don't they both have an obligation to safeguard the OWNER'S property and interests? If the sysadmin refuses to hand over the password to sensitive equipment & systems to a (perceived) inept superior-- as long as that guy DOESN'T own the company-- isn't he actually performing his responsibility to the real owner? Which in this case would be the city, and the personification of the city would be the mayor-- and that's exactly who he DID give the passwords to. So it seems to me like he did precisely what he was supposed to do in terms of safeguarding the network and sensitive equipment. Of course he should probably be then fired for failing to keep backups, conops, continuity planning, etc. But that's a different matter.
Use nmap to scan the city's IP block for the port that responds to "terminal server" protocols.
It wasn't a server, but we did have some maintenance guys once stick a new wall over a bunch of switches and important patch panels. We were rather pissed when we found out, they were rather pissed when we tore down their wall (luckily with the permission of management, which was pissed at the maintenance guys and not us).
In soviet Russia mystery devices hunt for you!
The thing is, it may have an IP address, but the purpose of such a thing is mainly to connect to other devices. If they know neither IPA nor hostname and only know to search for ``a device'' that may or may not exist, well... maybe they're looking for the wrong thing. Think radius/tacacs+. Or rely on the privateness of the council's internal telephone notwork: Most Real Routing Hardware is or can be equipped with a modem. Not too difficult to make your PBX ``forget'' about those numbers, ie only allow them to be used from certain other numbers.
Doesn't change that I too think management over there is horribly incompetent, and have no pity for them now that they get some of what they deserve.
and changed the MAC address to C0:FF:EE:C0:FF:EE
or
FE:ED:C0:ED:BA:BE ...
Just saying
No, he doesn't mean your the city. He means, "My, The City."
"Ah The City. My, The City." - The Tick
<erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
Get your own free personal location tracker
I went to a boarding school in Kenya for high school. The system of bells ran across the campus of several hundred acres and many buildings in a closed loop, with all the bells in series. The system ran through the main office, with the Super Secure Bell System locked in a cabinet there so nobody could access it. Penalty for messing with the system of bells was said to be expulsion.
The problem was, that all you had to do to get all the bells on campus to ring was to wire the loop back into the mains.
We took a clock from the darkroom in the photo lab, and ran two wires through the face plate. We then ran another strip of wire along the minute hand, so whenever the minute hand swept by a certain point on the clock every hour, it would complete the circuit for about 30 seconds and ring every bell on campus.
We then hid this contraption under a pile of wood in the attic of the wood shop. Right after convocation when I could no longer be expelled, I ran into the building and turned it on.
Apparently the bells rang off and on mysteriously for most of the next month of holiday until they managed to follow the loop and find the device. Good times.
www.clarke.ca
It sounds like the city hired someone that actually knew what they're doing. This is an obvious mistake, it's usually required that you have to be related to someone and totally incompetent to get a city job. I was a volunteer at Interop Las Vegas 08 and the day before the show went live the ticket desk manager had myself and another volunteer go out and hide Fluke Etherscopes in vendor booths. We'd plug it in, radio in the IP and generate a little bit of traffic with a ping. Within 2-3 minutes they'd call us back and tell us which booth we were in.
This urban legend will be updated were a network admin is found sealed in the wall.
It could even be a Honeypot...
If it's hidden in the raised floor it probably wouldn't look good for this guy -- but let's be serious here -- it's a terminal server. There are many, many legitimate uses for them.
... although I have no idea why he has 'personal property of' banners on it. If the management really was as incompetent as he says they were, maybe he got fed up, and shelled out his own cash for a Cyclades/Xyplex/Digi or whatever other brand box. (and I don't care that Xyplex is now MRV-- they still use Xyplex in product names)
Build it, and they will come^Hplain.
hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
It appears that the idiot "boss" is attempting to generate support for the claim that this guy is a "problem" by paying unreasonable amounts to "repair" the "damage" he did.
It's difficult to "prove" that a guy did millions of dollars of "damage" ... without a bill for millions of dollars of "repairs".
Any competent network admin could map out the network and document it for FAR less than the hundreds of thousands of dollars that is being thrown about.
They will never be able to find it, cuz it isn't on at all times, only when it is being used for admin stuff. Track the guy who set it up for a year, then recap all ip addresses to hops and tracert...then you might get lucky...other then that, it might be off at the moment...
Hissssss
Do Or Do Not, There Is No Spoon, There Is Only Zuul. Everything in the above post is probably opinion.
I'm out of mod points at the moment and this needs a +1 Informative at least.
Knowing that the boss was badgering the guy after he was fired changes the story 100%.
Honestly, this sounds extremely fishy to me. If the remaining people there are unable to find such a device, it's safe to say they are at least adequately incompetent. I think it's more likely that they have a much bigger problem on their hands that they have not yet revealed, and have perhaps introduced this "rogue device" to help divert attention from it, or to allow them to put the blame elsewhere when it is discovered.
I know it sounds paranoid, but this story is just a bit far-fetched, taken at face value. An alternate explanation might make more sense.
When users ask for Admin privilages, they should be told to go fsck themselves. No matter who they are.
I'm a software developer. For the first few weeks working here IT wouldn't give me admin rights on my own box. I couldn't install software.
So I sat here and did nothing. Not because that's what I wanted. But because that's all I could do, until they gave me permissions on my machine.
Generally speaking, you're right. Most people in a business should be locked down. But not everyone. Depends on the person - depends on the work they're doing.
Weaselmancer
rediculous.
Sorry to commit the solesism of replying to myself, but I (gasp!) just read TFA.
...being held in a jail cell on $5 million bond, also happens to be a former felon convicted of aggravated robbery and burglary stemming from charges over two decades ago, which the city knew when it hired him as a city computer engineer.
Childs, who has worked for the city for five years but faced firing for alleged poor performance...
Illuminating, but mostly in that it shows all parties in a very dim kind of light. Under the circumstances, I would have hesitated to employ the guy in this capacity anyway...
Is it really that hard to add a blinking LED?
Clear, Dark Skies
Seems to me that attempting to locate 1 network device on a network of that size would be a pretty difficult task. Kind of like trying to find a needle in a hay stack. I would that one that could be done is temporarily decrease the size of that network to only the pc's and devices that are absolutly needed. In other words manually power off the unneeded devices & PC's. That should make the haystack a bit smaller. If all of sudden the black box goes silent then the device might be a virtual machine. Bring up the devices one by one until you find your black box. This guy probably put some thought in to it and would expect the device to looked for using tools like tracert, etc..
I can already sense it coming another SOX compliance type initiative but geared towards IT for accountability, documenting, etc... all written up by people not in the IT industry.
Excessive checks and balances for the change management and auditors for the internal auditors.
aichee...
"All they have to do is look for the small black box with a lone, onerous blinking red LED."
Not to be a grammar/word-choice "Nazi", but I think you meant "ominous".
But, after all this time, one might expect that the NSA would have been on top of this. Anytime a city government fails to locate rogue devices that could compromise local/state/federal/international investigations, the criminals and the undercover agents/officers, and witnesses, as well as payroll and other HR information, the FBI, NSA, and other agencies should take over that aspect where the locals prove incompetent.
Previously: "Linux... Toward the Sunrise..." Now: "Linux... Toward the-- No, now, part of Every Sunrise"
In a report filed before the city disclosed the hidden router, a court-appointed expert witness for the defense wrote that DTIS could easily prevent Childs from accessing the networks. "I have seen no evidence that Mr. Childs is a 'computer hacker,' and by taking a number of simple steps, DTIS could block access by Mr. Childs to San Francisco networks," wrote Doug Tygar, a University of California, Berkeley computer science professor.
In other words, a vindictive city is looking for excuses to keep Childs in prison.
Clear, Dark Skies
Why dont they just port scan the whole network for a machine that is replying on RDP port maybe? That should at least tell them what subnet it is on find the router then trace it to the switch then trace it to the computer ...
Being a bully is not the same as being competent
It might take some time, but with managed switches and SNMP it is possible to pull the bridge tables off of each switch as well as ip to mac correlation from the routers. The switches/routers know where the device is or you would never see it on the network. You just have to know how to make the switches tell you. Even if it's on a virtual machine you will know what switchports are forwarding for what mac addresses which will narrow down the search quite a bit.
/* Insert some overused slashdot quote here */
In a big network I could see this happening. I know--computer rooms are supposed to be pristine with every wire perfectly aligned and in place with everything perfectly labeled and mapped--NOT! Most computer rooms I've been in, including my own, are somewhat less than ideal. They kind of grew with no plan. Need more space? Run a jumper. One of the Field Engineers who worked on one of our minis just laughed and said we weren't really that bad--you should see banks--they're the worst. In other words, poor housekeeping is widespread and tolerated. A typical terminal server could be 1RU or even a blade, or a box sitting loose on top of the rack where you can't see it. If I were really devious I would put a small terminal server in a bigger box. If this were intentionally hidden it could be in the ceiling hooked to a 128 port hub in the rafters itself and you'd never even know it. It's a bird's nest of Cat5 around a hub, all looking the same. I'll just bet it's a Class B network, so you've got a tremendous number of possibilities. And if you used virtual networks on Cisco hubs or did some bizarre subnets that simply confounds matters. I feel very confident that I could hide a box in my building that even the pros would have a hard time finding. Of course you could start turning off power until the device disappeared to try to pin down its location, but my guess is no one wants to do that just because someone lost a box. Too funny.
How about a moderation of -1 pedantic.
Is it possible this device was built with a MAC/translation table to monitor devices, and -- if it heuristically senses it is being hunted down --temporarily change to their MAC, and hide, probably by instructing other devices to propagate a false MAC?
Isn't this technically possible, to create virtual NICs and MACs that change on sniffing detection?
Previously: "Linux... Toward the Sunrise..." Now: "Linux... Toward the-- No, now, part of Every Sunrise"
Thanks for the heads up. I'm scanning the subnet right now for port 31337
they start out as network admins making dumb comments on slashdot about complex things that they don't understand.
$10 says it is a game console $20 says it's a dreamcast
Well, if it's on a managed network then the IP needs to be mapped to a MAC address (and port on a switch) and the port turned off. Once that's done, tracing the cable to a physical port should take no more than a day.
If it's on an unmanaged switch things get a little more annoying, but you should still be able to track an IP and MAC address to the switch using any open source network tool like WireShark. Find the switch. Pull the cable out of the port, or if you're feeling really adventurous you could bring a replacement switch in and start playing "Is that it?" until you find the bugger.
Must be a slow day for something this ho-hum to make it on /.
... nobody calls it "San Fran" ...
Triangulate the position based on the ping from a trace-route.
Even if ping or trace route is disabled they should still be able to get a latency based on what they know about the device.
Basically, if they know it exists, then it must be sending out some sort of keep-alive or something. So they would have a latency. Which you can use to find it's approximate location. Or at LEAST which LAN it's on.
This is kids stuff. People in SF are really stupid. Believe me, I live too close to them.
You mean Salt Lake City.
OMFG. If the city IT staff cannot find a terminal server on their own network, the entire lot of them should be fired on the spot. I still cannot get over the fact that they would put "terminal server" in quotes and call it mysterious. Just wait until they discover the inscrutable "virtual machine" on which the mysterious "terminal server" is running. I bet it's using some impenetrable "TCP protocol" over an unfathomable "RF medium" and running incomprehensible "network services".
Maybe they should call in Agent Mulder.
Note: Yes, I know "TCP protocol" is redundant, and that is why it is one of my favorite "managerisms".
The "hero" is the guy who rushes in in the middle of the day to fix the "problem" that is costing the company so much money.
Never mind that he is the one who caused the problem in the first place.
The ninja does the upgrades during the night/weekends and the users never see any difference.
Now, which admin is seen as more valuable?
The lesson is that even ninjas need to market themselves to their organizations.
If they can ping the device or attempt a log-in, it should not take a competent
operator more than, say, an hour, plus driving time if there are different
locations, to find this thing. Of course that assumes a map of the network
is available. Seems to me this network is run by complete morons.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I'm reminded of a conversation I had some 25 years ago with a co-worker IBM mainframe technician. IBM management was incensed that uneducated morons turning screwdrivers could make 70k a year. Back then as much as what they were paying top MBA stuff shirt types. They were on a mission to get salary levels down to "reality" paying these screwdriver wielding monkeys what they were (in their minds) really worth.
Attitudes have changed but not a lot. 93% of companies that loose their data center for 10 days or more due to a disaster filed for bankruptcy within one year. 50% filed bankruptcy immediately (National Archives & Records Administration in Washington). One can't say the same thing about those over paid MBAs.
It may be awhile before IT matures into a "profession" like doctor or lawyer however I personally believe we're holding the keys. The world can't function now without us.
-[d]-
As in, "This begs the question, did a rouge sysasdmin plant virii that caused us to loose control of our boxen?"
http://www.qdb.us/5273
A City of snakes, apparently.
We did that at school...
(Gotta love Eddie Izzard)
already ? you know, the one who disclosed 200+ passwords and usernames to court as 'evidence', totally proving validity of sysadmin's thesis.
how can MORONS like that, and i really, literally mean, MORON, can be allowed to work in public service DESPITE the sheer ignorance and lack of capacity they are displaying ? noone sues these pieces of shit ?
Read radical news here
It shouldn't be that difficult to find a piece of h/w on a network.
Interrogate the switches to find the IP/MAC address corresponding to the device you are trying to log on to. In the event that this Childs guy is deviously smart (i.e. patched the switch software to conceal a particular device) one can still use a stand-alone sniffer to trace packets through a system.
Its possible that the 'terminal server' might be virtual, just an app. running on some other piece of hardware that doesn't necessarily have "ACME Terminal Server" and a wining LED on the front. But tracing the network to that particular box isn't difficult (maybe time consuming).
If these people are really that dumb, I can understand why Childs kept them off the system. Reading some of the stories about him, it wouldn't surprise me if he left a bunch of 'dead ends', like phony terminal servers that nobody could find. Or wireless access points not plugged into anything but plastered inside a wall to drive security auditors nuts.
Have gnu, will travel.
The novell server walled up in the closet for years is well documented. http://www.techweb.com/wire/story/TWB20010409S0012 for example.
It was a Novell version 3.11 Netware box. If it'd been VMS it would never have been found, since it would have always worked perfectly ;) .
That's no Mystery Device. Itâ€(TM)s a space station.
The login prompt tells them some device exists, but supposedly they can't find it physically or by the network?
Why start by looking for it physically? The article doesn't say if it's a wired or wireless device, but an even partially wireless system could be very hard to find physically. All it takes is tucking it away in an older building with lots of odd niches and cubbys - something I'm fairly confident San Fran's government has plenty of.
The prompt claims that the device is the administrator's. Why not look at the financial records first, and see if the city bought this device or the administrator presumably did?
Why not study the prompt to see how much info about the device it gives? Knowing that the prompt shows it to be some sort of router isn't much. Most devices, and certainly Cisco's, have changes in the prompt appearance with different models. The city should be able to figure out what make and model they are looking for, and related factors, such as wireless range or number of connections.
For wireless, figure out what devices are talking to it, and which ones are out of range to connect directly. If they know where the rest of the systems are, something like old fashioned triangulation should do wonders.
Who is John Cabal?
Hidden device is hidden.
malicious, but im on the side of the ex employee. if the device is his, i hope he uses it. it seems like theres alot of incompetence and coverup going on at the sanfran city network. this story has stunk since they decided to imprison him for not giving the password. "unknown wireless device" just further confirms theres a good chance nobody know what the hell theyre doing, and this guy could have been right.
Good people go to bed earlier.
This is just the old "Linux sucks because it can't X" gambit! Rather than pay for expensive contractors, these guys are pocketing the money, making news of this so it ends up on /. and grepping..er... Ctrl-Fing the post for "Dumbasses. They should just do...." to find the answers
You're all suckers!!!!
Did they look in their shoes? Sometimes when I can't find things they are in my shoes.
It might be a stretch, but wonder if its setting on his desk.
These guys dont want to look, they want to create cost and confusion. Lawyers will not look, they just file papers, to force payments for somebody to look.
Possible but unlikely. How is it going to know that someone is investigating? The investigation (ie, packet capture) is not going to tell the rouge device that someone is watching.
You are being MICROattacked, from various angles, in a SOFT manner.
only it was an IBM main frame not a sun box.
A continent away, logged into every other computer, instantly, invisibly, wirelessly. Don't you watch movies?
yes | awk '{system(sprintf("ifconfig eth0 hw ether %02x:%02x:%02x:%02x:%02x:%02x; sleep 1", int(255 * rand()), int(255 * rand()), int(255 * rand()), int(255 * rand()), int(255 * rand()), int(255 * rand())))}'
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
Cheeky: What's the password ... ...
Dildo: Yeah, that's what I said, what's the password
Bobo: Why are you asking me, I'm asking you
Cheeky: You're asking who?
Dildo: Hu doesn't know
Hilarity ensues??
Network Admin, you are technically correct, the best kind of correct.
They should turn off everything but the main server. The mystery device will then be the only thing running. Then maybe send an ascii "beep" code to it and listen for beeping (if employees can be stopped from flapping their gums for five minutes). Best to do this on a Friday (or Mon-Thur) when no govt. employees are working.
Basic network security is to disable undocumented ports on your switches in order to prevent people from attaching rogue devices to your network. Nevermind that any good network administrator should be able to track down pretty much device using ARP tables in order to disable the port in question while they look for the device.
If security is important, and they are using non-managed switches? They are frickin idiots.
Exactly right....
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
They are probably looking for a "real" server since by all means it may be a terminal server.
They should be looking for a box thats likely rack mounted, has many serial cables coming out of it that go to the console port of every network device in the server room.
Probably one of these:
http://www.picotux.com./ Good luck to them. :-)
Paul Venezia digs a little deeper into this so-called "terminal server" today in his blog:
"From what I can see, it's a device running Cisco IOS that was accessed via telnet. I could generate an identical screenshot to the one entered into evidence in about five minutes using an elderly Cisco 2924-XL Ethernet switch -- a device that's certainly not a terminal server. It's completely unclear to me how they could have possibly come to the conclusion that this is a "terminal server" -- the evidence presented to the court certainly does not support that theory."
Venezia also uncovers additional technical errors in the prosecution's case, which appears to be unraveling with the recent news that the DTIS Datacenter Supervisor Ramon Pabros will testify on Childs' behalf. Since coming forward, Pabros has announced he will be retiring from the DTIS, effective Sept. 17. Coincidence?
We have just gotten a wake-up call from the Nintendo Generation.
The only thing new in this world is the history that you don't know.[Harry Truman]
so a bunch of management idiots get fired or moved upstairs
SF loses a million which is rounding error for a city that size
a total asshole gets bad free food and prostate massage for seven years - maybe even gets to talk linux file systems with a fellow punk
and eventually the mystery server story gets posted again on slashdot
How do you know it's red?
His actions were extremely stupid, but I fail to see why this idiot's relatively non-disruptive actions rise to the level of criminal prosecution.
One thing I wonder about though, knowing government (especially higher levels of government with deep pockets), is the whole scenario that led up to this. What if you knew your boss was doing something wrong (say, illegal), and that you were likely going to be the one blamed for it after being canned. What if by giving up access you'd give them plenty of ways to nail you with blame - and a lawsuit - after the fact?
I really wonder what the whole story is here. Certainly if it hadn't been made into such a debacle then nobody would have heard about it, so maybe that's the point of it all.
when they should be looking for a device with several serial cables coming out of it and going directly into their network devices console ports.
Sometimes you inherit the fires. Oftimes they may be created by other people, and frankly, without enough co-operation by management (either dealing with consistent firestarters or by hiring supporting staff), you cannot make yourself redundant.
There's only so much time in the day for a given person to do a given set of tasks.
Hilarity ensues??
Hell yes, I lol'd my ass off. Parent needs a good modding up.
Well, don't some malicous/sophisticated virus and Trojan code have the ability to know they're being hunted? If the rogue router has a packet sniffer, it only needs to heuristically determine it's being hunted. If it has a map of all the known devices (before and after it itself was planted in the network), it can listen for addresses being culled by some number of devices, how many polls are in play, and then manipulate the detected collector. If the collector/vacuum/detector device is immune to the rogue router, then the router can be commanded (in advance or remotely if there is a remote player involved) to self-destruct, or to wreak havoc on the ferret device, or wreak last-kiss-goodbye/kiss-of-death havoc on the LAN and tributary sites...
OTOH, maybe my imagination is too wild?
Previously: "Linux... Toward the Sunrise..." Now: "Linux... Toward the-- No, now, part of Every Sunrise"
Don't they have a network map? If its got an IP, and they know it ( which is implied here ) they can narrow it down to a physical location and work from there floor by floor, room by room.
---- Booth was a patriot ----
More of the same here. Malice and stupidity are the correct terms for things like these. Always misrepresent what's happening so that your POV can reign supreme.
All it would take is some scripting to find the mac address on the network to find out what switch it is connected to. First shut down the port it's connected to. They know where the switch would be so that would be a huge clue as to where the device is. We're not getting the whole story here...
Maybe that's an over simplification but if the RIAA and MPAA can find a mom to sue over for file sharing then then they shouldn't have a hard time with this.
Someone loaded vmware server on their desktop that has an extra network card.
---- Booth was a patriot ----
citrix. What they need to look for is a box with several serial cables coming out of it and going to the console ports of their various network devices. Please send $300 to the nigerians who desperately need your help transferring money.
UNC in 2001
Best Slashdot Co
Didn't any of you hamburgers notice the date on this? meta name="keywords" content="Security,security, data breach, Pilz, insider threat, General Dynamics, ArcSight, "/> meta name="contenttype" content="News,,"/> meta name="publicationDate" content="2008-07-16"/> meta name="articletype" content="News"/>
are they not able to find the physical location? Using MAC and tracerouting, it should be fairly simple to find a router.
TOP DSLR Cameras Reviews of the top DSLRs
security, haha, networking, it, idiots
I used this once to track down which server room a system was located in and while it's not perfect for all occasions, it might help.
Ok, first if you can get an IP for the device, perform a traceroute from 3 or 4 separate sites. Identify it's Gateway if possible, also if find see if you can determine from the traceroutes if it has a common parent node that it's traffic is going through.
Once you've found the most common system talking to it, go to that system and perform ping tests to other systems where you know their physical location in proximity to the system your at, and are only 1 hop away (if possible). The key here is to make sure that all of your samples share as much of the same route as possible to minimize signal noise in your data set you're going to build.
See if you can develop a correlation between ping times and amount of network cable to your sample set. Compare that data to the ping times on your mystery device and you *potentially* have a physical range now in hand to perform your search.
I'll be the first to admit that this approach has limited success based on how your infrastructure is built, but it might help.
No, it's "Frisco" or "Fairyland".
They have their own perception of reality there that has little or no relation to what anyone else does.
I must be missing some key information here, but if the thing has an IP address, they should be able to track it down to the nearest router/switch and follow the cabling, no ? It's not like the thing is sitting in some guy's closet.
-Billco, Fnarg.com
Gumstix seem to have more options.
This story doesn't seem right to me.. this guy wouldn't reveal his password so they fired him? There are perfectly good reasons why it would be right for a network admin not to reveal a password to someone. OR alternatively they fired him and THEN he wouldn't reveal passwords.. ya know, I might forget too if I lost my job all of a sudden.. It's their job to have a comprehensive security plan, that means using user accounts and not root for a case like this but no no.. The evil "hacker" Childs is to blame.
However they did not just fire him.. they ARRESTED him labeled him a "hacker" and put a $5 million bond on him.
In the original case it says "He is accused of creating a password allowing him root access to e-mail, law enforcement documents and other sensitive info." Damm right the Systems Admin should have the "secret" root password to the mail and database server.
Now the city's IT boobs in attempting to do an audit, found some unknown system on their network (might even be a firggen laptop someone brought from home) and they can't figure out how to map a MAC address to the Cisco Switch port.. So they claim Childs installed some sort of "secret server" that they need to find. How absurd.
It may very well be the City is abusing its powers prosecuting this guy.
Bringing liberty to the masses. - http://freetalklive.com/
Just place the terminal server behind a NAT box (say a Cisco router) that looks like every other router on the network and all of a sudden, the hidden terminal server becomes much harder to find. Fake its MAC address to be that of a typical switch behind the router, and you are now looking at a physical search to find it.
Good luck with that, as they say around here.
I wonder if they tried entering 'admin' as the password and leaving the user name blank?
Because I'm a...dumbass and didn't think about it....
Or could it be that you just didn't care, Lord Apathy?
db
I am literally 3000 tokens away from the chaotic crossbow --Stephen
Dig up a film called "Demon Seed"
see what that gets you..
every day http://en.wikipedia.org/wiki/Special:Random
who cares? root's always right.
login: root
password: ************
sorry, but morons and salesdroids should'nt have access to the superuser account. you will just cause a big mess not knowing how to clean up afterwards.
The thing that amazes me about this story is how willing the press are to spout on and on in a defaming way about things they know jack about.
I seem to have missed the part where it was stated the admin installed this thing they are looking for. Sounds like people get the scariness of such a thing being out there, and just assume he did it.
The technical "details" in the articles I have read about this story sound like voodoo, because the people who wrote them have no idea what theyy are talking about. They use words like server, switch, router and network interchangeably (ie, calling the fiber WAN "the city's servers").
Regardless of how much of an ass this guy may be, they have pretty much tried and convicted him in the media already, in the same way a native might think those nasty explorers cast spells on them when they brought smallpox over.
It is truly disconcerting to realize how unconcerned some people can be with the effects of their uneducated asshatery.
Add to that a large city that provides services that people's lives depend on, which hasn't taken the time to ensure it has more than one person in IT who can figure something out(seems they recently got rid of THE one), and you have the makings of a truly IT bent version of The Office I think. Probably at least 8 seasons worth.
"I'm sorry Dave... I can't let you do that."
*kill*
So this thing isn't on a TCP/IP network, right? 'Cause otherwise, methinks one could use some layer 2 info to start digging.
I would laugh my ass off if it was a virtual server running on the admins comp.
The city IT personnel must be inept.
I used to do this kind of mystery work for our help desk to find PC's if they were infected with a virus, had a web server running, or a variety of other problems before the wild wild west days there ended and they got some management tools, policies, centrally managed virus scan, etc in place.
it's not like you shouldnt even be able to narrow it down even closer than that with a properly segregated network, unless you have a class A network doing the broadcast storm of doom; if you know its there you must have an IP address no? by then you should know its hanging off one of a series of switches; look at the arp tables and then at the mac address tables; you should be able to determine which port the mac address is hanging off of; without too terribly much effort; after that you trace the ethernet cable and go see where it's at...
i was doing this with extreme networks switches eons ago, but it should be just as doable with just about any switch
Why would the Feds want the city to find the secret box they installed?
In other words, never attribute to incompetence that which can be easier explained by malice...
it's a very big LED.
Oh. Of COURSE. (Blinks very slowly, right?)
It's on top of the triple-pronged antenna platform on the hill over by the Castro district.
They probably can't find it because they're re-hacking the antennas for digital TV and are moving all the equipment around.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
"But did you have to blow up the whole planet?!"
"Well it was a lot easier than trying to find the one guy who sold me this lousy watch."
I figure by 2030 or so my 6-digit UID will be something to brag about.
Instead of common sense and ability.
I killed da wabbit -Elmer Fudd
Packet sniff to find out it's IP, exploit it, then have it eject the cd-rom... then just go look for a computer on a rack with a cd drive open.
I'll Find You Peer, If It's The Last Thing I Do!!!!
Sure, two or three decent guys could rappel in from helicopters, flash-bang the problematic idiots and have that network back up on it's feet in a couple of days.
The problem is it's a municipal network. Those bureaucrats would rather play turf-war than send successful pings any day of the week.
The one million dollar figure isn't about accurately reporting the damage done. It's about citing a figure (pinky in mouth, mind you) to inspire shock and awe.
Now they need Robert Wagner to remind them, it really isn't all that much any more...
Does Kevin Rose still live there? Just asking, not implying. :)
it's a cisco network...
check the arp table for a mac address, and trace the mac through the network.
"show mac-address-table | include "
telnet/ssh to the next hop, rinse and repeat until it's on a local port. then go trace the wiring.
No wonder the network admin was scared to give his passwords up if these guys can't even locate a device on the network...
Big Brother did it.
ECHELON is a name used in global media and in popular culture to describe a signals intelligence (SIGINT) collection and analysis network operated on behalf of the five signatory states to the UK-USA Security Agreement (Australia, Canada, New Zealand, the United Kingdom, and the United States, known as AUSCANZUKUS).[1]
The system has been reported in a number of public sources.[2] Its capabilities and political implications were investigated by a committee of the European Parliament during 2000 and 2001 with a report published in 2001.[3]
In its report, the European Parliament states that the term ECHELON is used in a number of contexts, but that the evidence presented indicates that it was the name for a signals intelligence collection system. The report concludes that, on the basis of information presented, ECHELON was capable of interception and content inspection of telephone calls, fax, e-mail and other data traffic globally through the interception of communication bearers including satellite transmission, public switched telephone networks and microwave links. The committee further concluded that "the technical capabilities of the system are probably not nearly as extensive as some sections of the media had assumed".[3]
http://en.wikipedia.org/wiki/ECHELON
Here is the link to the server behind the wall;
"The University of North Carolina has finally found a network server that, although missing for four years, hasn't missed a packet in all that time. Try as they might, university administrators couldn't find the server."
http://www.techweb.com/wire/story/TWB20010409S0012
The truth shall set you free!
. . .is that the user/password is
admin/password
if they know the IP address of the offending equipment, they should be able to find it on a switch/router and what port its connected to and then disable that port, or even just change the VLAN of that port to a blackhole.
if its wireless, can't they just change the SSID and/or encryption key and change their existing machines to move over to the new wifi network?
It seem's either of those could be done within 1 business day, then they can work on finding it.
Am I missing something?
maybe, after all what has been said about the compentence about the non-fired admins, it turns out that the mystery device has the ip 127.0.0.1 and is able to hide its real MAC adresss...
I'm suprised no-one has proposed waterboarding for locating the rogue device - One wonders how many N-Taps are installed also.
As far as I am concerned, SF should fire the entire supervisory chain, from the Admin's supervisor up to the CIO.
Its apparent IT Management is lost on the organization. Here, we have poor change management and poor seperation of duties resulting in a complete loss of network control.
While the direct costs may add up to a million, I am sure the indirect costs of a slipshod IT organization without a managerial clue run quite a few million, every year.
And sure, SF got oWn3d - but by a nice guy - had this been an organized criminal element - they still would not know they are owned.
Also, if you have vmware you could do a hot P2V and then do whatever your heart desired to the new VM.
They should go examine each and every router starting at the core, and check if the AUX port or console port is plugged into something. In fact all serial interface types should be examined somewhat; since a router at the core of a fibre WAN should have serial interfaces only for management networks or console server access.
If neither is plugged into anything, then it is not connected to a terminal server. Put a temporary label on it that says "unmanaged", until it can be connected to a known terminal server or modem.
If a router's connected only to known terminal servers and modems put a label on it that says "managed by (terminal server id)". Don't stop until all devices are labelled managed or unmanaged.
If an untagged box is seen with a pair of octal cables, that's a dead-giveaway that it's a terminal server.
If a cable on the serial, AUX, or Console port is connected to an unknown device, then trace the cable as far as possible.
The cable should end at either: (a) the terminal server, (b) a modem, (c) a router, (d) a patch panel/punch down block
In case of (a); stop, you found the terminal server, now take full control over it, tag it as a known terminal server, and update the tag on the connected routers to indicate they're properly managed.
In case of (b); the router must be configured to use the modem, or the modem is deadweight. Verify secure configuration of the router, that only authorized personnel can login via the serial link. Tag the router as managed by modem, if all serial lines are known.
In case of (c); in all likelihood, that router is the terminal server. Examine conf to verify.
In case of (d); this reverts to the typical situation of tracing network cables.
Serial cables do not have a very long distance they can travel before the signal is unusuable; it should be possible to trace.
But everyone who supports more government ought to take a look at the incompetence here.
Right. Because there's no incompetence in private corporations. (Or "conflict of interest", whatever that is.) We'd be much better off farming out responsibility for civic infrastructure to the private sector.
SIERRA TANGO FOXTROT UNIFORM
<erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
"Good news, everyone!"
This isn't the first time that a server has been lost. Or found.
http://groups.google.co.uk/group/alt.folklore.computers/browse_thread/thread/6289e24b593eaf16/17ac734391deebbb?lnk=gst&q=server+behind+drywall#17ac734391deebbb
Don't any of these people remember reading newsgroups 7 years ago? It's not rocket science.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
I am the Team Manager of Systems Management at the company I work for and I do NOT know the administrator password.
Why? Well if you've been reading the story and the comments it's because I don't have to know the administrator password. I have x<10 people in our whole company's IT departments world-wide that need to know that password and they are all people that know what they have to do.
You may be thinking: oh look , just another manager that isn't capable of pulling his load.
Wrong: I once did know the password and we got hacked. Now try tracing problems if 25+ people know the admins password.
Ahhh. Yess, that's the answer. I do have Admin privilege but only on a second account (not my primary user account). With that account I pull my weight and I am trackable as are the other x<10 gurus.
As for the guy in SF. Kudos go to him. I would deffinitely not let my boss or his boss (CIO) know the Admin password.
As for the DA. How to prosecute somebody who "hijacked" a network but didn't? Who wanted to crash the network, but was in fact asking "are the core network components affected?"-"No!"?
As for all you guys chipping in to help the "in-your-terms" incapable leftover SysAdmins. Why do it? Let them solve their problems themselves. They will realize that learning by doing is the best way to learn and it might teach them that there are people who you just have to rely on. I lost one SA recently to cancer. He took a lot of knowledge with him when he left. He isn't unreplacable, but it will take a hell of a lot of time.
As for the SF IT dept. Why?.................
-hot2use
----------
Searching a knowledge base is like picking your nose. You never know what you will find.
Why don't they try something simple like say the Ping of Death. Try the simple things first and work your way up to the more complicated.
Am I the only one who shouted that out in the theater at the scene in Matrix 3, when Agent Smith was using Trinity as a human shield in a hostage-faceoff with Neo?
David Gould
main(i){putchar(340056100>>(i-1)*5&31|!!(i<6)<< 6)&&main(++i);}