Slashdot Mirror
San Fran Hunts For Mystery Device On City Network
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
Power cycle it with a city-wide EMP.
Life would be easier if I had the source code.
From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.
Given that they cannot hunt down a single device on the network, I'd have to agree with that assessment.
MAC address ... switch port ... it should be easy.
Um, do what any network admin does with a rouge device. Search out what port its MAC address is connected to and then start tracing the cable?
I'm fairly certain most all current managed switches allow for this. Even with unmanaged ones you can hunt down which unmanaged switch it is connected to and snoop from there.
------
"And may your days be long upon the earth."
<erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
if the answer isn't violence, neither is your silence / freedom of expression doesn't make it alright
Hey! Fyodor! They need your number!
Fyodor spent much of this summer scanning tens of millions of IPs on the Internet (plus collecting data contributed by some enterprises) to determine the most commonly open ports. Nmap now uses that empirical data to scan more effectively.
Zenmap Topology and Aggregation features were added, as discussed in the next news item.
Hundreds of OS detection signatures were added, bringing the total to 1,503.
Seven new Nmap Scripting Engine (NSE) scripts were added. These automate routing AS number lookups, "Kaminsky" DNS bug vulnerability checking, brute force POP3 authentication cracking, SNMP querying and brute forcing, and whois lookups against target IP space. Many valuable libraries were added as well.
Many performance improvements and bug fixes were implemented. In particular, Nmap now works again on Windows 2000.
With just nmap, my old buddies at Farm9 could have sussed this out in a few hours. I think they are still around - as Red Siren / Getronics.
Ahh. I miss running netcat at 3 AM!
"Flyin' in just a sweet place,
Never been known to fail..."
Man, the more I read about this story, the more inclined I am to believe the network admin.
He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.
As Indy deciphered the symbols, he found the correct sequence of tiles to push. The huge stone door slowly opened. Indy grabbed a torch and headed inside. At the end of the long room, there it was on the throne: A massive server. It was archaic, and it appeared to be attached to a punch card reader. Along the sides of the room, there were two rows statutes of archers pointed at the center. Indy made his way slowly to the monitor and keyboard of the server. He brushed away the dust and hit the spacebar. The screen turned on slowly and it displayed:
SCO Server 1.0
Your license has expired. You owe use $699.
>_
Suddenly the archers rotated positions and were aimed at Indy.
"Oh boy."
Well, there's spam egg sausage and spam, that's not got much spam in it.
I recall hearing a story about a Sun Sparcstation 2 at my old college that had accidentilly got sealed inside a wall by construction folks when re-working the building the CS lab was in to eliminate a few closets for structural support reasons.. nobody could find it (shock!), but kept using it as a DNS server for another six years. It was found about 2 years after it stopped responding to ping when some component (nvram?) let out, and it started beeping after a power flicker.
2> It's easy to find wireless devices... I've personally been doing it since the 1980's.. it's called a fox hunt here in the Chicago area. We used to get 1 minute of transmission every 5... with WiFi you can just ping the dang thing... how easy is that?
--Mike--
You think they've learned anything about the gear since then? No wonder they're having problems.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
Why is Slashdot linking to stories that paint the network administrator as a bad guy when he's so obviously surrounded by morons? These are the same people who published all of their user names and passwords. That puts the cost of this "hijacking" into perspective. The cost of trusting their employee with the powers required to do the job was zero.
Friends don't help friends install M$ junk.
If the city can't even complete one of the most basic network administration tasks of finding a physical device on a network, I think they have absolutely no right to accuse anyone of "hijacking" their network. I hope the defense attorney for Terry Childs brings this up.
1) They were firing the guy, so he was no longer in the employ of the city, so his boss, was no longer his boss.
2) You don't know what you're talking about. Every IP address on the network should be known. Either through DHCP or static IP address map. A ping sweep should reveal any IP address in use, that shouldn't be. From the ping sweep, one can arp the unknown IPs to get a MAC address, and do a lookup on the Manufacturer code to know what KIND of device the MAC could be. one could use NMAP to try to discover type of device as well. Then you start going to every port on every switch with rogue IPs hanging off it, and manually looking at what is attached at the other end.
As for wireless access points, if you don't have control over them, you pull the freakin plug. Unsecured Access points and open access points should be VLANed off from administrative networked, including not allowing VPN tunnels from unsecured and open wireless access point.
If the boss allows crap like that on the network, he is an idiot, and shouldn't have the Passwords and access codes to anything.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
your employer's passwords are NOT yours, no matter how stupid you think your boss is.
Refusing to give out passwords to higher-ups is not always the wrong thing to do. If you are the network admin, and your job is to maintain security of the network, wouldn't it be reasonable to refuse to hand out passwords to people outside of the network administration roles?
Although I can say that an admin can make that choice at his or her own peril. After all, the higher-ups can always opt to fire the admin and replace him or her with someone who is willing to seek security of their job over security of the network they are paid to administer.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Did they try the Rouge Admin's office. It's probably that beige box under his desk... Either that or he made up the device and it does not exist, he's laughing at them ripping the place apart trying to find it :D
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
I'd like to add that while the way he handled being surrounded by idiots was wrong, he was clearly surrounded by idiots.
No documentation?
No change control?
No diagrams?
What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.
I worked one place where a guy with a great deal of responsibility died. (here today dead tomorrow kind of thing) His peers blamed *everything* on him simply because they could. This sounds like the same thing.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
No no. "The City" is quite clearly "The City of London". And no where near San Francisco. (I wonder if they use Cisco hardware though, which might make the San Fran - Cisco more apt)
Huh? London is only about 142 miles SE from San Francisco and with a population of about 2000 people barely qualifies as a city, let alone "The City" moniker.
I'm a consultant - I convert gibberish into cash-flow.
If you find that you are "holding the place together", IT-wise, you are likely part of the co-dependency and are part of the problem.
IT and the other management have both agreed to ignore each other, literally or otherwise, allowing each (and the individual personalities) to do things "their way"; damn the best practices, good management, logical, financial, or even legal issues.
Except when things go wrong.
Like a breakup, they can get ugly. And, as the IT guy, you will always lose for it is not your Business, but theirs. You are simply hired help.
Your London may be inferior. Ours definitely warrants a 'City' moniker. Especially when The City of London is distinct from the conurbation that is known as London. And the City of London is actually fairly small - almost exactly a square mile - but ... well, you know what they say. It's not the size, it's how you use it.
http://weblog.infoworld.com/venezia/archives/018376.html
An insider claims that the power outage that Terry Childs was accused of using to sabotage the San Francisco network was not a planned outage.
TAGS: Problems, San Francisco's FiberWAN, Terry Childs
If you've been following the Terry Childs case to any degree, you probably know that one of the key allegations keeping him in prison on $5 million bail is that he had willfully planned to cause the network to fail during a planned power outage at the DTIS One Market Plaza Datacenter on July 19th. According to credible information I've recently received, that power outage was only going to affect the cubes and offices in that building, but not the datacenter itself.
Thus, there never was a plan to power down the network core. Thus, there's no way that Childs could have tried to engineer the failure of the network during this planned power outage, since the network core would not have lost power.
[ Follow the Terry Childs saga with InfoWorld special report: Terry Childs: Admin gone rogue. ]
The evidence supporting this claim comes from someone certainly in a position to know: Ramon Pabros, the DTIS Datacenter Supervisor himself. Pabros has been employed by San Francisco's DTIS for a surprising 41 years. He's been the Datacenter Supervisor since 1984. He's been running datacenters for the City of San Francisco since Ronald Reagan's first term, the introduction of the Macintosh, and the second season of The A-Team. It's probably safe to say that he knows what he's doing.
According to my source, he will testify to the fact that he discussed the power outage with Childs several weeks before the outage, and at least 10 days before Childs' arrest. He will also state that Childs specifically asked for confirmation that the datacenter itself would not be affected, and was reassured that it would not lose power.
With this statement, the City's allegations that Childs planned to cause the failure of the FiberWAN basically collapse.
Now, I'm admittedly a stranger to San Francisco politics, and am certainly not a lawyer, but if the DA was going to make these accusations against Childs, shouldn't they have talked to Pabros? If the OMP Datacenter was not going to lose power on that date, then this charge against Childs is essentially the same as charging someone with planning to burgle a store that doesn't exist.
But then again, this is the same DA's office that placed valid group usernames and passwords into the public record, and an IT department that ran public, unprotected websites containing internal emails, core network details, as well as usernames and passwords.
I suppose I really shouldn't be surprised at all.
UPDATE: It appears that Pabros has just announced he will be retiring, effective next Wednesday. I can't help but wonder if one event has anything to do with the other. I do know that there have been a number of odd layoffs from San Francisco's DTIS in the past two weeks.
Posted by Paul Venezia on September 8, 2008 08:48 AM
There are now dozens of cars packed full of cheetos cheap laptops and foul smelling individuals travelling near, or perhaps at the speed limit, towards san francisco. They're full of people thinking the same thing, "Shit if they can't find a wired device, they sure as hell can't find a wireless one!"
All they have to do is look for the small black box with a lone, onerous blinking red LED.
I find it difficult to understand how a blinking red LED would constitute a heavy burden.
What would you think of a doctor who, because some exec somewhere decided he should, pushed the WRONG medication / procedure to you?
Where does your ethical responsibility end and the boss's desires begin?
To me there isn't even a question. Fire me. Go ahead. I will get another job.
Who is actually the OWNER of the system? The boss? Isn't he employed by the same company as the sysadmin? Don't they both have an obligation to safeguard the OWNER'S property and interests? If the sysadmin refuses to hand over the password to sensitive equipment & systems to a (perceived) inept superior-- as long as that guy DOESN'T own the company-- isn't he actually performing his responsibility to the real owner? Which in this case would be the city, and the personification of the city would be the mayor-- and that's exactly who he DID give the passwords to. So it seems to me like he did precisely what he was supposed to do in terms of safeguarding the network and sensitive equipment. Of course he should probably be then fired for failing to keep backups, conops, continuity planning, etc. But that's a different matter.
Oh yeah, let's give him a break. Oops, he's been by hit by a bus. Where's his disaster recovery plan? That's right, there isn't one.
My bet is, it's sitting right in the middle of his old desk blotter, in a fat manila folder marked "Disaster Recovery and Service Continuity Plans". These clowns would never find it there in a million years. The infamous missing passwords are probably in a letter-size envelope in the top left desk drawer, too.
and changed the MAC address to C0:FF:EE:C0:FF:EE
or
FE:ED:C0:ED:BA:BE ...
Just saying
I went to a boarding school in Kenya for high school. The system of bells ran across the campus of several hundred acres and many buildings in a closed loop, with all the bells in series. The system ran through the main office, with the Super Secure Bell System locked in a cabinet there so nobody could access it. Penalty for messing with the system of bells was said to be expulsion.
The problem was, that all you had to do to get all the bells on campus to ring was to wire the loop back into the mains.
We took a clock from the darkroom in the photo lab, and ran two wires through the face plate. We then ran another strip of wire along the minute hand, so whenever the minute hand swept by a certain point on the clock every hour, it would complete the circuit for about 30 seconds and ring every bell on campus.
We then hid this contraption under a pile of wood in the attic of the wood shop. Right after convocation when I could no longer be expelled, I ran into the building and turned it on.
Apparently the bells rang off and on mysteriously for most of the next month of holiday until they managed to follow the loop and find the device. Good times.
www.clarke.ca
It appears that the idiot "boss" is attempting to generate support for the claim that this guy is a "problem" by paying unreasonable amounts to "repair" the "damage" he did.
It's difficult to "prove" that a guy did millions of dollars of "damage" ... without a bill for millions of dollars of "repairs".
Any competent network admin could map out the network and document it for FAR less than the hundreds of thousands of dollars that is being thrown about.
Hissssss
Do Or Do Not, There Is No Spoon, There Is Only Zuul. Everything in the above post is probably opinion.
When users ask for Admin privilages, they should be told to go fsck themselves. No matter who they are.
I'm a software developer. For the first few weeks working here IT wouldn't give me admin rights on my own box. I couldn't install software.
So I sat here and did nothing. Not because that's what I wanted. But because that's all I could do, until they gave me permissions on my machine.
Generally speaking, you're right. Most people in a business should be locked down. But not everyone. Depends on the person - depends on the work they're doing.
Weaselmancer
rediculous.
Sorry to commit the solesism of replying to myself, but I (gasp!) just read TFA.
...being held in a jail cell on $5 million bond, also happens to be a former felon convicted of aggravated robbery and burglary stemming from charges over two decades ago, which the city knew when it hired him as a city computer engineer.
Childs, who has worked for the city for five years but faced firing for alleged poor performance...
Illuminating, but mostly in that it shows all parties in a very dim kind of light. Under the circumstances, I would have hesitated to employ the guy in this capacity anyway...
I'm reminded of a conversation I had some 25 years ago with a co-worker IBM mainframe technician. IBM management was incensed that uneducated morons turning screwdrivers could make 70k a year. Back then as much as what they were paying top MBA stuff shirt types. They were on a mission to get salary levels down to "reality" paying these screwdriver wielding monkeys what they were (in their minds) really worth.
Attitudes have changed but not a lot. 93% of companies that loose their data center for 10 days or more due to a disaster filed for bankruptcy within one year. 50% filed bankruptcy immediately (National Archives & Records Administration in Washington). One can't say the same thing about those over paid MBAs.
It may be awhile before IT matures into a "profession" like doctor or lawyer however I personally believe we're holding the keys. The world can't function now without us.
-[d]-
Who modded this insightful? Part of the reason he was getting canned was because he was PUSHING for the sort of documentation and recovery plans you're snarling about. None of the PHBs wanted to put their names on it because if they came up short, it would be their asses on it.
It shouldn't be that difficult to find a piece of h/w on a network.
Interrogate the switches to find the IP/MAC address corresponding to the device you are trying to log on to. In the event that this Childs guy is deviously smart (i.e. patched the switch software to conceal a particular device) one can still use a stand-alone sniffer to trace packets through a system.
Its possible that the 'terminal server' might be virtual, just an app. running on some other piece of hardware that doesn't necessarily have "ACME Terminal Server" and a wining LED on the front. But tracing the network to that particular box isn't difficult (maybe time consuming).
If these people are really that dumb, I can understand why Childs kept them off the system. Reading some of the stories about him, it wouldn't surprise me if he left a bunch of 'dead ends', like phony terminal servers that nobody could find. Or wireless access points not plugged into anything but plastered inside a wall to drive security auditors nuts.
Have gnu, will travel.
Paul Venezia digs a little deeper into this so-called "terminal server" today in his blog:
"From what I can see, it's a device running Cisco IOS that was accessed via telnet. I could generate an identical screenshot to the one entered into evidence in about five minutes using an elderly Cisco 2924-XL Ethernet switch -- a device that's certainly not a terminal server. It's completely unclear to me how they could have possibly come to the conclusion that this is a "terminal server" -- the evidence presented to the court certainly does not support that theory."
Venezia also uncovers additional technical errors in the prosecution's case, which appears to be unraveling with the recent news that the DTIS Datacenter Supervisor Ramon Pabros will testify on Childs' behalf. Since coming forward, Pabros has announced he will be retiring from the DTIS, effective Sept. 17. Coincidence?
UNC in 2001
Best Slashdot Co
I used this once to track down which server room a system was located in and while it's not perfect for all occasions, it might help.
Ok, first if you can get an IP for the device, perform a traceroute from 3 or 4 separate sites. Identify it's Gateway if possible, also if find see if you can determine from the traceroutes if it has a common parent node that it's traffic is going through.
Once you've found the most common system talking to it, go to that system and perform ping tests to other systems where you know their physical location in proximity to the system your at, and are only 1 hop away (if possible). The key here is to make sure that all of your samples share as much of the same route as possible to minimize signal noise in your data set you're going to build.
See if you can develop a correlation between ping times and amount of network cable to your sample set. Compare that data to the ping times on your mystery device and you *potentially* have a physical range now in hand to perform your search.
I'll be the first to admit that this approach has limited success based on how your infrastructure is built, but it might help.
I must be missing some key information here, but if the thing has an IP address, they should be able to track it down to the nearest router/switch and follow the cabling, no ? It's not like the thing is sitting in some guy's closet.
-Billco, Fnarg.com