Slashdot Mirror
Researcher Resigns Over New Cisco Router Flaw
An anonymous reader writes "Michael Lynn, formerly a researcher for Internet Security Systems resigned today rather than conceal his research into serious new flaws in Cisco routers, according to stories at Washingtonpost.com and CRN.
Interestingly, Cisco says the the problem is not a security vulnerability, although it chided Lynn for not going through proper vulnerability disclosure channels. Both stories note that Lynn is in danger of being sued by Cisco for revealing the information, details of which were pulled at the last minute from the materials handed out to Black Hat attendees." Update: 07/28 12:23 GMT by Z : SimilarityEngine writes "Cisco and ISS are filing a law suit against Michael Lynn and the management of the Black Hat Conference, following Lynn's presentation discussing a vulnerability in IOS."
It's ok, really it is. Karl Rove gave him the information.
FLR
As dependent on as our economy is upon routers, and Cisco in particular, it seems that his disclosure was definitely in the public interest, and if he isn't entitled to whistleblower protection, we need to mount a campaign to get him protected. Write your Congressoid.
"The mind works quicker than you think!"
but couldn't he at least have waited a few weeks to see how Cisco responds
Cisco seems to suffer from the same stupidity that most other large corporations do. They'll take a report, and sit on it for weeks, and sometimes months. Full Disclosure is usually the only way to get them to actually fix the issues in a timely manner.
In TFA, Cisco themselves said that he did not disclose any new vulnerabilies... so... what is the BFD?
Later, Cisco said it was all bent out of shape because they follow an "industry established disclosure process" and because Mr. Lynn "illegally" obtained the information...
Hey, Cisco, I have news for you. "Industry established disclosure process" != "Law"
Get over yourselves, admit that you're a bunch of fuckups that can't make secure networking equipment, and move along..
Companies like Cisco, Microsoft, etc. are generally made to look really bad when security flaws are exposed in their products.
The way they prefer it to go is that someone contacts them secretly, tells them the hole, and they can have it fixed all up by the time the vulnerability is published.
Then they get to look super-secure, since they were "too quick" for the bad hackers.
Some people, however, think that the only thing that'll get companies to take security more seriously is if they are actually made to look really bad, and maybe some of their products actually get hacked.
Unfortunately, when you're dealing with some giant businesses cost/benefit analysis, the only thing that can get them to take notice is a little carnage.
Is it worth it? I dunno, but it's certainly arguable.
The thing is (from what the articles say) it's not about one particular flaw. It's that ANY overflow flaw can be exploited to take control of Cisco IOS, which is bad news. Add Cisco's plan to abstract the hardware from IOS and then you've got a major problem. Basicly, it's about time Cisco implimented some form of DEP protection offered by current Intel and AMD processors + software, to prevent this from being an issue. Or check their bloody code of course.
Yes, he could. But then again, I suspect he already did. The traditional approach was to tell the vendor, and announce the flaw publicly 28 days later. That gave a vendor sufficient time to code and test a patch. However, many vendors (and Cisco seem to be particularly bad about this) sit on problems like this for several months and take no immediate action. I'd be far from surprised to hear Cisco were notified of this 3 months ago, hence Lynn's frustration and his decision to publicly talk about the flaw. I don't actually know what happened, and the above is just speculation. I suspect there's more than a grain of truth to it, though.
"The invisible and the non-existent look very much alike." -- Delos B. McKown
I must have missed the "master password" thing.
As far as Cisco going down hill I don't really agree with that. Currently Cisco is expanding their product offerings into new unexplored territories such as IP Telephony. I have installed and supported several of these systems. As long as you follow thier design, install, and support guidelines they are as robust and as problem free as any other platform that i've worked with.
I think most people on Slashdot understand the complexities of the internet world. A minor change here can have a huge, uexpected, impact across the network or application. However, if time tested procedures for upgrades and testing are followed nothing has really changed. I think what may be giving a Cisco a bad name is all of the under qualified people out there installing their systems. The MS world of patch it, reboot, and go about your business does not fly when you critical systems are involved.
I must have missed the "master password" thing.
That was from a while back. They had set up a master "backdoor" password in a version of IOS and ended up getting ridiculed for it quite heavily.
Yes, he could. But then again, I suspect he already did.
From the article:
"The decision was made on Monday to pull the presentation because we wanted to make sure the research was fully baked."
In other words, the research was not even finished yet. Isn't that a little impatient, and might there be a little chance that the researcher in question would have liked the attention he would've gotten if he presented this information at Black Hat, which was part of why he made the decision to pull out the information anyway ?
- Leon Mergen
http://www.solatis.com
He told them in April, according to BoingBoing, and they still hadn't fixed the problem totally.
The articles cited are light on details. But nowhere do the articles suggest that Cisco was burying the flaw. In fact, the opposite is indicated. ISS and Cisco are apparently working on a fix. In my mind whistle blower protection is valid if the whistle blower is uncovering corruption. Which does not appear to be the case here. Based on the information presented, the system was working on the problem, he just wasn't happy with it.
How do you apt-get hardware?
The point of buying a router is efficiency. Otherwise get a switch and a 386 running BSD or Linux... Having hardware move packets is almost certainly going to be faster (and efficient) then having a general purpose processor do it.
That said you have firmware that controls the hardware which could be "apt-get" though in reality I'd rather see an open source firmware that was also provided as binary images you could just upload.
Do you really want some MCSE throw-back building a firmware image when they can hardly manage cmd.exe?
hehehee sick.
Tom
Someday, I'll have a real sig.
Well if you worked for the Secret service and knew that the president was having young girls kidnapped so he could rape them would you keep your mouth shut? It's about scruples. These flaws seriousally bother this man to the point that he is willing to give up his career and life as he knows it to get the information out.
this means it is very big, probably one of those one person can disable the whole net easily or snoop on all internet traffic without traceability.
I know of people that quit their jobs to blow the whistle and these men and women need to be held up as the heros of our time as they are the ones who not only have lots more guts that the rest of us, but are certianly more driven to not violate their core values.
I commend this man, he should be look up to.
Do not look at laser with remaining good eye.
I agree that disclosure, in general, is clearly in the public interest, but this cannot always be the case.
We simply do not have enough details here to declare this disclosure "good" or "bad." Although Cisco is claiming the information was on vulnerabilities that have been fixed, that could be a PR move to stave off a stock plummet or put a stop to proliferation of the information to those that may want to use the vulnerability to bad ends.
We also can't be sure of what "fixed" truly means. How tested are these fixes? Are they complete fixes or do some variations on the vulnerabilities revealed still exist? The questions go on and on.
I'm all for protecting Whistleblowers, but only if they have done all they could to ensure that they are not causing more damage by revealing information that can still be used against current users. I'm not saying that this is clearly not the case here, only that we need more time before we declare this guy our champion.
How long should it take?
0 7/update_to_cisco.html
http://blogs.washingtonpost.com/securityfix/2005/
The injunctions filed against him state that ISS and Cisco had been working together on the flaw for the past four months, and that up until earlier this week, a Cisco executive was slated to co-present the findings with Lynn at Black Hat.
Our friend Mojgan Khalili is the Cisco employee mentioned in the article, who said the security researcher broke the law -- "It is especially regretful, and indefensible, that the Black Hat Conference organizers have given Mr. Lynn a platform to publicly disseminate the information he illegally obtained."
If you'd like to write to Mojgan and say that you don't like their attitude toward full disclosure, or their attack on the guy who's working hard to make things secure, here is his information.
If nothing else, you could ask him "what law did the guy break, biatch!?!"
Mojgan Khalili
Cisco Systems, Inc.
978-936-1297
mkhalili@cisco.com
http://www.thebricktestament.com/the_law/when_to_
I know, I know. Mod me redundant. This is slashdot. The editors are on crack. Who Rs TFing A? But really. Not a security flaw? No, Cisco said it wasn't a NEW security flaw, but an extension of older ones. There's kind of a difference between "Not" and "Older-but-born-again". Mod me into oblivion now.
I can't help but wonder, if this in the end really about gaining some publicity and in the end making more money.
Cisco is actually very upfront and cooperative when you report things which might be a vulnerability (I have personally dealt with PSIRT). The people who work there are actually so polite, it's kind of annoying (I have been thanked about 2 dozen times for reporting a very minor finding).
They do however expect you to play by the rules. Even if you are the person who found a bug, you are expected to let Engineers fix the bug before you release the information.
Also, there is policy in place, which makes sure major ISPs (Carriers) are informed first, so they can do upgrades before the PSIRT release is made public.
All that makes sense, since we are really talking about essential infrastructure.
Of course, all that kind of takes away the coolness of reporting a vulnerability and you will get a lot less publicity (cisco credits you) than what you would get, if you just post to some mailing list.
If he really released information he researched at ISS without consent, well, he should face consequences. Because I obviously was to gain from it (getting a new job, making a name or himself). Hopefully he wasn't just doing it for the publicity.
"Mommy, mommy! The garbage man is here!" "Well, tell him we don't want any!" -- Groucho Marx
Okay, this sounds pretty simple. Michael Lynn finds a (new) explit of Cisco routers and its a doosey. He informs ISS, who informs Cisco. Cisco management can't believe that such a serious flaw exists, since they've know about the possibility, but its been written off as minor in the past. Lynn presses his case to his supers, and they get down and dirty with Cicso. Cisco craps its pants because the flaw is everywhere, and it's going to cost real money to fix, and could hurt company Q results.
Cisco agrees with ISS taht they're going to do something about it, but it's going to take a bunch of resesarch and time. They'll keep it quiet for a few years while they put th fix in the pipline for new models. They'll work on a firmware fix, but its back burner as long as the explot isn't public. If ISS keeps its mouth shut, they can still do work for Cisco.
Lynn hears that his research is to be hush-hush, and that Cisco will work on it, but it could be a while before there's an actual patch. No arguing that the flaw is critical will make ISS management, with a financial gun to its head, budge.
Lynn flips ISS the bird, 'cause he thinks its a major security issue, and presents his research anyway. Cisco and ISS claim they're working ont it, and that its and old flaw, and nothing really serious. And they're quietly looking for a man to fir Lynn with concrete shoes for blowing their cover.
Seems pretty clear to me.
Is it just my observation, or are there way too many stupid people in the world?
Contradiction?
... and I can never find out why pople can get sued for disclosure of something dangerous to a lot of costumers.
Quote: "It is important to note that the information Mr. Lynn presented was not a disclosure of a new vulnerability or a flaw with Cisco IOS software. Mr. Lynn's research explores possible ways to expand exploitations of existing security vulnerabilities impacting routers."
Quote: "... Mr. Lynn a platform to publicly disseminate the information he illegally obtained."
If his research regards known and exsisting vulnerabilities how could they be illegal obtained? This can only happen if Cisco sits on the vulnerabilities for some time. If this is the case its a poor excuse by Cisco to state that its not a new vulnerability.
In my humble opinion its new when first made public.
If I use their routers I would like to know if they can be hacked. If they can get hacked I would like the oppotunity to take them offline if I need to protect my business.
If I don't have that oppotunity - and I loose data/values/etc due to an attack, I'll have to keep Cisco responsible.
-:) Oh no - not again.
www.rednebula.com
I dont believe in keeping an exploit away from the public until the vendor gets his thumbs out of the dark place that smells funny. First of all i really think much more work needs to be put down into securing the systems before they are released, this includes various linux vendors. Its insane today with the user being the Q&A and security department for the vendors.
Full disclosure is a nice cushion for people who really didnt do their job in the first place. It doesnt in no way help the users. Before the exploit is released publicly you can bet your backside its used for company spying and other shoddy activities.
A company shouldnt be afraid of scriptkiddies, theyre harmless compared to their competitors armed with their most secret info. Full disclosure makes it possible for a company to atlest try to mitigate that threat. Other disclosure puts them in the whims of the vendors.
HTTP/1.1 400
The way they prefer it to go is that someone contacts them secretly, tells them the hole, and they can have it fixed all up by the time the vulnerability is published.
... And this happens in the Open Source world too. Mozilla, for instance, has "classified" bugs, which are not opened up to the public until a fix (or whatever) is available. Take for instance, the Windows chrome:// bug from a few months to a year ago. They sat on it for over a year (and it was classified, of course), and didn't do anything until an exploit appeared in the wild. The fix was issued right away. "Too quick" for the hackers, indeed.
Then they get to look super-secure, since they were "too quick" for the bad hackers.
What I'm getting at is don't say that this sort of behavior is limited solely to closed source software. No one wants to have the pressure of handling a security fix WHILE an exploit is out in the wild. Would you rather have the opportunity to fix a security flaw while no one else (but the person who discovered it) knew about it, or would you prefer the person who discovered it announce it to the world and release an exploit first?
Umm you do know that Black Hat is a security conference? Mostly attended by security professionals?
You're a prick. RTFA. He waited 4 (in words FOUR) months for Cisco to fix this until he finally made it public.
"The more prohibitions there are, The poorer the people will be" -- Lao Tse
-Mark
that would keep all parties happy, is a modification of the current craze for bug-bounties.
Flaw is reported, accepted and cash is paid on a daily/weekly basis until the issue is resolved.
Submitters would get more for a complex bug that involves more work to fix it and the can happily keep their gobs shut from announcing the problem as they're getting paid to be quiet.
Just a thought..
Let the Cisco network defend itself. Just like on 24.
Personally, I'd probably rather the bank/hospital had a few weeks to establish a plan, rather than have to bang something out in an emergency, and whilst the records have already been made much more vulnerable.
Here's what I do: Bitty Browser & Andromeda
The point of buying a router is efficiency. Otherwise get a switch and a 386 running BSD or Linux... Having hardware move packets is almost certainly going to be faster (and efficient) then having a general purpose processor do it.
What do you think a Cisco router is? Traditionally, an underpowered general purpose CPU running a somewhat-specialized operating system.
Unless you're talking about the "big boys" (Catalyst switches, Cisco 10000s, etc) switching is not done in hardware.
Would you similarly welcome the disclosure of a security flaw at your bank, hospital, etc. that granted access to your private/personal records?
Actually, yes I would. I'd much rather they fix or at least stopgap the issue instead of it sitting there wide open for all to see and/or exploit for months.
That was true a few years ago, but its rarely the case these days. Once you contact the correct people at the vendor they generally move fairly quickly to resolve the issue. Independant researchers can contact CERT and they'll handle all of this legwork for you and make sure you get the credit. Of course the patching process still takes time for development, porting across platforms, and regression testing. So you do have to cut the vendors some slack.
In the case of ISS there's almost no excuse for not getting some serious cooperation from the vendor. ISS has the weight and all the contacts they need to notify the vendors and get a fairly quick response. This was either an extreme circumstance, or Michael had another job lined up and he wanted to exit with a big splash. For that matter, he may have just made enough noise about his Blackhat presentation that he didn't want to have to pull it back.
On an entertaining side note, Blackhat actually reburned all the CD's and cut his section out of the convention notes. Cisco must have come down pretty heavy for them to pull such a strong CYA move.
Months? There are outstanding issues on their 2900 switches that have been unfixed there for years.
I don't buy cisco gear anymore.
"It is our belief that the information that Lynn presented at Black Hat this morning is information that was illegally obtained and violated our intellectual-property rights,"
Lynn decompiled Cisco's software for his research and by doing so violated the company's rights, Noh said. [emphasis added]
So basically, Cisco is claiming that decompiling their object code is illegal.
Isn't it a greater violation of the customer's rights to prohibit them from decompiling the code on their own equipment to check for security vulnerabilities?
We've come to the point where corporations believe they have the right to impose conditions of operation on equipment they no longer own. If Cisco sells someone a router, the customer now owns it. Cisco doesn't have any right to impose any conditions of use on the new owner, because they no longer legally own the product. The owner has the right (and some would claim even the responsibility) to decompile their router's code to check for potential vulnerabilities.
It seems that Cisco believes that even after they've sold it to you, they still own your router. And who knows, maybe this vulnerability was deliberately placed so they could own your router anytime they pleased...
The society for a thought-free internet welcomes you.
The filing in US District Court for the Northern District of California asks the court to prevent Lynn and Black Hat from "further disclosing proprietary information belonging to Cisco and ISS," said John Noh, a Cisco spokesman. "It is our belief that the information that Lynn presented at Black Hat this morning is information that was illegally obtained and violated our intellectual-property rights," Noh added.
Ok, let's look at this objectively, shall we? Proprietary information belonging to Cisco and ISS is nonsense. That information should belong to the customers who bought the router so they can take the appropriate steps; for example, a customer should be able to replace an affected router with something else if they're concerned about the problem, or modify the software on the router to alleviate the problem itself (and this is again another example of where OSS is so important).
In terms of violating intellectual property rights, what about violating the property rights of the people who own the router? What rights do they have in this whole situation? Are they expected to sit their with their collective thumbs up their collective asses and wait randomly for a fix? Don't the people who use the routers have the right to uninterrupted network services? What happens if this router belongs to a large ISP and a DoS attack brings the router down? Are they supposed to be stuck with the bill? I'll tell you this much - if this happened, Cisco would never credit them with the cost of service refunds to their end customers. Of course, this would be hypocritical on Cisco's part for obvious reasons, but I digress.
But it only became "wide open" with the public disclosure of exactly how to exploit it.
c'mon... you're telling me that out of 5+ billion people on this planet, that only the person that found the exploit is the one that knows about it?
surely you're not that niaeve?
It seems like a pretty basic concept, but I guess it should be pointed out that just because an exploit hasn't been presented by a security professional at Black Hat doesn't mean there aren't some sleazy Croatian identity thieves (for example) who are abusing this vulnerability left and right.
As long as it's a secret that only a few seriously malicious hackers know, the cost to Cisco is virtually nill. "Oh, your network got hacked? Well, it sure wasn't through your Cisco routers: check it out - we've got zero unpatched known vulnerabilities!" When security holes remain a secret, there is DEFINITELY a cost, but it's shouldered by the users of the product, not the designers. In general, the best way to get the designers to care is to demonstrate to the general public that Cisco is putting their networks at risk.
Not hypothetically, not a month ago, but now. Your networks are being hacked right this minute because Cisco hires sloppy firmware programmers.
Sad, but true.
I'd probably rather the bank/hospital had a few weeks to establish a plan, rather than have to bang something out in an emergency, and whilst the records have already been made much more vulnerable.
Your preference suffers from the flawed (although typically wide-spread) assumtion that only one person is smart enough to discover the flaw.
If a white hat can discover it, then a black hat can too - and black hats are constantly looking. Vulnerabilities need to be *FIXED*, not discussed for weeks in private meetings.
"The injunctions filed against him state that ISS and Cisco had been working together on the flaw for the past four months"
Four months qualifies as a "few weeks" in my mind.How can he be sued if "the problem is not a security vulnerability"
Way to go, Cisco.
The global economy is a great thing until you feel it locally.
And D.J.Bernstein considers immediate release to be the correct way.
All the more reason to wait.
-mkb
c'mon... you're telling me that out of 5+ billion people on this planet, that only the person that found the exploit is the one that knows about it?
We know, from the last time a story about this topic was posted, that Cisco was alerted to the issue and had supposedly "been working on a fix" during that time.
So, no, we aren't that dumb -- what's dumb is that they believe that they can threaten people with lawsuits to keep them quiet.
This is nothing but a corporate scare tactic to keep people from disclosing issues w/their shit in the future.
I'd be far from surprised to hear Cisco were notified of this 3 months ago, hence Lynn's frustration and his decision to publicly talk about the flaw.
Exactly. IIRC from another article this morning, the flaw was disclosed a while ago, I think in April. He publicly announced it on Wednesday July 27th. That's indeed around 3 months.
Using any buffer overflow or similar flaw, he showed how you could take control of the IOS (the OS on the router?). The IOS is supposed to be abstracted from the hardware and immune to this type of flaw.. this wasn't supposed to be possible before. So this flaw isn't tied to a specific low-level buffer-exploit vulnerabilty, so it's not enough to patch that vulnerabilty, because as soon as another is discovered, the IOS will be vulnerable too.
From other posts, it seems Cisco is usually quite reactive to flaw disclosure. Maybe this flaw was bigger and tougher to fix than the usual, but according to a Wired article. CISCO wanted to keep the flaw secret until next year, when a patched IOS beta would be released.
Lynn found this outrageous.
Outrageous enough to quit his job on the spot, burn himself from the industry's eye, and expose himself to a lawsuit from Cisco. Doesn't that make you think?
Misleading titles? Inflammatory blurbs? Keep in mind that Slashdot is a tabloid.
From the (update) article:
"It is our belief that the information that Lynn presented at Black Hat this morning is information that was illegally obtained and violated our intellectual-property rights," Noh added. Lynn decompiled Cisco's software for his research and by doing so violated the company's rights, Noh said.
So, he reverse engineered their software (presumably using demonstrable decompilation techniques) to obtain all or part of the source code which he then studies to ascertain any potential vulnerabilities. Oh dear, this is a violation of their intellectual property.
Please enlighten us Cisco:
Much obliged, do take your time...
Would you consider 5 people with this knowledge "wide open"? 5000?
Here's what I do: Bitty Browser & Andromeda
I'd also like to add that knowing the seriousness and amount of flaws helps a consumer make an informed buying decision. Of course Cisco all but owns the networking market but there are quite a few vendors making inroads. Soon people will have a choice and people will need to rely on more then some PR marketing material supplied by the company to make a decision on what equipment to buy. Responsible disclosure (definition of which varies widely by opinion) is good for consumers and helps to maintain a good balance of power between the users and vendors.
Using the legal system or using any type of mask to prevent or limit disclosure only helps the bottom line of the vendor.
Bad boys rape our young girls but Violet gives willingly.
Ridiculed? They built a backdoor into their product that was such a security flaw that it made IT professionals worldwide look at Cisco in awe. Who the hell would use a master password for a product that's going to be in the server rooms of a thousand businesses?
I don't think "ridiculed" is the right word at all. They deserved the attention that was directed at them, as a master password is no small oversight. That'd be like Windows shipping with a master password.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
This is not a problem of disclosing a major vulnerabilty before the vulnerable company could react.
The flaw had been privately disclosed a few months ago. Cisco, for its own reasons, didn't intend to distribute a fix before long (next year!). Too major a flaw? Publicity? Too much work already? Internal politics?
Obviously, Michael Lynn couldn't live with the idea of leaving this flaw open, and decided to disclose it publicly, thus forcing Cisco to aknowledge it and fix it. Also obviously, this wasn't the only reason. He seemed disgusted by the industry's approach to this kind of problem.
Misleading titles? Inflammatory blurbs? Keep in mind that Slashdot is a tabloid.
I'm always amazed that companies think they have, or do have the right to sue someone for pointing out a flaw in their product. "Only in the software industry". If Chevy sells a new pickup that has seatbelts that don't work properly in a crash, and I find out, damn straight i'm telling the whole world. And if chevy tried to sue me for it they'd get laughed out of court. There should be absolutely no legal grounds for a company to sue someone over pointing out the flaws in their product. It's their own damn fault for not making a secure product in the first place.
I can't remember whether I saw this from the Outer Limits or some other Sci-Fi series, but it was about a guy who discovered that cold-fusion bombs were feasible, and built one. Eventually he was killed, but at the end, some other person also stumble upon the same solution.
I much rather have the security flaw be exposed, and they get to scrambled into a more heightened mode and fix the problem then let it be silent. He discovered the problem publicly, but that doesn't prevent other hackers from knowing the exact same thing.
Can't say for sure. But two points:
A clear case of corruption would be if Cisco tried to "kill the messenger", bury the problem,conceal its existence, so they wouldn't have to spend more resources dealing with it.
I'm not inclined to believe Cisco would do that. Rather, they'd attack the problem with as many resources as they think it deserves.
But in the real world of shades of gray it's hard to determine whether Cisco is working on the bug with all necessary and sufficient expeditious diligence, or they are needlessly and carelessly dragging their feet because fixing the problem looks to be an expensive proposition.
Personally, I think the annual reports of companies like Cisco, MS, Oracle, IBM, Sun, etc. should be required to provide an after-the-fact one-year history of their bug handling, notification, fix, distribution (with all the legal baggage that financial reporting and auditing requires), and how many of their customers' systems were vulnerable, and actually exploited (anonymous is OK there). That kind of full disclosure would provide potential customers with at least the historical information they need to make an informed decision in a functioning free market.
"Provided by the management for your protection."
I'm not assuming that at all. I explained the process in more detail in my previous post (http://it.slashdot.org/comments.pl?sid=157252&cid =13184604 ) but I didn't want to repeat myself. I suppose I should have should have thrown the link in.
The funniest thing though, is that this isn't even a true vulnerability in the strict sense. It demonstrates how to circumvent certain protection mechanisms to build a more reliable exploit for an existing vulnerability. What's more, Cisco was very obviously trying to address the concern, but resolving the issue was taking time. With that in mind, I'm not sure how you can even make the argument that full disclosure was necessary at this time.
you had me at #!
There are various protocols that are directly used by VoIP - these would include things like SIP, UDP connections for the streamed audio and other fairly mundane stuff. For videoconferencing (a related technology), you'd probably use IGMP to set up the multicast conference.
Of these, IGMPv3 (the newest version of IGMP) is the only one the router would really need to talk. It is also a variable-length structure, which means crappy implementations may be subject to buffer overflow. On a liklihood scale of 0-10, where 0 is impossible and 10 is a certainly, I'd put this at a 2 or 3.
There are also indirect protocols used with VoIP. Most VoIP setups that want any decent quality will use bandwidth management schemes, such as QoS. Cisco routers support a number of QoS functions. Some are local, but IIRC, some will propogate between Cisco routers. It could be there is something exploitable in such a mechanism. On the same scale as before, I'd put this at a 4, as I doubt the QoS code has been as extensively tested by consumers or by crackers.
A third option is that it is only tangental to VoIP. The easiest way to secure VoIP is to set up IPSec tunnels. Could there be a flaw in IPSec that can be exploited? There are two candidate areas here - one would be a flaw that made it possible to spoof legit connections without the Cisco router being able to tell. The second, and more serious for Cisco, would be if there's a bug in IKE/ISAKMP where a malformed and/or oversized packet did Really Nasty Things.
Again, IPSec isn't widely deployed so the bulk of the testing it will have received will have been from Cisco itself and not from users (who are always much more creative in creating bizare network scenarios). Of all of the options I've outlined, it would also be the strongest candidate for a follow-on discussion after talking about the security of Internet Telephony. It is also the most complex, in terms of packet exchanges, putting it at a higher risk of having bugs. Again, on the scale I gave, I'll put this at a 6.
Finally, a lot of router technology (not just Cisco's products) are open to ARP cache poisoning, router table poisoning and the like. In a VoIP scenario, these could be used to redirect a call as a means of wiretapping it without duplicating it. This would fall in the category of VoIP security and router security. Normally, routers are set up so that they can't get routing information from anyone. However, one place I worked, I did see a fairly major ISP fry three of its routers with circular routes.
It is possible, then, that Cisco's handling of router-level traffic is suspect - perhaps there's a buffer overflow somewhere that allows escalated priviledges to another networked device. The problem here is that this IS in an area that has been extensively used and tested in the field by Joe Average Customer. And if Joe Average Customer cam crew up, they will screw up.
Knowledge of such a bug would not be kept under wraps, simply because too many people would be experiencing it first-hand. (Same reason Windows bugs aren't secret for long.) So although this is a well-known problem with networks, I would say that the chances of this being the bug Cisco is fighting tooth-and-claw with is about a 2.
The only way we'll know if I'm even remotely close, though, is if Cisco or the researcher says something definite. Either that, or some Black Hat skilled in the Dark Electronic Arts reverse-engineers the defect from what has been said and publishes their observations.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Well, you're right. But I don't think the Mozilla project is a shining star in the security department.
I rather like Daniel Bernstein's policies on his software... publish a verifiable exploit against my software and I'll give you $500.
Ok, how's this, its perfectly reasonable to put out publically his E-mail address at work, but I expect nobody to post photos or personal addresses or wife's name, or anything like that.
... "
*Personal* attacks should never be used, even against someone who might deserve it; it misrepresents our ideology.
However, a personal complaint about corporate policy is perfectly reasonable.
"Why is it that you, representing Cisco said that
- Michael T. Babcock (Yes, I blog)
I am Michael Lynn...I'd like to clarify things
Cisco was notified of the vulnerability in question many months ago and the issue has been patched for about 3 months now.
Furthermore I did not disclose the details of this vulnerability at all. The presentation was merely a demonstration that IOS was exploitable just like any other OS.
It's really funny to see that quote because they ALWAYS tell you to upgrade the IOS no matter what problem is reported to them... classic response from Cisco!
It's not the destination that matters, but rather the journey.
Not sure if you really are Mike, but your facts are 100% correct. It wasn't a new vulnerability, just a new way to exploit a known vulnerability which has already been patched. Also, if I read correctly, you need to be directly connected to the router to execute the vulnerability; it's a not a remote attack.
--- RFC 1149 Compliant.
Last time I looked, there is no such thing as "intellectual property rights". There is copyright law, patent law, and trademark law. These three are commonly grouped as "intellectual property" in the media, but that phrase has no legal standing.
As far as I can tell, no Cisco copyright was violated; no patents were infringed; and no trademarks were fraudulently used. Thus nothing illegal has occurred.
The only remaining possibility in the U.S. is a violation of the DMCA, which Cisco hasn't mentioned. The DMCA is pretty complex, but as far as I can see, the only way it would apply here is if Cisco had encrypted their information and Lynn had decrypted it for commercial purposes. I don't know if compiling source code to object code counts as encryption for the DMCA, and the purposes of the "decryption" are a fair stretch in that context anyway. So I don't see that as being a legal problem here either.
Horrible analogy. Cisco had months of advance notice. There didn't have to "bang something out in a hurry." They simply haven't gotten off their asses and fixed the problem. Microsoft is not the only lazy monopoly in town.
You often hear that, but I wonder if it's always a valid line of reasoning. Do you think it's more of a risk for a few malicious people to possibly know about an exploit while the company takes its time fixing the problem, or for the entire world to definitely know about it while the company scrambles to cobble together a quick fix?
Some security flaws require such detailed technical understanding of the systems involved that not many people are really likely to uncover them. If a professional security researcher with very specialized knowledge who works full time trying to uncover new exploits succeeds in finding something, it doesn't n necessarily follow that many other people will, or even that anyone else will. It's certainly possible that someone else will find it, but I think people should try to balance the possibility of some malicious people knowing about the flaw for a long time against the certainty of everyone knowing about the flaw for a shorter time.
But it only became "wide open" with the public disclosure of exactly how to exploit it.
He used an already patched exploit to show the vuln. He only showed how easy it would be were you to find a new, unpatched exploit.
Also, from an interview at security focus
"It has been confirmed that bad people are working on this (compromising IOS). The right thing to do here is to make sure that everyone knows that it's vulnerable."
The bad guys already know about this, Lynn believes it's time the rest of us found out.
you're all figments of my deranged imagination
It must be a *really* bad hole - they might just as well hang a "crack me" sign on their heads. Either that, or they've hired security experts from Microsoft.
"We are all geniuses when we dream"
- E.M. Cioran
[re "master password thing"]That was from a while back. They had set up a master "backdoor" password in a version of IOS
So since that didn't work, they put a backdoor into the hardware, then slapped a superficial patch on the first (of a number of possible exploits) that has come to public attention. And now they are persecuting the guy who has publicized the underlying flaw, which they have neither patched nor fixed.
So I think it is time for these questions:
I guess I'd better get myself a new tinfoil hat. This one is worn out...
If you read the article you can plainly see that ISS and Cisco have had a restraining order imposed; this is not a "law suit", but it certainly does not preclude them from doing that as well. Disclaimer: I am not a lawyer nor do I play one on TV nor did I stay in a Holiday Inn Select last night.
Please try to stay with the group.
Don't be an ass, turnstyle had a legitimate point. This used to be a problem that a "small number" of black hats could exploit, now it's a problem that a million script kiddies know about. Now don't get me wrong, I'm not trying to claim that cisco was fixing the issue promptly enough, but dissmissing people who point out the problems with full disclosure is just plain irresponsible.
What changed at the last minute?
Makes you kind of wonder who else has known about this vulnerability and told Cisco to dummy up about it.
So again,
BTW, if anybody in a trenchcoat asks, I'm just going for "funny" here... and don't tell them that I'm opening a discount store for tinfoil hats, okay?
Two words "Professional obligation".
There used to be two general ways to handle security flaws when you discovered them. Either you could privately exploit the hell out of them. Or you could just privately report them to the company involved and wait patiently for them to release a fix.
However there is a big problem with this particular model. The problem is that companies like Cisco, Microsoft, etc. don't really seem to think that exploits that allow people to remotely execute administrator level code are really that big of deal, and they figure that they can just create a patch when "we get around to it" or "next year".
Meanwhile, do you really think that you are the only person in the entire world who is guaranteed to find the exploit? The black hats of the world have probably already found the exploit anyway in many cases. It's just the customers who are suffering because a patch is not available.
This model of waiting around forever was a dismal failure. So, security professionals found that by publicly releasing their findings, they could force companies to take security more seriously. The responsible way to do this is to first inform the company privately of your finding, and give them a reasonable chance to fix it.
What you think is reasonable is up to you, *not* them. They are playing by your rules. You are not playing by theirs. Remember, that you are being nice to them by not just publicly releasing the exploit the day that you found it. So, they should respect that. If they do not, that is their problem. Still, as a professional, you should rise above them and try to give them a reasonable time to fix the problem.
Now in this case, what he did was he informed them 4 months ago of the vulnerability along with a proof of concept. They decided not to fix the problem. They claimed there was no problem. He waited patiently for *4 months*. They said that this wasn't really a vulnerability. Then, they knew well in advance of his presentation at Black Hat, and yet they still chose not to fix the problem.
So, what is he supposed to do? As a security professional, it is his ethical obligation to publicly disclose his findings at that point.
In conclusion, Cisco should spend more money on engineers instead of lawyers.
Randy.Flood@RHCE2B.COM
If it was already known and they've already fixed it then why are you being sued?
Any man who afflicts the human race with ideas must be prepared to see them misunderstood. -- H. L. Mencken
The rationale behind why public disclosure of a security flaw (knowing that the 'bad guys' will hear about it too) is based on the idea that (a) customers have a right to know that they are at risk and also need to apply a fix as soon as it's available, and (b) companies should face pressure (even extreme pressure) to prioritize the fixes for these bugs.
It's pretty much accepted across the industry that the disclosure that there is a vulnerability is a "good thing". Indiscriminately revealing the gory details about how to exploit the vulnerability is a "bad thing".
After reading all the articles, it sounds like the exploit was discovered months ago, the patch has been available for months, and though Mr. Lynn demonstrated that the exploit is real (usually required to establish credibility) he did not expose the gory details necessary to allow someone to exploit the attack on their own.
So what's the big deal?
I'm particularly annoyed with Cisco's comment about Mr. Lynn having "illegally" obtained his information. Frankly, it's in the best interest of the public, the Internet, and the security world that security researches will decompile code to search for exploits. The security indsutry accepts that "security through obscurity" is a very bad idea. Vetted code is deemed secure because the gory details have been explosed to a wide audience and *still* no exploits could be found -- NOT because nobody was allowed to know how it all worked.
The choice isn't between some malicious people possibly knowing, and the world definitely knowing. It's between some malicious people possibly knowing now, and some malicious people possibly knowing later.
We've seen this over and over again historically - if there is no disclosure, there is no urgency, so the problem remains unpatched until the worm hits, and then suddenly, after the fox is done raiding the henhouse, steps are taken to close the door.
I don't know if that is the case here - I really have no information at all about the vulnerability, and TFA doesn't tell us anything substantive. But that's the argument for rapid disclosure. The usual rule is to give the responsible party notice, and wait a while to see if they fix it. If they don't, disclose.
If that's what happened here, I'd say Mr. Lynn did the right thing. But again, we really don't know, at least based on TFA, whether that's what actually happened.
As you've already been told, Lynn did NOT work for Cisco, nor does ISS work "for / with" them. The mutual effort was a result of Lynn finding the flaw in the first place, and notifying them about it.
Four months ago.
However, the more damningly flawed portion of your argument is that 'now Cisco doesn't have time to fix the problem'. <snort>
Could you please provide proof that this flaw hasn't been actively exploited since even before the time at which Lynn found it?
It is, needless to say, impossible to prove a negative.
If you're not living on the edge, you're just taking up space!
you are assuming that the security professional is the first one to discover it.
For all he know, it's been exploited for weeks.
Ideally, we could say here is an exploit. In a week I'll release it to the public. Unfortuanatly, he would get sued, and the exploit would go unpatched for a while.
The Kruger Dunning explains most post on
I am in favor of full responsible disclosur (give the vendor a deadline and stick to it unless you KNOW they are moving on it)
Still, most exploits seem to be reverse-engineered from patches. Compare the patch to what came before and you have a serious clue to the problem.
That's in the public world; I don't claim to have any insight into privately held 0-day exploits. I suppose that a there are some blackhats as clever as the white, with equivalent labs.
what's dumb is that they believe that they can threaten people with lawsuits to keep them quiet.
What's dumb is that people sign NDA's and then reveal what they learn. Even if Lynn didn't have an NDA personally, ISS almost certainly did, and he would have been bound by it. In addition, some of the information may have been based on ISS trade secrets, and since he's no longer an employee, he would have no authority to discuss them. So, in this case, a civil lawsuit is absolutely appropriate.
If you and I have a contract that you won't disclose X without my permission, and you tell me you are going to disclose X, what should my reaction be?
Si vis pacem, para bellum
The only thing more annoying than a Libertarian is an (un|mis)informed Libertarian
(As I posted about a year or two ago...)
All corporations (I'm talking about large corporations with hundreds or thousands of employees) are like trains, planes, or other large pieces of equipment. They can not stop and/or turn on a dime. (As the saying goes.)
As in my previous posting on this subject - think of a bus which is going madly down the road at 100mph. Within a mile of where the bus (ie: the company) is is a bridge which has collapsed (ie: the problem). If you start a mile back from the bridge you can easily stop the bus and save everyone (ie: anyone who uses the company's product). If you wait until there is only 1/2 of a mile the bus can still be saved but they might have to slow down a lot faster and they could blow some tires and maybe have an accident. (Thus hurting some of their customers.) Or you could wait until there is only 1/4 of a mile and try to stop the bus. Here, since a bus travelling 100mph travels 100 * (5280ft/60/60) = 146.6666ft per second. It means that the bus has less than 10 seconds to stop. Most porbably, unless the bus driver causes the bus to fall over onto its side - the bus will most likely go over the bridge and kill everyone.
The same holds true for talking about problems in ANY WAY, SHAPE, or FORM when it comes to computer software or computer hardware. You can't just jump out there and start screaming there is a problem because the bus can't stop that fast to prevent disaster. Nor can you tell a company about a problem, wait a couple of hours, days, or even weeks and get mad because nothing has been done. It takes a while to bring the bus to a stop, pick up on what you have to say, and then to start back up again.
What's a good rule of thumb? Three to six months minimum depending upon how severe the problem is. If it is just a one or two line coding problem - three months. If it is a major change due to parts of a program having to be either completely re-written or major portions having to be changed - six months. And remember - that is a MINIMUM requirement. Normal length of time to fix? More probably two to three times those minimums. That's because you are not the only person who may have found a problem as well as the fact that they are trying to put in new features that have been requested. The same people work on both things at the same time.
So people who find problems need to think in months - not weeks, days, hours, minutes, or seconds. Because that is how long it will take to fix a problem. In fact, sometimes something that looks really simple turns out to be a real mess to fix. It all depends upon the way in which some software was originally written. So you can't base how fast the company fixes something by what you may think is a fair amount of time. You just need to be patient while the company does what it can to fix the problem.
Now, as for the company - it is extremely important for companies to keep everyone up-to-date on any/all progress made to fix a certain problem. This can even be automated somewhat. But it is very important not to try to hide the problem because as anyone knows - that is what gets a company in trouble. Trying to hide things that is.
Someone put a black hole in my pocket and now I'm broke.
Great. The problem is a flaw in BGP that affects every router that implements it. It allows certain messages to cause a DOS attack on certain IP addresses. Tell me how long it will take to fix. By the way, if you're wrong on the time estimate, everyone is going to jump all over you. And if the time period is too long, everyone is going to jump all over you. Also, you can't make everyone upgrade at the same time, so your solution has to be backwards and forwards compatible. Well? I'm waiting.
Intron: the portion of DNA which expresses nothing useful.
If you are him, I would like to ask you: why did you feel it was worth losing your job so that you could talk about an exploit that had already been fixed that, on top of that, could not be completed remotely? The odds of this being a real problem seem pretty low to me.
ZDnet reports that David Lynn and Cisco have agreed to a legal settlement. Lynn doesnt't talk about the matter at Blackhat or Defcon and returns all related material to Cisco. I suppose Cisco drops its charges against him, though that's not mentioned.
I'm glad for Michael Lynn that this affair ended quickly and not too harshly. Kudos to him for his courage.