Slashdot Mirror

← Back to Stories (view on slashdot.org)

Cisco Patches 'Black Hat' IOS Flaw

Posted by Zonk on from the spraypainted-white dept.

thursnick writes "eWeek is reporting that Cisco has finally issued a comprehensive fix for a critical IOS vulnerability that set off a firestorm of controversy at the Black Hat Briefings earlier this year. The patches come more than three months after former ISS researcher Michael Lynn quit his job to present the first-ever example of exploit shellcode in Cisco IOS (Internetwork Operating System), a presentation that landed him in legal hot water. Cisco's advisory effectively confirmed Lynn's summer warning that the flaw could be exploited by remote attackers to execute arbitrary commands or cause a denial-of-service on compromised routers."

66 comments

  1. Wow, quick turnaround... by Stormeh · · Score: 3, Funny

    Awesome, and it's only been how many months?

    1. Re:Wow, quick turnaround... by Conare · · Score: 1

      If I hear "Oh security doesn't matter because it's a special operating system" someone is going to get such a pinch. arcing!

      --
      Stop Continental Drift! Reunite Gondwanaland!
  2. Why not earlier? by ParrotAtSlashdot · · Score: 2, Insightful

    Why on earth did Cisco not release this earlier? It would save people alot of trouble.

    --
    ParrotAtSlashdot
    1. Re:Why not earlier? by Anonymous Coward · · Score: 0

      1. Which trouble exactly ? This is a method to exploit other vulnerabilities. Those other vulnerabilities (can) have been patched independently.

      2. Earlier > Look at the advisorty, more precisely the "Software verions and fixes"

    2. Re:Why not earlier? by scheme · · Score: 5, Informative
      Why on earth did Cisco not release this earlier? It would save people alot of trouble.

      If you read TFA, the bug involved system timers and how they were handled. Given that this probably affects most of the system functions, it's not surprising that it would take a while to make the changes and test it. Think about how long it took to fix the VM bugs in linux 2.4, this probably a change of similar magnitude.

      --
      "When you sit with a nice girl for two hours, it seems like two minutes. When you sit on a hot stove for two minutes, it
    3. Re:Why not earlier? by seva · · Score: 1
      This is a related problem, but it's not the same one:
      This advisory documents changes to Cisco IOS® as a result of continued research related to the demonstration of the exploit for another vulnerability which occurred in July 2005 at the Black Hat USA Conference. Cisco addressed the IPv6 attack vector used in that demonstration in a separate advisory published on July 29, 2005.
  3. Cisco vs. Microsoft by mandreko · · Score: 4, Funny

    looks like Cisco is trying to beat Microsoft for patch times

    1. Re:Cisco vs. Microsoft by Tackhead · · Score: 2, Funny
      > looks like Cisco is trying to beat Microsoft for patch times

      From: <billgatus.of.borg>
      To: <ceo.of.cisco>

      "Johnny, you're doin' a heck of a job!"

  4. Black hats, white hats, Red Hats by picz+plz+2 · · Score: 1, Funny

    When did geeks become gangsters? WTF dude! Today's secret word is: arcing. For the rest of the day, whenever anybody says the secret word, scream real loud!

  5. hrmmm by PetyrRahl · · Score: 1

    /me wonders if this is just described as "A patched undisclosed vuln. of low priority" or some such rot in the update... Petyr Rahl

  6. patching ciscos... by Anonymous Coward · · Score: 2, Informative

    So now we can all visit CiscoUpdate and have our routers automatically patched....?

    Or do we have to manually evaluate lengthy decision diagrams, check memory requirements, prove that we have legally bought the affected hardware and software, and hope that the monolythic IOS image will not introduce bugs into other areas that are being patched by this fix?

    1. Re:patching ciscos... by Anonymous Coward · · Score: 0
      So now we can all visit CiscoUpdate and have our routers automatically patched....?

      Or do we have to manually evaluate lengthy decision diagrams, check memory requirements, prove that we have legally bought the affected hardware and software, and hope that the monolythic IOS image will not introduce bugs into other areas that are being patched by this fix?

      the latter

    2. Re:patching ciscos... by m0rningstar · · Score: 1

      You have to follow that requirement, today.

      (Not that non-monolithic systems are necessarily exempt from the patch breaking other systems)

      However (while off topic), it should be noted that 12.2XR (7600 only, today, but where else are you going to see this level of change) is no longer monolithic. It's a HUGE change in the architecture brought about to address just the type of issues discussed.

  7. So much for by scenestar · · Score: 0

    My plans for world domination.

    (we are so dependant on these routers it's just scary)

    --
    perpetually dwelling in the -1 pits
  8. What ever happened... by Anonymous Coward · · Score: 5, Interesting

    So, what ever happened to Michael Lynn? He quit his job and made the presentation but, where is he today? Is he employed? Is he proud of what he did? Does he feel the price he paid was worth what he gave up for 15 minutes in the spot light? Would he recommend his "high road" choice to others in the future? Does he feel that he really made any difference in the end?

    1. Re:What ever happened... by Ckwop · · Score: 4, Informative

      He's alive and well as far as I know. I saw him at Toorcon this year, but didn't speak to him.. (He was a speaker and gave a good talk on Reverse Engineering)

      I know that he has a new job and I while I obviously can't speak for him, I got the impression that he felt as if he did his duty the security community. As an amateur member of that community, I'd thought that he put principle before pay and deserves our respect.

      Simon.

    2. Re:What ever happened... by Anonymous Coward · · Score: 0

      I saw a homeless guy that looked just like him outside the library. I'm not sure if it was him or not but I did see he was carrying a copy of the book "Tropic of Cancer" with him.

    3. Re:What ever happened... by Anonymous Coward · · Score: 5, Informative

      Mike is working at Juniper, and doing well (Juniper pays better than ISS, apparently, and their code is cleaner than Cisco's, plus they have some ethics). He feels he did the right thing. So do a lot of folks in the US military and intelligence communities, who are very very pissed off at Cisco for exposing them to a security risk of this magnitude and trying to cover it up. They consider Mike a hero, so he has some very useful new friends...

    4. Re:What ever happened... by Anonymous Coward · · Score: 0

      CAN'T STANDYA!

    5. Re:What ever happened... by Wellspring · · Score: 3, Insightful

      I'm glad. I love it when the right thing (for him) is also the Right Thing (ethically).

      The coverup is almost always worse than the crime in these kinds of things. Companies that aren't up-front and honest (trying to protect their reputation) end up trashing their reps. Cisco just created an anecdote for the next time a customer or regulator wants to take a deep, careful look at their security. We can't just take their word for it, and if I were buying routers right now, I'd be much more inclined to look at Juniper than Cisco, even though previously I wouldn't have even considered them.

      It's not magic pixie dust, but making the effort to bring hard-core ethics onstaff is important to me.

    6. Re:What ever happened... by Anonymous Coward · · Score: 0

      I just checked with a mutual friend who knows Mike. Apparently Mike's job is going over JunOS and Netscreen source code, cleaning it and securing it. More points in Juniper's favor, as far as I'm concerned.

  9. Re: by V_Pundit · · Score: 1

    arcing

    --
    that's how I see it anyway . . .
  10. Seems CISCO should be in hot water.. by TheCeltic · · Score: 1

    Seems Cisco should be in legal hot water not Lynn. Why would we "shoot the messenger"? Kinda like blaming the little boy in "the emperors new clothes".

    --
    =-=-=-=-=-=-=-= - The Celtic - =-=-=-=-=-=-=-=
    1. Re:Seems CISCO should be in hot water.. by egypt_jimbob · · Score: 1

      Lynn was (is?) in hot water because he signed Non-Disclosure Agreements with Cisco and ISS. When he said, "I'm making this public," Cisco said, "If you do, we'll sue" and ISS said, "If you do, you're fired." He did and they did and he was.

      --
      I am a leaf on the wind. Watch how I soar.
    2. Re:Seems CISCO should be in hot water.. by Anonymous Coward · · Score: 0

      He did and they did and he was.

      Not exactly.

      Lynn did sign NDAs but the validity of such NDAs WRT whistleblowing, especially when ordered to lie by his then employer, is in question. Keep in mind that the talk had been planned weeks (months?) in advance, with both Cisco's and ISS's blessings. Cisco didn't spring the surprise C&D until 48hrs before the conference.

      The criminal charges initiated by Cisco were dismissed with prejudice by the judge almost immediately. I believe that the civil charges were settled shortly afterwards.

      Lynn quit his job with ISS a few hours before he gave his talk at Blackhat; he was not fired. ISS later claimed that they fired Lynn, which is not true; that was simply PR spin and ass covering on the part of ISS.

  11. Initial problem already fixed? by rednalb · · Score: 1

    It looks like this patch adds countermeasures to the original patch for this problem back in July? Here was the initial patch for this problem.

    1. Re:Initial problem already fixed? by just_another_sean · · Score: 1

      Thank you. I was wondering about this. Jokes about "keeping up with Microsoft" aside I believe Cisco did provide an initial patch and this is a much more comprehensive patch that actually fixes the core problem.

      Everyone always bashes MS for releasing patches that simply prevent a working/known exploit w/o fixing the core issue. Cisco has done both now and even though I think they handled this very poorly on both the PR front, and with their behaviour towards Lynn, I do applaud them for taking the two step approach and actually fixing the core issue with their products.

      --
      Creationist Textbook Stickers Declared Unconstitutional by CowboyNeal
  12. The question is....... by 8127972 · · Score: 3, Interesting

    ..... Is this safe enough to deploy or should it be dropped into a test environment of some sort before deploying into a production environment? That assumes of course that admins have the luxury of delaying the deployment of this.

    --
    This is my opinion. To make sure you don't steal it, it's covered by the DMCA.
    1. Re:The question is....... by anticypher · · Score: 5, Informative

      The answer is.....

      This code has been out for a few months now, and many select beta sites have been testing it in production environments. The first few iterations had some serious (crash and reboot every few hours) problems, but it (12.2.15T1thru17) has been in production use on several edge routers for a month with no noticable problems. Cisco didn't just patch the one 'sploit published, they categorised the class of exploits and went about fixing many different possible attack vectors or watching for suspicious behaviour that could indicate a compromised system. That is what took several months even before Michael's talk, and its been in testing (and re-patching and recursion testing) since then. The announcement today is because they are confident their fix is solid, but anyone staying at the bleeding edge of IOS releases has been using it since at least June.

      I'd say its solid, but I'm not rolling out the latest version on everything until others add some real world stress testing. I'm sure there will be several more newly introduced bugs uncovered in the new few months, and the timer checks usually result in a panic reload, not optimal for stable systems with SLAs and big money riding on them.

      I'm also not in a rush to roll this out, because for the moment there are no known exploits running around. Maybe Effugas or some of the IOS engineers (I know you read /.) can add something to this thread.

      the AC

      --
      Hemos is like...sci-fi fans;he thinks technology is cool, but he hasn't bothered to understand the science it's based on
    2. Re:The question is....... by estebanf · · Score: 1

      so... you actually consider deploying a patch in production without testing it?.....Thanks god I'm not one of your lan users

      --
      DON'T STEAL MUSIC!
    3. Re:The question is....... by cat6509 · · Score: 2

      yeah you know, i have a *ton* of those spare $50,000.00 TEST routers to throw at the lab.....

      --
      "Tolerance is a virtue of a man without convictions." G.K.Chesterton
  13. Great by Atlantic+Wall · · Score: 3, Funny

    Great, Now how long before everyone implements this and all of the other patches that need to be done on the cisco routers. OK the patch is out, but when will they all be patched, probably another 3-6 mo. So this is a hackers last call sort of, if you have not exploited this yet, time is running out, soon. So get in ur haxoring.

    --
    To Hell with the Queen of England!
  14. Boy oh boy by MightyMartian · · Score: 2, Funny

    Do I feel bad about abandoning Cisco for Linux and IPTables. I mean, there's nothing quite as fun as upgrading Cisco's IOS. It's right up there with root canals in my book of things I like to experience.

    --
    The world's burning. Moped Jesus spotted on I50. Details at 11.
    1. Re:Boy oh boy by Anonymous Coward · · Score: 0

      From my experience, (i have hollow teeth, they tend to rot themselves away from the inside out) root canals over the past few years have gotten more and more painless. My first one back in the eighties took me out for about 4 days with pain. The last one i got a couple of years ago didn't really hurt much at all, took the first dose of medicine and didn't have any pain after that and didn't take any more painkillers.

      YMMV

    2. Re:Boy oh boy by soellman · · Score: 1

      heh.. hell of a lot easier than say.. linux? even an operation as simple as "emerge system" takes forever and could end up in a broken box.

      copy flash tftp
      copy run tftp
      copy tftp flash
      reload

      not too bad..

    3. Re:Boy oh boy by sharkey · · Score: 1
      You left out Steps 5, 6 and 7:

      Remove fan grill
      Insert penis
      Pray that the networking gods accept the sacrifice placed before them

      --

      --
      "Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
    4. Re:Boy oh boy by MightyMartian · · Score: 1

      And, if that fails, step 8 where you get to sit in Cisco's queue for six hours waiting for someone to offer you technical support, only to be told "Are you certain you've paid for support, sir?"

      --
      The world's burning. Moped Jesus spotted on I50. Details at 11.
  15. Two scary bits" Completely Compromised by fuzzy12345 · · Score: 2, Interesting
    From TfineA:

    "In many cases, a heap-based overflow in Cisco IOS will simply corrupt system memory and trigger a system reload when detected by the "Check Heaps" process, which constantly monitors for such memory corruption."

    Is anyone else bothered that Cisco figures heap corruption is common enough that a process is running full time on production routers looking for it? I suppose you could view this as proactive, but obviously the process can only look for nonmalicious corruption, and is only statistically likely to find corruption before it causes errors according to how much CPU you give it.

    "In some cases it is possible to overwrite areas of system memory and execute arbitrary code from those locations. In the event of successful remote code execution, device integrity will have been completely compromised,"

    Think about it. Once an exploit is executed against your router, reloading your firmware isn't an option, because that's a function of your firmware, which could be corrupted. Unlike a computer OS virus, which can be circumvented by rebooting and taking control before the corrupted OS does, there's no way to preempt the corruption here. For total peace of mind, you'd either have to replace the (probably not socketed) flash chips, or take the whole unit out back and burn it. Am I wrong? Of course, that's not going to be Cisco's recommended solution.

    --

    Everybody's a libertarian 'till their neighbour's becomes a crack house.
    1. Re:Two scary bits" Completely Compromised by fimbulvetr · · Score: 1

      Is anyone else bothered that Cisco figures heap corruption is common enough

      I'm bothered. In fact, this reminds me of how DJB's software almost always users the supervise daemon to ensure your process is running. It keeps track of all of djb's software, and you can run it with most other daemons.

      What happened to writing good software? Why should you have a daemon check for corruption? Why should you have a daemon that checks to see if other daemons die? Wait! Wasn't the author of the daemon-that-might-die software also the same author of the daemon-that-checks-if-it-dies software? That sounds pretty funky.

      Does cisco's corruption checking software check its own heap?

      Write good software the first time, and you won't have to kludge the kludges.

    2. Re:Two scary bits" Completely Compromised by gclef · · Score: 3, Informative

      Cisco doing heap checking is a mark of a reasonable system doing checks on itself. Why is this bad? They almost never use the stack, so they check the memory they are using a lot. It doesn't run often (Lynn found it running about once every 30 seconds or so), and it's a good thing to do. Why complain?

      As for reloading firmware, I don't think you understand Cisco stuff. There is a mini-firmware burned into ROM on all the Routers & Switches...it's called ROMMON mode on the ones that immediately come to mind. If your device firmware is totally thrashed (by a worm, by some damn fool tftp'ing up an image for the wrong router type, etc) you'd just use ROMMON mode to re-load a good image. Now, the real problem is that a worm could trash your flash storage.

      In that case, unless you've got one of the expensive boxes with removable flash cards, you've now got a very expensive paperweight.

    3. Re:Two scary bits" Completely Compromised by silas_moeckel · · Score: 1

      Memory coruption happens it's a function of radiation and ECC does not fix/catch all of it. Routers have uptimes counted in years unlike your average PC or windows server so yes it does make a bit of sence.

      Well most cisco routers have socketed and/or slot based flash. The slot based ones have these realy cute write protect switches on the end.

      --
      No sir I dont like it.
    4. Re:Two scary bits" Completely Compromised by Anonymous Coward · · Score: 1, Interesting

      Cisco doing heap checking is a mark of a reasonable system doing checks on itself.

      It's a mark of a *bad* system. Why? Because 1) it means they believe they haven't properly written their software and, more importantly, 2) it doesn't guarantee you anything except "the heap was consistent up to 29 seconds ago". Who cares? I need the heap to be consistent *all* the time.

      The best thing is to just write the code correctly, the next best thing is to place some kind of "barrier" (at the hardware level? Who knows) between all memory access and the program that guarantees things are okay 100% percent of the time.

      It's like a DBMS without consistency checks, and when you point it out to the programmer, he writes a script that checks the consistency every 30 seconds. Well, what does it mean when the script reports an error? It means you might be totally screwed! It doesn't *solve* the problem.

      Cisco has shown that they aren't up to the task, so I'm not surprised, but don't fool yourself, this isn't "defense in depth" or anything like that. Defense in depth (to me) means, you have multiple layers, each capable *by itself* of defending the system. For instance, run your software in a chroot jail AND set resource limits so it can't open any files AND use a string library that's immune to buffer overflows. It doesn't mean, write some crap and then check it every 5 minutes with Tripwire.

    5. Re:Two scary bits" Completely Compromised by DrSkwid · · Score: 1

      Supervise the process list.
      Process external input.
      Exceptions are exceptional.
      Expect the unexpected.

      --
      There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
    6. Re:Two scary bits" Completely Compromised by Anonymous Coward · · Score: 0

      Cisco routers have a bootloader that can load new firmware as well.
      The flash module holding the firmware can be replaced, but not always very easily.
      In the larger/newer models it is one of those "flash IDE disk" boards, in older routers it is a DIMM.

  16. "First-ever exploit" by Anonymous Coward · · Score: 1, Informative

    The patches come more than three months after former ISS researcher Michael Lynn quit his job to present the first-ever example of exploit shellcode in Cisco IOS (Internetwork Operating System), a presentation that landed him in legal hot water. Cisco's advisory effectively confirmed Lynn's summer warning that the flaw could be exploited by remote attackers to execute arbitrary commands or cause a denial-of-service on compromised routers."


    It was not the first-ever example of exploit shellcode in IOS, Phenoelit already made public some Proof-of-Concept IOS exploits in the past. Phrack 60 #7
  17. Cisco not aware of an exploit? by Alejo · · Score: 1

    "Cisco is not aware of any active exploitation of this vulnerability"
    Right.

    1. Re:Cisco not aware of an exploit? by Anonymous Coward · · Score: 0

      Can you read? They aren't denying the exploit they are saying they aren't aware of anyone activly using the exploit to compromise routers.. completely different and accurate imo.

  18. Am I affected? by gfilion · · Score: 1

    Hey, I'm all mixed up with this advisory.

    My router has version 12.4(2)T1, is it affected? The advisory says that all version are affected, but it seems to propose version 12.4(2)T1 as a fix.

    Could someone shed some light one this?

    1. Re:Am I affected? by estebanf · · Score: 3, Funny

      Give me your ip... i'll tell you :)

      --
      DON'T STEAL MUSIC!
    2. Re:Am I affected? by Guybrush19 · · Score: 2, Informative

      You aren't vulnerable. The bug was integrated in 12.4(2)T1, so you already have the fix. Older 12.4T versions will be vulnerable, such as 12.4(2)T.

  19. Alrightythen boys and girls... by kurbchekt · · Score: 0

    ARCING!!!

  20. Re: by Anonymous Coward · · Score: 0

    DORYAAAAAAAAAAAAAAAAAA!!!!

  21. And they make it so damn hard to upgrade! by Anonymous Coward · · Score: 0

    I've already wasted five hours on the phone with them trying to get an upgrade for our six routers. If you don't have SmartNet they make it such a pain to upgrade even though they claim they will always provide security upgrades for free. I just want the file so I can try booting with it via TFTP with one of the routers. So, anyone have a web page you can go to download the files? I've given-up on cisco's phone service. They used to be the best in the business.

  22. Great news by Anonymous Coward · · Score: 0

    At least now he works for Juniper. Thats great. Next time he gets that weird idea he calls "ethics" and starts publishing critical vulns without working with the vendor we got nothing to worry about. Juniper's market penetration is nowhere close to Cisco's. As for Lynn, i share that opinion http://technolustandsushi.blogspot.com/#1123076679 49080502

    1. Re:Great news by abaddon314159 · · Score: 2, Informative

      thats funny; it never fails to amaze how many people can't be bothered to read the actual body of an article before commenting on it...

      I'm Michael Lynn, so I know a thing or two about what went on...I DID NOT release any bug details, I DID work with the vendor, the bug in question was patched months before I went on stage as a result of my working with PSIRT, and when I went on stage I didn't disclose any details about any bug...all I did was prove it was possible to exploit bugs on IOS...

      If you don't believe me, then go and find out the exact nature of the vulnerability...you won't be able to do it (at least not without disassembling the thing yourself and rediscovering it) because I never disclosed it to the public...furthermore I disclosed it to the vendor months in advance, waited for them to get a fix out, worked with them all the way until about 48 hours before the talk...they were even going to co-present with me, then someone changed their mind and went into panic mode...

      --Michael Lynn

    2. Re:Great news by m0rningstar · · Score: 1

      As someone who works with Ciscos and works for a Cisco Gold Partner:

      Thanks for going public. I hope you get a chance to read this, but as far as I'm concerned and will tell anyone who asks that you did everything right and it was Cisco who screwed themselves.

      I've heard few people say differently, either.

  23. Hold on a second here... by schon · · Score: 1

    Think about how long it took to fix the VM bugs in linux 2.4

    They fixed the VM bugs in 2.4? :o) /me ducks

  24. Upgrade Procedure by green+pizza · · Score: 1

    Send an email to tac@cisco.com requesting the security update. They will reply with a short list of "REQUIRED INFORMATION". Email this back with the info requested (router serial number, current IOS version, your contact info) and they will send you a download link.

    At least that's how it worked for me this morning. The entire process took less than 2 hours from initial email to downloading the updated version of IOS.

    BTW: be sure to quote the advisory URL in all of your emails to Cisco.

    1. Re:Upgrade Procedure by Anonymous Coward · · Score: 0

      Thanks for the reply. I did that, but they don't have a version of IOS that's fixed that will run on any of our six routers. IOS is getting very bloated. I guess now I have to see about borrowing money to buy new routers. Expect to see a bunch of very cheap and useless ciscos for sale on Ebay RSN(real soon now). Either that or I might just wait and hope for the best. Damn cisco.