Slashdot Mirror


Major Flaw Found In Cisco IOS Devices

Joff_NZ writes "CERT has released an advisory regarding a serious flaw in all Cisco routers and switches which run IOS and process IPv4 packets (i.e. pretty much everything), which causes the device to stop processing inbound packets, and so: 'The device must be rebooted to clear the input queue on the interface, and will not reload without user intervention.' There are apparently no known exploits (yet), and Cisco have this advisory with a workaround and available fixes."

43 of 266 comments (clear)

  1. Yet... by jerw134 · · Score: 5, Insightful

    There are apparently no known exploits (yet)

    I say we start a pool on how long yet will actually be, now that CERT released the info.

    1. Re:Yet... by jamesh · · Score: 3, Insightful

      I couldn't glean from the article exactly what packet would cause the failure. The ACL that was given as a workaround permitted typical protocols (eg tcp, udp, icmp, etc) and blocked the rest. Presumably somewhere in 'the rest' lies the exploit but it's a big space to search.

    2. Re:Yet... by sleeper0 · · Score: 5, Funny

      Between this announcement and the microsoft one I know at least one of the fine readers out there has cancelled all of their appointments for the next three days and has a case of mountain dew and a copy of worms for dummies under their arm whistling happily.

    3. Re:Yet... by rf0 · · Score: 5, Funny

      I'm going to say an exploit by tommorow. End of the internet by Sat. All back to normal on Monday

      Rus

    4. Re:Yet... by Anonymous Coward · · Score: 3, Funny

      I call a book that can whistle pretty damn smart - knock off the 'dummies' stuff, mmkay?

    5. Re:Yet... by cscx · · Score: 4, Funny

      Yeah you would think more people would be using Juniper routers; however, I think they creepy-looking lady on their homepage is a deterrent.

    6. Re:Yet... by bigberk · · Score: 3, Funny

      Where the hell did you find that? On my new big LCD monitor it looks like this lady is in the room right beside me. I'm not going to be able to sleep.

    7. Re:Yet... by Anonymous Coward · · Score: 5, Interesting

      They have an awesome colletion of Anti-Cisco cartoons :)

      I think this one is on of the best:
      http://www.juniper.net/nettoons/03_1280.jpg

      (Just change the first number)

    8. Re:Yet... by Anonymous Coward · · Score: 5, Informative

      ok folks, here's how it works. A specially crafted packet is sent to an interface on a router. This packet takes up space in the queue on the interface. Once a few of these packets fill up that queue no more traffic is able to pass thru the interface. You won't see a high utilization on the CPU, it'll just throw'em away. It's important to understand that the packet has to be directed to the interface on the router, not just merely passing through it. After the queue fills up (around 4k I'm thinking)the only way to empty it is to reload, if I'm reading correctly. From what I can tell, the large back bones got the notice a few days ago. Some lower tier players received it yesterday. And public disclosure supposed to happen tonite around 21:00 EDT or so. However, several major internet players all of a sudden performing emergency maintenance, was a bit obvious. Especially when companies known to employ lots of Juniper didn't seem to do much. Well, guess it wasn't that OBVIOUS, but...net-eng people are worse than a small town knitting group.

    9. Re:Yet... by losmurfs · · Score: 3, Funny

      Just another example of using sex to sell products.

  2. It's days like this... by Nethead · · Score: 5, Funny

    It's days like this I'm REALLY glad that I'm a unemployyed network engineer! This looks like a very serious headache!

    --
    -- I have a private email server in my basement.
    1. Re:It's days like this... by rf0 · · Score: 4, Interesting

      I remeber the day Bind 8.2.2-P5 had an exploit come live. 24 hours and 56 servers later I finally managed to get to bed. Only to have to upgrade it all again a few days later.

      fun

      Rus

    2. Re:It's days like this... by Anonymous Coward · · Score: 4, Funny

      Pfft, you're a "network engineer" just like I'm a "computer surgeon" and that guy over there is an "electronics astronaut".

  3. Alternative by rf0 · · Score: 4, Funny

    This is why I always suggest alternatives to Cisco such as IP over Avian and actual implementaion on Linux

    Rus

    1. Re:Alternative by mrbill · · Score: 3, Funny

      Yeah, but with IP over Avian Carriers, your "denial of service" is called "bubba with a shotgun".

  4. Whoa, very interesting!! by Anonymous Coward · · Score: 3, Interesting
    CERT has released an advisory regarding a serious flaw in all Cisco routers and switches which run IOS and process IPv4 packets (i.e. pretty much everything), which causes the device to stop processing inbound packets, and so: 'The device must be rebooted to clear the input queue on the interface, and will not reload without user intervention.' There are apparently no known exploits (yet)
    I wouldn't be so sure of that. A couple of days ago, my cable modem (and others in the area) started having problems where the connection randomly drops and it takes awhile to get it back. As if maybe a router somewhere has gone down and needs rebooted..

    If I fire up Ethereal to peek at the traffic, I notice that the arp who-has requests are labeled with a source of "cisco_f2" ...

    Wonder if someone has been pointing this sploit at cablemodem routers ??
    1. Re:Whoa, very interesting!! by MrMickS · · Score: 4, Informative

      Cisco Cable Modems run a version of IOS. However they have private IP addresses on the cable side and pass thru the DHCP requests that your device(s) make to the providers DHCP server. Unless your cable provider's network has been compromised I doubt that this is related to your problem.

      --
      You may think me a tired, old, cynic. I'd have to disagree about the tired bit.
  5. No Exploits My A$$ by Anonymous Coward · · Score: 5, Interesting

    AT&T has been having problems all over the west coast the last 4 days. Ill bet even money this is why. There last 2 emails state they had no clue what was causing it and that random reboot's of routers were to be expected.

    Im not Anonymous, Just Lazy.
    Crackers`n`Soup

  6. Re:Disclosure of vulnerabilities by eskimoboy · · Score: 5, Insightful

    Sometimes, it's in the best interest of the public to have vulnerability information released directly when it is found out. It opens up the ability for hackers to create exploits before the manufacturers have a chance to find a way to stop it. Sure, releasing information on vulnerabilities for open source projects right away is usually a good idea, but that's due to the fact that with an open source project, the public has the ability to come up with a patch. In cases like these, perhaps it is best for the public to be left out until a proper solution or workaround has been developed by the vendors.

  7. At least it won't worm. by Valar · · Score: 5, Insightful

    At least it only freezes the device. If you could make it send the same packet to some of it's router buddies, then freeze, this could get real bad, real fast.

  8. The ACL "fix" is not a fix by jgaynor · · Score: 5, Interesting

    Here's the reccomendation for a temporary workaround using ACls:

    Cisco recommends that all IOS devices which process IPv4 packets be configured to block traffic directed to the router from any unauthorized source with the use of Access Control Lists (ACLs). Legitimate traffic is defined as management protocols such as telnet, snmp or ssh, and configured routing protocols from explicitly allowed peers. All other traffic destined to the device should be blocked at the input interface.

    Does "A rare sequence of crafted IPv4 packets sent directly to the device" mean a sequence utilizing one of these three protocols? If so then frigging tell us! If not, this is just a vague precautionary warning that really won't stop any user inside the network from exploiting the bug.

    The TRUE details of the bug, including which protocol it uses, would help us put a nail in the coffin regarding the ACL workaround, but the Cisco bug tool isn't returning any information for the bugs they're talking about - specifically CSCea02355 and CSCdz71127.

  9. Latest news .... by Snoopy77 · · Score: 5, Funny

    While the army took time to celebrate the discover and safe return of Major Flaw it still maintained the need to continue the search for other missing top ranking officials. We spoke with a member of the search and recovery team soon after Major Flaw was discovered.

    "It is great to have found Major Flaw but we are still very worried about the others. Our job here is not finished." said Private Data.

    Colonel Panic has been spotted from time to time but the army has not yet been able to pinpoint his exact position. But the most gravest of fears are held for General Protection-Fault. Sightings of the General have been few and far between in the last few years. Some conspiracy theorists say that he is not actually missing but has disguised himself. Private Data would not confirm wether they are searching for a man of similar build to General Protection-Fault but dressed all in blue.

    --
    "She's a West Texas girl, just like me" - G.W Bush Iraqis
  10. There ARE exploits in the wild by Anonymous Coward · · Score: 5, Interesting

    The claim that there are no exploits is false.

    Below is a note I received from my ISP about 2 hours before this was topic posted:

    =-=-=-=-=-=-=-=-=-=-=

    17/07/03 01.12 - 01.38 DOS Attack on Sydney PoPs

    Incident

    A DoS attack against the AN border router resulted in that router's CPU reaching 100% and triggering the same attack on the Perth gateway router which in turn brought down the Comindico Border router

    Action

    While all of the hardware remained 'up' nothing could be authenticated and therefore all traffic through the Sydney PoP ceased.

    Resolution

    Swiftel Engineering rebooted the Perth Gateway router clearing the DoS packets and that in turn allowed the Sydney routers to rebuild the BGP4 tables thus restoring the ability to process customer traffic.

    Result

    By 1.38 pm all traffic was flowing normally.

    Future Elimination Of This Problem

    The elimination of this type of new DoS attack has just been recognised and released by Cisco (today) and the workaround and fixes are documented in:

    http://www.cisco.com/warp/public/707/cisco-sa-20 03 0717-blocked.shtml

    We are considering whether to implement the workarounds which may impact traffic such as ICQ and some games or upgrade the IOS's in all of our Cisco equipment.

    We will inform you when that decision is made.

    1. Re:There ARE exploits in the wild by forged · · Score: 3, Informative
      You don't understand: this isn't a DOS attack that Cisco is warning about.

      Once the input queue is full of said packets, the router doesn't accept any more packets, then CPU utilization drops at 0% while the router idles waiting for more apckets (which of course never arrive once the device is blocked).

    2. Re:There ARE exploits in the wild by Florian+Weimer · · Score: 3, Informative

      According to Cisco, high CPU utilization is not a result of the present defect.

      The ISP was probably experiencing an ordinary DoS attack.

  11. Department of Homeland Security is interested! by dekashizl · · Score: 5, Funny

    This is actually good news for Cisco, because security holes like this appear to be a prerequisite for getting a large Department of Homeland Security contract.

  12. Re:Yikes... by Grizzletooth · · Score: 5, Informative

    No, the advisory states that non-contract customers can send an email to tac@cisco.com and get access to a "free upgrade".

  13. "A great disturbance..." by mino · · Score: 4, Funny

    "Like millions of sysadmins cried out in terror -- then were silenced."

  14. wow by revmoo · · Score: 3, Interesting

    Sounds pretty bad.

    I got this email earlier:

    Special Emergency Service Affecting Maintenance Dear Cogent Customer, With this message, we are notifying you of a special, emergency maintenance that will affect your service beginning at 3:00 a.m. tomorrow, Thursday July 17. The service outage you will experience is expected to be ten minutes or less. Cogent Communications takes very seriously its responsibility for maintaining a robust, well-performing network, and only due to extreme circumstances would we ask your indulgence for an emergency maintenance of this type. Please be assured that Cogent engineers will do everything possible to minimize your down time and its associated inconvenience. If you have any problems with your connection after this maintenance is complete, or if you have any questions regarding the maintenance at any point, please call Customer Service at 1-877-7COGENT and use this work order number: XXXXXX. We sincerely appreciate your patience and welcome any feedback. We apologize for the short notice.

    Thank you for being a Cogent customer.

    Sincerely,
    Customer Support
    Cogent Communications

    And was wondering what was up. Been hearing about a lot of router issues from various people. Lets hope this gets wrapped up quickly.

    --
    I would expect such blatant racism on Fark, but on Slashdot? Mods please ban this asshole.
  15. This has been discussed...... by flirzan · · Score: 5, Interesting

    ...on NANOG most of the day today. It looks like Cisco discovered the vulnerability in their own testing, notified major backbone providers (AT&T, Qwest, Sprint, L3, etc), who then scheduled emergency maintenance, which in turn tipped off savvy network engineers all over the place, who started wondering what was up, which in turn generated enough interest that bits and pieces leaked, and I bet Cisco figured better to release the advisory now and end the speculation than to wait till tomorrow. As for the "no exploit available", I had a router with an uptime of many many moons hang for no apparent reason tonight...while working on that I found the cisco advisory in my inbox. Could be a coincidence, but it's a strange one.

    --
    Twinkies sure taste good for something that is 68% air.
  16. Comcast has been having problems all day... by SlashChick · · Score: 4, Interesting

    I'm in the Bay Area, and my Comcast (formerly ATTBI) cable modem connection has been having issues all day. This router kept crashing earlier today:

    tbr1-p013601.sffca.ip.att.net [12.122.11.77] (hop #6 after my cable modem)

    I have no idea what the problem is or whether it's related to this exploit, but it really stinks to have the connection continually crash. I actually haven't had problems in the last few months... until today. I hope this isn't a harbinger of things to come...

  17. Will Homeland Security have kept it under wraps?? by Anonymous Coward · · Score: 5, Interesting

    What I really wonder wonder about is whether the vulnerability has been kept under wrap by the the Department of Homeland Security, just like they did with the Sendmail vulnerability of a short while ago, which was kept from the world for a couple of weeks. The US-military had at least a full week maintenance time before the rest of the world got it.

    As a non-american I found this quite disturbing, since certainly with the Sendmail vulnerability, there was a risk of this being exploited by the US-governement against foreign nations. NOw, I know I am just being paranoid, but it does freak me if this would become standard operating procedure: 1. Vulnerability discovered 2. US government given ample time to protect itself 3. US government makes use of vulnerability 4. Us gov releases it to friendly nations 5. You get notified.

  18. Just filter out packets with the evil bit by AaronW · · Score: 5, Funny

    Why not just filter out all the packets with the evil bit set? This should fix the problem.

    --
    This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
  19. Just got this from Internap: by flirzan · · Score: 5, Informative

    To all Internap customers:

    Cisco Systems has released to the public notification of a vulnerability
    in many versions of Cisco IOS which can create a Denial of Service on an
    affected router. The details of the advisory can be viewed at the
    following link:

    http://www.cisco.com/warp/public/707/cisco-sa-20 03 0717-blocked.shtml

    No exploits which target this vulnerability have yet been identified.

    Prior to the public notification, Cisco had contacted their major NSP
    customers including Internap to inform us of this vulnerability. Internap
    has identified IOS versions with the appropriate fix for the platforms in
    our network and scheduled upgrades to our routers. Customers will receive
    notification shortly of the window in which the routers you are homed to
    will be upgraded. Due to the severity of this vulnerability these
    upgrades are being performed as emergency maintenance.

    Customers with questions about the possible impact of this vulnerability on
    their own equipment are urged to read the notice at the link above or to
    contact Cisco directly.

    --
    Twinkies sure taste good for something that is 68% air.
  20. The cries of thousands of Cisco 827's by xQx · · Score: 3, Funny

    Wow, It's times like this I'm happy I'm not a sysadmin for a DSL service provider with heaps of customers with 827s around. ... oh wait.

    Boss. I'm at the pub.

  21. Yes it is by forged · · Score: 5, Insightful
    Actually, the proposed workaround works very well (it wouldn't be a workaround otherwise).

    Don't misunderstand traffic going THROUGH the router with traffic directed TO the router. You probably want to control the latter because as a good netadmin you should know that this is good practise.

  22. Dilbert by forged · · Score: 4, Funny
    Today's strip.

    Boss: Look what one of our engineers said to a reporter !
    Dogbert: (reading) "Our technology is putrid, but we compensate by ignoring complaints."
    Boss: You know what would be more fun than fixing those problems ?
    Dogbert: WITCH-HUNT !!!

  23. Interesting.... by slayer99 · · Score: 3, Informative

    "Obtaining Fixed Software

    Customers with contracts should obtain upgraded software free of charge through their regular update channels. For most customers, this means that upgrades should be obtained through the Software Center on the Cisco worldwide website at http://www.cisco.com/tacpage/sw-center/sw-ios.shtm l.

    Customers whose Cisco products are provided or maintained through prior or existing agreement with third-party support organizations such as Cisco Partners, authorized resellers, or service providers should contact that support organization for assistance with obtaining the free software upgrade(s).

    Customers who purchase direct from Cisco but who do not hold a Cisco service contract and customers who purchase through third-party vendors but are unsuccessful at obtaining fixed software through their point of sale should get their upgrades by contacting the Cisco Technical Assistance Center (TAC). TAC contacts are as follows.

    +1 800 553 2447 (toll free from within North America)

    +1 408 526 7209 (toll call from anywhere in the world)

    e-mail: tac@cisco.com"

    So, if I understand this correctly, if you've given Cisco money for hardware _and_ given Cisco money for a support contract, only then can you get hold of the fix. Neat.

    --
    Martin Brooks / Slayer99 #linux / UIN 2178117
    1. Re:Interesting.... by ckan · · Score: 3, Informative

      I had previously called tac@cisco.com for security patches for our Cisco devices not covered by any service contract. The response was quick, and the quality of service was very high. I got the patches I wanted very quickly without paying a cent! It was really a good experience.

  24. No Exploits? by grimani · · Score: 4, Insightful

    What does "no exploits" mean?

    No script kiddy tool for it yet?
    Nobody's used it yet to take down routers?

    Because the security advisory sure sounds like it's discovered an "exploit" on Cisco IOS routers to me.

    Any self respecting coder can whip up something homemade to take advantage of the issue.

    Is "no exploits" yet supposed to make us feel safer?

    If a security hole is there, it's vulnerable. Calling it "unexploited for now" is just misleading and confusing.

  25. Re:and no posting of the exploit code? by dillkvast · · Score: 3, Funny

    what the hell do you call CiscoWorks???

    Today "CiscoWorks" would probably be a contradiction in terms.

    --
    Scitne aliquis remedium potimum crapulae?
  26. homogeneous networks by martin · · Score: 4, Insightful

    Now let us step back a little.

    IF this had happened to our friends at Redmond (what do you mean 'if' :-) then we'd all be crying about how homogeneous networks/OS's etc are bad for security.

    Now it's happened to a vendor with probably more pieces of kit attached to the public internet than anyone else (by a long chalk IMHO).

    Do we cry, bad Cisco bad, no we just look at all the poor network admins who will get no sleep for the next 2 days....

    Perhaps NOW people wil start looking at alternatives to Cisco.

    Don't get me wrong I love Cisco kit, but I think the risk of Cisco everywhere is just about to hit home...

  27. Amazing by grayantimatter · · Score: 3, Interesting

    I think it's amazing how so many people posting here want to assume/believe that ANY slight hiccup on ANY network ANYWHERE in the last week is a direct result of this issue.