Slashdot Mirror
Major Flaw Found In Cisco IOS Devices
Joff_NZ writes "CERT has released an advisory regarding a serious flaw in all Cisco routers and switches which run IOS and process IPv4 packets (i.e. pretty much everything), which causes the device to stop processing inbound packets, and so: 'The device must be rebooted to clear the input queue on the interface, and will not reload without user intervention.' There are apparently no known exploits (yet), and Cisco have this advisory with a workaround and available fixes."
There are apparently no known exploits (yet)
I say we start a pool on how long yet will actually be, now that CERT released the info.
hmm... the cisco page shows up as 50 pages of text in lynx, with the first 20 being useful.
a four-hour timeout for IP-4 packets and you can do it
remotely to almost ANY cisco device except those that are run as purely IP-v6. Seems more like a nuisance DOS exploit and hope not to see it.
It's days like this I'm REALLY glad that I'm a unemployyed network engineer! This looks like a very serious headache!
-- I have a private email server in my basement.
This is why I always suggest alternatives to Cisco such as IP over Avian and actual implementaion on Linux
Rus
Cheap UK and US VPS
If I fire up Ethereal to peek at the traffic, I notice that the arp who-has requests are labeled with a source of "cisco_f2"
Wonder if someone has been pointing this sploit at cablemodem routers ??
Notice how both vulnerabilities, from Cisco and Microsoft, were not released to the public first. Instead the public announcement comes after the vendors have the patches.
Exploits, anybody?
AT&T has been having problems all over the west coast the last 4 days. Ill bet even money this is why. There last 2 emails state they had no clue what was causing it and that random reboot's of routers were to be expected.
Im not Anonymous, Just Lazy.
Crackers`n`Soup
At least it only freezes the device. If you could make it send the same packet to some of it's router buddies, then freeze, this could get real bad, real fast.
====
Crudely Drawn Games
Here's the reccomendation for a temporary workaround using ACls:
Cisco recommends that all IOS devices which process IPv4 packets be configured to block traffic directed to the router from any unauthorized source with the use of Access Control Lists (ACLs). Legitimate traffic is defined as management protocols such as telnet, snmp or ssh, and configured routing protocols from explicitly allowed peers. All other traffic destined to the device should be blocked at the input interface.
Does "A rare sequence of crafted IPv4 packets sent directly to the device" mean a sequence utilizing one of these three protocols? If so then frigging tell us! If not, this is just a vague precautionary warning that really won't stop any user inside the network from exploiting the bug.
The TRUE details of the bug, including which protocol it uses, would help us put a nail in the coffin regarding the ACL workaround, but the Cisco bug tool isn't returning any information for the bugs they're talking about - specifically CSCea02355 and CSCdz71127.
I suggest that African and European swallow are used as reference implementations.
While the army took time to celebrate the discover and safe return of Major Flaw it still maintained the need to continue the search for other missing top ranking officials. We spoke with a member of the search and recovery team soon after Major Flaw was discovered.
"It is great to have found Major Flaw but we are still very worried about the others. Our job here is not finished." said Private Data.
Colonel Panic has been spotted from time to time but the army has not yet been able to pinpoint his exact position. But the most gravest of fears are held for General Protection-Fault. Sightings of the General have been few and far between in the last few years. Some conspiracy theorists say that he is not actually missing but has disguised himself. Private Data would not confirm wether they are searching for a man of similar build to General Protection-Fault but dressed all in blue.
"She's a West Texas girl, just like me" - G.W Bush Iraqis
Are you sure you don't want to start a holy war?
Anyway yes Cisco has problems just like all the
other vendors. I have no idea how you are
trying to compare a hub and a switch attached
to different boxes for their performance. Maybe
there is a misconfiguration in the Catalyst.
A 4 port belkin hub is little less likely to have
that issue
Of course if you are really trying to use a catalyst to
ROUTE traffic maybe you just are using
the wrong tool.
YMMV
The claim that there are no exploits is false.
0 03 0717-blocked.shtml
Below is a note I received from my ISP about 2 hours before this was topic posted:
=-=-=-=-=-=-=-=-=-=-=
17/07/03 01.12 - 01.38 DOS Attack on Sydney PoPs
Incident
A DoS attack against the AN border router resulted in that router's CPU reaching 100% and triggering the same attack on the Perth gateway router which in turn brought down the Comindico Border router
Action
While all of the hardware remained 'up' nothing could be authenticated and therefore all traffic through the Sydney PoP ceased.
Resolution
Swiftel Engineering rebooted the Perth Gateway router clearing the DoS packets and that in turn allowed the Sydney routers to rebuild the BGP4 tables thus restoring the ability to process customer traffic.
Result
By 1.38 pm all traffic was flowing normally.
Future Elimination Of This Problem
The elimination of this type of new DoS attack has just been recognised and released by Cisco (today) and the workaround and fixes are documented in:
http://www.cisco.com/warp/public/707/cisco-sa-2
We are considering whether to implement the workarounds which may impact traffic such as ICQ and some games or upgrade the IOS's in all of our Cisco equipment.
We will inform you when that decision is made.
This is actually good news for Cisco, because security holes like this appear to be a prerequisite for getting a large Department of Homeland Security contract.
It is nearly midnite on the left coast and the updates for this bug are still not available for download.
Also interesting to note is that the top of the Cisco Advisory had a release date of 7/17 00:00 GMT. But the bottom said that it would not be published to the public until 7/17 21:00 GMT.
Why the release 21 hours ahead of schedule? Especially since you can't d/l the patches!!
No, the advisory states that non-contract customers can send an email to tac@cisco.com and get access to a "free upgrade".
"Like millions of sysadmins cried out in terror -- then were silenced."
Sounds pretty bad.
I got this email earlier:
Thank you for being a Cogent customer.
Sincerely,Customer Support
Cogent Communications
And was wondering what was up. Been hearing about a lot of router issues from various people. Lets hope this gets wrapped up quickly.
I would expect such blatant racism on Fark, but on Slashdot? Mods please ban this asshole.
...on NANOG most of the day today. It looks like Cisco discovered the vulnerability in their own testing, notified major backbone providers (AT&T, Qwest, Sprint, L3, etc), who then scheduled emergency maintenance, which in turn tipped off savvy network engineers all over the place, who started wondering what was up, which in turn generated enough interest that bits and pieces leaked, and I bet Cisco figured better to release the advisory now and end the speculation than to wait till tomorrow. As for the "no exploit available", I had a router with an uptime of many many moons hang for no apparent reason tonight...while working on that I found the cisco advisory in my inbox. Could be a coincidence, but it's a strange one.
Twinkies sure taste good for something that is 68% air.
With this exploit now out there(at least in theory anyway), I guess the question now becomes what can we expect from it. Assuming that a black-hat or someone else of an infamous nature figures out this exploit, what are the ramficiations that we can expect? Obviously, many routers are owned and run by compotent admins, but with all the Cisco routers out there, it's niaeve to believe that all of the routers will be fixed before someone exploits this. Given that, what does everyone suppose will happen to the internet as a whole? The core routers will most likely be fixed ASAP, but there's always the problem of the "oopps, I forgot that one" router. Will this exploit become the ever-lasting Code Red(in terms of network problems), or will its threat blow over just like Code Red?
I'm in the Bay Area, and my Comcast (formerly ATTBI) cable modem connection has been having issues all day. This router kept crashing earlier today:
tbr1-p013601.sffca.ip.att.net [12.122.11.77] (hop #6 after my cable modem)
I have no idea what the problem is or whether it's related to this exploit, but it really stinks to have the connection continually crash. I actually haven't had problems in the last few months... until today. I hope this isn't a harbinger of things to come...
Simpli - Your source for San Jose dedicated servers and colocation!
if this is what i think it is, it only takes about 10 frames (8+ is the word) to stop a switch (and i guess now the routers) in their tracks requiring a manual reboot.
-d
The above post so looks loke a troll, however in case the poster is serious and is in need of a whack with a clue stick...
Hubs / repeaters / fanouts work at the electrical signal level.
Bridges / Switches work at the frame level.
In limited circumstances i.e. there are only two devices talking on the segment the cheap hub may well be faster than any $20,000 switch.
Start throwing more devices, full-duplex operation etc. into the conversation and you will soon see the light.
Now when you have touched a network outside of your bedroom please feel free to respond
slashnik
What I really wonder wonder about is whether the vulnerability has been kept under wrap by the the Department of Homeland Security, just like they did with the Sendmail vulnerability of a short while ago, which was kept from the world for a couple of weeks. The US-military had at least a full week maintenance time before the rest of the world got it.
As a non-american I found this quite disturbing, since certainly with the Sendmail vulnerability, there was a risk of this being exploited by the US-governement against foreign nations. NOw, I know I am just being paranoid, but it does freak me if this would become standard operating procedure: 1. Vulnerability discovered 2. US government given ample time to protect itself 3. US government makes use of vulnerability 4. Us gov releases it to friendly nations 5. You get notified.
This thing right here
Is lettin all the geeks know
What CERT talks about
You know
The major flaws in hardware
Hahaha
Check it out
Ooh dat packet so scandalous
And ya know this router couldn't handle it
See ya shakin that fist cause you paid for it
With a look in ya eye so BOFHish
Uh
Ya liked it a lot till the processing stops
And ya job was secure till da connection dropped
No time to sleep you'll reboot a lot
Cuz the router's completely loca
The core dumps like a truck truck truck
Admins like what the f^@%t
Press the power butt butt butt
Uh
I think I need to reboot it again
The core dumps like a truck truck truck
Admins like what the f^@%t
All night long
Let me see that flaw
Why not just filter out all the packets with the evil bit set? This should fix the problem.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
Well, I can safely predict that alot of the 12xxx routers are going to reload/have reloaded already. At least if you don't have a Juniper sitting on your core, you most likely have 12xxx series one. And try to apply an acl on their interfaces... bye bye router :)
To all Internap customers:
0 03 0717-blocked.shtml
Cisco Systems has released to the public notification of a vulnerability
in many versions of Cisco IOS which can create a Denial of Service on an
affected router. The details of the advisory can be viewed at the
following link:
http://www.cisco.com/warp/public/707/cisco-sa-2
No exploits which target this vulnerability have yet been identified.
Prior to the public notification, Cisco had contacted their major NSP
customers including Internap to inform us of this vulnerability. Internap
has identified IOS versions with the appropriate fix for the platforms in
our network and scheduled upgrades to our routers. Customers will receive
notification shortly of the window in which the routers you are homed to
will be upgraded. Due to the severity of this vulnerability these
upgrades are being performed as emergency maintenance.
Customers with questions about the possible impact of this vulnerability on
their own equipment are urged to read the notice at the link above or to
contact Cisco directly.
Twinkies sure taste good for something that is 68% air.
Wow, It's times like this I'm happy I'm not a sysadmin for a DSL service provider with heaps of customers with 827s around. ... oh wait.
Boss. I'm at the pub.
Don't misunderstand traffic going THROUGH the router with traffic directed TO the router. You probably want to control the latter because as a good netadmin you should know that this is good practise.
Thinking about shutting down the Internet today ? I think I prefer to keep my job :-)
You can always contact the seller and ask them for the license :->
Boss: Look what one of our engineers said to a reporter !
Dogbert: (reading) "Our technology is putrid, but we compensate by ignoring complaints."
Boss: You know what would be more fun than fixing those problems ?
Dogbert: WITCH-HUNT !!!
Sorry if this is a dumb question, but are DSL and other broadband router devices running CBOS 2.x.x such as Cisco 675 and 678s vulnerable or is CBOS a different critter with different TCP packet handling code?
I find the African and European swallow to be a ppor refrence. I highly recomend the peregrine falcon as the refrence implementation due to it's higher overall speed.
"Obtaining Fixed Software
m l.
Customers with contracts should obtain upgraded software free of charge through their regular update channels. For most customers, this means that upgrades should be obtained through the Software Center on the Cisco worldwide website at http://www.cisco.com/tacpage/sw-center/sw-ios.sht
Customers whose Cisco products are provided or maintained through prior or existing agreement with third-party support organizations such as Cisco Partners, authorized resellers, or service providers should contact that support organization for assistance with obtaining the free software upgrade(s).
Customers who purchase direct from Cisco but who do not hold a Cisco service contract and customers who purchase through third-party vendors but are unsuccessful at obtaining fixed software through their point of sale should get their upgrades by contacting the Cisco Technical Assistance Center (TAC). TAC contacts are as follows.
+1 800 553 2447 (toll free from within North America)
+1 408 526 7209 (toll call from anywhere in the world)
e-mail: tac@cisco.com"
So, if I understand this correctly, if you've given Cisco money for hardware _and_ given Cisco money for a support contract, only then can you get hold of the fix. Neat.
Martin Brooks / Slayer99 #linux / UIN 2178117
What does "no exploits" mean?
No script kiddy tool for it yet?
Nobody's used it yet to take down routers?
Because the security advisory sure sounds like it's discovered an "exploit" on Cisco IOS routers to me.
Any self respecting coder can whip up something homemade to take advantage of the issue.
Is "no exploits" yet supposed to make us feel safer?
If a security hole is there, it's vulnerable. Calling it "unexploited for now" is just misleading and confusing.
------- Sent to ValueWeb customers -------
ValueWeb has been informed by a number
of our bandwidth providers (MCI, Sprint, and
Level 3) that they will be undergoing maintenance
that may cause widespread Internet slowness
between 3:00 and 6:00 am EST, July 17th, 2003
Well I guess this is why my isp has died twice today... Luckily they have junipers on their overseas links...
Now let us step back a little.
:-) then we'd all be crying about how homogeneous networks/OS's etc are bad for security.
IF this had happened to our friends at Redmond (what do you mean 'if'
Now it's happened to a vendor with probably more pieces of kit attached to the public internet than anyone else (by a long chalk IMHO).
Do we cry, bad Cisco bad, no we just look at all the poor network admins who will get no sleep for the next 2 days....
Perhaps NOW people wil start looking at alternatives to Cisco.
Don't get me wrong I love Cisco kit, but I think the risk of Cisco everywhere is just about to hit home...
OK. So we know that it's an IPv4 packet that is sent to the router, and that's it? Is there any way we can detect hacker attempts to exploit this bug.
If there _is_ an exploit out there, I for one would like to know if they're trying it on.
That is without dropping a Cisco router at my front door and waiting for it to get shot!
(Which would be a little dropping your daks at a Lorena Bobbit convention, and hoping no-one brought the garden shears with them!).
I have seen this behavior on several of my systems on interfaces where I *know* the customer is not intentionally sending bad packets. I resolved the problem by disabling fair-queueing on the interfaces where this tended to happen.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
(For Americans and others; the NHS is a country wide health service that treats everyone. It's on a WAN, called NHSnet. For that reason, any netowrk problems are very serious, as it means hospitals are almost totally unable to function.(for the record, internal IP's start with 10.1.xxx))
Toured a local NHS facility yesterday, when they were recovering from a total internal crash that cut off all internal network traffic, as well as external traffic in and out.
The crash was caused by a halt in network traffic(read: router failure) in which the error messages overwelmed the servers. (that was the reason I was given, at least... but I can't really belive that the servers would be that badly configured.)
The servers, for the record, are a mixture of VMS (insert unix plug here) and NT/2000's (insert windows flame here). (There's a effort to move them over to pure Windows... goverments. bah.)
Anyhoo, the upshot of all this was that the routers had to be restarted and the server's hard disks to be remerged...
Guess who's routers they were? Cisco.
This was before or shortly after the release of the warning... could the problem be a bit more serious than previously thought?
Up to 85% off Cisco 2501
Save on Used Cisco Equipment Routers, Gbics, Modules & more.
www.bizinetworks.com
Used Cisco Routers
Used/Refurbished Cisco Routers save up to 90% off retail price.
www.networkliquidators.com
Cisco Switches
Compare Prices and Save Money. Find the best deals at BizRate.com!
www.BizRate.com
*Ahem*! -Ocelot Wreak.
"I figure you're here 'cause you need some whacko who's willing to stick his finger in the fan. So who are we helping?
The title of the document states, and I quote "Cicso 7200 Series Routers..."
The problem only occurs in the 7200 series router!
PIX Firewalls actually run a OS called Finesse. It used to look a lot less IOS like than it does today. They do not appear to be vulnerable at this time
As Spock would say Fascinating the link is no longer there on CCO
Cisco routers are notoriously underpowered! Install some ACLs on a busy 3600 or 2600 series router and you'll start a DOS attack on your own router!
As this recent NetworkWorldFusion review shows, a 2651 starts to fall apart with 8 rules on only 2 T1s!
I would definitely NOT recommend adding ACLs willy-nilly to Cisco routers. BE CAREFUL and add a couple at a time during peak traffic times to make sure the router stays up.
And (experience talking here) be sure you're logged in on console and not telnetted into the Cisco when adding a bunch of ACLs. Your telnet session will not get priority over forwarded packets and you'll have no control over the router!
Cisco wants you to think ACLs are only "waaafer theeen"!
Oh, Juniper makes great routers, but they're all carrier (ISP) class, or at least they all were when I trained on them. You're not going to find them used as customer CPE very often... and individual companies have the most to lose by this exploit, especially small ones whose ISPs maintain their equipment for them, who aren't rolling out fixes for all those small Ciscos now.
Get off my launchpad!
Nope.... /. issue I assume (as it is happening to me too!)o ftware/ios121/121newft/121t/121t5/dtssm5t.htm#1021 424
his link is just badly formatted...a
The link is still there...
link - http://www.cisco.com/univercd/cc/td/doc/product/s
I think it's amazing how so many people posting here want to assume/believe that ANY slight hiccup on ANY network ANYWHERE in the last week is a direct result of this issue.
...something was up at lunch on Tuesday. Our Cisco SE said he couldn't say what it was until 5pm that day. Apparently about 20-30 big networks were contacted then and only a few (100) people at Cisco itself knew.
So much for appliances ... plug 'em in and leave 'em? NOT!
Tripping off soapbox now ...
I think they should upgrade my IOS version out of good faith ;)
While you are out there drinking your redbull and upgrading IOS on every Cisco router you own... Don't forget to upgrade all of your switches too...Read this http://www.packetstormsecurity.nl/filedesc/ciscoCa tOS.txt.html
Way to Go cisco.... How's that for the five nines.
NOTE: Juniper testing will be starting very soon at my job >:)
and route IPv4. Otherwise we'd be screwed.
cheap labor conservatives - they want to keep you hungry enough to be thankful for minimum wage.
From: Hembree, Daniel [mailto:Daniel.Hembree@Level3.com]
_ __ _________________
Sent: Thursday, July 17, 2003 10:39 AM
To: undisclosed-recipients
Subject: Level(3)
_______________________________________________
As you may be aware, Level 3 performed significant maintenance to Cisco
routers in our Network over the past two evenings. Due to restrictions in
our contract with Cisco, we were not at liberty to share with you the nature
or details of the pending work. Additional information can now be shared.
Level 3 Communications was notified by Cisco on the evening of Tuesday, July
15, of a potential software risk running on Cisco routers. In coordination
with Cisco, Level 3 Engineers worked to secure the Level 3 Network through
network modifications and router maintenance that evening. The remainder of
our core Network infrastructure was completed in the maintenance window last
evening.
We recognize that the timeframe and notification provided in this case have
not been consistent with standard practice. The decision to move forward
with work was based on a collective assessment of the potential impacts to
your services if the risk was not mitigated.
We will continue to conduct maintenance activities over the coming days as
we address issues associated with this specific exposure, and mitigate any
potential remaining risk. We will provide specific maintenance notifications
to Customers on the associated services we would impact in those follow-on
maintenance activities.
For more information regarding the vulnerability please visit:
-- Jack
Flamebait maybe, absolute fact (cisco sucks), yes...
Just Received the following message from incidents@securityfocus.com mailing list:
3 0717-blocked.shtml
" Cisco has updated the advisory to include details on the exploit.
http://www.cisco.com/warp/public/707/cisco-sa-200
Cisco routers are configured to process and accept Internet Protocol version 4 (IPv4) packets by default. A rare, specially crafted sequence of IPv4 packets with protocol type 53 (SWIPE), 55 (IP Mobility), 77 (Sun ND), or 103 (Protocol Independent Multicast - PIM) which is handled by the processor on a Cisco IOS device may force the device to incorrectly flag the input queue on an interface as full, which will cause the router to stop processing inbound traffic on that interface. This can cause routing protocols to drop due to dead timers. "
After hearing the sploit it only took me a couple hours to get the code.... Some real Geniouses out there so I will take Saturday before it is ported to windows for the script kidddddies