Duke Wireless Problem Caused by Cisco, not iPhone

jpallas writes "Following up to a previous Slashdot story, it now turns out that the widely reported problems with Duke University's wireless network were not caused by Apple's iPhone. The problem was actually with their Cisco network. Duke's Chief Information Officer praises the work of their technical staff. Does that include the assistant director for communications infrastructure who was quoted as saying, "I don't believe it's a Cisco problem in any way, shape, or form?""

Comments

deficient

[Anonymous Coward]

Still others seem to imply that Duke's network was deficient in some way because the problem had not been encountered more broadly.

I would say that the network was deficient until the patch was applied. For him to say otherwise implies that there was no problem to begin with.

Re:deficient

[Sillygates]

ahh, the battle of the iPhones

idiots

[SolusSD]

this is just another example of hair trigger IT morons with nothing but an MIS (Management Information Systems) degree and experience working on their mom's computer. I can hear it now, "but they are cisco certified!!!". Yeah- certification.. spend a few hours studying some high level networking material, take a test-- now your an *expert*. always blaming whatever is new touching their "pristine" networks.

Re:idiots

[Anonymous Coward]

A decent CCNP or CCIE probably would have done packet dumps and debugs and sat there reading RFCs and following the state-flow diagram. Thats what I do, and I'm just a lowly CCNA.

Why do developer types always have be hating on your friendly IT folk? Developers are some of my best friends, and we ask each-other for advice all the time. I don't meet one stupid developer who I have to explain to what NAT, proxies or TCP options are and say 'ugh, those damn CS geeks!'

Re:idiots

[fotbr]

Because there are so many BAD IT folks with certificates that aren't worth the paper they're printed on, that its given the entire field a bad name.

I know maybe a half dozen IT people that are worth their weight in gold pressed latinum, but I know many times that number more that are useless, and cannot figure out how to do simple things without having to have their hand held through the entire process, for something they do on a weekly basis. But they have certificates out the wazoo, so they look good on paper, which gets them hired.

Re:idiots

[ejtttje]

I don't meet one stupid developer who I have to explain to what NAT, proxies or TCP options are That's because we already studied them in our Networking class and wrote our own implementations from scratch for homework assignments while we were getting four-year degrees... ;-P

Just pulling your leg (mostly) ;) I have a feeling you know a lot more by now than anyone finishing a one semester course... and a lot of CS grads get through without taking a networking course in the first place... but the trick is, we both know a lot more than the new recruits wearing their certification diapers and intoning "cisco can do no wrong, buy nothing but cisco, cisco or bust" (because of course, the certified on a specific brand's interface, and have no idea what they're actually doing.)

For example, I love the story of when we visited a university with a cisco-sycophant net admin, and we wanted to put up a linksys wireless access point in the lab for the robots to connect to, and he was resistant until we pointed out cisco had recently bought linksys, so it was actually a cisco device too, and then it was OK. Siiiigh.

But anyway, 15-441 ftw! :)

Re:idiots

[nosilA]

I used to work with the "hair trigger IT moron." He has a CS degree from one of the best CS schools in the country, he has been running college networks since 2000, and he does, in fact, know what he's doing.

I will admit though, that he has been known to get ahead of himself. When he looked at the logs and saw a bunch of iPhone MAC addresses spewing garbage, but no other devices are, he assumed that it's an iPhone problem. The quote in Network World is unfortunate, but he is no "hair trigger IT moron." He continued working on getting to the root of the problem and solved it yesterday.

Re:idiots

[FDDIcent]

Sounds like a fun class, I was a CS minor, but I never got around to taking Networks, just OS/Datastructures/AI etc. Bummer. Your admin does sound like an idiot. The reason to say 'no' to a Linksys in an environment like mine have mostly to do with support (4 hour RMA), interference with other devices (our Cisco APs all talk to each-other), and management (I want to enforce security policy on you). Who cares if Cisco bought Linksys ;) Its the same reason why our server admins want all HPs, because they already have a shop full of HPs, support agreements etc. Not because HP is magical. There's always the option to change, but in a large enterprise, a homogeneous environment greatly reduces cost IMHO. Then again if I worked in an academic environment I'd be a lot less of a hardass, I work in Healthcare.

Re:idiots

[eipo]

Your assumption at the skill of the network folks at Duke is sadly mistaken. The people at Duke are very intelligent and experienced group of folks that detected a problem on the network that seemed to be related to the iPhone. In turn they contacted BOTH Apple and Cisco and began running dumps to try to figure out what was going on. In the beginning it did appear to be caused by the iPhone and only after a lot of testing and help from Cisco was the true problem discovered. They had Cisco network that functioned perfectly until iPhones started popping up, it wasn't a far stretch to suspect the new device introduced into a working system.

The only thing they did poorly was fail to realize how much the techie world is hot and bothered over ANY news about the iPhone. Had the cause seemed to have been the the latest Crackberry this would have never sweep through the iPhone loving media/techie-verse this quickly.

So come off your superiority complex a bit and cut them some slack. They managed to detect and solve this issue within a week on a massive University network with half the tech world breathing down their collective necks. It wasn't the work of inexperienced MIS folks but group of talented network professionals that had the misfortune of publicly grappling with the iPhone juggernaut and half million know-it-alls on forums like this.

Re:idiots

[Anonymous Coward]

Hate to break it to you, but almost every industry sector is like that, be it software developers, construction workers, furniture makers, restaurant staff, and even medical staff.

It's actually rare to find an industry where almost everybody is top-notch, simply because most companies don't want to pay the premiums for these folks. I can only think of a few off the top of my head: NASA, Google, most engineering firms...

Re:idiots

[gb506]

He continued working on getting to the root of the problem and solved it yesterday.

Well, he HAD to continue to work the problem, that's his job, he didn't really have the option of simply ignoring the situation, did he? Not sure that constitutes a pat on the back.

The quote in Network World is unfortunate, but he is no "hair trigger IT moron."

It does, in fact, show that he's willing to drop statements to the press such as "I don't believe it's a Cisco problem in any way, shape, or form," quite prematurely and without factual foundation. In the end a.) he should have kept his mouth shut until he definitively knew what was happening, and b.) if she was at all concerned that the guy was prone to getting ahead of himself, Tracy should have muzzled him from the moment the issue first came to light. Any story involving a potential iPhone problem at this juncture is going to get mega exposure, and Miller and Futhey should have realized that.

Re:idiots

[jayhawk88]

Uh huh. Because you've never made a mistake or misdiagnosed a problem when something is broken and your entire customer base is screaming at you to fix it.

Jesus, I love how you all are posting here like you single handly created the first router and invented TCP/IP. Let's try and look at this from the Duke IT perspective: 1. Wireless network is (presumably) working great. 2. iPhone is released, students start showing up with it. 3. Wireless starts getting slammed. Yes it was a wrong conclusion and faulty logic but come on, was it really that horrible? When something breaks the first thing you ask is "What has changed", in this case iPhones were introduced to the network. I guarentee that would have been the first thing I would have looked at.

Re:idiots

[peragrin]

go back and read the slashdot article on the subject when this first came out. Dozen of slashdot guys were reporting that cisco routers and WAP's have a flaw that would enable just such a solution and that you had to patch the routers with a patch that Cisco already had made.

Cisco makes some solid equipment, but when they let flaky stuff loose it's really flaky. It is also not something you announce to the world first, without throughly checking out your own equipment first, especially when the iPhone was working perfectly fine with tens of thousands of other access points around the country.

Re:idiots

[Slackus]

Yeah- certification.. spend a few hours studying some high level networking material, take a test-- now your an *expert*

While this holds true for most IT certifications, I have to say that Cisco's CCIE certification is all but a few hours study. Ask anyone who has gone through the CCIE certification.

Re:idiots

[Anonymous Coward]

Any story involving a potential iPhone problem at this juncture is going to get mega exposure, and Miller and Futhey should have realized that.

I bet it's precisely because they realized that they went public in the first place.

Re:idiots

[EGSonikku]

Perhaps they should have waited a week then and not announced prior to discovering the true issue that the iPhone was at fault.

Re:idiots

[eeyoredragon]

Calling someone a moron is over the top. However, it is annoying when you work in a Windows only world, and when something non-Windows of yours screws up, it's because it's not Windows. "I'm having trouble connecting to our VPN." "Must be a mac thing." " We don't support Apple." etc. It seems everywhere I go this is the pat answer I get, and they get annoyed when my response is: That's nice, but you see, I just tried in OS X on my laptop, Windows XP on my laptop, and Windows Mobile on my cell phone. Is there any configuration you do support?

Re:idiots

[SkyDude]

.....nothing but an MIS (Management Information Systems) degree and experience working on their mom's computer. I can hear it now, "but they are cisco certified!!!". Yeah- certification.. spend a few hours studying some high level networking material, take a test-- now your an *expert*. always blaming whatever is new touching their "pristine" networks.

You've said what I've been thinking for years. Maybe with a little more force than I would have said it, but your point is on target.

When I certified on Novell in 1998, I assumed most of my classmates were already in the IT field as techs. Come to find out, many of them had never used a PC before Windows 98 or 95. One character was given a floppy disc with the Novell client on it and waited for it to run the install routine for him. On another occasion, I demonstrated ANSI color codes in a batch file to the class (the instructor and I got along well)and they were amazed.

I've said it before and nothing here has disproved my view - degrees or certifications are nothing more than showing the world you can take classes, accumulate facts and regurgitate what's been learned on a test. Out of every 100 degrees or certifications earned, I'd be willing to bet less than a third actually understand what they've studied. This may not apply to every cert or degree, but as one progresses up the degree tree, I think the number may actually decrease.

Re:idiots

[magneticstorm]

I also used to work with this "hair trigger IT moron" at the university where we both got our top-tier Computer Science degrees. Hands down, he is brilliant. I have seen him at his best, which includes: writing an entire network registration software suite for use at large institutions, diagnosing bizarro network problems, and managing network security issues. It is rather unfortunate that he was quoted as saying he was confident that it wasn't a Cisco-related problem. While that ended up not being the case, I have no doubt in my mind that he had reasonable cause to think so originally. I have always seen Kevin as a rising star, and I am confident that he will continue to do great things.

Re:idiots

[rollthelosindice]

Actually, that's exactly what it was. A far stretch. If you have a campus wide network and as few as ONE or TWO CELLPHONES, can disrupt it, you need to make sure you are looking at your information properly. The amount of requests per second that they were reportedly getting flooded with would require more processing power than both or those cellphones had combined. Even if they were buggy to shit and doing nothing but flooding with requests they couldn't have been doing that many requests.

It's like saying that an old car has lots of problems with its engine and was seen going 200 miles per hour.

Re:idiots

[operagost]

I hate that phrase: "any way, shape, or form." What's wrong with "in any way?" Do issues have a shape or form?

Re:idiots

[Maniac-X]

I guess this truly puts me in the minority, because I do both IT and development stuff pretty equally well, though i do favor development a little (don't tell anyone)

Of course there are arguments to work in a homogenous environment, I encountered the same arguments in my first year IS classes. My teacher insisted it was much more efficient to buy mass amounts of generic-branded PCs because the "support was better" in case of hardware failure. Of course I argue that if I build them myself, I already know by the time each one is deployed that the hardware is not a lemon (burn-in testing), and it's probably going to last quite some time.

In my opinion, it is far better to spend a little more on in-house support, mix and match quality hardware for the best results (performance and reliability-wise), than it is to go all-out generic and be RMAing whole machines all the time. Yes, it's more work in the end, but overall there seem to be less problems. Besides, the biggest upside is, no matter who you hire onto your IT team, they WILL learn how things actually work, and how to actually fix them, after a while. The RMA-everything method discourages learning by experience, imo.

Re:idiots

[ktappe]

Let's try and look at this from the Duke IT perspective: 1. Wireless network is (presumably) working great.

That word "presumably" was pretty close to the heart of this entire debacle. It's an assumption and those are the first things you should throw out when performing logical troubleshooting.

2. iPhone is released, students start showing up with it. 3. Wireless starts getting slammed. Yes it was a wrong conclusion and faulty logic but come on

Come on what? Logically fixing this problem is their primary job. Not publicly pointing the finger at Apple before that logic had run its course.

was it really that horrible? When something breaks the first thing you ask is "What has changed", in this case iPhones were introduced to the network.

What they essentially did was the same as if an engineer blamed a person walking across a bridge for the bridge collapsing. The iPhones were making completely normal use of infrastructure yet they were blamed when that infrastructure failed. Engineers (be they civil or network) need to make sure their house is clean before they start casting aspersions. Speaking as an OS engineer myself, I do not blame the user when the OS crashes; I examine the OS. The user should not be able to make it crash just as iPhones should not be able to take down a wireless network.

Re:idiots

[g-san]

What, and miss out on all that iPhone hype fueled publicity? Inconceivable! I'm considering swallowing one just to get on the front page of the local paper. Probably make MSNBC too.

Re:idiots

[Anonymous Coward]

Gentlemen:

I'm a technical writer. Stop correcting him. I need the work!

Re:idiots

[CryBaby]

The only thing they did poorly was fail to realize how much the techie world is hot and bothered over ANY news about the iPhone.

No, what they did poorly was their job as problem solvers. They made the classic mistake of trying to solve a problem in reverse -- they started out with an assumption and then looked for evidence to support that assumption. For whatever reason, they wanted to rule out Cisco as the cause, so they did. This prevented them from finding the real cause as quickly as they might have.

Had they kept an open mind, they would have looked for more evidence before making a determination. For example, they could have asked some other universities (who undoubtedly now have iPhones on their wireless networks) whether or not the same type of problem was occurring there. With the answer being "no", they would have learned that the problem must have something to do with the combination of the iPhone and their specific network. That would have opened the doors to start looking at network configuration and/or faulty networking equipment. Obviously, that's exactly what happened in the end, but my point is that they erected a barrier in the problem solving process by "trusting" their Cisco equipment rather than suspecting it along with everything else.

I don't think the Duke IT people are incompetent, unintelligent, lacking in education, etc. Rather, I think this is an interesting little case study that illustrates how even highly competent people can allow their preconceptions to undermine their problem solving efforts. After all, our instincts, gut reactions, feelings, etc. are extremely useful when diagnosing a problem. They are often correct or at least highly informed on a level that is difficult to quantify. So, it's not easy to consider that your instincts may be completely wrong -- that you may be looking at an entirely new and surprising situation in which your instincts only serve to mislead you. Effective problem solving requires creativity, deliberate role-playing (e.g. "playing devil's advocate") and a certain amount of (forced) objectivity. Unfortunately, too few technical professionals display these traits when attempting to diagnose a problem and fail to understand that problem solving, in a general sense, is a discipline unto itself.

Re:idiots

[hol]

Also, keep in mind, that Cisco has been known to deny clearly existing problems in the past. I personally sat with a Cisco engineer and showed him what frames were wrong on an LACP problem, and he continued to insist it was this "freeware OS stuff" that was causing the problems with the switch. Too bad the ethereal scrapes were from a Windows server...

Re:idiots

[Anonymous Coward]

"When he looked at the logs and saw a bunch of iPhone MAC addresses spewing garbage, but no other devices are, he assumed that it's an iPhone problem."

-- and this is a lazy technician level response I would expect from the guy installing my cable at the house not anyone running anything. He is the guy who is supposed to ask the questions and *stop* this kind of "analysis" from becoming an action item.

I've worked with this guy too - maybe smart, but a bit of an opinionated cowboy; not one to let facts get in the way of his high opinion; his mouth engaged before his brain had sealed-the-deal;

Re:idiots

[RzUpAnmsCwrds]

My teacher insisted it was much more efficient to buy mass amounts of generic-branded PCs because the "support was better" in case of hardware failure. Of course I argue that if I build them myself, I already know by the time each one is deployed that the hardware is not a lemon (burn-in testing), and it's probably going to last quite some time

You've never worked in a large-scale IT environment. At my company, we deploy over 7000 machines per year (1/3 of the entire infastructure) in hundreds of sites around the world.

Are you going to build and "burn in" 20 machines per day? How many people are you going to hire (probably at least two dedicated employees, which is at least $300k/year in expenses)?

Who's going to handle packaging and shipping the machines (HINT: Dell/HP/Lenovo spend a LOT of time testing to make sure the PCs arrive intact)?

When there's a problem, are you going to be able to repair them locally, or will you have to ship them back to headquarters? You can't have a dedicated tech for a 10-man site, but major manufacturers can offer support pretty much anywhere in the world.

How do you know that your images are going to work? You don't want to find out that some chipset mismatch on 2% of your PCs is causing kernel panics.

When you have a problem, who's going to fix it? HP/Dell release BIOS updates for years to fix bugs. Good luck getting ANY support out of AsusTek/ECS/Tyan/Biostar/MSI/Gigabyte/Whoever after even 1 year.

Where do you dispose of your PCs? HP/Dell have extensive recycling programs in place.

How do you handle your purchase orders? HP/Dell are very good at working with your accounting department. It's not as simple as "put it on the Visa".

Of course I argue that if I build them myself, I already know by the time each one is deployed that the hardware is not a lemon (burn-in testing), and it's probably going to last quite some time.

Of the 7396 PCs (desktop and notebook) we deployed in 2005, 143 have failed (1.9%). Generally, we find that the lifetime failure rate is below 3%. You're not even going to get close to that by building them in-house. One of my friends runs a custom-built PC business, and he sees a failure rate closer to 5%, with a large percentage being damaged during shipping.

As for "lasting quite some time", this indicates that you've never worked in a large IT environment at all. All major IT environments have some sort of lifecycle in place, typically 3 years but sometimes 4 or 5. A typical employee costs the company $150,000 per year (salary + benefits + taxes + etc) - if you replace a $1500 PC every three years, you're only spending $500 per year on the PC. It makes precisely zero sense to stick your $150,000 employee with old technology - if the new PC makes them even 0.5% more productive, you are saving $750 per year.

You may think that the big manufacturers just throw together parts, but nothing could be further from the truth.

Re:idiots

[tgibbs]

The only thing they did poorly was fail to realize how much the techie world is hot and bothered over ANY news about the iPhone.

This hardly seems like a minor error. They apparently went public with a premature conclusion about a new product before they actually knew what was going on, thereby holding themselves and Duke University up to ridicule.

And it is an unfortunately typical knee-jerk reaction reflects the arrogance typical of many IT departments: "The problem isn't with our network; it must be your computer or software that is screwing things up."

Re:idiots

[Anonymous Coward]

Of course yes... Just because they had never had a problem before they should have suspected their current infrastructure.

Their only mistake was to go public with it, the fact the iPhones had shown up was a logical jump to say they were at fault.

Take a factory, all the workers are working fine each is given a cup of a specific brand of coffee in the morning. There is an accident at a powerstation nearby, and productivity in the factory immediately ceases. Should we blame the fact the factory is out of power or maybe that the workers don't like the coffee and have stopped work until they get a different brand?

The powerstation is more likely, but you would obviously not discount the coffee... OK fair you shouldn't discount it but the power is much more likely since that is what changed.

I wish they had been right and proved the iPhone was at fault, at least it would have shut these fanboys up.

Re:idiots

[eli pabst]

He continued working on getting to the root of the problem and solved it yesterday.
See, that's what he gets for not reading Slashdot. If he would've just sat there eagerly refreshing his browser, he would've seen several people post the solution to their problem last week and could've taken the weekend off. Hope this is a lesson.

Re:idiots

[Retric]

Don't forget Dell also pays people to do build systems and do support. So when your buying a Dell your paying for the HW, SW, man power to build it, shipping costs, man power to support it, and some R&D.

Granted Dell gets huge discounts due to their size but you can build systems in house for around the same cost as a rebuilt Dell if you build a lot of systems. (AKA Assuming you don't pay full retail cost for windows.)

Don't forget Support Contracts make Dell a lot of cash and they outsource a lot of this Support. So you could higher the same company's and get that slice of the pie.

PS: 7000 machines per year is on the small side so you may be better off working out a good deal with Dell but if you start to grow it may not be such a great idea any more.

Re:idiots

[Thrudheim]

You forgot this point: "4. Why is it that our network is getting slammed but similar networks at other universities are not?"

One does not need a technical background to know that if the iPhone caused problems for these kind of networks, we should be seeing them all over the place. A simple, logical process of elimination would soon cast strong doubt on the iPhone as the cause. It had to be the way that particular network interacted with the iPhone. Hence, it was the network and not the phone.

Re:idiots

[AmberBlackCat]

So if you get hired at a new company and a bunch of stuff gets broken or stolen, it's okay to blame it all on you and then figure out what happened, right?

Re:idiots

[tenman]

Okay.. I don't even know if mod points still exist in slashdot.... I haven't logged into this account in so long I don't even know what the rules are any more... this you comment made me chuckle. When i was in IT, I'd search google for it first... the use google to search slashdot for it second. Slashdot has provided me with many fixes over the years. It's to bad I don't have enough time to read it much any more. I'd add +1 funny.

Re:idiots

[Ohreally_factor]

Jesus, I love how you all are posting here like you single handly created the first router and invented TCP/IP. There can be only one Al Gore.

Re:idiots

[zippthorne]

Scott Adams wrote a whole chapter about people who add redundancy or nonsense to strengthen their statements.

Although I suppose I should say that they elucidate a plethora of needless added extra redundancy to give more force, pizazz and oomph to their otherwise mundane, banal and uncertain statements to elicit illicit confidence.

Re:idiots

[Bjorn_Redtail]

"Well, he HAD to continue to work the problem, that's his job, he didn't really have the option of simply ignoring the situation, did he? Not sure that constitutes a pat on the back." Were sorry, the iPhone is not supported on the University network. Block iPhones from accessing the network. Problem solved. Very much an option, at least in their eyes.

Re:idiots

[gb506]

That response would last about one week into the new school year. Dude, there were over 150 of the darn things ALREADY on campus, and it's a ghost town compared to September when the kids return!

Three things:

1.) if you think that Duke students will put up with an IT staff too closed minded to resolve a problem with their own network, which would result in making their $500 iPhones useless on wifi, you're crazy. The kids pay the big $$$ to go there, IT toads can be easily replaced.

2.) Duke U. has been out front in embracing iPhone's parent, iPod, as a learning tool, going so far as to issue them to incoming freshmen a few years back, if you think that some pasty cheeto eater in IT would be able to gloss over his own network problem by banning iPhone when Duke has embraced iPod in the classroom, you're friggin stoned.

3.) Lastly, and maybe most importantly so far as the iPhone itself is concerned, there are ALREADY over 150 known iPhones on the Duke campus accessing the network. The iPhone has been out for only 3 weeks. How many do you think there will be when the population on campus grows exponentially with the start of the school year? What that says is that the iPhone is ALREADY a massively disruptive technology, and that anyone who does not understand that we are seeing a massive shift away from traditional computing is either deluded or stupid.

I fart in your general direction!

Re:idiots

[arth1]

Also, even if they suspected the iPhone of not playing by the rules, they should still look at their own network and its configuration first, instead of shooting for the symptom. If a small handful of phones could possibly have brought their network down, that's a much bigger problem with their network.

Hindsight is 20/20, but in this case, it's clear that they lacked any kind of foresight too. If they truly have constructed their network in a way that's vulnerable to a few slow devices sending more traffic than they should, which is what they believed was the case, then they have a problem on the design and implementation level, in addition to the problems that were discovered.

Re:idiots

[Anonymous Coward]

No, if someone is demanding an answer right away (your boss perhaps) you state that you have not found the root of the problem yet. Placing blame anywhere without having any evidence is likely to backfire on you. As some of the previous posters mentioned, one of the first things a network engineer (or network support or whomever was troubleshooting the issue) should have done is look at the traffic. Capture some pcaps and analyze whats going on and look at some router logs or wireless logs. I'm sure part of the reason people were quick to blame the iPhone devices was because it is *easy* to blame a device that you don't own or don't have control of. Not to mention that it is new to your network. I used to see the same behavior at work with idiots you blame other peoples products.

It just boils down to people doing their jobs right and not making assumptions.

Re:idiots

[Tony Hoyle]

Cisco don't seem to test against Windows - when they put the dhcp server into ios 12.something I was bitching it didn't work.. sent loads of packet dumps to cisco, and eventually they came back 'windows is using deprecated DHCP requests. not our problem.'

Eventually they fixed it when I pointed out that the Cisco IP phone I had on the desk was sending exactly the same 'deprecated' requests.

Re:idiots

[LKM]

I agree with you, except for one thing: how the hell did an IT issue at a university turn into a national news story after a few days of debugging with quotes from the techies involved? There's nothing wrong with what they did from a tech side. They thought it was the iPhone. No problem, it was the logical conclusion at that time. But how did the media get a hold of this? Who thought it was a good idea to let the media interview the techies?

Re:idiots

[LKM]

Uh huh. Because you've never made a mistake or misdiagnosed a problem when something is broken and your entire customer base is screaming at you to fix it.

I never gave interviews on national media about it.

The Information Age

[Anonymous Coward]

We live in a such a great time. Information can spread from any source to the entire globe in minutes. However, no one actually makes sure it is correct. I'm going to go put tin foil around my bee hives.

Re:The Information Age

[janrinok]

However, no one actually makes sure it is correct.

Isn't that the entire basis for wikipedia?

Re:The Information Age

[The_Fire_Horse]

No, Wikipedia is peer moderated.
While not perfect, obvious mistakes or blatant lies are eventually picked up and corrected - this is why you shouldnt quote Wiki's, but use the references they cite.

Re:The Information Age

[janrinok]

Whoosh. Sorry, I wasn't being serious.

Re:The Information Age

[dakameleon]

However, no one actually makes sure it is correct.

Isn't that the entire basis for wikipedia?

[citation needed]

I'll feel bad...

[CCFreak2K]

...for the poor guy who said it wasn't a Cisco problem when he starts getting those Apple fanboy death threats.

Re:I'll feel bad...

[Anonymous Coward]

Nothin' scarier than a couple of Apple-lovin' nerds

Re:I'll feel bad...

[samkass]

when he starts getting those Apple fanboy death threats.

You mean when hack journalists start reporting unsubstantiated rumors of death threats.

Re:I'll feel bad...

[Anonymous Coward]

Threatened by a Mac user? What are they going to do, stab him with a Stylu... oh wait...

Re:I'll feel bad...

[Anonymous Coward]

Duke IT underling: We figured out the problem. Turns out it was a Cisco problem, not an iPhone problem.
Duke Chief Information Officer: Sweet! Let's hire some strippers.
(Allegedly heard at a Duke IT meeting.)

Re:I'll feel bad...

[Anonymous Coward]

I *already* feel bad for the Nukem Forever dev team as this scope creep continues.

Could we please..

[Anonymous Coward]

..get some statements and reflections from the usual crowd all-knowing trolls who instantly pointed at Apple in the previous, related thread? Just for some reference, ofcourse.

Re:Could we please..

[Anonymous Coward]

Do we ever get such statements from the lunix and apple zealots after we find out that *gasp* Microsoft was actually not responsible for something wrong that happened?

Oh, wait... slashdot never posts those stories.. I forgot... /. only vindicates apple and lunix.

Re:Could we please..

[Anonymous Coward]

Do we ever get such statements from the lunix and apple zealots after we find out that *gasp* Microsoft was actually not responsible for something wrong that happened? Microsoft probably was involved. Cisco uses Microsoft software internally and it is causing no ends of problems, particularly with respect to defect tracking.

(posted anonymously because I work within Cisco).

More information?

[physicsnick]

I'm curious to find more information on this. TFA just says "Cisco has provided a fix". What nature of fix was this? Was it actually a flaw in the routers, or did someone just configure them wrong?

Given the widespread use of Cisco routers compared to the isolated nature of the problem, it sounds a bit like Duke is just trying to save face.

Re:More information?

[Anonymous Coward]

Cisco equipment definitely has it's share of bugs, and while it's usually solid I've definitely seen some highly broken crap.

I want a bug id!

Re:More information?

[Anonymous Coward]

They did this for my university too. And then made us sign an NDA so we wouldn't talk about their hardware sucking. Not me, though, so I don't care...I was just the guy that couldn't do my job until the techies figured out a solution.

http://hardware.slashdot.org/comments.pl?sid=25112 9&cid=19886053

Our other routers and access points work perfectly. For instance, I have a dozen PCs with Intel network cards that when set to autonegotiate, they get pretty much crippled speeds (feels like dialup)...I have to set the speed to something completely different than our network engineers tell me is right to get any decent speed. The throughput is around a third of the others (100bT) on the Cisco routers. On the old HP routers, it worked perfectly. They work perfectly when put back on other networks. I don't really care to try to fix it at this point as these make up a small amount of my machines and unless I'm pushing images, I really don't care about the speed. So it was never just an Apple problem. Cisco makes substandard products these days and don't seem to mind living off their name and reputation of the past.

Re:More information?

[Anonymous Coward]

I'm sure that is was less of a firmware fix and more of a configuration provided by TAC. With the wide-spread use of aironet APs, I'm sure more than just Duke would have had this. So it gets down to how did Duke have their wireless network configured. It could be as simple as they dump unauthenticated wireless clients into a guest VLAN and that VLAN didn't have enough IPs available in that subnet. And if they use like LWAPP APs and force people to use a login before getting out to the internet, it could be that their guest network ran out of IPs allowing no one else to get on to authenticate.

Re:More information?

[Anonymous Coward]

Ok, thanks to the AC, I learned that slashdot has a hard-on for cisco equipment like they do anything apple or linux.

That's why there are so few comments that have been modded up in this article. The mods simply can't decide which side they're supposed to root for. :-p

Re:More information?

[mikkelm]

So because a handful of identical NICs didn't play well together with whatever Cisco equipment they were interfacing with, instead of investigating the problem and finding out whether the NICs were too liberal in their interpretation of the autonegotiation specifications of 802.3u, or the Cisco equipment somehow messed up a process that should be identical for all NICs with -only- those specific NICs, you chose to just blame Cisco and call their products sub-standard.

I don't know about anyone else, but I'm not really convinced by your anecdotal evidence that seems to be solely based in assumptions.

Re:More information?

[ZWithaPGGB]

"Given the widespread use of Cisco". So Windows must be pretty good too, right?

Cisco is the Microsoft of networking gear. Their stuff is complete crap compared to the alternatives in every category. It's also overpriced.

People buy Cisco for the same reason Chambers used to be able to get them to buy IBM Front End Processors (where he cut his teeth as an exec), because No-one gets fired for buying what everyone else buys. They SHOULD be, because they are just buying on inertia, but they don't.

Re:More information?

[jaypeg]

I hear they put a together a crack IT team. Then they all went to the local Starbucks and asked them how to set up a wireless router.

Re:More information?

[skoaldipper]

hmmm...

Take your pick. That's just for one controller series. Who knows which Cisco hardware Duke is using, but the problems associated with LEAP (lightweight extensible authentication protocol) are of particular interest.

I'm not pointing fingers either. I have to many broken of my own.

Re:More information?

[physicsnick]

"Given the widespread use of Cisco". So Windows must be pretty good too, right? You misunderstood. I wasn't implying anything about the quality of Cisco routers.

Suppose Duke University (and only Duke university) suddenly has problems with all of their Windows boxes. Do you think it's a Windows problem? Given the widespread use of Windows compared to the isolated nature of the problem, it's far more likely that they themselves configured something incorrectly, otherwise all universities should be encountering similar problems.

This isn't to say that there aren't such problems; just as you said, both Cisco and Windows have widespread flaws that affect all universities. But for THIS particular problem, it's more likely to be just a misconfiguration, simply because of the fact that it's localized to Duke.

Re:More information?

[Anonymous Coward]

oh REEEEEALLLYYY?

like you have anything better, mr smartypants! well? do ya, punk?

Re:More information?

[macdaddy357]

They sent in the Cisco Kid!

Cisco Kid was a friend of mine
The Cisco Kid was a friend of mine
He drink whiskey, Poncho drink the wine
He drink whiskey, Poncho drink the wine

We met down on the fort of Rio Grande
We met down on the fort of Rio Grande
Eat the salted peanuts out of can
Eat the salted peanuts out the can

The outlaws had us pinned down at the fort
The outlaws had us pinned down at the fort
Cisco came in blastin', drinkin' port
Cisco came in blastin', drinkin' port

They rode the sunset, horse was made of steel
They rode the sunset, horse was made of steel
Chased the gringo last night through a field
Chased the gringo last night through a field

The Cisco Kid was a friend of mine
The Cisco Kid he was a friend of mine
The Cisco Kid was a friend of mine
The Cisco Kid was a friend of mine
Cisco Kid was a friend of mine
The Cisco Kid he was a friend of mine
The Cisco Kid was a friend of mine
The Cisco Kid was a friend of mine
Cisco Kid he was a friend of mine
The Cisco Kid he was a friend of mine
The Cisco Kid he was a friend of mine

Re:More information?

[bryan1945]

I've been out of the networking world for a while. Could you tell me which companies are better in the different categories? (such as home, small business, large business, really big ass deployment)

Or even some sites that do these kind of comparisons?

Thanks in advance.

Re:More information?

[ZWithaPGGB]

Juniper for routers. Extreme for Network Switches. Juniper/Netscreen, Fortinet, or even Checkpoint for firewalls. Intruvert for IDP. Aventail for VPN. Aruba for Wireless.

Even a Vyatta or other OSS router is as good as or better than all but the biggest, and most horribly expensive, Ciscos.

But you knew that, because you couldn't point to any evidence that refuted my opinion that Cisco has more than just market share in common with MS.

Re:More information?

[ZWithaPGGB]

But it isn't Duke, and only Duke, or even iPhones, and only iPhones, that have been having problems with Cisco APs. It's anything other than Windows clients. Cisco has just done a good job of hushing it up by requiring that people sign NDAs to get the fix.

Re:More information?

[HockeyPuck]

Cisco is the Microsoft of networking gear. Their stuff is complete crap compared to the alternatives in every category. It's also overpriced. I think you hit the nail on the head. Alternatives in every catagory. Which means you have 500 different vendors. From core routers, to access switches to firewall appliances, to Content/Caching engines to telephony to wireless, heck Cisco even makes storage switches. If there's a nework problem, you call up ONE company. You sign one large support contract, makes it very easy to have 'one neck to choke' when there's an issue.

When you build a server (not a hobbiest linux box at home) would you rather buy all the parts (cpu, ram, disk, etc..) from ONE vendor, or would you rather buy each component from someone else? You'd call up IBM/hp/dell/sun and order a server, so when the ram breaks you call the same vendor as when the CPU breaks.

While cisco gear may not be the best in every catagory, the solution as a whole is pretty good and there's not a networking vendor that can provide an 'end to end' solution. Plus there's something to be said for being able to put firewall/content/PoE/WAN modules in a single chassis.

Integration and consolidation does save power.

Re:More information?

[HockeyPuck]

I'm not saying that there aren't vendors with single produts that are better, but NOT all companies/customers are looking for 500 different vendors. You wouldn't build a server farm from 50 different linux servers b/c "IBM w/redhat is better at dns, HP w/Ubuntu is better at samba, Dell/slackware is better at sendmail..." you'd go outta your mind supporting such a hetergenous infrastructure.

Cisco/MSFT have plenty in common. All religions aside, when you hire someone it's much easier to find someone that is familiar (CCIE) with a broad range of cisco products than to find one that has (as you put it), "Juniper for routers. Extreme for Network Switches. Juniper/Netscreen, Fortinet, or even Checkpoint for firewalls." The same holds true if you were hiring someone with office skills. It's much easier to find someone that is well versed in MS-Office than it is to find someone that has the same skillset in lotusnotes, wordperfect, etc...

Building an IT infrastructure is more than just having the 'fastest, best out there'. It's building the best solution for YOUR environment. I work with plenty of clients that have Juniper in the core and cisco at the access/distribution layer.

Re:More information?

[Anonymous Coward]

Those cisco NDAs are hidden in area 51!!!!

Re:More information?

[Anonymous Coward]

Perhaps you could back up those statements?

Why/how is Juniper better for Routers?

Why/how is Extreme better for switches?

Same question for your FW's, IDP,VPN, Wireless

For every document you find saying that vendor X is better, I'm 100% positive I can find one saying that Cisco is better.

TO keep everything even, No Gartner(major anti-cisco bias) , No meircom (major pro-cisco bias)

Lets see what you come up with Mr "Everything Big Business does is crap and OSS developers are l33t h@ck3r$ with mad skillsss"

Re:More information?

[Anonymous Coward]

this is true (atleast in case of Aruba and Netscreen based on personal experience. Both are best in class). However when your network goes down or you have basic connectivity issues who do you call and in what order? The reason people buy all Cisco is what another poster mentioned: when shit hits the fan there is one call you make and one support contract. You dont bounce among vendors or have them point fingers at each other...

Re:More information?

[ZWithaPGGB]

Except the finger was pointed @ Apple in this case until a smart engineer @ Duke figured it was Cisco. Where was the benefit in the big company standing behind the stuff then?

Where is the benefit to everyone when the fix was subject to NDA?

In my experience, Cisco almost always pushes back with the same thing as Microsoft any time there is an interoperability problem: "We ARE the standard, so it must be them."

Old joke:
Q: How many big_dominant_company engineers does it take to change a Light Bulb?
A: None, they just declare darkness the standard.

Re:More information?

[profplump]

There are benefits to having only one provider -- like consolidated support and supposed interoperability.

There are also costs, like lock-in -- not only are in a position to be taken advantage of by your single provider in terms of price, but you're actually likely to dimiss technically superior solutions if they don't come from your provider, and your solutions will be inflexible outside the bounds set by your provider.

Take Exchange email as an example. It's not a terrible way to do mail folders, and the integrated calendar is handy. But it means that only Outlook can read your mail, and Outlook only runs on Windows machine and Windows only runs on x86 hardware. Those may all be reasonable choices, but it's not reasonable to make them all simply because you want to run Exchange.

I know Exchange supports IMAP, but there are companies (like my current employer) that won't turn on the IMAP interface because it doesn't fit their one-provider toolchain. I develop on a company-provided linux system for a product that runs entirely in linux, but I have to launch a VM to check my email because of single-provider lock-in.

Re:More information?

[hxnwix]

Yes, the "you're the only one having this problem" defense.

Tell me, do you work for Oracle's support dept? Seriously, I think we've spoken...

Re:More information?

[amper]

You sign one large support contract, makes it very easy to have 'one neck to choke' when there's an issue.

Unfortunately, the "one neck" often turns out to be yours, rather than the vendor's...

The reason Cisco's gear is dominant in the networking marketplace has nothing to do with superior hardware, software, or service. It does, however, have quite a lot to do with the fact that Cisco was one of the first players in the IP router market, with products that frequently failed to interoperate with other brands of equipment. The large WAN providers, having a large installed base of Cisco gear, therefore practically forced all their clients into using Cisco routers, since that was the only thing they could guarantee would work with their network. Those of us who knew better chose other brands, and supported it ourselves, at least until it became difficult to so because of mergers and marketplace attrition.

So, yes, Cisco *is* the Microsoft of networking. It's sad, really.

Re:More information?

[Anonymous Coward]

You have obviously never been on a 3-way conference call trying to diagnose a problem involving 2 pieces of equipment from different vendors...

Also anybody that thinks that an "OSS router is as good or better" obviously uses virtually none of the routing features that Cisco (or Juniper) offers.

If all you need is ethernet to ethernet routing with NAT, check out IPRoute, circa 1993.

Re:More information?

[ZWithaPGGB]

Go troll elsewhere mister AC.

If you really want answers to your questions, use Google and read the independent product tests from Tolly, Network World, or just about any other non-marketing group.

Re:More information?

[Anonymous Coward]

He wasn't calling Cisco substandard - just reporting the facts as he knew them
1) Stuff works with previous HP equip
2) Stuff doesn't work with Cisco equip
3) Cisco makes them sign NDA to make stuff work with Cisco equip.

_Then_ he may have, and many others do, call Cisco substandard, overpriced crap - Windows ME, Mars Climate Orbiter, Itanic, "Jingle All the Way", EDS, SAP, Denver Airport Baggage-handling System, Zune crap.

People who have to deal with network equipment, whether they are "network people" or not, don't come to despise Cisco based on price alone. They despise it based on quality, price, unwillingness to be forthcoming about problems (and prosecuting those who do come forward to expose problems) but esp. lack of interoperability.

Please get real, use google. Problems with cisco and autoneg are notorious and pervasive, since then cisco has learned the way corporations learn - don't improve quality in the future, instead become better at covering up problems and silencing critics.

http://en.wikipedia.org/wiki/Autonegotiation "Even Cisco recommended its customers not to use autonegotiation."

Re:More information?

[iceT]

This is funny. You just highlighted the business strategy behind Microsoft too.

Of course, this is why people pay for for a loaf of bread at 7-11. Convenience.

What good is one throat to choke when you spend all your time trying to choke it?

Dont you people know

[Anonymous Coward]

its NEVER the network! Cisco is untouchable! :)

Correlation is not cause and effect

[PIPBoy3000]

This is unfortunately a common issue with people. When two events happen at about the same time, people assume they're somehow connected. The autism and vaccine link, for example, is one of those things where they get their shots and soon afterwards, they notice their child is acting strangely. Then there's the old "this coincidence must be a sign of the divine" theory.

We run into this all the time when doing server administration. For example, one of our developers found that web pages were slower on our new virtual servers. The obvious thought is that virtualization=slow. It turns out that compression hadn't been turned on for those servers. Since he was going over a slow VPN connection, it made a fairly significant difference. Once switched on, they worked about the same as real servers.

Re:Correlation is not cause and effect

[kevorkian]

We run into this all the time when doing server administration. For example, one of our developers found that web pages were slower on our new virtual servers. The obvious thought is that virtualization=slow. It turns out that compression hadn't been turned on for those servers. Since he was going over a slow VPN connection, it made a fairly significant difference. Once switched on, they worked about the same as real servers.

Yea , but it was still 'something' related to the change that was made.

The dev may not know all about what was done. All he knows is that "before the change it was fast" and "after the changes it was slow". His only information about the change is that it was new VM servers.

Because of the fact that his knowledge of the change was limited. His observations are no less valid. to him it IS the vm servers that are slower.

You mention server administration , so I assume that you do something tech like as work. When you walk in the door on Monday and there is a problem, do you start trouble shooting the whole system ? or do you first ask "what has changed" and start looking at it from that point ? 9 times out of 10 if something has changed , thats the cause of the problem.

The good thing about this story is about how apple and cisco were able to come together to find the problem. In my experience , cisco is one of the few company's that will admit when its there stuff thats broken. At least once you get through the first levels of support. And duke most likely has a ccie on staff , or a provider contract with cisco to gain access to real support.

Re:Correlation is not cause and effect

[Zebra_X]

In your virtualization example - there are mnay more varibales to isolate before you can declare "virtualization is slow". To conclude this based on a remote employee on a new server with a configuration that may or may not be the same is a bit of reach.

If there is an exisiting network that "works" and then a new device is put into use on the network - and then the network breaks... it is reasonable to conclude either the device is a potential source for the issue.

While details are sparse I suspect the Apple network stack was trying to do something completly valid, and the cisco routers were screwing it up.

However, the iPhone was involved as he cites thanks to Apple and Cisco tech support.

Re:Correlation is not cause and effect

[trb]

Speaking of correlation, if I were at Duke, before blaming Apple or Cisco, I'd ask myself the question - is this problem happening anywhere else? If not, why not? Then how likely is it that the problem is with my basic system rather than with the way I've configured it?

Re:Correlation is not cause and effect

[farker haiku]

For example, one of our developers found that web pages were slower on our new virtual servers. The obvious thought is that virtualization=slow. It turns out that compression hadn't been turned on for those servers.

So how was he wrong? The virtual servers were slower.

Re:Correlation is not cause and effect

[sholden]

Because "virtualization=slow" is completely different than "The virtual servers were slower".

Re:Correlation is not cause and effect

[farker haiku]

actually, you'll notice that the original poster's word choice didn't indicate what the dev said exactly. He said "The obvious thought is that virtualization=slow. " That may be how they interpreted the dev's complaint, but since he didn't quote the dev exactly, we can't know for sure.

But, what we know now is that there is a definite correlation between pointing out someone's mistake, and them flaming you on slashdot. Oh, and modding you troll. That's also much more likely to happen apparently.

Re:Correlation is not cause and effect

[mercury83]

I get this all the time from my software engineer coworkers trying to resolve problems. They notice something that seems unusual and they also note a decrease in performance. They assume that the one caused the other immediately without any other reason and then spend hours trying to resolve the unusual condition that they first saw. It's amazing how often the unusual condition is completely unrelated. In fact, it seems like when something goes wrong, it's easy to spot a whole bunch of problems in your configuration even though none of them are causing the problem.

If you want to sound a little pedantic, this logical fallacy has a name: Post hoc ergo propter hoc.

Why don't they teach logic at these schools?

Re:Correlation is not cause and effect

[sholden]

The other bit I can't see that you seem to is where the dev was declared to be "wrong"?

Re:Correlation is not cause and effect

[jrumney]

Not that it makes a difference to your point, but the whole autism/MMR scare was not about individual parents noticing their children were autistic after the vaccination, it was about the fact that the recorded rates of autism started increasing about the same time that the MMR vaccine started being given to children. It was also disproven by the same method; Japan stopped giving MMR a couple of years after they had started, due to a problem with the mumps vaccine they were using (Europe and North America were using a different vaccine). But their recorded autism rate continued to grow in line with other industrialised countries.

There was another alarmist report recently (in the last month), but the report itself stated that the recent sudden rise in autism rates was due to reinterpretation of the statistics, which seemed to invalidate the report's conclusion that MMR was somehow involved. It turned out the report's authors were involved in the original discredited research.

Re:Correlation is not cause and effect

[LKM]

Well, Apple was involved since they thought it was the iPhone's fault. Whether or not it ended up that the iPhone actually was included is unclear (even though it seems likely).

So what was it

[Anonymous Coward]

I read in another article that some Gartner group guy speculated that it was a problem with the wireless network at Duke's security settings, that they were using LEAP (Lightweight Extensible Access Protocol). Since that was an unusually technical speculation by a Gartner-ite, I'm curious if anybody can confirm/deny that.

Re:So what was it

[freeweed]

Take anything Gartner says and apply the old adage: a stopped clock is correct twice a day. That seems to just about cover their accuracy in most technical matters.

Re:So what was it

[sammy baby]

I prefer Charles Wang's assessment of the folks at Gartner.

"I want to choose my words carefully here, so I'm not misunderstood," he said. "They're a bunch of fucking idiots."

Sorry I can't provide an authoritative cite... but even if it's apocryphal, it's so perfect that I can't care.

Re:So what was it

[antifoidulus]

but even if it's apocryphal, it's so perfect that I can't care.

Are you sure you don't work for Gartner?

Not the iPhone but the iPhone

[tepples]

So it wasn't Apple's iPhone but Cisco's Linksys iPhone that was causing problems, am I right?

Most Don't understanding networking

[henryhbk]

Many network IT folks just understand how to change settings on routers (what you learn to do in a "certification" course on a router) and understanding networking. Networking is more than just some router settings, and understanding the organic interdependent flowing nature of a network is critical to debugging problems. Just knowing something is causing a problem, and blaming the most recent change as the cause (as opposed to some underlying problem that this change simply brings to light). A senior IT official should, even if he doesn't know the exact problem, know that weird entworking problems are often way more complex than they seem, and should not jump to knee-jerk conclusions (especially based on some 1994 anti-mac bias about networking)

Re:Most Don't understanding networking

[TheEmptySet]

"weird entworking problems"

Damn those Ents and their slow decision making. First they nearly refused to act to stop the downfall of middle Earth and now, even worse, they are causing problems with Steve's divine creation. Personally I think we should ban them from having I phones if they are going to do this.

Re:Most Don't understanding networking

[j00r0m4nc3r]

Maybe that's why the entwives left

Re:Most Don't understanding networking

[Thundersnatch]

...especially based on some 1994 anti-mac bias about networking...

I managed the all-Mac network at a newspaper in 1994, and let me tell you, Macs did suck at networking back then. Unless you used a single, unbridged AppleTalk domain, which scaled up to exactly 5 workstations.

Getting Macs to reliably access the Internet or other TCP/IP resources back then was also pretty much impossible. Reboots were required approximately once every hour.

Praise?

[Anonymous Coward]

>>Duke's Chief Information Officer praises the work of their technical staff.

This sounds like every company in america... praise for mediocraty. The "technical" people on an
IT staff at a famous university can not figure out what is wrong with traffic on a network?? When
I saw the original post I thought it had to be a joke... Duke, not being able to figure out a network
problem?? They should not be praised, they should have been fired.

And yeah, don't go on with this "oh, but its a big network" retort either... If you work on a big
network, you should be able to fix it.

lol @ Cisco

[Neuropol]

I heard a *story form some one that was an old GECO programmer way back when. In the midst of telling me stories about 3 foot diameter platters on huge racks and resistors the size of fists, he went on to tell a story of Cisco and and imminent wireless network failure.

It seems that years ago, some where in Europe, there was an issue with Cisco equipment failure at intervals. Massive wireless network failures and completely indeterminable. After bringing in a team of engineers for some thing like 9 months to assess the situation, they were still unable to determine the cause of failure. Until some one had peered in the way packets were being labeled under VLAN. Apparently Cisco had a number algorhithm issue, by which, say, every 10,000th packet, was getting dropped after being mislabeled and misdirected. After a so many identifying packets were getting dropped, total failure would ensue. I believe there was speculation of a Linux hardware and Cisco compatablity thing happening. Same protocols, just one was taking the shorter route to completion and it just happened to be Cisco to which all fingers pointed.

*The details of this may not be exact, but the ideas and names will remain the same to point out the guilty.

Is a pattern emerging?

[Anonymous Coward]

Is Kevin Miller related to Mike Nifong? Beware the science coming out of this institution.

Everyone is a winner

[BillGatesLoveChild]

Cool. Cisco screws up, iPhone gets blamed, but nobody minds, because iPhones are so cool.

Boss: "Did you get those reports done?"
Underling: "Sorry Boss, I Couldn't. iPhone Congestion."
Boss: "iPhone? ... (smiles) iPhones are cool aren't they!"
Underling: "They sure are boss!"

Boss wanders off feeling good.
Underling returns to screwing around with his iPhone.

Cisco gear just isn't that good.

[CRC'99]

I think that after spending a number of years working in Cisco only networks, I'm constantly amazed at the generally poor compatibility and functionality of Cisco equipment.

This ranges from critical recovery steps being removed from the 7200 series G2 NPE (NEVER make one of these crash to ROMMON on boot. The fix is to RMA the NPE) for Xmodem recovery of bootloaders - something a basic 827 router has to their latest 7961 VoIP SIP phones that are apparently RFC compliant for SIP communications - but aren't.

There are MANY things that make Cisco equipment worse and worse as the years go by. Part of it I believe is the outsourcing of the people who write the software for these things now. Chances are that they weren't even around with Xmodem was in use - and I bet a lot of the coders have NEVER admin'ed a network of Cisco gear. This is the only thing I can think behind removing essential recovery procedures for $35,000AU routers.

There's a whole new direction that Cisco is heading, and with the stupid things missing from their new gear, I'm starting to wonder if it's a direction that will have huge impacts for the worse in the network admin side of life.

Re:Cisco gear just isn't that good.

[jkbull]

...(NEVER make one of these crash to ROMMON on boot. The fix is to RMA the NPE)...

I understand the ROMMON, RMA, and NPE acronyms, but what's NEVER stand for?

Re:Cisco gear just isn't that good.

[CRC'99]

...(NEVER make one of these crash to ROMMON on boot. The fix is to RMA the NPE)...

I understand the ROMMON, RMA, and NPE acronyms, but what's NEVER stand for? The NEVER stands for what I mean when I don't want to sit through 8+ weeks of rubbish from Cisco to get the thing RMA'ed (lucky it was in our testing phase and not live equipment). The TAC closed the case off and refused the warranty and it's been put on the account managers plate to fix. You can think of it as _never_ or never - which ever you like. I still refuse to use the flash tag though ;)

Re:Cisco gear just isn't that good.

[eblum]

I think the reason for some of the most expensive hardware being difficult to use, is only an excuse to charge a lot for training and certification. The real difference between a $50 router and a $9000 router is basically expansion capability for special modules, CPU power to apply thousands of rules to filter packets and of course $8950.

If you are just going to route packets from network A to network B based on the IP address, both routers will perform just fine.

Another example are some workstation computers that justify their price by building their case of thicker metal plate just to make them heavier. You wouldn't expect expensive work stations to be light and small as the cheap ones although they have the same internal components and OS. Wouldn't you?

Re:Cisco gear just isn't that good.

[jkrise]

I think that after spending a number of years working in Cisco only networks, I'm constantly amazed at the generally poor compatibility and functionality of Cisco equipment. There's a very simple explanation. Cisco, Oracle and Microsoft are 3 big gorillas who make a business of selling closed-source complicated implementations that are supposed-to-be standards-compliant. Unfortunately they are not truly standards-compliant, besides being overly complicated and obfuscated. Hence such compatibility issues.

Notice we have no clue in this particular incident, of what exactly triggered this response from Cisco equipment. It is like a Service Pack from Microsoft - we only get vague details on what it fixes, not how it does it, why the original problem was caused, and who was actually responsible.

Re:Cisco gear just isn't that good.

[technormality]

Are you saying that tftp is no longer support in ROMMON on a 7200/G2NPE? I have a few deployed and haven't had any crash yet nor had a reason to tinker in their ROMMON. So knowing more about this type of scenario would be helpful. Latest Cisco docs (last updated this month/link below) still shows TFTP recovery procedures. http://www.cisco.com/en/US/products/hw/routers/ps3 59/products_tech_note09186a00800a750e.shtml/

Re:Cisco gear just isn't that good.

[gblfxt]

Network Engineering Verifying Energy Remote

Re:Cisco gear just isn't that good.

[CRC'99]

ROMMON doesn't have ANY networking support. I think you may be talking about the bootloader. The G2 has 3 different layers when it boots.

1) ROMMON - this is responsible for initialising the device and loading the bootloader from the bootflash: device. The bootflash is soldered onto the NPE.

2) Bootloader - This starts up and initialises the network. It parses the startup-config and then launches the IOS image.

3) IOS - This again parses the config and loads the actual unit.

The bootloader does have networking functionality, however if ANYTHING ever happens to the bootloader or bootflash:, then you will have no networking, no Xmodem recovery, nothing. To make things worse, you cannot even load a bootloader image from the compact flash card.

Re:Cisco gear just isn't that good.

[skeeball]

Are you sure the problem was just the bad compact flash that Cisco received from another manufacturer? The solutions was the RMA the flash, not the NPE. Xmodem is a little slow considering how large IOS images have gotten, but you can TFTP boot the router, did you try that? Most engineers I run into that bitch and moan about Cisco simply don't know how to configure things correctly.

Sounds like you got bad TAC advice. Which is a shame because I seem to get better support from the Australian TAC than even San Jose.

Re:Cisco gear just isn't that good.

[CRC'99]

The ROMMON on the G2 has zero network functionality - therefore you can't TFTP anything across. You also can't Xmodem over the console.

The flash card was good, but the bootflash: image was deleted (part of my testing). From this point, it was impossible to recover the router.

I work for a very big Cisco shop

[Random Q. Hacker]

All of our 100mbit servers and the Cisco switches they connect to have to be nailed to 100/Full, because the Cisco hardware refuses to negotiate properly with HP Proliants and Sun hardware. Yet a $40 D-Link can manage just fine.

Not to mention how buggy IOS releases have become in recent years.

I wonder how many folks will start checking out Juniper.

Re:I work for a very big Cisco shop

[CRC'99]

Wow - I think my experiences are quite the same... Check out my post above yours :P

Re:I work for a very big Cisco shop

[cciechad]

I call BS on this. We have 1000's of HP and Sun boxes on my DC network that all auto-negotiate fine with my 6513's. In the far past there was a Sun bug that broke autonegotiation on certain Sun adapters that was caused by Sun not following the RFC's. I believe this has been fixed in Solaris 9. Chad

In Other News...

[Anonymous Coward]

/.ers list the total number of IT troubleshooters who have never made a bad call...

.. er, that's it.

Re:you're

[uglyduckling]

Shouldn't that be "fuck's sake" or "fucks' sake" - certainly not "fucks sake"..?

But...

[Anonymous Coward]

assistant director Kevin Miller said it was Apple's fault.

"Kevin Miller, assistant director, communications infrastructure, with Duke's Office of Information Technology, laid the blame for Duke's networking problems squarely on the iPhone.
"I don't believe it's a Cisco problem in any way, shape or form," Miller said firmly."

Yup.. he put the ASS in assistant director.

Keep this quiet!

[Anonymous Coward]

Hey guys,

Please keep this quiet until the first software update for the iPhone becomes available.

I told my CIO the same story. I'm reading up on networking now, and I think I can fix it.

Thanks,
Anonymous
Director of Networking

To be fair....

[Vacuous]

To be fair, who hasn't had an issue where you were SURE it wasn't one thing, when it actually was. I would imagine most of you, like me, have seen issues where you still can't explain how you fixed it.

Re:To be fair....

[kybred]

To be fair, who hasn't had an issue where you were SURE it wasn't one thing, when it actually was.

Exactly, so why would he make a public statement until the problem was resolved?

Re:To be fair....

[Vacuous]

One of 2 explanations, an enormously bad judgement call, or, what I suspect, he was under pressure to make a statement.

Re:To be fair....

[byronf]

To be fair, who hasn't had an issue where you were SURE it wasn't one thing, when it actually was. I would imagine most of you, like me, have seen issues where you still can't explain how you fixed it.

To be fair, who hasn't had someone blame you, publicly, for a problem that wasn't well understood yet, just to find out later it had nothing to do with you.

The criticism is not for making a mistake, we all do it... But rather for prematurely assigning blame. Saying in so many words, I know it can't be Cisco, the iphone is the only logical conclusion, and Apple isn't cooperating! sniff sniff... smells like agenda.

Meh.

[morari]

They should still ban iPhones... and all mobile telephones. Annoying junk!

Re:Meh.

[Anonymous Coward]

Wow! Your comment was remarkably insightful. You clearly put a lot of effort into that. I haven't seen such originality in years. It's a shame that Slashdot shouldn't allow users with IDs higher than 1 million to post. Annoying twits!

Re:Meh.

[gelfling]

and electricity too and the damn horseless buggy. you kids get off my lawn

Re:Meh.

[foniksonik]

Yea and while their at it get rid of those dang newfangled auto-mobiles... I can never find a place to park my buggy, and them new quarter eatin' hitchin posts aren't worth the quarter... my horse just riggles his lead free in two minutes, too slick I tell ya... a good ol fashioned railroad tie is the way ta go.

Obviously a Cisco Problem All Along

[smack.addict]

The sick thing is that it was OBVIOUS it was a Cisco problem from the start. If you make the assumption that the iPhones are somehow defective, it's still a Cisco problem because any defective behavior from an iPhone would be indistinguishable from malicious behavior from a student. The fact that the iPhone was involved really was a non-issue all along.

It was terribly irresponsible of them to go off blaming Apple and, worse, absolving Cisco of responsibility.

Re:Obviously a Cisco Problem All Along

[Anonymous Coward]

this comment ends the discussion. there is absolutely nothing more to see here. please move along and have a nice day.

Re:Obviously a Cisco Problem All Along

[Slashcrap]

The sick thing is that it was OBVIOUS it was a Cisco problem from the start. If you make the assumption that the iPhones are somehow defective, it's still a Cisco problem because any defective behavior from an iPhone would be indistinguishable from malicious behavior from a student.

It's a wireless network. If wireless client devices spam the network to the extent that all of the available bandwidth is consumed, then there is not the slightest fucking thing that the wireless routers can do to keep the network available.

When you say that it was obvious from the start that the Cisco routers were to blame, you're not doing so because you know the technical details and have diagnosed the issue. You're saying it because as far as you're concerned, Apple are always blameless.

Re:Obviously a Cisco Problem All Along

[smack.addict]

Or, you are an idiot who does not know what he is talking about...

Re:Obviously a Cisco Problem All Along

[Bill, Shooter of Bul]

Good point. I agree they shouldn't have exonerated Cisco, but you have to keep in mind the Denial of service angle. You can very well destroy all wireless network activity with enough noise at the right frequency. There really is no defense for that.

Re:Obviously a Cisco Problem All Along

[Anonymous Coward]

Very childish.

Poor Sake

[Oyume]

Why is everyone picking on Sake? I mean, sure it's served slightly warm and in tiny cups, but it's not all that bad to drink, is it?

Re:Poor Sake

[Maserati]

Sake is only "bad to drink" until you figure out how wildly dangerous hot liquor can be. Once you get used to the difference, it's great. Just don't forget to take the cap off the bottle before you heat it up.
</pedantic>

Re:Poor Sake

[drinkypoo]

Sake is only "bad to drink" until you figure out how wildly dangerous hot liquor can be.

The tradition of heating sake started because it makes shitty sake more drinkable. Those who care still drink it cold.

And those who don't care, but prefer unfiltered sake, also drink it cold.

It Was the Stripper's Fault

[Doc Ruby]

Everyone knows that Duke's got problems with open access. It's the access point's fault, not the new units that play a little rough with seasoned pros.

Though the expert officials blaming the wrong party should find a new line of work. I suggest politics.

Re:It Was the Stripper's Fault

[Anonymous Coward]

I guess you missed the part where the state (after lengthy investigation) dropped the case and declared the lacrosse players innocent?

Nifong used the case to get reelected. The "victim" was clearly lying (which should have been obvious by the 5th time her story changed.)

Re:It Was the Stripper's Fault

[Doc Ruby]

I guess you missed the part in my post where I said exactly that.

Or the part where I linked to how the prosecutor was fired and disbarred (though it looks like the CNN article to which I linked is now broken).

Anonymous illiterate Coward can't even think straight when they agree with me. Maybe something's jamming the plate in their head, like fevered lust to join the Duke lacrosse team, though they're wedged between a gamer's chair and their keyboard, tied by a gamer's catheter to their mom's basement.

If you're turning off autonegotiate...

[msauve]

and forcing 100/full, you had better be doing it at both ends.

If you aren't, then the devices will come up as half duplex (assuming they properly implement the standards), you have a duplex mismatch, and you _will_ have network problems. 802.3u requires an end which is set to autonegotiate to assume half duplex if the other end will not autonegotiate.

Except, some Suns can not be forced and will only autonegotiate, in which case you MUST set the switch port to half duplex if you're forcing.

Re:If you're turning off autonegotiate...

[ivan256]

The whole thing wouldn't be an issue if Cisco hadn't patented their auto-negotiation process. If we had reliable auto-negotiation nobody would have to force anything.

First blame the user

[metoc]

Typical reaction. Something goes wrong, blame the user.

After all you and your staff would never mess up, your high status, reputation and salaries are at stake. The fact that you have outstanding trouble tickets, perform patches and upgrades without testing are coincidental.

So something goes wrong, you blame the user, remove them and claim problem solved. In the background you quietly find the problem and fix it (as part of routine maintenance). Your reputation is intact and all is good with the world. If only their were no users it would all be perfect.

Re:First blame the user

[Anonymous Coward]

As an IT staff member at a state university, I stand by Randall Graves' statement:

"This job would be great if it weren't for the customers."

tag!

[jeffro69]

Might want to tag this as an apple story, since the previous article was in the apple section.

Re:idiots.. But it is true...

[Fallen Kell]

I and my group have experienced this at work all the time almost whenever a new person is hired into the network team. Cisco gear do NOT play nice with Sun Microsystems, be it their desktop workstations or their servers. The Cisco gear refuses to properly auto-negotiate with the equipemnt causing issues such as duplex/simplex mis-matches (i.e. the workstation thinks it is connected at 100 Full duplex, while the switch thinks it is connected at 10 Half duplex). Needless to say this causes all kinds of collisions, IErrors, OErrors, etc., on the system and the network. All the Sun gear must have their associate network partner's port forced to 100 Full, and we do the same for the system as well. How do I know the problem is with the Cisco gear? Because the workstation/server works fine if you use a HP, Xylan, Baynetworks, or other switch. The net network engineers immediately believe it is the Sun equipment because they have been brainwashed into believing that Cisco can't make a mistake or a poor product. It usually takes us to demonstrate using 2 or more other switches that the problem only happens on the Cisco. Cisco still denies that there is a problem as well.

Oh and if you don't believe me, do a google "Cisco problems with Sun"...

Blaming Apple is par for the course

[dpbsmith]

I wish I had a nickel for every jerk who has instantly pointed the finger at Apple for any IT issue in which Macintoshes were involved.

Or described the NuBus "proprietary" and the AT bus and Micro Channel as "standard."

Or claimed that the ASCII standard defines CR+LF as the proper character combination for a line break.

Or asked me to "go to the 'START' button" on my Mac and, when I said I didn't have one because I was on a Mac, told me that Macs weren't supported even though their website says that they are.

Re:Blaming Apple is par for the course

[Biff98]

Dude, you're in the minority... deal with it...

Your micro channel point -- weak, it was never REALLY really standard (think ISA)

CRLF CR LF -- Who cares, character sets are more of a challenge. Should everyone just speak and write English so you don't have to think/code so hard?

And lastly, you're like 2% of the world computer users that use a Mac. When you call up for tech support on something that "no one" uses, you're going to find someone who is clueless when you demand support on your Mac.

Get a life...

(Typed on a MacBook Pro)
-Steve

How obvious is this?

[bdraw]

Let me get this straight, Cisco blamed the outage on the iPhone? How could any single device be allowed to take down the network? It's just a WiFi client, the network shouldn't let this happen. Of course it's Cisco's fault.

Jumping to conclusions

[faloi]

Seems to be all the rage at Duke. One would think they'd learn from their past mistakes.

Re:Jumping to conclusions

[PM4RK5]

There's a reason they have a world-renowned business school. I'm pretty sure Jumping To Conclusions is a senior-level course.

*ducks*

Re:Jumping to conclusions

[alan_daniel]

The problem is that Duke has become this adopted child by the media. The media seems to dig for stories to report about Duke. Duke, of course, loves the attention, because it's a free way of advertising (even if it might be in a negative story, it gets their name out to more people).

Re:Jumping to conclusions

[Anonymous Coward]

The media hasn't had to dig too far for Duke stories. Has anybody from the media (or Duke) apologized to the falsely accused?

Re:you're

[foo fighter]

Depends. If you're telling a story about your friend Forest having sex with alcoholic beverages during a trip to Japan, then "For fucks sake." is appropriate.

tag it "duh"

[Anonymous Coward]

duh...

When this story was first reported it was pretty obvious that it was misconfigured Cisco products, and not the iPhones, that were causing the flooding.

Duke's wireless network were not caused by iPhone

[xeniast]

The story here is Duke blaming the innocent before looking into the problem

hound

[Gary W. Longsine]

What about a publicity seeking IT hound, then?

Haha Called It

[g-san]

Smelled like a network problem.

rm999, blindbat, Doctor Memory, and Funk_dat69, you all owe me a beer. Go drink it for me and think about what you've done. ;)

Re:Haha Called It

[rm999]

One beer, 10-1 odds, I'll drink 10 for you ;)

Re:idiots.. But it is true...

[TooMuchToDo]

This problem isn't restricted to Suns. In the last hosting environment I managed, all of the Cisco gear had to be hard-coded to full duplex/100Mb in the IOS, as auto-detect was busted. All of the Dell networking gear worked like a champ though. Cisco gear is overrated.

Something is still unexplained though

[lena_10326]

In the original article:

18,000 requests per second from iPhones knocking out dozens of access points at Duke Universit

I don't understand why no one thinks it's a problem that the iPhone didn't back off. It still generated thousands of requests (or responses) to the broken router. It should have detected that and backed off. But then, I'm not very familiar with how ARP works.

Re:idiots.. But it is true...

[Big Jason]

In my experience this is only true with the older interfaces: hme, qfe, eri. The newer ones such as bge, ce, and e1000g work just fine with Cisco equipment using auto-negotiation.

first think, then post

[chriss]

It's kind of funny: most posts here complain about the Duke IT staff, either about their lack of competence or that they didn't wait until they had all the facts before claiming that this was an iPhone related problem.

Some people here who know the IT staff at Duke defended them and objected the claim of lacking competence, and there is no reason not to believe them, since everybody else is just guessing.

So most posters rushed to explain what happened without having seen the whole picture, didn't look into the details and therefore show exactly the behavior that they are attacking the Duke staff for.

Re:first think, then post

[Anonymous Coward]

Good point. Of course, most of the people who READ these posts did not "explain what happened without having seen the whole picture." Rather, they came to learn about how the iPhone network problems at Duke, that they had read about in /. as well as the mainstream press, had been resolved.

If there is a problem at Duke it is that they allowed their comments to address the quality of the Apple iPhone in the popular press. And thusly, they disseminated their denigration of the iPhone and Apple support way beyond the set of people knowing anything about either the iPhone or networking. They allowed themselves to be a shill for detractors of the iPhone.

My mother used to tell me "when you don't have anything good to say, say nothing at all." The IT staff at Duke ought to spend their time locating and addressing their network problems, not involving themselves in the fan-boy wars. That is, their chief error was in disseminating their, errant, view of the cause of a networking problem to the wider population intensely interested in the iPhone. Will the NY Times, the Washington Post, the Wall Street Journal and the others, who ran stories trumpeting a networking flaw in the iPhone, run a retraction by the staff at Duke on the same pages as they ran the original stories? Not likely.

And, it is that lack of the responsibility --- to disseminate factual information --- that is the flaw in the Duke response. Duke owes the academic community (not to mention the larger world) a more reasoned analysis. After the apology to Apple.

Re:idiots.. But it is true...

[amorsen]

This problem isn't restricted to Suns. In the last hosting environment I managed, all of the Cisco gear had to be hard-coded to full duplex/100Mb in the IOS, as auto-detect was busted. All of the Dell networking gear worked like a champ though. Cisco gear is overrated.

I just don't get this. Auto-detect has never failed me, except when I messed up myself. When one end is hard-coded to full duplex and the other end is autonegotiating, the other end picks half duplex. That's what the standard says must happen, brain-dead though it is. The fix is to set both ends to autonegotiate, or if you like debugging weird problems, hard code both ends to full duplex. Anyway, the problem is gone with gigabit ethernet, because autonegotiation is enforced.

Precisely

[Anonymous Coward]

They had Cisco network that functioned perfectly until iPhones started popping up, it wasn't a far stretch to suspect the new device introduced into a working system.

As a fellow computer professional, I can assure you that this is exactly how computers work. When I write a function that works perfectly for all the test cases I can think of, and then other people start using it and discovering bugs, it's only logical to assume that the problem is with their code, not mine. After all, my function was a perfectly functioning system until they started using it.

If you read slashdot regularly, you see this all the time. Somebody comes up with a new way to "break" Microsoft Windows security, and then Microsoft points out that it's not really their fault, but the guy who wrote the "virus" was doing things he wasn't supposed to. They don't even need to provide a patch, because it's not their fault, but they do, anyway. (Isn't that nice of them?)

It was the iPhone...

[Anonymous Coward]

...just not the one from Apple!

what's the problem?

[anton_kg]

I don't understand why so mach noise around this problem. If your browser crashes on a website it's something wrong with your browser, not a website. If your system goes down because of couple new mobile phone it's something wrong with your network, not a phone.

Good Old Duke

[q256]

At least they didn't go for a conviction this time.

Re:idiots.. But it is true...

[drinkypoo]

Cisco is notorious for autodetect failures and some of their equipment has been known to fail to autodetect when connected to other examples of their equipment.

Re:idiots.. But it is true...

[tenman]

Notorious? You sound like you've done your homework on this one. What are some examples that make these problems "notorious"?

Those responsible for the earlier comments...

[NateTech]

... have been sacked.

As someone in another forum pointed out, and it's a good point...

Cisco provided a NEW patch, or just finally got Duke's IT staff off their ass and over to a patch set that's been readily available for some time?

Re:Those responsible for the earlier comments...

[deets]

I could be mising something, but aren't Cisco patches really IOS upgrades, meaning downtime?
No, I did not RTFA.

Re:Those responsible for the earlier comments...

[NateTech]

Well, these are apparently patches for the Access Points, which don't really run "IOS" usually.

Duke? Duke? Where have we heard ... Dear iPHONE

[Rockin'Robert]

Duke? Duke? Where have we heard ... that before?
Dear iPhone, Welcome to the the asylum.
GUILTY! GUILTY! GUILTY! UNTIL PROVEN INNOCENT? (Of OMG being white ... surely not?)
ROUNDLY CONDEMNED BY THE STAFF and the entire 'LEFT' BY HYSTERICAL CONSENSUS.
Notwithstanding an opportunist (now EX-)prosecutor.
RR
TROLL CREDO: "I don't caree how you mod me, just as long as you do."

Re:idiots.. But it is true...

[drinkypoo]

Notorious? You sound like you've done your homework on this one. What are some examples that make these problems "notorious"?

I haven't seen the problems in a while, because I've been on the applications side for some time, but once upon a time you could see this problem just by connecting autodetecting, 100Mbps+FDX ports on two catalyst 5000 switches together. Sometimes they would figure things out and play nice. Sometimes they wouldn't, and bad things happened.

Beyond that concrete example (cat5ks aren't generally even in service any more because the backplane was somehow non-y2k-compliant) I've simply lost track of the number of network admins who have warned me not to do something or who have simply complained about it. So even if it weren't true (might not be any more, maybe they have it figured out now?) it's become one of those computer axioms.

You don't have to take my word for it though; google for "cisco autonegotiation failure" and you'll find more references than you'll ever need. Many of those will track back down to people not following the spec, because that's life, but many of them also go to clear examples of cisco being lame.

Cisco isn't the best. They're like McDonalds. You can get the same burger in any country :P

Re:idiots.. But it is true...

[tenman]

Okay, now I'm not being antagonistic, I'm really curious (in a survey sort of way)... Cisco isn't the best, but what is your opinion what is?

Who puts together the best router solution?

Who's switches would run your ideal network?

Who kills at the SANS device space?

Who's VoIP gateways are seen as the best?

Who's Security devices rule?

What manufacturer offers all these type devices?

Who supports their products better?

Re:idiots.. But it is true...

[cthulhu11]

My experience with bge's and e1000g's is different. They tend to come up at 100/hdx until fixed. The ce's seem to be better, but then the only ce's we're using are fiber cards.

They Confused the Symptom with the Problem.

[Stephen Samuel]

Essentially, the problem probably showed up first with iPhones, because they were relatively numerous, going back and forth between different nodes , and running on a relatively continuous basis. Because the problem showed up just about the time that iphones did, and iphone users were the most common complainers about the problem, they assumed that iPhones were also causing the problem.

An understandable, but fallacious, intermediate conclusion.

It was an iPhone problem

[Anonymous Coward]

It was a problem with the iPhone that caused the Cisco WAP's to lock up. The iPhones like to ARP for IP's that aren't on the network, so the Cisco's don't respond, and the iPhones hammer the AP's with ARP requests and bring them down. I would know because I work Telecom at a prominent University and we had the same problem.

Re:idiots.. But it is true...

[n1ckml007]

Adtran, check them out.

Not that bad...

[bill_mcgonigle]

Even a Vyatta or other OSS router is as good as or better than all but the biggest, and most horribly expensive, Ciscos.

But you knew that, because you couldn't point to any evidence that refuted my opinion that Cisco has more than just market share in common with MS.

I don't know many Cisco haters, and I know plenty of Microsoft haters. My experience with Cisco gear goes like: unbox the layer 3 switch, connect the serial cable, configure IP addresses on a couple ports or VLAN's, tell it 'ip route', go get some lunch. Setting up a Windows server requires a 60 page guide from the NSA and four days of work to get something that should have been usable out of the box.

Now, their warranty, contracts, support, and pricing are all unreasonable (HP seems to be the better bet), but I've been known to order up a dozen refurbed Cisco VLAN-savvy switches and light up a new office building for pretty reasonable money when that fit the requirements, and the stinking things just work forever.

I'm not aware of any OSS routers that can handle 24 vlan'ed ports for $500, but I'd be happy to learn about them. The case is certainly there for some of their other products - I frequently sell clients a matched set of 1U redundant firewalls (freebsd-based) that beat the pants off PIX on price, reliability and ease of use.

I also have good luck installing WRT-54GL's in schools to get them on wireless. It's all a matter of picking the right tools from the right vendors, and occasionally Cisco has one.

Sounds like the techs did everything right

[Anonymous Coward]

I'm bewildered that so many people seem to think there's fault in what the tech department did. From everything I read so far, they followed the best course of action as far as the diagnoses is concerned and solved the problem.

The access points are crashing in groups. Analysis of the traffic shows this occurs when iPhone spews what appears to be garbage to them. Along side further diagnoses, they open tickets with Cisco and Apple saying there a suspicion that a combination of iPhone and their Cisco network is causing downtime and provide the evidence that shows this to be likely. They work with both groups, and, lo and behold, this IS what was happening (though the garbage was not actually garbage). iPhone stressed the network and brought out a flaw (a known one I think) in the Cisco equipment they had. They fixed/patched it. A week later and the network is working.

Yes, tech people can make mistakes and do so all the time, but I fail to see that here. About the only thing I didn't hear that I would have liked to hear is whether or not they did web searches for the problem to speed up the process.

Overall, the fact that there's a story at all and this whole snafu seems to be a result of either the media, the schools relationship department or other upper managers or a combination thereof. Really, they made it sound like it was an iPhone problem (and technically it was!, it just wasn't where the core fault lied) which is actually a reasonable suspicion given the circumstances. Beyond that, though, I doubt the story was worthy of reporting in the first place. The story was information, not news.

My two cents.
--Dave Romig, Jr.

Cisco wireless arp storm patch out

[Anonymous Coward]

Likely related.

http://www.cisco.com/warp/public/707/cisco-sa-2007 0724-arp.shtml