Slashdot Mirror
Duke Wireless Problem Caused by Cisco, not iPhone
jpallas writes "Following up to a previous Slashdot story, it now turns out that the widely reported problems with Duke University's wireless network were not caused by Apple's iPhone. The problem was actually with their Cisco network. Duke's Chief Information Officer praises the work of their technical staff. Does that include the assistant director for communications infrastructure who was quoted as saying, "I don't believe it's a Cisco problem in any way, shape, or form?""
Still others seem to imply that Duke's network was deficient in some way because the problem had not been encountered more broadly.
I would say that the network was deficient until the patch was applied. For him to say otherwise implies that there was no problem to begin with.
...for the poor guy who said it wasn't a Cisco problem when he starts getting those Apple fanboy death threats.
"Beware of he who would deny you access to information, for in his heart he dreams himself your master."
I'm curious to find more information on this. TFA just says "Cisco has provided a fix". What nature of fix was this? Was it actually a flaw in the routers, or did someone just configure them wrong?
Given the widespread use of Cisco routers compared to the isolated nature of the problem, it sounds a bit like Duke is just trying to save face.
This is unfortunately a common issue with people. When two events happen at about the same time, people assume they're somehow connected. The autism and vaccine link, for example, is one of those things where they get their shots and soon afterwards, they notice their child is acting strangely. Then there's the old "this coincidence must be a sign of the divine" theory.
We run into this all the time when doing server administration. For example, one of our developers found that web pages were slower on our new virtual servers. The obvious thought is that virtualization=slow. It turns out that compression hadn't been turned on for those servers. Since he was going over a slow VPN connection, it made a fairly significant difference. Once switched on, they worked about the same as real servers.
So it wasn't Apple's iPhone but Cisco's Linksys iPhone that was causing problems, am I right?
Many network IT folks just understand how to change settings on routers (what you learn to do in a "certification" course on a router) and understanding networking. Networking is more than just some router settings, and understanding the organic interdependent flowing nature of a network is critical to debugging problems. Just knowing something is causing a problem, and blaming the most recent change as the cause (as opposed to some underlying problem that this change simply brings to light). A senior IT official should, even if he doesn't know the exact problem, know that weird entworking problems are often way more complex than they seem, and should not jump to knee-jerk conclusions (especially based on some 1994 anti-mac bias about networking)
Isn't that the entire basis for wikipedia?
Have a look at soylentnews.org for a different view
A decent CCNP or CCIE probably would have done packet dumps and debugs and sat there reading RFCs and following the state-flow diagram. Thats what I do, and I'm just a lowly CCNA.
Why do developer types always have be hating on your friendly IT folk? Developers are some of my best friends, and we ask each-other for advice all the time. I don't meet one stupid developer who I have to explain to what NAT, proxies or TCP options are and say 'ugh, those damn CS geeks!'
Cool. Cisco screws up, iPhone gets blamed, but nobody minds, because iPhones are so cool.
... (smiles) iPhones are cool aren't they!"
Boss: "Did you get those reports done?"
Underling: "Sorry Boss, I Couldn't. iPhone Congestion."
Boss: "iPhone?
Underling: "They sure are boss!"
Boss wanders off feeling good.
Underling returns to screwing around with his iPhone.
I think that after spending a number of years working in Cisco only networks, I'm constantly amazed at the generally poor compatibility and functionality of Cisco equipment.
This ranges from critical recovery steps being removed from the 7200 series G2 NPE (NEVER make one of these crash to ROMMON on boot. The fix is to RMA the NPE) for Xmodem recovery of bootloaders - something a basic 827 router has to their latest 7961 VoIP SIP phones that are apparently RFC compliant for SIP communications - but aren't.
There are MANY things that make Cisco equipment worse and worse as the years go by. Part of it I believe is the outsourcing of the people who write the software for these things now. Chances are that they weren't even around with Xmodem was in use - and I bet a lot of the coders have NEVER admin'ed a network of Cisco gear. This is the only thing I can think behind removing essential recovery procedures for $35,000AU routers.
There's a whole new direction that Cisco is heading, and with the stupid things missing from their new gear, I'm starting to wonder if it's a direction that will have huge impacts for the worse in the network admin side of life.
Sendmail is like emacs: A nice operating system, but missing an editor and a MTA.
Take anything Gartner says and apply the old adage: a stopped clock is correct twice a day. That seems to just about cover their accuracy in most technical matters.
Endless arguments over trivial contradictions in books written by ignorant savages to explain thunder in the dark.
Shouldn't that be "fuck's sake" or "fucks' sake" - certainly not "fucks sake"..?
Just pulling your leg (mostly)
For example, I love the story of when we visited a university with a cisco-sycophant net admin, and we wanted to put up a linksys wireless access point in the lab for the robots to connect to, and he was resistant until we pointed out cisco had recently bought linksys, so it was actually a cisco device too, and then it was OK. Siiiigh.
But anyway, 15-441 ftw!
I used to work with the "hair trigger IT moron." He has a CS degree from one of the best CS schools in the country, he has been running college networks since 2000, and he does, in fact, know what he's doing.
I will admit though, that he has been known to get ahead of himself. When he looked at the logs and saw a bunch of iPhone MAC addresses spewing garbage, but no other devices are, he assumed that it's an iPhone problem. The quote in Network World is unfortunate, but he is no "hair trigger IT moron." He continued working on getting to the root of the problem and solved it yesterday.
To be fair, who hasn't had an issue where you were SURE it wasn't one thing, when it actually was. I would imagine most of you, like me, have seen issues where you still can't explain how you fixed it.
Sounds like a fun class, I was a CS minor, but I never got around to taking Networks, just OS/Datastructures/AI etc. Bummer. Your admin does sound like an idiot. The reason to say 'no' to a Linksys in an environment like mine have mostly to do with support (4 hour RMA), interference with other devices (our Cisco APs all talk to each-other), and management (I want to enforce security policy on you). Who cares if Cisco bought Linksys ;)
Its the same reason why our server admins want all HPs, because they already have a shop full of HPs, support agreements etc. Not because HP is magical. There's always the option to change, but in a large enterprise, a homogeneous environment greatly reduces cost IMHO.
Then again if I worked in an academic environment I'd be a lot less of a hardass, I work in Healthcare.
Sorry I can't provide an authoritative cite... but even if it's apocryphal, it's so perfect that I can't care.
The sick thing is that it was OBVIOUS it was a Cisco problem from the start. If you make the assumption that the iPhones are somehow defective, it's still a Cisco problem because any defective behavior from an iPhone would be indistinguishable from malicious behavior from a student. The fact that the iPhone was involved really was a non-issue all along.
It was terribly irresponsible of them to go off blaming Apple and, worse, absolving Cisco of responsibility.
Why is everyone picking on Sake? I mean, sure it's served slightly warm and in tiny cups, but it's not all that bad to drink, is it?
Your assumption at the skill of the network folks at Duke is sadly mistaken. The people at Duke are very intelligent and experienced group of folks that detected a problem on the network that seemed to be related to the iPhone. In turn they contacted BOTH Apple and Cisco and began running dumps to try to figure out what was going on. In the beginning it did appear to be caused by the iPhone and only after a lot of testing and help from Cisco was the true problem discovered. They had Cisco network that functioned perfectly until iPhones started popping up, it wasn't a far stretch to suspect the new device introduced into a working system.
The only thing they did poorly was fail to realize how much the techie world is hot and bothered over ANY news about the iPhone. Had the cause seemed to have been the the latest Crackberry this would have never sweep through the iPhone loving media/techie-verse this quickly.
So come off your superiority complex a bit and cut them some slack. They managed to detect and solve this issue within a week on a massive University network with half the tech world breathing down their collective necks. It wasn't the work of inexperienced MIS folks but group of talented network professionals that had the misfortune of publicly grappling with the iPhone juggernaut and half million know-it-alls on forums like this.
and forcing 100/full, you had better be doing it at both ends.
If you aren't, then the devices will come up as half duplex (assuming they properly implement the standards), you have a duplex mismatch, and you _will_ have network problems. 802.3u requires an end which is set to autonegotiate to assume half duplex if the other end will not autonegotiate.
Except, some Suns can not be forced and will only autonegotiate, in which case you MUST set the switch port to half duplex if you're forcing.
"National Security is the chief cause of national insecurity." - Celine's First Law
Hate to break it to you, but almost every industry sector is like that, be it software developers, construction workers, furniture makers, restaurant staff, and even medical staff.
It's actually rare to find an industry where almost everybody is top-notch, simply because most companies don't want to pay the premiums for these folks. I can only think of a few off the top of my head: NASA, Google, most engineering firms...
Well, he HAD to continue to work the problem, that's his job, he didn't really have the option of simply ignoring the situation, did he? Not sure that constitutes a pat on the back.
The quote in Network World is unfortunate, but he is no "hair trigger IT moron."
It does, in fact, show that he's willing to drop statements to the press such as "I don't believe it's a Cisco problem in any way, shape, or form," quite prematurely and without factual foundation. In the end a.) he should have kept his mouth shut until he definitively knew what was happening, and b.) if she was at all concerned that the guy was prone to getting ahead of himself, Tracy should have muzzled him from the moment the issue first came to light. Any story involving a potential iPhone problem at this juncture is going to get mega exposure, and Miller and Futhey should have realized that.
Uh huh. Because you've never made a mistake or misdiagnosed a problem when something is broken and your entire customer base is screaming at you to fix it.
Jesus, I love how you all are posting here like you single handly created the first router and invented TCP/IP. Let's try and look at this from the Duke IT perspective: 1. Wireless network is (presumably) working great. 2. iPhone is released, students start showing up with it. 3. Wireless starts getting slammed. Yes it was a wrong conclusion and faulty logic but come on, was it really that horrible? When something breaks the first thing you ask is "What has changed", in this case iPhones were introduced to the network. I guarentee that would have been the first thing I would have looked at.
and electricity too and the damn horseless buggy. you kids get off my lawn
go back and read the slashdot article on the subject when this first came out. Dozen of slashdot guys were reporting that cisco routers and WAP's have a flaw that would enable just such a solution and that you had to patch the routers with a patch that Cisco already had made.
Cisco makes some solid equipment, but when they let flaky stuff loose it's really flaky. It is also not something you announce to the world first, without throughly checking out your own equipment first, especially when the iPhone was working perfectly fine with tens of thousands of other access points around the country.
i thought once I was found, but it was only a dream.
I and my group have experienced this at work all the time almost whenever a new person is hired into the network team. Cisco gear do NOT play nice with Sun Microsystems, be it their desktop workstations or their servers. The Cisco gear refuses to properly auto-negotiate with the equipemnt causing issues such as duplex/simplex mis-matches (i.e. the workstation thinks it is connected at 100 Full duplex, while the switch thinks it is connected at 10 Half duplex). Needless to say this causes all kinds of collisions, IErrors, OErrors, etc., on the system and the network. All the Sun gear must have their associate network partner's port forced to 100 Full, and we do the same for the system as well. How do I know the problem is with the Cisco gear? Because the workstation/server works fine if you use a HP, Xylan, Baynetworks, or other switch. The net network engineers immediately believe it is the Sun equipment because they have been brainwashed into believing that Cisco can't make a mistake or a poor product. It usually takes us to demonstrate using 2 or more other switches that the problem only happens on the Cisco. Cisco still denies that there is a problem as well.
Oh and if you don't believe me, do a google "Cisco problems with Sun"...
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
Perhaps they should have waited a week then and not announced prior to discovering the true issue that the iPhone was at fault.
- "Scientia non habet inimicum nisp ignorantem"
Seems to be all the rage at Duke. One would think they'd learn from their past mistakes.
"It is a miracle that curiosity survives formal education." -Albert Einstein
I also used to work with this "hair trigger IT moron" at the university where we both got our top-tier Computer Science degrees. Hands down, he is brilliant. I have seen him at his best, which includes: writing an entire network registration software suite for use at large institutions, diagnosing bizarro network problems, and managing network security issues. It is rather unfortunate that he was quoted as saying he was confident that it wasn't a Cisco-related problem. While that ended up not being the case, I have no doubt in my mind that he had reasonable cause to think so originally. I have always seen Kevin as a rising star, and I am confident that he will continue to do great things.
I guess this truly puts me in the minority, because I do both IT and development stuff pretty equally well, though i do favor development a little (don't tell anyone)
Of course there are arguments to work in a homogenous environment, I encountered the same arguments in my first year IS classes. My teacher insisted it was much more efficient to buy mass amounts of generic-branded PCs because the "support was better" in case of hardware failure. Of course I argue that if I build them myself, I already know by the time each one is deployed that the hardware is not a lemon (burn-in testing), and it's probably going to last quite some time.
In my opinion, it is far better to spend a little more on in-house support, mix and match quality hardware for the best results (performance and reliability-wise), than it is to go all-out generic and be RMAing whole machines all the time. Yes, it's more work in the end, but overall there seem to be less problems. Besides, the biggest upside is, no matter who you hire onto your IT team, they WILL learn how things actually work, and how to actually fix them, after a while. The RMA-everything method discourages learning by experience, imo.
(A)bort, (R)etry, (I)gnore?_
"We can categorically state we have not released man-eating badgers into the area." - UK military spokesman, July 2007
Camping on quad since 1996.
No, what they did poorly was their job as problem solvers. They made the classic mistake of trying to solve a problem in reverse -- they started out with an assumption and then looked for evidence to support that assumption. For whatever reason, they wanted to rule out Cisco as the cause, so they did. This prevented them from finding the real cause as quickly as they might have.
Had they kept an open mind, they would have looked for more evidence before making a determination. For example, they could have asked some other universities (who undoubtedly now have iPhones on their wireless networks) whether or not the same type of problem was occurring there. With the answer being "no", they would have learned that the problem must have something to do with the combination of the iPhone and their specific network. That would have opened the doors to start looking at network configuration and/or faulty networking equipment. Obviously, that's exactly what happened in the end, but my point is that they erected a barrier in the problem solving process by "trusting" their Cisco equipment rather than suspecting it along with everything else.
I don't think the Duke IT people are incompetent, unintelligent, lacking in education, etc. Rather, I think this is an interesting little case study that illustrates how even highly competent people can allow their preconceptions to undermine their problem solving efforts. After all, our instincts, gut reactions, feelings, etc. are extremely useful when diagnosing a problem. They are often correct or at least highly informed on a level that is difficult to quantify. So, it's not easy to consider that your instincts may be completely wrong -- that you may be looking at an entirely new and surprising situation in which your instincts only serve to mislead you. Effective problem solving requires creativity, deliberate role-playing (e.g. "playing devil's advocate") and a certain amount of (forced) objectivity. Unfortunately, too few technical professionals display these traits when attempting to diagnose a problem and fail to understand that problem solving, in a general sense, is a discipline unto itself.
You've never worked in a large-scale IT environment. At my company, we deploy over 7000 machines per year (1/3 of the entire infastructure) in hundreds of sites around the world.
Are you going to build and "burn in" 20 machines per day? How many people are you going to hire (probably at least two dedicated employees, which is at least $300k/year in expenses)?
Who's going to handle packaging and shipping the machines (HINT: Dell/HP/Lenovo spend a LOT of time testing to make sure the PCs arrive intact)?
When there's a problem, are you going to be able to repair them locally, or will you have to ship them back to headquarters? You can't have a dedicated tech for a 10-man site, but major manufacturers can offer support pretty much anywhere in the world.
How do you know that your images are going to work? You don't want to find out that some chipset mismatch on 2% of your PCs is causing kernel panics.
When you have a problem, who's going to fix it? HP/Dell release BIOS updates for years to fix bugs. Good luck getting ANY support out of AsusTek/ECS/Tyan/Biostar/MSI/Gigabyte/Whoever after even 1 year.
Where do you dispose of your PCs? HP/Dell have extensive recycling programs in place.
How do you handle your purchase orders? HP/Dell are very good at working with your accounting department. It's not as simple as "put it on the Visa".
Of the 7396 PCs (desktop and notebook) we deployed in 2005, 143 have failed (1.9%). Generally, we find that the lifetime failure rate is below 3%. You're not even going to get close to that by building them in-house. One of my friends runs a custom-built PC business, and he sees a failure rate closer to 5%, with a large percentage being damaged during shipping.
As for "lasting quite some time", this indicates that you've never worked in a large IT environment at all. All major IT environments have some sort of lifecycle in place, typically 3 years but sometimes 4 or 5. A typical employee costs the company $150,000 per year (salary + benefits + taxes + etc) - if you replace a $1500 PC every three years, you're only spending $500 per year on the PC. It makes precisely zero sense to stick your $150,000 employee with old technology - if the new PC makes them even 0.5% more productive, you are saving $750 per year.
You may think that the big manufacturers just throw together parts, but nothing could be further from the truth.
This hardly seems like a minor error. They apparently went public with a premature conclusion about a new product before they actually knew what was going on, thereby holding themselves and Duke University up to ridicule.
And it is an unfortunately typical knee-jerk reaction reflects the arrogance typical of many IT departments: "The problem isn't with our network; it must be your computer or software that is screwing things up."
See, that's what he gets for not reading Slashdot. If he would've just sat there eagerly refreshing his browser, he would've seen several people post the solution to their problem last week and could've taken the weekend off. Hope this is a lesson.
You forgot this point: "4. Why is it that our network is getting slammed but similar networks at other universities are not?"
One does not need a technical background to know that if the iPhone caused problems for these kind of networks, we should be seeing them all over the place. A simple, logical process of elimination would soon cast strong doubt on the iPhone as the cause. It had to be the way that particular network interacted with the iPhone. Hence, it was the network and not the phone.
That response would last about one week into the new school year. Dude, there were over 150 of the darn things ALREADY on campus, and it's a ghost town compared to September when the kids return!
Three things:
1.) if you think that Duke students will put up with an IT staff too closed minded to resolve a problem with their own network, which would result in making their $500 iPhones useless on wifi, you're crazy. The kids pay the big $$$ to go there, IT toads can be easily replaced.
2.) Duke U. has been out front in embracing iPhone's parent, iPod, as a learning tool, going so far as to issue them to incoming freshmen a few years back, if you think that some pasty cheeto eater in IT would be able to gloss over his own network problem by banning iPhone when Duke has embraced iPod in the classroom, you're friggin stoned.
3.) Lastly, and maybe most importantly so far as the iPhone itself is concerned, there are ALREADY over 150 known iPhones on the Duke campus accessing the network. The iPhone has been out for only 3 weeks. How many do you think there will be when the population on campus grows exponentially with the start of the school year? What that says is that the iPhone is ALREADY a massively disruptive technology, and that anyone who does not understand that we are seeing a massive shift away from traditional computing is either deluded or stupid.
I fart in your general direction!