Slashdot Mirror


Cisco Looking To Make Things Right With West Virginia

alphadogg writes "Cisco has offered to 'take back' routers it sold to West Virginia if the state finds they are inappropriate for its needs, according to a post on wvgazette.com. The offer is in response to a state auditor's finding (PDF) that West Virginia wasted $8 million — and perhaps as much as $15 million — in acquiring 1,164 ISR model 3945 branch routers from Cisco in 2010 for $24 million in federal stimulus funds, or over $20,000 per router. The auditor found that hundreds of sites around the state — libraries, schools and State Police facilities — could have been just as suitably served with lower-end, less expensive routers."

38 of 182 comments (clear)

  1. Worth more than any car? by erroneus · · Score: 3, Insightful

    A router?! A computer that is dedicated to the purpose of moving data along a network path and/or deciding which network paths based on some rules and protocols.

    Maybe I'm missing something here, but it seems to me one of the industry's biggest shams is the gross overvaluation of Cisco networking. Is it really so much better than all the others or are they cloaked in so much brand naming and the hallowed process by which people become "certified" that people forget what the actual purpose of Cisco's stuff is?

    1. Re:Worth more than any car? by chrylis · · Score: 3, Informative

      Both, depending largely on the particular devices in question. In recent years, general-purpose CPUs have gotten so fast and buses so efficient that a quad-core Xeon running a Linux-based routing system (such as Vyatta) can allegedly handle 10G line speed for a few ports, and PCI cards are widely available for DSx and other interfaces that used to require standalone routers. That said, you can't do line-speed 10G to 720 ports without serious custom hardware, and while Cisco's stuff is still overpriced for the capability compared to HP or Juniper, it's not the sort of outrageous ripoff that the ISR series is.

    2. Re:Worth more than any car? by AK+Marc · · Score: 2, Interesting

      In a sense, they're the networking industry's redmond. If you're not getting 40% off from list you're not doing very well. And, of course, they sell industry certification that does much what redmond's certification does: Generate an army of vendor lovers that have been taught the answers that generate them the most money are the right ones.

      The difference is that the "Cisco"CNA/CCDA/CCNP/CCDP/CCSP/CCIE certifications aren't "cisco" only. I'm a BCNE. The Brocade test could be passed by anyone who could score a 90%+ on CCNA. There was a "Cisco to Brocade" test I took. There wasn't a single question on the test that was Brocade specific. Cisco pushes EIGRP every chance. Brocade has FSPF for an STP replacement/enhancement, but I didn't have a single question on it. The command line is identical, aside from some things you can pick up from contextual help.

      The result is that a Cisco certified something can run a Juniper, Brocade, Alcatel, Huawei, etc. A Microsoft Certified anything can't do much on Windows, let alone anything else.

      Sure, it's easier when you've spend years messing with Cisco's proprietary WRED and EIGRP, and maybe still consider ISL as a trunk type, even though even Cisco has officially depreciated it, to just select Cisco so you don't have to mess with anything new.

    3. Re:Worth more than any car? by SomePgmr · · Score: 2, Insightful

      Cisco claims they were instructed to provide a quote for routing devices with features like, "redundant power supplies", and just provided a list of the devices that qualified. The state denies these requirments.

      Put simply, they put together a sheet with 1,164 of the same exact device. One for every location, and wrote off the gross oversizing to future-proofing. That meant a big municipal facility would get one of these $20k machines, which was probably unnecessary, but the one room shack they call a "library" in rural virginia also got one... in case they ever did a high speed haul out there .

      It's absoutely nuts. And the worst part is that someone signed off on this, even after Cisco had the balls to propose it.

    4. Re:Worth more than any car? by Robert+Goatse · · Score: 2

      I don't think Cisco is at fault here. They're not a charity. I blame the poor sap that signed off on the PR. Of course Cisco is going to try and patch this up as it's generating bad publicity for not doing anything wrong.

    5. Re:Worth more than any car? by sheehaje · · Score: 3, Insightful

      Looking at Cisco from a hardware perspective, yes they are overvalued and there are less expensive, comparable options out there.

      However, I will say a few things in Cisco's defense - I've worked with Cisco, Dell PowerConnect, ProCurve, Avaya and Nortel -- hands down, when I do run into problems, Cisco is the easiest to troubleshoot for. Mainly finding documentation/community help is much easier. Finding technicians that actually know what they are doing is easier.

      The other thing I would like to say is that Cisco is not always as expensive as people want to portray them. A lot of time, things like West Virginia happen - the options aren't investigated properly, and you end up with a 20K router... A great example is before I got to my job, they were buying all 3550 switches for the wiring closets.. We didn't need a layer 3 switch in a closet, so we started ordering 2560 (the next gen model in that series) and significantly cut costs.

      Another example of ours, we had implemented Cisco Wireless in one of our locations, but for another location were sold an Avaya on the promise that it performed just as well and would be cheaper. The later proved true - but by a small margin. Performance and support has been an issue since day 1. Trying to find engineers inside Avaya that know their own devices like a comparable Cisco engineer is few and far between.

      The last thing people don't realize - you don't always need a smartnet.. We don't order them for all our wiring closet switches anymore - we just keep our latest round of switches on SmartNet. Cisco Catalyst does have a LIFETIME warranty on the hardware... The same thing that HP Procurve tries to sell customers hard... Core switches, we absolutely keep on 24/7 4 hour Smartnet ... Wiring closets, and branch routers... nah... we can just keep a spare or two, they are cheap enough. Replace when needed, send back for lifetime warranty...

      With this said, I'm not always rosy on Cisco. We did a VoIP project about 3 years ago, and going with another vendor (Mitel in our case) gave us significant savings. I'm just saying that they get the overvalued label a lot, and yes, if you are just looking from a hardware perspective yes. If you are looking at the whole training, support, community and logistics angle - Cisco definitely has the leg up on any other networking company.

    6. Re:Worth more than any car? by amorsen · · Score: 2

      The problem is that the many of those sites were getting fiber out there. The state wanted a single device that could handle both the legacy T1's and the new fiber connections. Cisco really ought to have told them to go with whatever their cheapest T1 model is these days and then replace the router when the fiber is actually installed. Cisco is certainly to blame for not doing anything to help out.

      However, the state is certainly to blame for not letting someone with a little bit of experience take a look at the bid.

      --
      Finally! A year of moderation! Ready for 2019?
    7. Re:Worth more than any car? by chrylis · · Score: 2

      Skipping the former-Linksys-style low-low end, the ISRs have an unusual hybrid processing strategy; most routing in even a 2900 is done in custom hardware rather than on the processor (which is, IIRC, a PowerPC 700-series), which couldn't handle the throughput that the ISRs can. This does have the advantage of lower power consumption/heat and thus greater reliability, but if someone starts producing a generic TCAM-based forwarding plane that can be programmed via OpenFlow, Cisco's low-end lunch is eaten.

    8. Re:Worth more than any car? by silas_moeckel · · Score: 4, Informative

      You apparently have never dealt with government RFP's, you have to meet the specs and have no input on them or visibility as to what they are for. They specked a single device that could run voip with PSTN fallback, wan acceleration (WAAS is cisco's version of that same), and an embedded managed switch with POE. The device they came back with is the only one that fits all those requirements. The issue squarely lies with the people that wrote the RFP.

      --
      No sir I dont like it.
    9. Re:Worth more than any car? by cbiltcliffe · · Score: 2, Funny

      And if it dies then I have to live without pr0n for a day. In industrial and business settings, that's not often acceptable.

      I want to work where you work.....

      --
      "City hall" in German is "Rathaus" Kinda explains a few things......
    10. Re:Worth more than any car? by drinkypoo · · Score: 2

      Cisco really ought to have told them to go with whatever their cheapest T1 model is these days and then replace the router when the fiber is actually installed. Cisco is certainly to blame for not doing anything to help out

      I have not looked into the matter at all but there may have been some budgetary consideration for spending all the money at once. None of this justifies spending so very much money.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  2. Re:Should Virginia settle with a "take back" offer by Anonymous Coward · · Score: 3, Insightful

    They bought massively overspecced routers. And this is Cisco's fault... how?

    Cisco is only offering to take them back because the cost of taking them back and reselling them is way less than the cost of the bad publicity of a government agency whining that they spent way too much on a big-iron router for a library with two computers...

  3. the right thing by kermidge · · Score: 2

    For whatever the reasons Cisco makes this offer, it's the right thing to do. Just as sucking the Federal teat (hey, it's just bidness, everybody does it) was the wrong thing to do. To really make things right, they'd also offer to find the state suitable routers, at cost, and set'em up as well.

    If I was the state, I'd be taking a close look at conscientious civil servant who approved the original deal. "Misappropriation of public monies" has a nice ring to it on a résumé.

    1. Re:the right thing by mysidia · · Score: 4, Insightful

      . To really make things right, they'd also offer to find the state suitable routers, at cost, and set'em up as well.

      Cisco's not a charity -- the management who approved the mistaken design, and the firm that designed and selected inappropriate router choices, should have to deal with this.

      It's not Cisco's job to stop you from buying equipment that can do more than what you need it to do right now.

    2. Re:the right thing by gmhowell · · Score: 2

      It's best to check before correcting someone in public.

      Why? In case 'anonymous coward' does harm to his reputation?

      --
      Jesus was all right but his disciples were thick and ordinary. -John Lennon
  4. Re:Should Virginia settle with a "take back" offer by AK+Marc · · Score: 2

    Or that unified platforms across 1000+ sites makes for cheaper support costs when the configs and hardware is identical. Though, given how it went down, they likely paid for Smartnet and Verizon gold-plated latinum level support contracts.

  5. The key is who you sell to by EmperorOfCanada · · Score: 5, Interesting

    A good salesman will get all the tech people convinced that they need his cool stuff that will work well for a good price. A great salesman goes right to the top and convinces the top(non technical) people (with white papers like this week's pole) A truly great salesman will even eliminate the tech people and replace them with his own so that the new tech people will not only support every suggestion but will become a sales force in their own right.

    I am willing to bet that no serious tech person had anything to do with this and if they did that they are Cisco certified up the ying yang. Just a guess but that the decision to purchase these came from very near the very top and the person was totally chuffed to be running a multi-million dollar project and was convinced that their tech wienies would be way out of their "depth" on this one.

    Assuming some tech guy did protest they were probably told that their suggested routers were mere toys and that to play with the big boys that you needed serious hardware.

    One of the greatly overlooked solutions is that your networking demands are so small that quite old solutions can be very effective. As long as the system can be remotely administrated you would be hard pressed to buy old hardware that didn't meet the rest of the system's requirements. 100,000 users you need the big guns. 100 users you probably need one step up from a home router.

  6. Re:Should Virginia settle with a "take back" offer by sjames · · Score: 2

    Read again, Cisco sold them and it is Cisco offering to take them back. You did read that right?

    Of course not, why would anyone read about an issue before they comment.

  7. Re:Should Virginia settle with a "take back" offer by Bearhouse · · Score: 3, Informative

    Ahem, from the article:

    " State auditors concluded that Cisco's sales staff showed "wanton indifference to the interest of the public." "

    Seems pretty clear to me.

  8. Cisco looking for federal stimulus money by PolygamousRanchKid+ · · Score: 2, Informative

    Cisco, and others, were specifically looking for government pork: http://www.networkworld.com/community/node/40380

    Cisco is looking for about $1 billion in federal bailout money, according to a report in the Raleigh, N.C., News & Observer. The company expects the U.S. government to fork over $47 billion to high-tech.

    Bruce Klein, a Cisco senior vice president, is charged with making sure Cisco gets that share of the money. Cisco can't receive it directly, but only through projects tied to local and state governments that are financed by the stimulus funds, the N&O reports.

    So Klein put together teams across Cisco to identify business opportunities with local and state government agencies and other public sector organizations.

    Cisco is not alone in looking to capitalize on the influx federal stimulus funds. General Electric and IBM are also lining up stimulus-backed government contracts, the N&O reports.

    But should companies shipping jobs to offshore facilities and contractors be eligible to bid on contracts financed by federal stimulus funds?

    --
    Schroedinger's Brexit: The UK is both in and out of the EU at the same time!
  9. Re:Should Virginia settle with a "take back" offer by amorsen · · Score: 5, Informative

    The requirements were that the sites had legacy T1's and similar and were being upgraded to fiber. Therefore the router had to have both legacy interfaces and high performance. That combination is awfully expensive and the 3945 is not an unreasonable choice.

    It would have been much cheaper If the requirements had allowed for temporarily having two routers on the sites until the legacy T1's were taken down or alternatively allowed for an extra visit to the site to replace the router.

    Trying to avoid an extra trip to each site is not stupid. Requiring both legacy and high speed interfaces is not stupid. Going for a unified platform is not stupid. However, a joint meeting with the pre-qualified bidders would likely have revealed the potential cost savings of making a compromise on the requirements. Alternatively, an independent consultant with just a little experience in the area should have spotted it.

    The same thing happens in many of bids, not just in the IT sector. Seemingly reasonable requirements together mean that only very few vendors can bid and that they need their most expensive solutions to handle it.

    --
    Finally! A year of moderation! Ready for 2019?
  10. Re:Should Virginia settle with a "take back" offer by amiga3D · · Score: 4, Interesting

    You know, if you are buying something and have no clue what you need then this is what happens. They should take a long hard look at whoever decided to buy these things as they are the ones responsible for wasting taxpayers' money. Cisco is on the hot seat right now but if you went through what states buy line by line I'd be willing to bet big money that you'd find a lot more stuff like this. When people spend other peoples' money there sometimes is a tendency not to worry about it.

  11. Re:Should Virginia settle with a "take back" offer by Anonymous Coward · · Score: 2, Insightful

    I can't comment at all on WV, but my experience is that most IT staff, incl. specialized or "specialized" IT staff, don't have much influence over such things.

    A general trend in the industry is to de-prioritize internal expertise, esp. more specialized expertise, and to depend on outside support, esp. as it becomes more specialized. Where there is still expertise within internal IT staff, their concerns are easily ignored.

    In some organizations, the people who make the final decision often have no technical background, or a limited background, or a "worked in IT 10 years ago" background, or a "specilized in IT management" background... or whatever... They are often people who do something because "this is how things are done in business". Even if such decisions are ostensibly made by competent people with a technical background, they may have been told how to make the decision by somebody else.

    As said, I don't know about the VW situation but it wouldn't surprise me if a number of VW's IT networking staff saw the waste of money and disagreed with it.

  12. Salespeople as nannies by unixisc · · Score: 4, Informative

    I know I'm probably gonna get modded down for this, but what the heck....

    Since when is it the responsibility of a company to tell customers what exactly they need? To use the famous Slashdot car analogy, if I am in a Lexus showroom and I am buying a car just for the sake of doing my usual daily stuff. I'm not a CEO or a VP trying to impress my company employees, I'm not a Hollywood star trying to do the same, I'm just someone from the street who's totally clueless on cars, and while I could just as easily have walked into a standard Toyota showroom, I happened to walk into one that sold Lexus.

    Now, is it the moral duty of the sales guy there to tell me that I have no business buying a Lexus, and should instead look at a Camry? The parents suggestion seems reasonable, except that we're now expecting salespeople to sell people what they need, rather than what they want. Since when is it the role of salespeople to spoonfeed customers? What next - someone in Safeways who's checking out a coke being told that it's bad for him by the checkout clerk? Or being told not to buy gourmet bread from the store's bakery since that's more than what he needs, and instead being told to make do w/ standard items in the breads section.

    In the above case, I understand that people shopping for the government of WV didn't have a clue. But that's where they could have used consultants to advise them on what to shop for. As it is, various governments make use of IT outsourcing services from various companies, and can easily ask them to (for a fee) advise them on the most appropriate equipment to buy, and from whom: WV could have done likewise. People look at middlemen as a scourge, but sometimes, when the stakes are high, it makes sense to use them to determine how to extract value for money. Like normally, I wouldn't bother asking someone how to shop for a computer or even a car. But if I were shopping for something I was unfamiliar w/, I'd either do the research myself, or if I was still not confident, I'd ask people I consider better than me at it how to go about it. Seems like this is something obvious that the WV government should have done.

    Anyway, since Cisco has decided to do damage control in the PR perceptions, they might as well offer alternative replacements, as opposed to just cash, for overpriced equipment.

    1. Re:Salespeople as nannies by anagama · · Score: 2

      I would say "it depends" based on what the seller knew of the buyer's use case. If the seller didn't know, then it had no moral duty. If it did, then, it had a moral duty. Obviously, most businesses aren't going to do the moral thing, but that is merely a societal convention we've come to expect. We don't actually have to accept that it is OK for business to act in a wholly amoral fashion and in fact, we don't. Our air quality would look like Beijing's if we did.

      --
      What changed under Obama? Nothing Good
    2. Re:Salespeople as nannies by penix1 · · Score: 2

      Since when is it the responsibility of a company to tell customers what exactly they need?

      Since it was a part of their contract as well as the law in WV. ( See: http://www.legis.state.wv.us/wvcode/ChapterEntire.cfm?chap=05a&art=3&section=33D )

      That was the code section of WV law that the auditor's office is recommending the Purchasing Division look into.

      --
      This is a sig. This is only a sig. Had this been an actual sig you would have been informed where to tune for more sigs.
  13. Re:Should Virginia settle with a "take back" offer by sjwt · · Score: 4, Insightful

    Government contracts don't work like that, You bid to meet the requirements, if you can not tick off every box as requested it is good by!

    If you handed in a contract and it said "we can do as you requested and it will cost $15m, but if you do this it will only cost $2m" your submission may be thrown out, as its not your job to tell the government what to do. Government contracts are made to sound fair, but in reality it usually means the little guys got 0% and the big guys going to *have* to mark up to cover what the government thinks they need.

    --
    You have 5 Moderator Points!
    Which Helpless Linux zealot/MS basher do you want to mod down today?
  14. Re:Should Virginia settle with a "take back" offer by penix1 · · Score: 5, Informative

    What happened to the WV's bid process?

    I work in government too and more specifically in WV in the office where this occurred. I'll tell you what happened to the bid process. The incompetence of the state purchasing division is what happened. Their process is so painful and long that state agencies do everything they can to avoid using them. Even the former governor Joe Manchin got caught stringing contracts to avoid them when he was in office. I've had contracts languish over there for over a year.

    In this case, an existing contract the state has to purchase minor items with Cisco was used for these big ticket items. So technically it was bid out. It just wasn't bid out for these routers. The agency got dinged for this misuse of the system and the spirit of the law.

    Having said that, the whole process here in WV needs to be overhauled. It is too complex and way too lengthy to be useful especially when the funding is on a tight timeline like the stimulus funding was. That complexity and duration is what makes purchasing something to be avoided. It is only human nature to try to avoid the pain. I don't have a choice but to use them and dread it every time I do.

    --
    This is a sig. This is only a sig. Had this been an actual sig you would have been informed where to tune for more sigs.
  15. Re:Should Virginia settle with a "take back" offer by Anonymous Coward · · Score: 3, Funny

    Geez. You no, for all intensive proposes, your just picking on are grammer. If you has a problems with are education system just right a letter. You'd feel better than.

  16. Re:Should Virginia settle with a "take back" offer by scotts13 · · Score: 4, Insightful

    Government contracts don't work like that, You bid to meet the requirements, if you can not tick off every box as requested it is good by!

    If you handed in a contract and it said "we can do as you requested and it will cost $15m, but if you do this it will only cost $2m" your submission may be thrown out, as its not your job to tell the government what to do. Government contracts are made to sound fair, but in reality it usually means the little guys got 0% and the big guys going to *have* to mark up to cover what the government thinks they need.

    THIS. I spent many years bidding equipment into the Education marketplace, and many, many, MANY times I had to meet bid specs that made no technical or financial expense. The mechanism for asking to have the spec revised is nonexistent or dangerous (as in, your company is dropped from consideration for trying to tamper with the bidding process). All through coverage of this story, I've never seen enough of the actually bidding process to make a determination - I've have to read the paperwork. But I strongly suspect Cisco did absolutely nothing wrong. They simply made the decision to make money for the company (however much), rather than making nothing.

  17. Re:are they forgetting something here? by 6ULDV8 · · Score: 2

    Installation and labor might be covered in the monthly recurring cost of connectivity that Verizon is supplying.

    --
    Pull my finger for my public key.
  18. Re:Should Virginia settle with a "take back" offer by Stormthirst · · Score: 2

    I suspect it's more about the negative publicity than "Please don't sue us". Cisco has incredibly deep pockets (mostly cost they sell $20k routers to 2 person part time libraries), and could tie anything like that up on court till the cows come home.

  19. GOVERNMENT IS THE PROBLEM HERE by Jeremiah+Cornelius · · Score: 3, Funny

    We need to eliminate monopolistic, nany-state actors, like the "State Auditor" - who's sole purpose is meddlesome interference and disruption of a free-market system.

    This case is a great example, illustrating that the enlightened self-interest of all parties will ensure a fair market of desired outcome, if we remove the coercive influence of Government.

    --
    "Flyin' in just a sweet place,
    Never been known to fail..."
    1. Re:GOVERNMENT IS THE PROBLEM HERE by Jeremiah+Cornelius · · Score: 2

      "WHOOOSH!"

      --
      "Flyin' in just a sweet place,
      Never been known to fail..."
  20. $20k router, can someone explain the cost? by Nyder · · Score: 2

    How exactly does a router cost $20k? Granted this was in 2010, but how exactly is it that expensive? Special government price? How much would it cost someone else to buy that router?

    And honestly, it's time to check all the books of all the states and start punishing the people who overpriced and sold stuff to the government, and also punish the idiots who accepted those prices and purchases. Corporations needs to be put in check.

    --
    Be seeing you...
  21. Re:Should Virginia settle with a "take back" offer by mabhatter654 · · Score: 2

    Have you tried to do ANYTHING at a remote site based on waiting for the PHONE company???

  22. Re:Should Virginia settle with a "take back" offer by amorsen · · Score: 2

    A single device should be cheapest.

    Why? T1 routers these days are legacy devices with very few units sold. Of the few units shipped, most are likely low-performance devices like the 1900. If you buy something non-mainstream, it is usually more expensive than a mass-market item.

    You are completely right about the ASR 901 though. It would have been a much better choice than the ISR 3945. However, the routers were purchased in 2010 and it seems the ASR 901 was not announced until 2011.

    --
    Finally! A year of moderation! Ready for 2019?
  23. Re:Should Virginia settle with a "take back" offer by sjames · · Score: 3, Informative

    It's not an uncommon arrangement. Cisco sold the routers to WV but the transaction went through a sales channel (in this case Verizon) to complete the deal. It was Cisco sales engineers that speced it out, drew up all of the particulars (model numbers, etc) and handed the deal off.