Slashdot Mirror

← Back to Stories (view on slashdot.org)

Cisco's Network Bugs Are Front and Center in Bankruptcy Fight (bloomberg.com)

Posted by msmash on from the buggy-bugs dept.

Reader Dharkfiber writes: Bloomberg is covering a story today about a hosting business that is now filing chapter 11 due to bugs in a switch. Good, bad, or ugly, is it time to admit that business really can't continue without IT? When will IT training become formal curriculum in schools?An excerpt from the Bloomberg report: There's buggy code in virtually every electronic system. But few companies ever talk about the cost of dealing with bugs, for fear of being associated with error-prone products. The trial, along with Peak Web's bankruptcy filings, promises a rare look at just how much or how little control a company may have over its own operations, depending on the software that undergirds it. Think of the corporate computers around the world rendered useless by a faulty update from McAfee in 2010, or of investment company Knight Capital, which lost $458 million in 30 minutes in 2012 -- and had to be sold months later -- after new software made erratic, automated stock market trades. Peak Web, founded in 2001, had worked with companies including MySpace, JDate, EHarmony, and Uber. Under its $4 million-a-month contract with Machine Zone, which began on April 1, 2015, it had to keep Game of War running with fewer than 27 minutes of outages a year, court filings show. According to Machine Zone, the hosting service couldn't make it a month without an outage lasting almost an hour. Another in August of that year was traced to faulty cables and cooling fans, according to the publisher.

18 of 103 comments (clear)

  1. When will IT training become formal curriculum by Anonymous Coward · · Score: 3, Insightful

    Never? That's all being outsourced, duh.

    1. Re:When will IT training become formal curriculum by petes_PoV · · Score: 2
      It will never hit the curriculum because schools could never retain IT competent teachers. As soon as they were sufficiently highly skilled to teach any sort of IT class that was relevant, they'd be off to work in IT, rather then remain a teacher.

      This is the exact same reason why companies don't train their (IT) staff. What is the point in spending money to make it easier for them to leave you?

      --
      politicians are like babies' nappies: they should both be changed regularly and for the same reasons
    2. Re:When will IT training become formal curriculum by JosephDoeden · · Score: 2

      High School is not to train kids. If it were then McDonalds, Walmart and AppleBees would be the top classes. That's not the proper way to educate the future of your nation.. stop being so stupid and argumentative just to be stupid and argumentative. One thing we don't need is more angry little trolls who want to hate on everything and be critical without taking the time to think or knowing how to use words well. We should have more coding and computer classes, which I think most people would call an IT class. IT is not just about configuring switches, most people would consider that coding. I think we have a coding class because coding is a field that will grow and grow. We should ALSO have a class for compouter and office apps use, just like most high schools have had for years now. I had a coding class in BASIC in elementary school. I think most average or better schools have IT classes now, they just aren't standardized. Lets get MS and Cisco to give billions more to schools to get kids interested.. whats wrong with that? You think GYM is a class, but not programming? WTF peopel

    3. Re:When will IT training become formal curriculum by roc97007 · · Score: 2

      You are right, for processes that can be described in a small enough number of steps. You definitely don't want to take up the time of a big brain with lots of experience doing the same operation over and over. But I would submit that a modern Enterprise installation is just too big and too varied to expect any practical set of procedures to cover, say, 80% of issues. The fall back in my experience is typically to (a) apply patches, and (b) hope the problem goes away.

      I think this is why, when a company decides they've had enough and insource their IT, they leave the helpdesk offshore. Because that really *is* a set of procedures that are easy enough to document. Communication issues are another matter, but that's not germane to this discussion.

      --
      Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
  2. business-critical means you need hot sparing by swschrad · · Score: 2

    or at least a cabinet full of new plug-ready parts. that means the HDAs need to be pre-formatted, for instance. cables tested. configurations stored on a server for tftp loading behind your firewall.

    things that cost money. things that suits have no clue about.

    --
    if this is supposed to be a new economy, how come they still want my old fashioned money?
  3. All Cisco users had this problem? by HornWumpus · · Score: 5, Insightful

    All systems have bugs, not all data centers have this kind of crap uptime record.

    Smart IT people build data centers out of heterogeneous hardware and set it up to degrade gracefully when something fails. You won't get this if you just hire A+/Net+ staff.

    Blame the PHB/CTO not the hardware.

    --
    John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
    1. Re:All Cisco users had this problem? by Salgak1 · · Score: 5, Interesting

      Agreed. I've worked at places that kept five nines of availability. By "normal" IT standards, it was massively overbuilt: multiple sets of gear, clustered in failover mode, with a separate redundant setup elsewhere in the data center, on an entirely different power feed. . . (as I recall, we had at LEAST 4 independent power feeds)..

      We also had cabinets full of spare parts, entire full pieces of gear on the shelf, and an entire library of config files on the TFTP server. Plus duplicated on a laptop that lived in one of the cabinets. Took a LOT of labor and gear, and was not cheap,

      And we constantly had to explain the man-hour and spares costs to the suits . .

    2. Re:All Cisco users had this problem? by silas_moeckel · · Score: 2

      I work with and build them all the time. Mind you I realy no longer think you can get any complex service into 5+ 9's without the application being part of the solution, reliability is not a bolt on thing it's baked into the design.

      The story is laughable, fault fans, buggy firmware on the nexus 3k's. Those are TOR switches should be extremely easy to replace and always used in redundant setups. They probably got suckered into VPC and similar, guess what I dont care what they say all stacks share a single failure domain, dont get me wrong they are great but you need at least A+B stacks. These guys cited cable issues, it realy sounds like PHB;s trying to blame the vendor because they picked the lowest bid not the right one and failed to test every failure mode they could come up with before going into production. 7 9's work is hard and your never going to just bolt it onto somebody else's design.

      --
      No sir I dont like it.
    3. Re:All Cisco users had this problem? by ewhac · · Score: 3, Funny

      And we constantly had to explain the man-hour and spares costs to the suits . .

      Suit: "Explain the man-hour and spares costs to me."

      Engineer: "Certainly." (*brains him with a fried 24-port managed switch*) "Would you like it explained again?"

      --
      Editor, A1-AAA AmeriCaptions
  4. Read the warranty card. .. by Salgak1 · · Score: 2

    "Disclaimer of Liabilities - Limitation" Page 16, states that (condensed) : all liability shall not exceed the price paid for the software, or of the price of the product which includes the software.

    And to use the equipment and Cisco software, you agree to the terms of service.

    http://www.cisco.com/c/en/us/t...

    So, at best, they can recover the costs of the switches involved. . .

  5. This is timely by roc97007 · · Score: 5, Interesting

    I'm a photographer, and I sell my work through a web service. They bring together the finishing providers (prints, calendars, t-shirts, etc) and take care of payment, and all I have to do is provide content and manage sales. When I finish post-processing on a new photo, the tool I use (Adobe Lightroom) automatically uploads to the web service in the album I select. I cover events, so there's often a massive number (600 or so) of photos to upload.

    Yesterday I was getting sporadic "service not available" messages from the service. After doing some triage to verify the problem was not at my end, I contacted customer support. Mind you, this was 10:30 PM PST. But that's the way it is with photographers -- we often take photos during the day and process them at night, which is somewhat the opposite of a standard use case. (And should be borne in mind when said services schedule maintenance. Just sayin'.)

    Browsing the service's forum, I saw others were seeing the same error message, and people were starting to get excited. (This is our livelihood, after all.)

    I got an answer to my service ticket in less than 30 minutes, that they were struggling with with network problems with one of their service providers (probably a cloud service). I got a followup shortly after that they thought the service was up now but they were still testing. And I got another followup at 6:30 AM that the problem had been resolved and they had put steps in place to insure it would not happen again. They also implemented a "status page" that we could consult in the future (which should have already existed, but live and learn).

    Now, *that's* the way to handle an incident like this. Very commendable. But it does point up the problems a business sometimes has when they rely too much on external services. Just my opinion, but the main difference I can see between in-house and outsourced is one of motivation. If you're providing an online service, your employees realize in their heart of hearts that outages can easily result in business failure and loss of jobs. But if you're renting all the pieces of your service from outside vendors, you soon find that those vendors may be concerned about their contract with you, and the money they make off you, which isn't at the same level in the hierarchy of needs as the live-or-die situation you are in.

    --
    Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
  6. IT in schools? by Kwyj1b0 · · Score: 2

    Good, bad, or ugly, is it time to admit that business really can't continue without IT? When will IT training become formal curriculum in schools?

    Good, bad, or ugly, is it time to admit that business can't really continue without Patents/Accounting/Negotiations/Advertising/Sales/1000 other things?
    When will patent law/banking/economics/marketing of these become formal curriculum in schools? That's about the time when IT should become a part of the formal curriculum as well.

    High school shouldn't be about training for a job that only a fraction of the students will eventually do. If businesses can't survive without IT, then they hire people who are specially trained in IT - a HS course won't be train people enough to solve any hard IT problems anyway.

    1. Re:IT in schools? by JosephDoeden · · Score: 2

      We should be teaching coding, especially scripting and automation in school. They are universally useful in all fields. Many fields WISH they had more field specific coders.. aka engineers who are also coders or scientists who are also coders. These people can help you build better apps faster than most anyone else. Since most of the important professions use computers we want to train kids to be good at computers and coding. Robotics, Automation and Coding is the future. Business and management is still an awesome field too. Accounting and Business Math have been part of High School educations for awhile. Chemist and Physics are as well. Do we expect kids to come out of school and solve chemistry problems? NO.I don't see where you really have a point in the words you've put together. We can offer kids more vocational options, but High School itself should continue to have classes that 'only a fraction of students will eventually do'. Dumbing down high school even more is a very bad idea. I don't know how any American could suggest we don't need computer training in high schools. It's been in high schools for 30 years now. We need IT and Coding, we need Chemist and Biology and Calculus even though 'only a fraction of students will eventually do'. Can we not have a bit better computers classes with more serious coding as standard in all high schools? Whats wrong with that? Stop being so narrow minded or you will live in a society of narrow minded people and you will no like it one bit.

    2. Re:IT in schools? by JosephDoeden · · Score: 2

      Do you guys not remember High School? GYM is a fucking class. You're sitting here, like argumentative clowns, telling people we don't need more computers classes in high school because that's not real education or that's not going to payoff? Yeah but Home Economics, Gym, Cheerleading, Marching Band, shop class are all totally practical. If you want to TRAIN people to fill a role instead of educating them to live up to their potential, you have a serious lack of understand of how society really works and that masses of stupid people are dangerous and costly. I guess we will be importing yet more college educated people if more American are thinking public education is yet another bloated government program they can cut. Why not just like McDonalds and Walmart make their own classes if you're going to do that?

  7. IT training? by ilsaloving · · Score: 3, Insightful

    People will figure out IT training is important, when they realize that they can't make stupid statement like "IT training" as if it means something.

    What even IS I.T.?

    Are you talking about server management? Network Managment? DevOps? What skill sets do you need?

    It's like saying we need more brain surgeons, so we need MOAR BIOLOGY TRAINING!

    MBAs, or people in general, will never appreciate just how complex some work can be, because of Dunning-Kruger. They don't know or understand how complex IT is, therefore they are unable to *appreciate* how complex IT is. Just like they are unable to appreciate anything else that is complicated, whether it's medicine, physics, etc.

  8. "faulty cables and cooling fans" by bmk67 · · Score: 2

    If they're contractually bound to deliver that sort of uptime, and their system isn't designed to tolerate these kind of failures, they deserve to fail.

  9. Don't depend on what you can't see by T.E.D. · · Score: 2
    The basic moral here is not to bet your life on systems you have no capability of fixing, if you can at all avoid it.

    The company’s Nexus 3000 switches began to fail after trying to improperly process a routine computer-to-computer command, and because Cisco keeps its code private, Peak Web couldn’t figure out why.

    ...

    Finally, late in October, came the 10 hours of darkness. Three people familiar with Peak Web’s operations say the lengthy outage gave the company time to deduce that the troublesome command was reducing the switches’ available memory and causing them to crash. The company alerted Cisco.

    So they ended up black-box debugging the vendor's own problem for them. I wish I could say I am unfamiliar with that...

  10. Re:Systems, not IT by I4ko · · Score: 2

    Most likely it never touched systems and was put in by some sales shmuck or bean pusher after everything on the RFP response was reviewed and engineering told them not to. Someone asked for 99.995% uptime (26min and a change; there is one more 9 than you put in), sales decided that planned outages and packet loss don't count in, hilarity ensues.