Slashdot Mirror


Post Mortem of GunnAllen IT Meltdown

CowboyRobot writes "The story begins when GunnAllen, a financial company, outsourced all of its IT to The Revere Group. Before long, it was discovered that 'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.' In addition to the obvious security concerns of sending information such as bank routing information and driver's license numbers, the act violated SEC rules because the routed information was not being logged. Regardless of whether the cause was negligence, incompetence, or sabotage, the matter was swept under the rug for a time until unpaid SQL Server licenses meant threatening calls from Microsoft as well. The rest of the story is one of greed, mismanagement, and neglect, and ends with the SEC's first-ever fine for failure to protect customer data."

56 of 192 comments (clear)

  1. Trusted Advisor? by Frosty+Piss · · Score: 3, Informative

    Wow, according to the The Revere Group website:

    WHEN TRANSFORMING THEIR BUSINESS, TOP PERFORMERS TURN TO A TRUSTED ADVISOR

    Guess that's not The Revere! Group

    --
    If you want news from today, you have to come back tomorrow.
    1. Re:Trusted Advisor? by tautog · · Score: 2

      Been had a little horsy named Paul Revere
      Just me and my horsy and a quart of beer
      Riding across the land, kicking up sand
      Sheriff's posse on my tail cause I'm in demand

      Burmashave?

    2. Re:Trusted Advisor? by T.E.D. · · Score: 4, Funny

      Wow, according to the The Revere Group website:

      WHEN TRANSFORMING THEIR BUSINESS, TOP PERFORMERS TURN TO A TRUSTED ADVISOR

      ...but they are too expensive, so they then turn to the Revere Group.

    3. Re:Trusted Advisor? by tiptone · · Score: 2

      Beastie Boys. GTFO

      --
      Please don't read my sig.
  2. Outsourced by Anonymous Coward · · Score: 5, Interesting

    Yeah keep outsourcing the responsibility of something so crucial that IT people hold the keys to the kingdom.
    This is nothing new in the world of IT. Save a dime to lose a million dollars.
    I am in a comany right now where they hired IT consultants for well over 3 years and come to find out so called "Experts" where just patching the system but never really fixing the real issues. It's amazing to see what these contractors were selling to a company who had the money to buy great gear only to discover pure incompetence at implementing it. I am no expert by any means but I can smeel bullshit when I see a network in need of a lot of TLC.

    1. Re:Outsourced by AK+Marc · · Score: 5, Insightful

      For the same reason they don't oursource their upper management. After all, CEOs cost money, why not outsource CEO to a management company and cut costs. After all, they are a finance company, not a management company, so all their management should be outsourced.

    2. Re:Outsourced by JDG1980 · · Score: 4, Insightful

      They are a finance company. Not an IT one

      If you run any business beyond the level of a mom-and-pop restaurant, you are in the IT business whether you want to be or not. The only question is whether you will leverage IT as a strategic asset or be outcompeted by those who do.

    3. Re:Outsourced by AK+Marc · · Score: 4, Insightful

      Consultants are often used for outsourcing blame, rather than outsourcing capability. "Oh, our consultant recommended that."

    4. Re:Outsourced by Rhinobird · · Score: 4, Insightful

      Eventually the people in charge are going to realize that any kind of financial institution is basically a database on the internet that holds and exchanges account information. And then they're going to turn ghostly white as they realize all these strangers are touching the equipment that, in a very real sense, IS the bank, er, financial whatever...or worse, those strangers OWN the equipment that IS the financial gobstopper.

      And then, at least in finance, outsourcing IT will be seen as a form of insanity.

      --
      If Mr. Edison had thought smarter he wouldn't sweat as much. --Nikola Tesla
    5. Re:Outsourced by Anonymous Coward · · Score: 3, Interesting

      With the revolving door nature of CEO and other top level jobs these days, you could argue that upper management is already outsourced away from the actual company. Just that they compete on paying the most instead of the least.

    6. Re:Outsourced by tibit · · Score: 2

      Because, obviously, in an "X company" everyone does X. Accountants do not know how to do clerical office work, they should just outsource their secretaries abroad, right? Right? Let's cut the bullshit. A company is ultimately people and processes. People you can get, and processes you can learn. I don't buy that IT consulting companies have some magical process powder that makes their people so much more effective than the same people, were they to be hired thy the "X company". It seems to be a ploy everyone falls for, but without any backing in fact. Most IT outsourcing is a disaster that's only used to postpone the inevitable technical debt blowup, and is used, demonstrably, only to temporarily boost share values.

      --
      A successful API design takes a mixture of software design and pedagogy.
  3. Sigh... by Black+Parrot · · Score: 4, Interesting

    A financial company outsourcing its IT ought to be considered criminal negligence.

    (Though an own employee could do the same thing, in this case.)

    --
    Sheesh, evil *and* a jerk. -- Jade
    1. Re:Sigh... by DigiShaman · · Score: 5, Insightful

      Agreed. I work in the MSP (Managed Service Provider) sector which is a fancy way of saying that we are outsourced IT. We focus on the SMB market where a company is too small to have a dedicated IT department, but just large enough that they place a trouble ticket in our queue once a week. Sometimes once a day. Anything ranging from tier 1 to 3 support.

      However, once you as a company get involved with needing to be HIPPA, PCI, or SOX compliant, that should be synonyms with "dedicated in-house IT dept".

      --
      Life is not for the lazy.
    2. Re:Sigh... by Charliemopps · · Score: 2

      I'd have to disagree. We have our own in house IT department... but a small part of our business is providing outsourced IT. And our stuff ridiculously overbuilt and robust. I doubt anyone could do it in house better. But it's expensive as hell, and not very flexible. If you're not getting too creative with your needs, and you have the money, you can get something very robust. But if you want to go on the cheap and still get some crazy ass system no ones ever tried before to work, then I think you're shit out of luck no matter who you go with.

    3. Re:Sigh... by girlintraining · · Score: 5, Insightful

      A financial company outsourcing its IT ought to be considered criminal negligence.

      Outsourcing IT isn't the problem. A failure to oversee the IT services provided was the problem; A complete lack of auditing and process control. I wish people would stop looking at outsourcing as somehow evil; It makes sense in a lot of cases. Most corporations have other companies contracted to replace and maintain printers. Most office printers have the ability to retain all documents printed from it, locally, to a harddrive inside it. That isn't a problem by itself -- unless you don't know that the functionality is enabled, and don't audit or remove the drives before the printers are rolled out the front door with all your confidential data... that you thought was secure because you had a contract to shred all your documents.

      The story of GunnAllen's criminal negligence starts with the CTO and board of directors -- who fired people for coming forward with security problems, and had a very obvious closed-door policy. Nobody with the parent company wanted to hear about problems, and it's no surprise that the firm they contracted with heard that loud and clear -- and propagated the same attitude right on down the line. "See no evil, hear no evil" often leads to a lot of people doing evil.

      GunnAllen's story is one being repeated by the thousand every morning of every workday across our industry. Managerial incompetence leads to otherwise trivial problems becoming fines, bankrupcy, and lawsuits. This story is not about the failures of IT -- IT was involved, but it was not that failed. It was the people at the top... and when the extent of the damage was finally discovered by the government, they tried to pin it all on former employees and the people under them. I'd like to know where those managers are now; Because I know they'll eventually find themselves in another position of power at another company. Whereas all the engineers and people who actually worked for a living, well... we all know what happened to them, whether the article says so or not.

      You want to fix problems like this: Start with accountability.

      --
      #fuckbeta #iamslashdot #dicemustdie
    4. Re:Sigh... by slashmydots · · Score: 2

      A financial company outsourcing its IT ought to be considered criminal negligence.

      (Though an own employee could do the same thing, in this case.)

      I worked at a hospital with around 1000 computers and IT was onsite but contracted from a 3rd party. So, that's odd but get this! They outsourced the support calls to Mexico! Yeah, you could walk right down to the damn IT office yourself on floor 1 and get your problem taken care of or you could call Mexico. You could even simply get an extension of someone in IT and call that...or call Mexico! MEXICO! AT A HOSPITAL! By the way, I was there on a 6 month PC replacement project from a different contractor that the other contractors hired. Oh and they all got fired 4 months later when the hospital didn't renew their contract.

    5. Re:Sigh... by LordLucless · · Score: 3, Insightful

      I'd have to disagree. We have our own in house IT department... but a small part of our business is providing outsourced IT. And our stuff ridiculously overbuilt and robust.

      It's not about robustness in these instances. It's about power and accountability. When you have hugely sensitive information (medical records, credit card details or financial records) you must be in control of your own systems. While downtime sucks, downtime is often better than data compromise in these cases.

      --
      Just because you're paranoid doesn't mean there isn't an invisible demon about to eat your face
    6. Re:Sigh... by mjwx · · Score: 2

      Outsourcing IT isn't the problem. A failure to oversee the IT services provided was the problem;

      Which is difficult to impossible to do unless you're directly managing the technicians. In which case, why are you paying another company A$200 an hour when the same techs would jump at being directly offered A$35-60 an hour (consulting rates in Oz).

      So we're back to outsourcing being the problem. There may be more to it than that, but if you need 100% control, you cant get that by going through third party.

      --
      Calling someone a "hater" only means you can not rationally rebut their argument.
    7. Re:Sigh... by drinkypoo · · Score: 2

      That sounds grossly naive. What company over the size of 0 employees doesn't have one of the following: Medical records, Credit card details, or financial records? Every single company has those, even companies that have 1 part time person in it. I seriously don't think there is enough IT professionals in the entire world worth a damn that you could have 1 at every single company.

      You're being grossly obtuse. We're talking about a bank here. They are directly responsible for customer data, and they are explicitly on the hook in the case of data breaches. It is a gross failure of responsibility not to maintain IT in-house when your entire business is built on IT, which is the case in banking today. They can't do anything for you if the computers are down, except take a deposit and give you a handwritten slip in exchange. And if I walk into my bank and their computers are down in this day and age of clustering and high availability, I'm probably going to go to some other bank and open an account with that check.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    8. Re:Sigh... by bluefoxlucid · · Score: 2

      Well, you have a small shop of specialized people to handle a handful of sparkly-resume folks to hopefully do the job right. To hopefully improve chances of doing the job right, you hire a guy who seems pretty smart and has a sparkly resume to act as a manager of these sparkly-resume college kids. For all this, you get to cover logistics and take on the liability of managing them; and when things get out of hand, they outsource to the Internet or friends from college, leaking hints of your internal operations to untrusteds trying to get a grasp on things--which may be wobbly.

      Or, you pay someone whose small shop of specialized people handles payroll for a big shop of IT people. They have specialized legal department that knows more than YOUR legal department about YOUR requirements for SOX and HIPPA and SAS70. They not only handle the logistics of finding you a sparkly-resume college kid, but they supply wide-spanning, far-reaching technical expertise comprising both management and consulting to make sure the sparkly-resume college kid is either REALLY good or can be collared and handled properly; if he's kind of fresh they'll put him somewhere less-critical and give support.

      Speaking of support, even the veterans have a fallback that's legally bound to contract and confidentiality--grabbing some help from coworkers from your parent contracting agency is better than blabbing around on Web forums. Even when you're a fully-placed shop (i.e. hired to give site support, you don't know your coworkers), your contracting manager can find you people to talk to--mine hooked me up with some Windows guys and Unix guys under the same contract in other parts of the building, once upon a time, when I needed specialized help.

      It's very expensive to do it right once or twice. Some of these places do it right hundreds or thousands of times a month. 'Doing it right' means hiring or properly training the right people, something that is logistically hard. How do you find a manager you know knows IT? You must know IT to know he's not a moron. A shop that specializes in IT is, by nature, well prepared to weed out the morons.

    9. Re:Sigh... by tibit · · Score: 2

      Yeah, but how do you know that the IT shop you're hiring is not in fact full of morons? There seems to be plenty of such IT shops around, big ones, even. You need some sort of external references or vetting if you lack your own know-how no matter whether you're hiring a shop or "just" some employees. In light of seemingly endless snafus masterminded by big consulting firms, I'd much rather hire the right people even if initially just as consultants and have direct managerial oversight. I mean, you must be good at something -- even if you don't have pre-existing technical know-how, as a good manager you should be able to figure out if a project is running correctly, and whether the manager under you is doing their job. I do see your point, but no matter how polished the airport billboards are, well run and competent IT consulting shops are few and far between. You may be quite lucky to be in one.

      --
      A successful API design takes a mixture of software design and pedagogy.
  4. Easier to Read Article by cluedweasel · · Score: 4, Informative
  5. Re:HAHA by Frosty+Piss · · Score: 2

    Well, you know, he had RoadRunner... In 2005, that was pretty wicked! If he had set up two or three accounts and load balanced them...

    --
    If you want news from today, you have to come back tomorrow.
  6. Re:HAHA by El+Puerco+Loco · · Score: 5, Insightful

    'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

    That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

  7. Wait a minute... by damn_registrars · · Score: 5, Funny

    Are you trying to tell me that the SEC has rules? That they enforce? I don't believe this. This does not reflect the US that I live in; are you perhaps talking about some other country with more reasonable laws about this kind of thing - maybe you meant to say it happened in Armenia, not America?

    --
    Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
    1. Re:Wait a minute... by jamstar7 · · Score: 3, Interesting

      Of course the rules get enforced, if you're small enough to where you can't outlawyer the Feds. Why you think none of the big brokerage houses faced prosecution? For every lawyer the DoJ fielded, the brokerages fielded *5* or more.. And it didn't help that a Republican-controlled Congress cut their funding to the point where the DoJ was damned near useless.

      --
      Understanding the scope of the problem is the first step on the path to true panic.
    2. Re:Wait a minute... by khallow · · Score: 3, Interesting

      And it didn't help that a Republican-controlled Congress cut their funding to the point where the DoJ was damned near useless.

      Even with funding, the DoJ would be pretty useless. I'll just trot out the current Republican talking points about Fast and Furious since they'll illustrate a good reason why the Republicans wouldn't be inclined to fund the Department of Justice.

      Here, you have a pretty much cut and dry case. ATF agents allowed roughly two thousand fairly high quality guns to pass to Mexican drug cartels with no attempt made to track those weapons. Since those weapons have turned up at many crime scenes, including the murder of a US border agent (which is what finally shut down Fast and Furious). Further, the ATF agents involved knew for a few months before that final murder that these weapons were turning up at crime scenes, including murders. So a prosecutor has a pretty good case that someone committed a bunch of acts of accessory to murder (with reckless disregard for human life) and other crimes, plus the murder of a federal law enforcement officer. So what is the Department of Justice doing with this case? Hiding the agents involved in Washington DC. When will they investigate this?

      This is why the "more funding" argument doesn't work. If the Department of Justice isn't going to do its job, then it doesn't really matter how much they're paid so might as well make it a little rather than a lot. The SEC is particularly notorious for providing the illusion of security for novice investors, or in other words, helping keep the marks from getting scared off before they can be fleeced.

    3. Re:Wait a minute... by Bill_the_Engineer · · Score: 3, Insightful

      Even with funding, the DoJ would be pretty useless. I'll just trot out the current Republican talking points about Fast and Furious since they'll illustrate a good reason why the Republicans wouldn't be inclined to fund the Department of Justice.

      You could but then again I could just trot out the bananas can't be considered oranges.

      I hate to be the one to break it to you, but the reason politicians love to underfund enforcement is to offset the showboat regulations that they pass in order to be re-elected. This way they said they passed laws that are designed to protect us from harm, while at the same time the chances of that law actually being used is low enough not to piss off the people who actually fund the politicians campaigns.

      Pointing to incompetence or the occasional misstep brought on by the underfunding of enforcement as an example of why we should fund government law enforcement is part of their plan. You don't actually think they would point out the overwhelming majority of things that the government does right? That would discredit the fairytale that they are trying to sell you.

      This is why the republicans in particular have been doing a shitty job. If the government is seen as doing the right thing then they wouldn't have a platform to run on. The number one reason that a republican filibusters every single bill of significance is to prevent the democrat president from looking good. Never mind that shitty legislation was passed with overwhelming support when there was a republican president. During the Bush years the attitude of the republicans was that it was okay to borrow money in order to keep taxes low because the interest being paid was offset by the nation's GDP. The day after a democrat is president, those same republicans immediately are concerned that we are borrowing too much money and selling our children to China. The amount of hyperbole that is spewed is ridiculous.

      I just find it laughable that someone would vote for a candidate that is more concerned with what would make his party look good than what is good for the nation. One key sign that this is taking place is the more they try to hurt the country to prop themselves up, the more they wrap themselves in the American flag and claim to be patriotic.

      Beware of the politician that campaigns on the platform that government sucks and reelect him and he'll keep it that way.

      --
      These comments are my own and do not necessarily reflect the views or opinions of my employer or colleagues...
    4. Re:Wait a minute... by DeadCatX2 · · Score: 4, Informative

      Here, you have a pretty much cut and dry case. ATF agents allowed roughly two thousand fairly high quality guns to pass to Mexican drug cartels with no attempt made to track those weapons

      From what I read it's not really that cut and dry. The officials involved DID want to track the guns and did try, but the bureaucracy did them in.

      http://features.blogs.fortune.cnn.com/2012/06/27/fast-and-furious-truth/

      --
      :(){ :|:& };:
  8. Re:HAHA by cbhacking · · Score: 5, Interesting

    It's not mentioned in the summary, but the first sign of the rerouting was, as you'd expect, their network slowing to a crawl. That earned the IT guy responsible for it a reprimand. A reprimand, for routing an entire company's trading data through his home modem for a week!

    There's other gold in there too, like the time the guy pulled the cable on a production rack in order to create a catastrophe so he wouldn't have to travel to a business meeting, or his habit of remoting into IT infrastructure (Blackberry and Exchange servers were mentioned) on the weekends to fuck up their configuration, just so he could "magically" fix it on Monday morning.

    He was, apparently, eventually fired.

    --
    There's no place I could be, since I've found Serenity...
  9. Re:HAHA by the_B0fh · · Score: 5, Funny

    I worked at a place where the Exchange admin - every so often - would have to heroically worked 72 hours or whatever to rescue the mail servers and we only have 2 days of downtime, etc etc, and the CIO would praise him for his hardwork.

    I asked my boss if I should also reboot the firewalls every now and then - just to heroically bring them back up again, and get thanked for my hardwork. He gave me a nasty look...

  10. Re:HAHA by Anonymous Coward · · Score: 3, Informative

    If not given the resources to have Exchange load balanced, and if it happens to crash and requires a 200GB Store restored...72 hours sounds about right. The 2 days downtime should have been 4 hours (time to investigate and bring a backup VM online). Without a backup VM, it should have been down 1 day.

  11. Negligence, Incompetence, or Sabotage? by techsurvivorman · · Score: 5, Interesting

    I say Sabotage. I'm presently a NOC engineer at an IT managed services provider. Before, I worked for a well-known financial market data provider. The most demanding client we have is a financial company. Everyone once in a while, they get unhappy with our service for whatever reason and decide to blast the blame-thrower. During the most recent hissy-fit episode, they threatened to not renew the service contract. Moreover, their CIO dropped in on the conference call and said not only are they not gonna renew the contract but he was gonna have us blacklisted with other financial companies that we were looking to grow business with. It's been my general impression that financial clients tend to be some of the most high maintenance, demanding, and nasty assholes. I've a hunch that a similar reason could be a factor In explaining this network engineer's actions.

    1. Re:Negligence, Incompetence, or Sabotage? by GSloop · · Score: 4, Insightful

      It sucks on your end but on the other end you always get great service by demanding more for less.

      I have news for you. People have the most ingenious ways of paying back arseholes. Thus, you don't always get great service by demanding more for less.

      As a matter of fact, you may [meaning almost certainly WILL] get pretty bad service when you treat people badly - by continually demanding more for less, past the point of reasonableness and fairness.

  12. Just for fun... by Anonymous Coward · · Score: 4, Informative

    Go to http://www.reveregroup.com/ and search for anything in the top right search box. You'll get a licensing error. These guys are on the ball...

    1. Re:Just for fun... by jaxtherat · · Score: 2

      Or how about the fact that they promote their non existent twitter profile on their main page:

      https://twitter.com/revere_group

      --
      http://www.zombieapocalypse.tv/
    2. Re:Just for fun... by Chris+Mattern · · Score: 2

      While it's not as out-and-out broken as their search box or twitter link, I also like their main page selection. Because everybody wants web navigation that induces motion sickness! Complete with mystery meat selections, too.

  13. Milton in the Middle by Mr.+Lwanga · · Score: 2

    Why would senior network engineer need to send traffic home to verify his routing patterns? Yeah right, he scammed millions and they covered it up to avoid more fines. Now, he and his red stapler, are at some beach resort complaining about the Mai Tais.

  14. Re:HAHA by ackthpt · · Score: 2

    'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

    That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

    He probably cooked lobsters in his dishwasher, too.

    --

    A feeling of having made the same mistake before: Deja Foobar
  15. Second paragraph has all you need to know by Stiletto · · Score: 4, Informative

    Over a period of roughly seven business days, traffic had slowed to a crawl at the Tampa, Fla.-based firm, which had outsourced its IT department to The Revere Group. GunnAllen's acting CIO, a Revere Group partner, asked a member of the IT team to investigate.

    Well, here we go! The CIO of the company outsourced the IT department to..... his own personal company. No conflict of interest there!

    1. Re:Second paragraph has all you need to know by Chris+Mattern · · Score: 2

      Not his own personal company; he was a Revere Group employee. At one point in the narrative one IT minion discusses how he went to the CIO's Revere Group superiors. When they outsourced IT they outsourced the CIO position along with it.

  16. Unions can be a big help in stopping BS like this by Joe_Dragon · · Score: 5, Insightful

    Unions can be a big help in stopping BS like this from happening.

    When you have people purposefully break things just to look good for the bosses that's bad even worse is sweeping security and other issues under the rug.

  17. Hard time reading train wreck stories by HangingChad · · Score: 3, Insightful

    It's hard reading IT train wreck stories, especially when the damage is self-inflicted. And yet I saw that same attitude, on both sides of the transaction, acted out over and over.

    A long time ago a CIO I worked for said he wasn't worried as long as he had a throat he could choke if things went sideways. The only thing he cared about was having somewhere to cast blame.

    Those were the days I naively cared about doing a good job.

    --
    That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
    1. Re:Hard time reading train wreck stories by dbIII · · Score: 4, Interesting

      A long time ago a CIO I worked for said he wasn't worried as long as he had a throat he could choke if things went sideways

      There seems to be a lot of that attitude with the cloud outsourcing. I put an example up here earlier of 25k email accounts inaccessible for a week due to a DNS typo and a long job queue to do the two second fix, but people seemed to think it was OK to have that so long as there was someone else to blame. In that case it was Microsoft doing the hosting so good luck in getting anywhere with blaming them, a customer with twenty-five thousand email accounts is ignorable small fry and legal action is pointless.

    2. Re:Hard time reading train wreck stories by Turminder+Xuss · · Score: 3, Insightful

      The five stages of IT projects: 1. Wild Enthusiasm 2. Cold Reality 3. The Hunt for the Guilty 4. Bayoneting the Wounded 5. Promoting the Absent

      --
      You seem to regard science as some kind of dodge... or hustle.
  18. Re:HAHA by dbIII · · Score: 4, Insightful

    However no jail time. Refusing to disclose a password in case it's used by such an incompetent carries jail time, but being deliberately criminally incompetent does not. It's a pretty nasty lesson we are teaching the next generation.

  19. Re:jeez, exchange is still used? by PsychoSlashDot · · Score: 4, Interesting

    Yeah yeah we know it does work, mostly, and is probably written in VBscript or cobol.

    But damn, you can afford a EX licence, but cannot afford a high end intel 512G SSD x 2.

    Restore in 5mins.

    Hardrives, puhhhh.... so 90s, like C64 tapes. Get with the future dude.

    Sure. So you restore in minutes but that's when you realize that the information store is - by definition - backed up dirty because it's in use. A moment later you discover that Exchange insists on you running some nice ISINTEG routines to mark the database as clean before it can be mounted. Those routines joyfully take a minor eternity, even on SSD if you have a huge database. Like... 450G. When you're done with ISINTEG, if you're really lucky you can have a bonus round of ESEUTIL followed by ISINTEG again if it turns out there was any minor database structural issues you didn't know about.

    High I/O absolutely helps, but don't write this off as if massive database restores are trivial just because someone follows your advice. For businesses that are big enough to accrue huge amounts of data but not big enough to afford redundant servers, TIME is the cost they pay.

    --
    "Oh no... he found the .sig setting."
  20. Re:Unions can be a big help in stopping BS like th by Joe_Dragon · · Score: 2

    No the NON unions american airlines el salvador maintenance works did it.

  21. Re:OOHHH GOD!! by Anonymous Coward · · Score: 3, Funny

    Sure, but first, show me an exchange installation that actually works.

  22. Re:HAHA by tibit · · Score: 3, Insightful

    Protip: the world is full of people who do stupid shit for apparently no rational reason at all. There.

    --
    A successful API design takes a mixture of software design and pedagogy.
  23. Parent needs some mod points. by mu51c10rd · · Score: 2

    Perhaps one of the greatest comments ever seen regarding I.T. projects...

  24. Re:Unions can be a big help in stopping BS like th by T.E.D. · · Score: 4, Informative

    Just look at what happened at American Airlines. Some maintenance worker loosened up a bunch of seats, and bingo within a week the Pilot's union has a new contract after over a year of negotiating. Some coincidence!

    No the NON unions american airlines el salvador maintenance works did it.

    Exactly. It was only after it happened *twice* that they sent everything to the union shop (right here in Tulsa) to get it fixed right. Then they settled with the union (and *still* shipped some more of their jobs to El Salvador, just not as many as they'd been trying to).

    I'd really like to see the AC's story about the union NFL referees. The non-union refs are comically bad for weeks, then blow a game-changing call on Monday Night football, and bingo within a week the Referee's union has a new contract after over a year of negotiating. Some coincidence!

  25. Re:HAHA by V+for+Vendetta · · Score: 2

    The thing I'm really struggling with is why on Earth would anyone do such a thing

    As per TFA:

    [...] and we found out that he'd sent the traffic home to ensure that his routing patterns at work were correct," Saccavino told InformationWeek in a recent interview. But after a week, Saccavino said, he'd forgotten to turn it off.

    But given the rest of the story, I'm not sure if that's the only reason.

  26. Re:Unions can be a big help in stopping BS like th by cusco · · Score: 2

    Never worked in a Union shop, have you? The difference is that in a Union shop you will get fired 'For Cause', rather than just because your boss doesn't like redheads or Asians. And in this case there was abundant cause.

    --
    "Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
  27. Regulators? by whitroth · · Score: 2

    I, too, love that they outsourced their IT - they got what they apparently deserved.

    But then there's the part in the article where it doesn't appear that before things came down that they'd *never* been audited.

    Oh, that's right, most of this happened between '01 and '08, when Bush & Cheney were in charge, and All Republicans Love Deregulation, and if you can't deregulate, strangle the budget of the regulating agency so they can't do their job.

    And before you libertarians here jump on me, tell me what you would have done if *you* had invested with them.

                          mark "that's right, you *ain't* rich, or you wouldn't be spending time reading comments on slashdot"

  28. Re:Unions can be a big help in stopping BS like th by mcgrew · · Score: 2

    On the contrary, union workers can be fired easily for what this guy is accused of.