Post Mortem of GunnAllen IT Meltdown
CowboyRobot writes "The story begins when GunnAllen, a financial company, outsourced all of its IT to The Revere Group. Before long, it was discovered that 'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.' In addition to the obvious security concerns of sending information such as bank routing information and driver's license numbers, the act violated SEC rules because the routed information was not being logged. Regardless of whether the cause was negligence, incompetence, or sabotage, the matter was swept under the rug for a time until unpaid SQL Server licenses meant threatening calls from Microsoft as well. The rest of the story is one of greed, mismanagement, and neglect, and ends with the SEC's first-ever fine for failure to protect customer data."
Wow, according to the The Revere Group website:
WHEN TRANSFORMING THEIR BUSINESS, TOP PERFORMERS TURN TO A TRUSTED ADVISOR
Guess that's not The Revere! Group
If you want news from today, you have to come back tomorrow.
Yeah keep outsourcing the responsibility of something so crucial that IT people hold the keys to the kingdom.
This is nothing new in the world of IT. Save a dime to lose a million dollars.
I am in a comany right now where they hired IT consultants for well over 3 years and come to find out so called "Experts" where just patching the system but never really fixing the real issues. It's amazing to see what these contractors were selling to a company who had the money to buy great gear only to discover pure incompetence at implementing it. I am no expert by any means but I can smeel bullshit when I see a network in need of a lot of TLC.
A financial company outsourcing its IT ought to be considered criminal negligence.
(Though an own employee could do the same thing, in this case.)
Sheesh, evil *and* a jerk. -- Jade
Here's the printer friendly page. The whole article on one page; http://www.informationweek.com/security/attacks/exclusive-anatomy-of-a-brokerage-it-melt/240008569?printer_friendly=this-page
Well, you know, he had RoadRunner... In 2005, that was pretty wicked! If he had set up two or three accounts and load balanced them...
If you want news from today, you have to come back tomorrow.
'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.
That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.
Are you trying to tell me that the SEC has rules? That they enforce? I don't believe this. This does not reflect the US that I live in; are you perhaps talking about some other country with more reasonable laws about this kind of thing - maybe you meant to say it happened in Armenia, not America?
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
It's not mentioned in the summary, but the first sign of the rerouting was, as you'd expect, their network slowing to a crawl. That earned the IT guy responsible for it a reprimand. A reprimand, for routing an entire company's trading data through his home modem for a week!
There's other gold in there too, like the time the guy pulled the cable on a production rack in order to create a catastrophe so he wouldn't have to travel to a business meeting, or his habit of remoting into IT infrastructure (Blackberry and Exchange servers were mentioned) on the weekends to fuck up their configuration, just so he could "magically" fix it on Monday morning.
He was, apparently, eventually fired.
There's no place I could be, since I've found Serenity...
I worked at a place where the Exchange admin - every so often - would have to heroically worked 72 hours or whatever to rescue the mail servers and we only have 2 days of downtime, etc etc, and the CIO would praise him for his hardwork.
I asked my boss if I should also reboot the firewalls every now and then - just to heroically bring them back up again, and get thanked for my hardwork. He gave me a nasty look...
If not given the resources to have Exchange load balanced, and if it happens to crash and requires a 200GB Store restored...72 hours sounds about right. The 2 days downtime should have been 4 hours (time to investigate and bring a backup VM online). Without a backup VM, it should have been down 1 day.
I say Sabotage. I'm presently a NOC engineer at an IT managed services provider. Before, I worked for a well-known financial market data provider. The most demanding client we have is a financial company. Everyone once in a while, they get unhappy with our service for whatever reason and decide to blast the blame-thrower. During the most recent hissy-fit episode, they threatened to not renew the service contract. Moreover, their CIO dropped in on the conference call and said not only are they not gonna renew the contract but he was gonna have us blacklisted with other financial companies that we were looking to grow business with. It's been my general impression that financial clients tend to be some of the most high maintenance, demanding, and nasty assholes. I've a hunch that a similar reason could be a factor In explaining this network engineer's actions.
Go to http://www.reveregroup.com/ and search for anything in the top right search box. You'll get a licensing error. These guys are on the ball...
Why would senior network engineer need to send traffic home to verify his routing patterns? Yeah right, he scammed millions and they covered it up to avoid more fines. Now, he and his red stapler, are at some beach resort complaining about the Mai Tais.
'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.
That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.
He probably cooked lobsters in his dishwasher, too.
A feeling of having made the same mistake before: Deja Foobar
Well, here we go! The CIO of the company outsourced the IT department to..... his own personal company. No conflict of interest there!
Unions can be a big help in stopping BS like this from happening.
When you have people purposefully break things just to look good for the bosses that's bad even worse is sweeping security and other issues under the rug.
It's hard reading IT train wreck stories, especially when the damage is self-inflicted. And yet I saw that same attitude, on both sides of the transaction, acted out over and over.
A long time ago a CIO I worked for said he wasn't worried as long as he had a throat he could choke if things went sideways. The only thing he cared about was having somewhere to cast blame.
Those were the days I naively cared about doing a good job.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
However no jail time. Refusing to disclose a password in case it's used by such an incompetent carries jail time, but being deliberately criminally incompetent does not. It's a pretty nasty lesson we are teaching the next generation.
Yeah yeah we know it does work, mostly, and is probably written in VBscript or cobol.
But damn, you can afford a EX licence, but cannot afford a high end intel 512G SSD x 2.
Restore in 5mins.
Hardrives, puhhhh.... so 90s, like C64 tapes. Get with the future dude.
Sure. So you restore in minutes but that's when you realize that the information store is - by definition - backed up dirty because it's in use. A moment later you discover that Exchange insists on you running some nice ISINTEG routines to mark the database as clean before it can be mounted. Those routines joyfully take a minor eternity, even on SSD if you have a huge database. Like... 450G. When you're done with ISINTEG, if you're really lucky you can have a bonus round of ESEUTIL followed by ISINTEG again if it turns out there was any minor database structural issues you didn't know about.
High I/O absolutely helps, but don't write this off as if massive database restores are trivial just because someone follows your advice. For businesses that are big enough to accrue huge amounts of data but not big enough to afford redundant servers, TIME is the cost they pay.
"Oh no... he found the
No the NON unions american airlines el salvador maintenance works did it.
Sure, but first, show me an exchange installation that actually works.
Protip: the world is full of people who do stupid shit for apparently no rational reason at all. There.
A successful API design takes a mixture of software design and pedagogy.
Perhaps one of the greatest comments ever seen regarding I.T. projects...
Just look at what happened at American Airlines. Some maintenance worker loosened up a bunch of seats, and bingo within a week the Pilot's union has a new contract after over a year of negotiating. Some coincidence!
No the NON unions american airlines el salvador maintenance works did it.
Exactly. It was only after it happened *twice* that they sent everything to the union shop (right here in Tulsa) to get it fixed right. Then they settled with the union (and *still* shipped some more of their jobs to El Salvador, just not as many as they'd been trying to).
I'd really like to see the AC's story about the union NFL referees. The non-union refs are comically bad for weeks, then blow a game-changing call on Monday Night football, and bingo within a week the Referee's union has a new contract after over a year of negotiating. Some coincidence!
As per TFA:
But given the rest of the story, I'm not sure if that's the only reason.
Never worked in a Union shop, have you? The difference is that in a Union shop you will get fired 'For Cause', rather than just because your boss doesn't like redheads or Asians. And in this case there was abundant cause.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
I, too, love that they outsourced their IT - they got what they apparently deserved.
But then there's the part in the article where it doesn't appear that before things came down that they'd *never* been audited.
Oh, that's right, most of this happened between '01 and '08, when Bush & Cheney were in charge, and All Republicans Love Deregulation, and if you can't deregulate, strangle the budget of the regulating agency so they can't do their job.
And before you libertarians here jump on me, tell me what you would have done if *you* had invested with them.
mark "that's right, you *ain't* rich, or you wouldn't be spending time reading comments on slashdot"
On the contrary, union workers can be fired easily for what this guy is accused of.
Free Martian Whores!