Slashdot Mirror
Post Mortem of GunnAllen IT Meltdown
CowboyRobot writes "The story begins when GunnAllen, a financial company, outsourced all of its IT to The Revere Group. Before long, it was discovered that 'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.' In addition to the obvious security concerns of sending information such as bank routing information and driver's license numbers, the act violated SEC rules because the routed information was not being logged. Regardless of whether the cause was negligence, incompetence, or sabotage, the matter was swept under the rug for a time until unpaid SQL Server licenses meant threatening calls from Microsoft as well. The rest of the story is one of greed, mismanagement, and neglect, and ends with the SEC's first-ever fine for failure to protect customer data."
Yeah keep outsourcing the responsibility of something so crucial that IT people hold the keys to the kingdom.
This is nothing new in the world of IT. Save a dime to lose a million dollars.
I am in a comany right now where they hired IT consultants for well over 3 years and come to find out so called "Experts" where just patching the system but never really fixing the real issues. It's amazing to see what these contractors were selling to a company who had the money to buy great gear only to discover pure incompetence at implementing it. I am no expert by any means but I can smeel bullshit when I see a network in need of a lot of TLC.
'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.
That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.
Agreed. I work in the MSP (Managed Service Provider) sector which is a fancy way of saying that we are outsourced IT. We focus on the SMB market where a company is too small to have a dedicated IT department, but just large enough that they place a trouble ticket in our queue once a week. Sometimes once a day. Anything ranging from tier 1 to 3 support.
However, once you as a company get involved with needing to be HIPPA, PCI, or SOX compliant, that should be synonyms with "dedicated in-house IT dept".
Life is not for the lazy.
Are you trying to tell me that the SEC has rules? That they enforce? I don't believe this. This does not reflect the US that I live in; are you perhaps talking about some other country with more reasonable laws about this kind of thing - maybe you meant to say it happened in Armenia, not America?
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
It's not mentioned in the summary, but the first sign of the rerouting was, as you'd expect, their network slowing to a crawl. That earned the IT guy responsible for it a reprimand. A reprimand, for routing an entire company's trading data through his home modem for a week!
There's other gold in there too, like the time the guy pulled the cable on a production rack in order to create a catastrophe so he wouldn't have to travel to a business meeting, or his habit of remoting into IT infrastructure (Blackberry and Exchange servers were mentioned) on the weekends to fuck up their configuration, just so he could "magically" fix it on Monday morning.
He was, apparently, eventually fired.
There's no place I could be, since I've found Serenity...
I worked at a place where the Exchange admin - every so often - would have to heroically worked 72 hours or whatever to rescue the mail servers and we only have 2 days of downtime, etc etc, and the CIO would praise him for his hardwork.
I asked my boss if I should also reboot the firewalls every now and then - just to heroically bring them back up again, and get thanked for my hardwork. He gave me a nasty look...
I say Sabotage. I'm presently a NOC engineer at an IT managed services provider. Before, I worked for a well-known financial market data provider. The most demanding client we have is a financial company. Everyone once in a while, they get unhappy with our service for whatever reason and decide to blast the blame-thrower. During the most recent hissy-fit episode, they threatened to not renew the service contract. Moreover, their CIO dropped in on the conference call and said not only are they not gonna renew the contract but he was gonna have us blacklisted with other financial companies that we were looking to grow business with. It's been my general impression that financial clients tend to be some of the most high maintenance, demanding, and nasty assholes. I've a hunch that a similar reason could be a factor In explaining this network engineer's actions.
A financial company outsourcing its IT ought to be considered criminal negligence.
Outsourcing IT isn't the problem. A failure to oversee the IT services provided was the problem; A complete lack of auditing and process control. I wish people would stop looking at outsourcing as somehow evil; It makes sense in a lot of cases. Most corporations have other companies contracted to replace and maintain printers. Most office printers have the ability to retain all documents printed from it, locally, to a harddrive inside it. That isn't a problem by itself -- unless you don't know that the functionality is enabled, and don't audit or remove the drives before the printers are rolled out the front door with all your confidential data... that you thought was secure because you had a contract to shred all your documents.
The story of GunnAllen's criminal negligence starts with the CTO and board of directors -- who fired people for coming forward with security problems, and had a very obvious closed-door policy. Nobody with the parent company wanted to hear about problems, and it's no surprise that the firm they contracted with heard that loud and clear -- and propagated the same attitude right on down the line. "See no evil, hear no evil" often leads to a lot of people doing evil.
GunnAllen's story is one being repeated by the thousand every morning of every workday across our industry. Managerial incompetence leads to otherwise trivial problems becoming fines, bankrupcy, and lawsuits. This story is not about the failures of IT -- IT was involved, but it was not that failed. It was the people at the top... and when the extent of the damage was finally discovered by the government, they tried to pin it all on former employees and the people under them. I'd like to know where those managers are now; Because I know they'll eventually find themselves in another position of power at another company. Whereas all the engineers and people who actually worked for a living, well... we all know what happened to them, whether the article says so or not.
You want to fix problems like this: Start with accountability.
#fuckbeta #iamslashdot #dicemustdie
Unions can be a big help in stopping BS like this from happening.
When you have people purposefully break things just to look good for the bosses that's bad even worse is sweeping security and other issues under the rug.