Post Mortem of GunnAllen IT Meltdown

CowboyRobot writes "The story begins when GunnAllen, a financial company, outsourced all of its IT to The Revere Group. Before long, it was discovered that 'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.' In addition to the obvious security concerns of sending information such as bank routing information and driver's license numbers, the act violated SEC rules because the routed information was not being logged. Regardless of whether the cause was negligence, incompetence, or sabotage, the matter was swept under the rug for a time until unpaid SQL Server licenses meant threatening calls from Microsoft as well. The rest of the story is one of greed, mismanagement, and neglect, and ends with the SEC's first-ever fine for failure to protect customer data."

Comments

Trusted Advisor?

[Frosty Piss]

Wow, according to the The Revere Group website:

WHEN TRANSFORMING THEIR BUSINESS, TOP PERFORMERS TURN TO A TRUSTED ADVISOR

Guess that's not The Revere! Group

Re:Trusted Advisor?

[Anonymous Coward]

Been had a little horsy named Paul Revere
Just me and my horsy and a quart of beer
Riding across the land, kicking up sand
Sheriff's posse on my tail cause I'm in demand

Re:Trusted Advisor?

[tautog]

Been had a little horsy named Paul Revere
Just me and my horsy and a quart of beer
Riding across the land, kicking up sand
Sheriff's posse on my tail cause I'm in demand

Burmashave?

Re:Trusted Advisor?

[T.E.D.]

Wow, according to the The Revere Group website:

WHEN TRANSFORMING THEIR BUSINESS, TOP PERFORMERS TURN TO A TRUSTED ADVISOR

...but they are too expensive, so they then turn to the Revere Group.

Re:Trusted Advisor?

[tiptone]

Beastie Boys. GTFO

Re:HAHA

[Anonymous Coward]

I agree. This is the funniest thing I've heard today. Through his home cable modem! HAHAHAHAHAHA!

Outsourced

[Anonymous Coward]

Yeah keep outsourcing the responsibility of something so crucial that IT people hold the keys to the kingdom.
This is nothing new in the world of IT. Save a dime to lose a million dollars.
I am in a comany right now where they hired IT consultants for well over 3 years and come to find out so called "Experts" where just patching the system but never really fixing the real issues. It's amazing to see what these contractors were selling to a company who had the money to buy great gear only to discover pure incompetence at implementing it. I am no expert by any means but I can smeel bullshit when I see a network in need of a lot of TLC.

Re:Outsourced

[bbelt16ag]

indeed, no expert here either, but i know enough to be dangerous. this is a complete lack of understanding of even the basics of what is going on in their networks, or how to identify who is a good engineer or a bad one.

Re:Outsourced

[Anonymous Coward]

IT doesn't make money. It COSTS money! Why should shareholders invest in IT when they receive no return on investment like you would with top talent or a bigger sales and marketing force to increase its value?

They are a finance company. Not an IT one so wouldn't an outsourcer provide a better job as it is its own strength?

Re:Outsourced

[AK Marc]

For the same reason they don't oursource their upper management. After all, CEOs cost money, why not outsource CEO to a management company and cut costs. After all, they are a finance company, not a management company, so all their management should be outsourced.

Re:Outsourced

[Anonymous Coward]

Search Gogle for management consulting.

Re:Outsourced

[JDG1980]

They are a finance company. Not an IT one

If you run any business beyond the level of a mom-and-pop restaurant, you are in the IT business whether you want to be or not. The only question is whether you will leverage IT as a strategic asset or be outcompeted by those who do.

Re:Outsourced

[AK Marc]

Consultants are often used for outsourcing blame, rather than outsourcing capability. "Oh, our consultant recommended that."

Re:Outsourced

[Rhinobird]

Eventually the people in charge are going to realize that any kind of financial institution is basically a database on the internet that holds and exchanges account information. And then they're going to turn ghostly white as they realize all these strangers are touching the equipment that, in a very real sense, IS the bank, er, financial whatever...or worse, those strangers OWN the equipment that IS the financial gobstopper.

And then, at least in finance, outsourcing IT will be seen as a form of insanity.

Re:Outsourced

[Anonymous Coward]

With the revolving door nature of CEO and other top level jobs these days, you could argue that upper management is already outsourced away from the actual company. Just that they compete on paying the most instead of the least.

Re:Outsourced

[Anonymous Coward]

No CEOs through their billiant ideas are what make products and things happen. Look at Nokia and Microsoft right now for what a bad CEO will do? An IT guy ... well he just plays with computers and costs money. Not making those sales or boasting share prices. Accountants do not know how to do IT stuff. IT companies do.

Re:Outsourced

[Anonymous Coward]

If your system is down for a week because you outsourced your experienced IT and replaced them with cheap idiots, how much does that cost your business in terms of lost/delayed orders, pissed off customers, stock dives, and the finger-pointing blame game?

Re:Outsourced

[Anonymous Coward]

I dread working with these companies, they have one outsourcing group for the desktops, one for the servers one for the network and another for the firewalls and load balancers. It is a pointing party every time we work on something.

Re:Outsourced

[alexgieg]

IT doesn't make money. It COSTS money!

Oh! Another Peter Drucker fan stuck in 1940's MBA theory. Time to upgrade to 2000's version, yes?

Re:Outsourced

[tibit]

This whole "cut costs" thing is just stupid. Even in a fairly large company, the expensive IT engineering side can be just a couple of people. If they seriously earn so much money that outsourcing is cheaper, then perhaps I can apply for the job at a 25% discount. I wouldn't mind that $250k/year salary, after all.

Re:Outsourced

[tibit]

Because, obviously, in an "X company" everyone does X. Accountants do not know how to do clerical office work, they should just outsource their secretaries abroad, right? Right? Let's cut the bullshit. A company is ultimately people and processes. People you can get, and processes you can learn. I don't buy that IT consulting companies have some magical process powder that makes their people so much more effective than the same people, were they to be hired thy the "X company". It seems to be a ploy everyone falls for, but without any backing in fact. Most IT outsourcing is a disaster that's only used to postpone the inevitable technical debt blowup, and is used, demonstrably, only to temporarily boost share values.

Re:Outsourced

[Anonymous Coward]

That's exactly why I got out of consulting, there was an almost religious aspect to prolonging problems rather than fixing them.

Re:Outsourced

[AwesomeMcgee]

You presume these people aren't blinded by the number of 0's in their paychecks into believing that they are the ultimate asset and whatever all those IT people are doing for the company is totally unimportant and unnecessary.

Re:Outsourced

[HornWumpus]

Dogfood.com 'leveraged IT as a strategic asset'. Didn't protect them from bad strategy. Did they put other pet food stores out of business?

The third alternative answer is focus on your core and out compete those who incompetently execute their IT leveraging.

In my experience most small business IT isn't all that competent. The part that is competent stays focused on specific narrow tasks. There is only a narrow subset of business that can even potentially 'out compete' based on IT.

Re:Outsourced

[HornWumpus]

They sell a product that is essentially identical to their competition's. Of course marketing will run the business.

Working in IT in banking or insurance and expecting to be treated as anything other then a cost is insane. If you can't differentiate your companies product on the strength of your teams work accept you are just a digital janitor and focus your energy appropriately. Get a better job.

There is a difference between necessary and critical.

Re:Outsourced

[Anonymous Coward]

No, it SAVES money. How many extra employees would they need to process the amount of data financial companies process nowadays using pen, paper, typewriters and mechanical adding machines of even pocket calculators, telephone, mail, messengers? Thousands? Tens of thousands? Millions perhaps with the speed the financial world operates with nowadays? When I started my first job as a programmer for a bank in 1985 that company was fully aware that their investment in an expensive mainframe computer and expensive people to operate it and write software for it saved much more money than it cost. And then it was ridiculously expensive compared to today. Only everyone seems to have forgotten that data doesn't store, process, communicate and backup itself, that automated systems don't build and maintain themselves, and that the investments in automating it are a fraction of what it would cost to do it manually.

Of course they can't afford to spend much more than their competitors do if they want to stay in business, and that pressure has made people forget that what they spend is far less than what they save by spending it. But if you forget that your feet aren't on the ground anymore. IT does NOT only cost money. Automating routine work means humans don't have to do it, and even a slow computer can perform MANY more routine data processing actions per second than any human can, and with a much higher reliability. That should not be forgotten.

You write GunnAllen is not an IT company. But their core business is to process information, their IT is their primary production line. It's stupid to neglect that.

Re:Outsourced

[Bryansix]

Are you done sensationalizing? I work for an MSP who has financial advisors as clients. These places are never going to be able to hire a full time person to perform their IT duties. Even if they could afford that they would have to pay even more for all the monitoring, backup, etc. Instead with us they get all of that at a fraction of the cost since we can solve almost any situation remotely but will also drive to fix things that require on-site. In the time we have had these clients, security has improved as we brought them in compliance with things like the GLBA etc. with whole hard drive encryption and encrypted email communication. Our company has had a security leak exactly zero times. I guess the point is this. Its not important that you outsource or don't outsource; its important WHO you have doing the work. Inside or out, these people CAN cause a lot of headaches but hire the right group and you'll be fine.

Re:Outsourced

[Anonymous Coward]

Gotta have someone to blame shit on if things go terribly wrong.

Re:Outsourced

[swalve]

It depends. Outsourcing goes bad when the people doing the outsourcing can't (or won't) understand whether they are getting good results or not. Simple example: the supply division switches over to refilled toner cartridges. Their costs go down. Yay! Meanwhile, the people fixing the printers see an uptick in service calls and the end users have more downtime. It's easy to measure those increases, but it's harder to correlate them and then prove it to the decision makers.

There are lots of situations where outsourcing IT makes sense, but only if an org is lucky enough to have a good provider and smart enough to retain some people to oversee it properly.

Re:Outsourced

[swalve]

It's all about economies of scale. Every situation is different.

Re:Outsourced

[Macgrrl]

Not getting into an argument regarding the quality of candidates available for direct hire vs outsourcing, but sometimes there are other issues at hand.

I used to work as a consultant for an large outsourcing division of a name brand company. We specialised in providing office automation services for back of house functionality which was often not core business for our customers.

In many respects the biggest beneficiaries of our services were not the companies but the staff we took over. Most of them were in what the companies considered dead end jobs with no career paths (how many people get promoted to management from the mailroom in reality these days). We gave them training, mentored them for team leader and management positions, gave them the opportunity to be promoted to higher performing teams based at other clients' sites or at our own offices running training or process improvement initiatives. One of the most important things it did was give them a peer group they could network with and relate to.

Quite a few of the head office staff in operations roles had come from staff we'd acquired taking on mailrooms, printrooms or imaging centres. They had the same opportunities to apply for positions as people who had come in as graduates directly.

Most of our clients wouldn't have known an efficient performing printroom operator vs a poor one. They had no incentive programs to motivate them. They provided no opportunities for improvement. We had performance benchmarks, tailored job descriptions that went beyond 'other duties as required', offered training and peer review.

Large IT consultancies CAN offer similar benefits. I'm not saying that all do, or even try to. But done right there can be benefits to both the client and the staff.

Re:Outsourced

[tibit]

Quite interesting. I didn't think of that :) The occasional slashdot gem post -- thanks!

Sigh...

[Black Parrot]

A financial company outsourcing its IT ought to be considered criminal negligence.

(Though an own employee could do the same thing, in this case.)

Re:Sigh...

[DigiShaman]

Agreed. I work in the MSP (Managed Service Provider) sector which is a fancy way of saying that we are outsourced IT. We focus on the SMB market where a company is too small to have a dedicated IT department, but just large enough that they place a trouble ticket in our queue once a week. Sometimes once a day. Anything ranging from tier 1 to 3 support.

However, once you as a company get involved with needing to be HIPPA, PCI, or SOX compliant, that should be synonyms with "dedicated in-house IT dept".

Re:Sigh...

[Charliemopps]

I'd have to disagree. We have our own in house IT department... but a small part of our business is providing outsourced IT. And our stuff ridiculously overbuilt and robust. I doubt anyone could do it in house better. But it's expensive as hell, and not very flexible. If you're not getting too creative with your needs, and you have the money, you can get something very robust. But if you want to go on the cheap and still get some crazy ass system no ones ever tried before to work, then I think you're shit out of luck no matter who you go with.

Re:Sigh...

[cbhacking]

Actually, given the specific expertise and experience required for such compliance (at least, for doing it right), I can see an argument for specialized IT services companies that handle the needs of companies up to a certain size (bigger than you were talking about, though not necessarily by much; still too small to make it worth hiring a team of such people). The problem is, you've got to assign responsibility along with that contract. LOTS of responsibility, as in no-feasible-way-in-hell-you-could-save-more-money-from-negligence-that-puts-our-compliance-at-risk-than-you'd-have-to-pay-for-breach-of-contract levels of responsibility. If the outsourced company has a serious stake in the matter, then it shouldn't be a problem... yeah, they could still screw up and be grossly incompetent or have a malicious insider, but the same is true of in-house people.

Not that I disagree that outsourcing such critical roles is a terrible idea in general... but sometimes, it really is the only economically practical option, and that shouldn't mean you can't do business at all. Besides, just because the current way the outsourcing is done is broken, that doesn't mean you have to throw the whole idea out; it may be possible to fix it instead.

Re:Sigh...

[girlintraining]

A financial company outsourcing its IT ought to be considered criminal negligence.

Outsourcing IT isn't the problem. A failure to oversee the IT services provided was the problem; A complete lack of auditing and process control. I wish people would stop looking at outsourcing as somehow evil; It makes sense in a lot of cases. Most corporations have other companies contracted to replace and maintain printers. Most office printers have the ability to retain all documents printed from it, locally, to a harddrive inside it. That isn't a problem by itself -- unless you don't know that the functionality is enabled, and don't audit or remove the drives before the printers are rolled out the front door with all your confidential data... that you thought was secure because you had a contract to shred all your documents.

The story of GunnAllen's criminal negligence starts with the CTO and board of directors -- who fired people for coming forward with security problems, and had a very obvious closed-door policy. Nobody with the parent company wanted to hear about problems, and it's no surprise that the firm they contracted with heard that loud and clear -- and propagated the same attitude right on down the line. "See no evil, hear no evil" often leads to a lot of people doing evil.

GunnAllen's story is one being repeated by the thousand every morning of every workday across our industry. Managerial incompetence leads to otherwise trivial problems becoming fines, bankrupcy, and lawsuits. This story is not about the failures of IT -- IT was involved, but it was not that failed. It was the people at the top... and when the extent of the damage was finally discovered by the government, they tried to pin it all on former employees and the people under them. I'd like to know where those managers are now; Because I know they'll eventually find themselves in another position of power at another company. Whereas all the engineers and people who actually worked for a living, well... we all know what happened to them, whether the article says so or not.

You want to fix problems like this: Start with accountability.

Re:Sigh...

[AK Marc]

I worked for a VAR for a while, and we sold a lot of wireless gear to a hospital and set up the system, with HIPAA compliance and all. For a relatively large hospital. There wasn't much of an IT department. I think IT people don't like working for doctors, I'm not sure anyone likes working for doctors.

Re:Sigh...

[slashmydots]

A financial company outsourcing its IT ought to be considered criminal negligence.

(Though an own employee could do the same thing, in this case.)

I worked at a hospital with around 1000 computers and IT was onsite but contracted from a 3rd party. So, that's odd but get this! They outsourced the support calls to Mexico! Yeah, you could walk right down to the damn IT office yourself on floor 1 and get your problem taken care of or you could call Mexico. You could even simply get an extension of someone in IT and call that...or call Mexico! MEXICO! AT A HOSPITAL! By the way, I was there on a 6 month PC replacement project from a different contractor that the other contractors hired. Oh and they all got fired 4 months later when the hospital didn't renew their contract.

Re:Sigh...

[LordLucless]

I'd have to disagree. We have our own in house IT department... but a small part of our business is providing outsourced IT. And our stuff ridiculously overbuilt and robust.

It's not about robustness in these instances. It's about power and accountability. When you have hugely sensitive information (medical records, credit card details or financial records) you must be in control of your own systems. While downtime sucks, downtime is often better than data compromise in these cases.

Re:Sigh...

[hairyfish]

I'd have to disagree. We have our own in house IT department... but a small part of our business is providing outsourced IT. And our stuff ridiculously overbuilt and robust. I doubt anyone could do it in house better. But it's expensive as hell, and not very flexible.

I bet you can't even see the irony of your post. If it expensive and inflexible then it's quite easy to do it better don't you think? The problem you haven't addressed is that every business has different requirements and not all of them require super-robustness. I worked both sides of the fence, and MSP has it's place but it isn't the solution for everyone (as TFA quite nicely demonstrates).

Re:Sigh...

[KingMotley]

That sounds grossly naive. What company over the size of 0 employees doesn't have one of the following: Medical records, Credit card details, or financial records? Every single company has those, even companies that have 1 part time person in it. I seriously don't think there is enough IT professionals in the entire world worth a damn that you could have 1 at every single company.

Outsourcing isn't the problem with data breaches. Outsourcing to companies that back up their promises with financial guarantees and fines is the problem. These companies that do outsourcing need to have a well trained staff that can actually do security well and they need to have a vested financial interest in doing it well. You would find less data breaches by having all the worlds "sensitive" information in a few dozen companies who actual responsibility to keep to safe than to have it in hundreds of thousands of companies who are totally incompetent.

Re:Sigh...

[mjwx]

Outsourcing IT isn't the problem. A failure to oversee the IT services provided was the problem;

Which is difficult to impossible to do unless you're directly managing the technicians. In which case, why are you paying another company A$200 an hour when the same techs would jump at being directly offered A$35-60 an hour (consulting rates in Oz).

So we're back to outsourcing being the problem. There may be more to it than that, but if you need 100% control, you cant get that by going through third party.

Re:Sigh...

[drinkypoo]

You want to fix problems like this: Start with accountability.

Yes, and you start with accountability by keeping your IT in house, where you have some control over the IT workers. In fact, outsourcing is primarily a vehicle for disposing of accountability; as long as the company you're outsourcing to claims responsibility, you get to avoid it. And then you have situations like this. Anyone outsourcing their IT is a dumbfuck. The only businesses who should ever hire anyone external to do any computer work are those whose business is too small to justify a full-time IT employee, because computing infrastructure is now critical to business.

It would have been a step in the right direction to audit the outsourced services, but you need an IT employee you can trust for that. And if you're only going to have one, is he going to sit around twiddling his thumbs any time he's not performing an audit for you? Or are you going to have an IT department of your own to handle the critical services upon which your business depends? It seems like a no-brainer to me, but then, it also seems like there's a lot of people with no brain, especially in management.

Re:Sigh...

[drinkypoo]

That sounds grossly naive. What company over the size of 0 employees doesn't have one of the following: Medical records, Credit card details, or financial records? Every single company has those, even companies that have 1 part time person in it. I seriously don't think there is enough IT professionals in the entire world worth a damn that you could have 1 at every single company.

You're being grossly obtuse. We're talking about a bank here. They are directly responsible for customer data, and they are explicitly on the hook in the case of data breaches. It is a gross failure of responsibility not to maintain IT in-house when your entire business is built on IT, which is the case in banking today. They can't do anything for you if the computers are down, except take a deposit and give you a handwritten slip in exchange. And if I walk into my bank and their computers are down in this day and age of clustering and high availability, I'm probably going to go to some other bank and open an account with that check.

Re:Sigh...

[Anonymous Coward]

Outsourcing IS evil, because it is creating a giant race to the bottom where quality surrenders to profit. As long as there is a large income disparity between countries, companies will try to ship their "costs" over there to save a buck, never noticing or caring that it is costing them in so many other ways.

I would argue excessive, unregulated outsourcing is one of the largest reasons the world economy is in the toilet right now.

Coincidentally, I don't disagree with you that accountability is another huge problem. But it is pretty related.

Re:Sigh...

[bluefoxlucid]

It is a gross failure of responsibility not to maintain IT in-house when your entire business is built on IT, which is the case in banking today.

Why? Contractors are still people, just their payroll department is elsewhere. They live in your building, sit at your desks, use your computers. I mean hell, I worked at the Social Security Administration and most people at the NSA are contractors. Some of my coworkers WERE NSA at one job for a while; we worked in the same office, I wasn't cleared and they didn't work on secret projects in the same office because that office wasn't a secure room or else, you know, I wouldn't be allowed in it.

Re:Sigh...

[bluefoxlucid]

In the US, it's $35-$60/hr plus the cost of benefits plus compliance with EEO laws plus payroll taxes plus you actually have to run payroll and accounting for all that instead of dumping a brick of cash into a line-item on your accounting.

Re:Sigh...

[KingMotley]

The article is specifically about a bank, but lordlucless wasn't speaking about the bank specifically. He took one example then expanded it to encompass basically every company on the planet.

As for your post:
1) Banks are hardly "built on IT". Not much more so than any other company out there. You walk into a store and the "computers are down" (this included McDonalds), and either they do the same thing, take your money have hand you bank a handwritten slip, or they are just closed. Like most companies, the computers make doing things there more efficient, but they are hardly necessary to actually complete the work. Banks existed before them, and if they disappeared off the face of the planet, they would still be around (after everyone rioted and burned them to the ground).
2) There is very little reason why the day to day activities of a bank and it's tellers etc need to be run by an in house IT. The bank isn't in business of buying/selling computers. Sure you need a highly secure network for the money transactions, etc., but are you suggesting that billy bob down IT guy who grew up down the street from the bank is better than companies with experts in the field that build highly secure banking networks 1000 times before?
3) Banks do a hell of a lot more than just process your deposits and withdraws and keep a tally on them. Most do investments (Stocks, Bonds, IRAs, Treasury Certificates, etc), loans (Personal, Car, Mortgage), credit cards (Visa, Mastercard), etc etc.
4) If I'm banking at a small mom and pop bank (Actually I don't, but I did for many years, they just got sucked up into Chase -- but there are many reasons for wanting a smaller bank), then as someone who actually worked on clustering and high availability systems (building, and writing clustering drivers), I'd rather they spend it on other more important things like backups. I can deal with the infrequent computers being down problem. What I don't want it my account hacked, or financial records being completely lost (unless of course, I only have a mortgage with them, then please lose everything!)

Re:Sigh...

[tibit]

So I ask, then, what's the fucking point? Hire the people and be the one who's in control. Don't pay someone else's profits.

Re:Sigh...

[DennyK]

I used to work for a web hosting company, and it was amazing how many of our clients would submit support tickets demanding that we make their $15/mo shared web hosting accounts PCI compliant. We even had some actual *banks* hosting their web sites on our cheap shared accounts. I suspect a lot of the problem was that these customers had no IT staff or knowledge and didn't understand that their requests were ridiculous or what a terrible idea it was to store unencrypted financial data on a third party shared hosting platform. (Unfortunately, since we'd gotten out of the dedicated server business and only sold shared hosting, we weren't even supposed to tell these poor folks that it was a terrible idea and they really needed a dedicated self-managed system of some sort at a minimum, since that would mean we'd lose their account.)

Re:Sigh...

[bluefoxlucid]

Well, you have a small shop of specialized people to handle a handful of sparkly-resume folks to hopefully do the job right. To hopefully improve chances of doing the job right, you hire a guy who seems pretty smart and has a sparkly resume to act as a manager of these sparkly-resume college kids. For all this, you get to cover logistics and take on the liability of managing them; and when things get out of hand, they outsource to the Internet or friends from college, leaking hints of your internal operations to untrusteds trying to get a grasp on things--which may be wobbly.

Or, you pay someone whose small shop of specialized people handles payroll for a big shop of IT people. They have specialized legal department that knows more than YOUR legal department about YOUR requirements for SOX and HIPPA and SAS70. They not only handle the logistics of finding you a sparkly-resume college kid, but they supply wide-spanning, far-reaching technical expertise comprising both management and consulting to make sure the sparkly-resume college kid is either REALLY good or can be collared and handled properly; if he's kind of fresh they'll put him somewhere less-critical and give support.

Speaking of support, even the veterans have a fallback that's legally bound to contract and confidentiality--grabbing some help from coworkers from your parent contracting agency is better than blabbing around on Web forums. Even when you're a fully-placed shop (i.e. hired to give site support, you don't know your coworkers), your contracting manager can find you people to talk to--mine hooked me up with some Windows guys and Unix guys under the same contract in other parts of the building, once upon a time, when I needed specialized help.

It's very expensive to do it right once or twice. Some of these places do it right hundreds or thousands of times a month. 'Doing it right' means hiring or properly training the right people, something that is logistically hard. How do you find a manager you know knows IT? You must know IT to know he's not a moron. A shop that specializes in IT is, by nature, well prepared to weed out the morons.

Re:Sigh...

[andy1307]

They are directly responsible for customer data, and they are explicitly on the hook in the case of data breaches. It is a gross failure of responsibility not to maintain IT in-house

Right....because banks with in-house IT departments don't suffer from data breaches...

Re:Sigh...

[tibit]

Yeah, but how do you know that the IT shop you're hiring is not in fact full of morons? There seems to be plenty of such IT shops around, big ones, even. You need some sort of external references or vetting if you lack your own know-how no matter whether you're hiring a shop or "just" some employees. In light of seemingly endless snafus masterminded by big consulting firms, I'd much rather hire the right people even if initially just as consultants and have direct managerial oversight. I mean, you must be good at something -- even if you don't have pre-existing technical know-how, as a good manager you should be able to figure out if a project is running correctly, and whether the manager under you is doing their job. I do see your point, but no matter how polished the airport billboards are, well run and competent IT consulting shops are few and far between. You may be quite lucky to be in one.

Re:Sigh...

[swalve]

That seems like more of a reason, provided the company that provides the IT is more of an expert in compliance than the main company.

Re:Sigh...

[swalve]

If done correctly, their profits are only their cut of your savings. You save, they profit, everyone wins.

Re:Sigh...

[bluefoxlucid]

We could continue this back and forth, but really, I think I'll defer the question: why are you outsourcing your software? Can't you just hire a software team to write an operating system and in-house control software for your financials? Surely it's better to have your own techs write your encryption implementation than to use the mess that is OpenSSL?

Re:Sigh...

[tibit]

I think that's a false dichotomy. Sure you could hire a team if it made business sense, but usually it doesn't. But then I think that for core business it often does make such sense.

Re:Sigh...

[bluefoxlucid]

Same problem though. You're outsourcing critical systems (software). The software S&P and TradeKing run, for example, is a multi-layer application with a backend, middleware (in Java!), and frontend with so many security holes it's hilarious. The vendor hired their own security analyst--a direct hire, not a contractor--who basically wanders around doing a Dunning-Kruger, thinking he knows what he's doing and not realizing just how broke-as-fuck their software is. They tell their clients they have a specialized security guy; they're more oblivious than fraudulent, they have a guy who's supposed to be doing a job he's not doing.

I worked at a place that built its internal financial system. PCI2 compliant, SAS70, the works. I basically hacked it by accident--wanted to cancel a service I had, didn't feel like jumping through the hoops so I set my credit card to something invalid and let it auto-expire. When I set it to the invalid value, it gave me an SQL statement dump and a DB error... it passed all that crap verbatim and basically sent me a response like, 'Put a quote here, a select statement, and a ;--, and enjoy all credit card numbers' (enough debug that in one second I already knew the layout of the database and how to slip in an SQL injection AND that it would work--I didn't follow through, but that's a SIGNIFICANT amount of recon, something most hackers try to dig out intentionally). It ran the update, too, btw, but broke the shit out of their system until they fixed it. That much brokenness was standard--duct tape and chewing gum technology here.

Take your pick. IT outsourcing works better because system administration, basic engineering, networking, and security grow experience. Programming is often more a grind fest because it's a dog-and-pony show: if it works, it works, and minor problems (runs like shit, crashes sometimes, etc) aren't important. Other IT tasks not so much, information blindness (audit system doesn't cover everything) and poorly loaded networks (90% on switch A and 10% on switch B is wrong) get attention, analysis, and fixes. Also they're just easier (programming is HARD).

Easier to Read Article

[cluedweasel]

Here's the printer friendly page. The whole article on one page; http://www.informationweek.com/security/attacks/exclusive-anatomy-of-a-brokerage-it-melt/240008569?printer_friendly=this-page

Re:HAHA

[Frosty Piss]

Well, you know, he had RoadRunner... In 2005, that was pretty wicked! If he had set up two or three accounts and load balanced them...

BOOM!

[Anonymous Coward]

What a clusterfuck!

Seriously, this seemed like a good idea?

[Anonymous Coward]

GunnAllen, a financial company, outsourced all of its IT

I think I've found the first problem.

Re:HAHA

[El Puerco Loco]

'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

Wait a minute...

[damn_registrars]

Are you trying to tell me that the SEC has rules? That they enforce? I don't believe this. This does not reflect the US that I live in; are you perhaps talking about some other country with more reasonable laws about this kind of thing - maybe you meant to say it happened in Armenia, not America?

Re:Wait a minute...

[jamstar7]

Of course the rules get enforced, if you're small enough to where you can't outlawyer the Feds. Why you think none of the big brokerage houses faced prosecution? For every lawyer the DoJ fielded, the brokerages fielded *5* or more.. And it didn't help that a Republican-controlled Congress cut their funding to the point where the DoJ was damned near useless.

Re:Wait a minute...

[khallow]

And it didn't help that a Republican-controlled Congress cut their funding to the point where the DoJ was damned near useless.

Even with funding, the DoJ would be pretty useless. I'll just trot out the current Republican talking points about Fast and Furious since they'll illustrate a good reason why the Republicans wouldn't be inclined to fund the Department of Justice.

Here, you have a pretty much cut and dry case. ATF agents allowed roughly two thousand fairly high quality guns to pass to Mexican drug cartels with no attempt made to track those weapons. Since those weapons have turned up at many crime scenes, including the murder of a US border agent (which is what finally shut down Fast and Furious). Further, the ATF agents involved knew for a few months before that final murder that these weapons were turning up at crime scenes, including murders. So a prosecutor has a pretty good case that someone committed a bunch of acts of accessory to murder (with reckless disregard for human life) and other crimes, plus the murder of a federal law enforcement officer. So what is the Department of Justice doing with this case? Hiding the agents involved in Washington DC. When will they investigate this?

This is why the "more funding" argument doesn't work. If the Department of Justice isn't going to do its job, then it doesn't really matter how much they're paid so might as well make it a little rather than a lot. The SEC is particularly notorious for providing the illusion of security for novice investors, or in other words, helping keep the marks from getting scared off before they can be fleeced.

Re:Wait a minute...

[CodeBuster]

I think that both of you are missing the essential mater. While it's true that the SEC reserves criminal prosecution for the most egregious cases, relying more upon fines and plea bargaining, it can also be argued, and indeed it has been, that this general strategy really is the most effective use of limited taxpayer resources; allowing the most correction to be achieved for the tax monies spent. Sure, you could increase the enforcement budget of the SEC and expand the number of prosecutors, investigators and associated support staff but what would that accomplish? The courts dockets are already jammed and even tripling the budget of the SEC would allow only a small fraction of additional cases to be investigated and prosecuted. Meanwhile, the US government is still drowning in debt with no viable long term policy to put the financial house in order. Could the SEC do better with what they're given? Probably. Is spending a majority of agency resources on a few high profile prosecutions each year, while letting many smaller fish pass untouched, in the best interests of the American people and the investing public? Probably not.

Re:Wait a minute...

[Bill_the_Engineer]

Even with funding, the DoJ would be pretty useless. I'll just trot out the current Republican talking points about Fast and Furious since they'll illustrate a good reason why the Republicans wouldn't be inclined to fund the Department of Justice.

You could but then again I could just trot out the bananas can't be considered oranges.

I hate to be the one to break it to you, but the reason politicians love to underfund enforcement is to offset the showboat regulations that they pass in order to be re-elected. This way they said they passed laws that are designed to protect us from harm, while at the same time the chances of that law actually being used is low enough not to piss off the people who actually fund the politicians campaigns.

Pointing to incompetence or the occasional misstep brought on by the underfunding of enforcement as an example of why we should fund government law enforcement is part of their plan. You don't actually think they would point out the overwhelming majority of things that the government does right? That would discredit the fairytale that they are trying to sell you.

This is why the republicans in particular have been doing a shitty job. If the government is seen as doing the right thing then they wouldn't have a platform to run on. The number one reason that a republican filibusters every single bill of significance is to prevent the democrat president from looking good. Never mind that shitty legislation was passed with overwhelming support when there was a republican president. During the Bush years the attitude of the republicans was that it was okay to borrow money in order to keep taxes low because the interest being paid was offset by the nation's GDP. The day after a democrat is president, those same republicans immediately are concerned that we are borrowing too much money and selling our children to China. The amount of hyperbole that is spewed is ridiculous.

I just find it laughable that someone would vote for a candidate that is more concerned with what would make his party look good than what is good for the nation. One key sign that this is taking place is the more they try to hurt the country to prop themselves up, the more they wrap themselves in the American flag and claim to be patriotic.

Beware of the politician that campaigns on the platform that government sucks and reelect him and he'll keep it that way.

Re:Wait a minute...

[AwesomeMcgee]

Nice strawman. I can make one too, how's this:
You're an IT consultant on a team of IT consultants, due to cutbacks your team has dwindled to you and one other guy, and the company sold 40% of your networking hardware.

Will your network perform as well as it did pre-cuts? Now that you're working double shifts will your work output be of the same *quality* as pre-cuts?

People who always complain that departments or anyone isn't doing a good enough job after their resources were cut as an excuse to cut more resources is ignorant. I'm not saying if the DOJ hadn't had resources cut they would do a great job, maybe they're incompetent and deserve the cuts- I don't know, but the fact that the cuts already happened prior to this event you're referring to invalidates that event as an excuse for cuts. It's like cutting off someone's fingers then saying they didn't need them since they can't type, because they have no fingers.

Re:Wait a minute...

[Anonymous Coward]

Guns to kill people. People kill people. At least that's what Republicans used to say when they want anyone to be able to buy a gun and people said they were going to Mexican drug cartels. So I guess as long as it gun dealers selling the guns and not the government it's okay? WTF?

Re:Wait a minute...

[DeadCatX2]

Here, you have a pretty much cut and dry case. ATF agents allowed roughly two thousand fairly high quality guns to pass to Mexican drug cartels with no attempt made to track those weapons

From what I read it's not really that cut and dry. The officials involved DID want to track the guns and did try, but the bureaucracy did them in.

http://features.blogs.fortune.cnn.com/2012/06/27/fast-and-furious-truth/

Re:Wait a minute...

[khallow]

I glanced through the article in question and I must admit to being a bit puzzled. For an article about the "truth", it seems to have missed a few facts.

One of those facts is that two thousand firearms were smuggled into Mexico as a result of the program. Read through the article, you will see no mention of the number of firearms smuggled under the program. Another is that as of June 2010, a large number of those weapons had already turned up at crime scenes. Yet the program continued for another six months.

Last June, about nine months into the ATF operation known as "Fast and Furious," suspects had "purchased 1,608 firearms for over $1 million in cash transactions at various Phoenix-area gun shops," according to internal documents obtained by CBS News. The documents indicate ATF already knew that 179 of those very weapons had turned up at crime scenes in Mexico, and 130 in the U.S.

Another fact they missed is that the firearms had neither a means for being tracked (a previous program had RFID chips planted in the firearms). And they never informed the Mexican authorities. So the claim

Quite simply, there's a fundamental misconception at the heart of the Fast and Furious scandal. Nobody disputes that suspected straw purchasers under surveillance by the ATF repeatedly bought guns that eventually fell into criminal hands. Issa and others charge that the ATF intentionally allowed guns to walk as an operational tactic. But five law-enforcement agents directly involved in Fast and Furious tell Fortune that the ATF had no such tactic. They insist they never purposefully allowed guns to be illegally trafficked. Just the opposite: They say they seized weapons whenever they could but were hamstrung by prosecutors and weak laws, which stymied them at every turn.

sounds like bullshit to me. The evidence that came out in June 2010 pretty much indicates that these weapons were directly arming criminals in Mexico and the US. A fifth of their "walked guns" quickly showed up at crime scenes.

And it looks like I'm not alone.

Surveillance video in the interview shows straw purchasers leaving gun shops with boxes of weapons. Documents showed these guns were showing up at crime scenes in Mexico and ATF supervisors actually keeping track of this information. Agent Dodson and other senior agents confronted their supervisors over and over about this horrible operation.

Their answer? "If you're going to make an omlette, you've got to break some eggs."

Ms. Eban tries to downplay an email, now known as the "schism" email, sent by Mr. Voth to the team. While many say the email was about gunwalking Ms. Eban insists it was about everything but that. I'd like Mr. Voth to explain these parts (emphasis mine):

"Whether you care or not people of rank and authority at HQ are paying close attention to this case and they also believe we [Phoenix Group VII] are doing what they envisioned the Southwest Border Groups doing."

"We need to resolve our issues at this meeting. I will be damned if this case is going to suffer due to petty arguing, rumors, or other adolescent behavior."

"I don't know what all the issues are but we are all adults, we are all professionals, and we have a (sp) exciting opportunity to use the biggest tool in our law enforcement tool box. If you don't think this is fun you're in the wrong line of work -- period!"

Mr. Voth also needs to explain why they let go of their top suspect when they had him in custody. This is the man who purchased the guns found at Agent Terry's death scene. The guns that have been recovered have been ones found at crime scenes. 1,400 guns are still missing. Mr. Voth and the ATF never made an effort to interdict the weapons. None. The t

Re:Wait a minute...

[DeadCatX2]

Let's review here. You said it's "cut and dry", ATF agents just let criminals walk away with thousands of guns, and made absolutely zero attempt to prevent them from falling into the hands of criminals.

So do you have any evidence at all that the following passages from the Forbes investigation are false? Seems like it should be easy if it was just that "cut and dry", but I see this recurring theme about prosecutors saying these gun sales were legal and that ATF couldn't stop the guns or even arrest the purchasers...

By June 2010 the agents had sent the U.S. Attorney's office a list of 31 suspects they wanted to arrest, with 46 pages outlining their illegal acts. But for the next seven months prosecutors did not indict a single suspect.

[...]

Republicans who support the National Rifle Association and its attempts to weaken gun laws are lambasting ATF agents for not seizing enough weapons—ones that, in this case, prosecutors deemed to be legal.

[...]

By January 2010 the agents had identified 20 suspects who had paid some $350,000 in cash for more than 650 guns. According to Rep. Issa's congressional committee, Group VII had enough evidence to make arrests and close the case then.

This was not the view of federal prosecutors. In a meeting on Jan. 5, 2010, Emory Hurley, the assistant U.S. Attorney in Phoenix overseeing the Fast and Furious case, told the agents they lacked probable cause for arrests , according to ATF records.

[...]

It was nearly impossible in Arizona to bring a case against a straw purchaser. The federal prosecutors there did not consider the purchase of a huge volume of guns, or their handoff to a third party, sufficient evidence to seize them. A buyer who certified that the guns were for himself, then handed them off minutes later, hadn't necessarily lied and was free to change his mind. Even if a suspect bought 10 guns that were recovered days later at a Mexican crime scene, this didn't mean the initial purchase had been illegal. To these prosecutors, the pattern proved little. Instead, agents needed to link specific evidence of intent to commit a crime to each gun they wanted to seize.

[...]

prosecutors had determined, Voth says, that the "transfer of firearms" was legal. Agents had no choice but to keep investigating and start a wiretap as quickly as possible to gather evidence of criminal intent.

[...]

The wiretap represented the ATF's best—perhaps only— hope of connecting the gun purchases it had been documenting to orders from the cartels, according to Hurley. In Minneapolis, the prosecutors Voth had worked with had approved wiretap applications within 24 hours. But in Phoenix, days turned into weeks, and Group VII's wiretap application languished with prosecutors in Arizona and Washington, D.C.

[...]

Prosecutors repeatedly rebuffed Voth's requests. After examining one suspect's garbage, agents learned he was on food stamps yet had plunked down more than $300,000 for 476 firearms in six months. Voth asked if the ATF could arrest him for fraudulently accepting public assistance when he was spending such huge sums. Prosecutor Hurley said no. In another instance, a young jobless suspect paid more than $10,000 for a 50-caliber tripod-mounted sniper rifle. According to Voth, Hurley told the agents they lacked proof that he hadn't bought the gun for himself.

Voth grew deeply frustrated. In August 2010, after the ATF in Texas confiscated 80 guns—63 of them purchased in Arizona by the Fast and Furious suspects— Voth got an e-mail from a colleague there: "Are you all planning to stop some of these guys any time soon? That's a lot of gunsAre you just letting these guns walk?"

Voth responded with barely suppressed rage: "Have I offended you in some way? Because I am very offended by your e-mail. Define walk? Without Probable Cause and concurrence from the USAO [U.S. Attorney's Office] it is highway rob

Re:Wait a minute...

[DeadCatX2]

I glanced through the article in question and I must admit to being a bit puzzled. For an article about the "truth", it seems to have missed a few facts.

One of those facts is that two thousand firearms were smuggled into Mexico as a result of the program. Read through the article, you will see no mention of the number of firearms smuggled under the program.

This post is a special response to this particular piece, since the other one got kind of long filled with other quotes. TFA in question actually does in fact say that. In case you didn't see them in the quotes above, I will quote them once again here.

According to two people present, the ATF presented detailed evidence, including the fact that their suspects had purchased almost 2,000 guns, and pushed for indictments.

Correct me if I'm wrong, but that is in fact TFA in question (sub-heading: "An unusual alliance", first paragraph), stating the very fact that you said it did not mention. Come on, man, you didn't even bother ctrl-f'ing for 2,000 before running your mouth?

For that matter, the article is full of information about how many suspects had been purchasing how many guns by what dates.

By January 2010 the agents had identified 20 suspects who had paid some $350,000 in cash for more than 650 guns.

He wrote to colleagues in February 2010 that the prosecutor seemed "taken aback by some of the facts I informed him about"—by then, the Fast and Furious suspects had purchased 800 guns—"so I am setting up a briefing for him (alone no USAO 'posse') about this case and several other cases I feel he is being misled about."

After examining one suspect's garbage, agents learned he was on food stamps yet had plunked down more than $300,000 for 476 firearms in six months.

Re:Wait a minute...

[khallow]

According to two people present, the ATF presented detailed evidence, including the fact that their suspects had purchased almost 2,000 guns, and pushed for indictments.

I think it's worth noting at this point that is a different number. It tells us nothing about what happened to those weapons which the ATF supposedly was trying to keep out of circulation.

It just so happens that 2,000 guns is a number of guns that the ATF lost track of in Mexico, and which a large percent have already turned up at crime scenes in both Mexico and the US (which incidentally seems to be how these weapons are being recovered). If these numbers refer to the same weapons, then no effort of the ATF prevented those weapons from entering the hands of criminals. That in turn is a strong indication that no effort was made.

For that matter, the article is full of information about how many suspects had been purchasing how many guns by what dates.

It's not however full of information about how these weapons were used or who knew what when. For example, they take at face value that Voth was unaware of what was happening, basically, taking his word over a number of his employees who say differently.

Re:Wait a minute...

[khallow]

Keep in mind that this story is all about protecting Voth. Where's the testimony from whistleblowers in his department? Beware of any source that presents only one side as fact.

You're accusing Dave Voth, a former Marine, a man ATF named "outstanding law-enforcement employee of the year for dismantling two violent street gangs in Minneapolis", of being an accessory to the murder of a federal law enforcement officer.

Yes, I am with good cause. What I don't get is the indignation from you. We don't decide innocence or guilt based on whether someone was a former marine or possibly an outstanding law enforcement officer. We base it on what they have done. Here, it's time for an investigation of the ATF and specifically, Voth's contribution to murder and other crimes, not the bullshit we're getting where Voth and his associates get protected at the expense of justice.

Re:Wait a minute...

[khallow]

You're an IT consultant on a team of IT consultants, due to cutbacks your team has dwindled to you and one other guy, and the company sold 40% of your networking hardware.

But the analogy is incomplete.

The company also knows that you've blown off a lot of work you had the resources to do. So why pay you to do anything when they already know that you have a pattern of not doing your work?

Re:Wait a minute...

[khallow]

If you give someone a gun with the knowledge that they're going to use them in crimes, such as killing people. then that makes you an accessory to the crimes in question. One doesn't need to confuse oneself with ordinary gun control issues.

A key problem in the Fast and Furious case is that part way through there was clear evidence that a lot of firearms were getting to criminals who were using them to commit crimes and kill people. There was around 300 cases for 1600 lost weapons. At that point, the right thing to do would be to halt the program and try something new. Instead, they kept the program going for another six months until Brian Terry a border patrol agent died in a firefight in which two of these weapons were involved. That is accessory to murder right there.

At least that's what Republicans used to say when they want anyone to be able to buy a gun and people said they were going to Mexican drug cartels.

And how many people died so that you could be given this talking point?

Terrible article

[Sean]

The writing is so bad I can barely follow the story.

Woe is me

[Anonymous Coward]

Woe is me, shame and scandal in the family
Woe is me, shame and scandal in the family.

Re:HAHA

[cbhacking]

It's not mentioned in the summary, but the first sign of the rerouting was, as you'd expect, their network slowing to a crawl. That earned the IT guy responsible for it a reprimand. A reprimand, for routing an entire company's trading data through his home modem for a week!

There's other gold in there too, like the time the guy pulled the cable on a production rack in order to create a catastrophe so he wouldn't have to travel to a business meeting, or his habit of remoting into IT infrastructure (Blackberry and Exchange servers were mentioned) on the weekends to fuck up their configuration, just so he could "magically" fix it on Monday morning.

He was, apparently, eventually fired.

Sabotage

[girlinatrainingbra]

It seems a lot like "Backdraft", the movie in which the fireman is also the firebug arsonist.

_

The network engineer was sabotaging the system by logging in during the middle of the night and breaking servers such as the Blackberry server, etc., so that he could come in during the morning and be the hero by fixing everything as quickly as he wanted.

"The network would get screwy over the weekend ... then [he] would show up, and five minutes in on a Monday, he'd fix the problem," Saccavino said.

He got caught when they sent a different level of IT person to investigate the network slowdowns and who used a keylogger to catch the shenanigans.

_

The saboteur network engineer was also plain ol' lazy, he's also accused of

"purposely pulling a cable out of a production environment in order that you would not have to travel to Jacksonville to attend an HP event at the request of the CIO." As a bonus, Microsoft also threatened to revoke their licenses for their version of SQL because, get this, the CIO had not gotten around to paying the license fees. That part seems to be a management problem, and not the network engineer's fault. But obviously, if this is the first time for a stand-alone SEC fine, then there were very crazy things going on at this company.

Re:Sabotage

[Anonymous Coward]

I can't stand it I know you planned it
I'm gonna set it straight, this Watergate
I can't stand rocking when I'm in here
Because your crystal ball ain't so crystal clear
So while you sit back and wonder why
I got this fucking thorn in my side
Oh my god, it's a mirage
I'm tellin' y'all it's sabotage

Re:Sabotage

[AK Marc]

It seems a lot like "Backdraft", the movie in which the fireman is also the firebug arsonist.

way to ruin the ending, no spoiler alert.

But then, you could have used plenty of real-life examples, including firemen. http://en.wikipedia.org/wiki/John_Leonard_Orr

Re:HAHA

[the_B0fh]

I worked at a place where the Exchange admin - every so often - would have to heroically worked 72 hours or whatever to rescue the mail servers and we only have 2 days of downtime, etc etc, and the CIO would praise him for his hardwork.

I asked my boss if I should also reboot the firewalls every now and then - just to heroically bring them back up again, and get thanked for my hardwork. He gave me a nasty look...

Corruption in finance? Unpossible!

[Anonymous Coward]

Or not... it seems from the small investment firm to the core of our financial systems are manned by corrupt, lazy, money-comes-through-grift-not-work types.

Re:HAHA

[Anonymous Coward]

If not given the resources to have Exchange load balanced, and if it happens to crash and requires a 200GB Store restored...72 hours sounds about right. The 2 days downtime should have been 4 hours (time to investigate and bring a backup VM online). Without a backup VM, it should have been down 1 day.

Negligence, Incompetence, or Sabotage?

[techsurvivorman]

I say Sabotage. I'm presently a NOC engineer at an IT managed services provider. Before, I worked for a well-known financial market data provider. The most demanding client we have is a financial company. Everyone once in a while, they get unhappy with our service for whatever reason and decide to blast the blame-thrower. During the most recent hissy-fit episode, they threatened to not renew the service contract. Moreover, their CIO dropped in on the conference call and said not only are they not gonna renew the contract but he was gonna have us blacklisted with other financial companies that we were looking to grow business with. It's been my general impression that financial clients tend to be some of the most high maintenance, demanding, and nasty assholes. I've a hunch that a similar reason could be a factor In explaining this network engineer's actions.

Re:Negligence, Incompetence, or Sabotage?

[Billly Gates]

Give the finance company credit? They are rich because they are dirt cheap and compensate their profit centers well. They find the the best bang for the buck and punish those who under deliver. I know it sucks for you as these guys demand metrics and have 1 guy support 1,000 users (I know I interviewed for Citigroup and turned them down after learning about that) but that is how they get rich.

It sucks on your end but on the other end you always get great service by demanding more for less.

Re:Negligence, Incompetence, or Sabotage?

[Anonymous Coward]

"It's been my general impression that financial clients tend to be some of the most high maintenance, demanding, and nasty assholes"

It's such a surprise that Wall Street are entitled pricks!

Re:Negligence, Incompetence, or Sabotage?

[GSloop]

It sucks on your end but on the other end you always get great service by demanding more for less.

I have news for you. People have the most ingenious ways of paying back arseholes. Thus, you don't always get great service by demanding more for less.

As a matter of fact, you may [meaning almost certainly WILL] get pretty bad service when you treat people badly - by continually demanding more for less, past the point of reasonableness and fairness.

Just for fun...

[Anonymous Coward]

Go to http://www.reveregroup.com/ and search for anything in the top right search box. You'll get a licensing error. These guys are on the ball...

Re:Just for fun...

[jaxtherat]

Or how about the fact that they promote their non existent twitter profile on their main page:

https://twitter.com/revere_group

Re:Just for fun...

[Alex Belits]

Error Message: The license does not allow the use of this search interface.

lol

Use the Coveo search box inserted in the upper-right corner

wtf

of your Sharepoint sites.

BWAHAHAHAHAHAHAHAHA!!!

Re:Just for fun...

[Chris Mattern]

While it's not as out-and-out broken as their search box or twitter link, I also like their main page selection. Because everybody wants web navigation that induces motion sickness! Complete with mystery meat selections, too.

Re:Just for fun...

[Anonymous Coward]

Wow. Just... wow. That may be the worst website I've ever seen outside of a mySpace page. And this company is trying to sell IT services? How in the world can they stay in business? I guess this article should hammer in the last nail on their corporate coffin.

BOFH Strikes again

[Anonymous Coward]

FTFA:
  "He'd purposefully break things, then come in in the morning and be the hero,"
  "purposely pulling a cable out of a production environment in order that you would not have to travel to Jacksonville to attend an HP event at the request of the CIO."

Outsourced IT will bring down companies

[Anonymous Coward]

I'm in a decent position at my company. My particular skillset is luckily in decent demand, so I'm not worried if I do get outsourced.

However, I like my company. It has good benefits and the working conditions are not bad. We are looking at co-location of our data center and outsourcing some of our support.

The biggest problem I see is that the outsourcing company really sucks. Their engineers are crappy, have little skills, and know little about regulatory or other compliance requirements.

We have already begun to outsource some web development efforts to another company. Our internal IT had to bid against the external company. Apparently internal IT's costs and delivery date were not 'aggressive' enough. Long story short, the external company won the bid but are now at least two months behind and 50% over budget. That 50% translates to over $1M US. Not only that, the external company has pretty much ignored any compliance requirements (PCI, internal baseline standards, change control processes, etc.). Why can they get away with it when internal IT cannot? Simply because this is a critical project and normal controls are being relaxed. Yes, it makes absolutely no sense that the more critical a project is, the less it has to adhere to standards, but welcome to my company.

As I said, I like my company, but some idiots got sold on a vendor promise and we will end up paying for it in lost revenue and jobs.

disappointing

[slashmydots]

I mean it's disappointing that a title like that wasn't a story about someone from IT going completely berserk apeshit. It's bound to happen, lol.

Milton in the Middle

[Mr. Lwanga]

Why would senior network engineer need to send traffic home to verify his routing patterns? Yeah right, he scammed millions and they covered it up to avoid more fines. Now, he and his red stapler, are at some beach resort complaining about the Mai Tais.

Re:HAHA

[ackthpt]

'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

He probably cooked lobsters in his dishwasher, too.

Second paragraph has all you need to know

[Stiletto]

Over a period of roughly seven business days, traffic had slowed to a crawl at the Tampa, Fla.-based firm, which had outsourced its IT department to The Revere Group. GunnAllen's acting CIO, a Revere Group partner, asked a member of the IT team to investigate.

Well, here we go! The CIO of the company outsourced the IT department to..... his own personal company. No conflict of interest there!

Re:Second paragraph has all you need to know

[Chris Mattern]

Not his own personal company; he was a Revere Group employee. At one point in the narrative one IT minion discusses how he went to the CIO's Revere Group superiors. When they outsourced IT they outsourced the CIO position along with it.

Re:Second paragraph has all you need to know

[Stiletto]

Revere Group "Partner" implies an ownership stake.

Re:Second paragraph has all you need to know

[Anonymous Coward]

It could be worse. For example the CEO of Nokia could throw away all their OS development and turn to his previous (?) employer...

Re:Second paragraph has all you need to know

[Bryansix]

This is a common trick to guarantee income stream. If the company you are running has no income, tell them you are being nice and take zero salary. At the same time outsource their biggest cost to... yourself. Constant revenue stream.

Re:Sabotage spoiler

[girlinatrainingbra]

Sorry for the spoiler without the alert! ;>)

_

I meant to find a real example of another lazy network tech., sabotaging for the sake of self-aggrandization or for getting out of work, but I couldn't find an example easily, or think of the search-terms that would do it. ("Self-aggrandization" didn't lead to much..., though there are some good reads like http://www.metafilter.com/88359/Not-enough-women-have-what-it-takes-to-behave-like-arrogant-selfaggrandizing-jerks

http://www.shirky.com/weblog/2010/01/a-rant-about-women/

http://www.computerworld.com/s/article/9034438/Former_network_engineer_faces_jail_time_for_sabotaging_patient_data ) but that last one is more of a criminal sociapath.

. And there was the San Francisco City Network administrator who refused to hand over his password, even to his boss or the mayor until he was taken to court on a criminal charge.

If you know any other good tech example, I'd love to know about it.

Unions can be a big help in stopping BS like this

[Joe_Dragon]

Unions can be a big help in stopping BS like this from happening.

When you have people purposefully break things just to look good for the bosses that's bad even worse is sweeping security and other issues under the rug.

Re:HAHA

[Anonymous Coward]

Fuck exchange for stuff the like this, the worst part is that it probably did it on its "own". Granted, it shouldn't take 72 hours, but a good 8 can occur if you have an entire raid array failing or something, coupled with a bad MS update and someone who tried to install Office 2007 on the CAS.

Re:HAHA

[shvytejimas]

Sounds like this guy: http://www.bash.org/?500338

Re:HAHA

[Anonymous Coward]

Apparently Scott Adams can retire, even the pointy haired boss couldn't top this.

Re:Unions can be a big help in stopping BS like th

[Z34107]

Are you kidding? If he was union labor, they wouldn't have been able to fire him.

Hard time reading train wreck stories

[HangingChad]

It's hard reading IT train wreck stories, especially when the damage is self-inflicted. And yet I saw that same attitude, on both sides of the transaction, acted out over and over.

A long time ago a CIO I worked for said he wasn't worried as long as he had a throat he could choke if things went sideways. The only thing he cared about was having somewhere to cast blame.

Those were the days I naively cared about doing a good job.

Re:Hard time reading train wreck stories

[dbIII]

A long time ago a CIO I worked for said he wasn't worried as long as he had a throat he could choke if things went sideways

There seems to be a lot of that attitude with the cloud outsourcing. I put an example up here earlier of 25k email accounts inaccessible for a week due to a DNS typo and a long job queue to do the two second fix, but people seemed to think it was OK to have that so long as there was someone else to blame. In that case it was Microsoft doing the hosting so good luck in getting anywhere with blaming them, a customer with twenty-five thousand email accounts is ignorable small fry and legal action is pointless.

Re:Hard time reading train wreck stories

[Anonymous Coward]

I remember an issue with PCs used in controlling automatic equipment where the plant manager ranted about how we should 'Get Microsoft in here because we use 400 copies of their software." The tone of the meeting went downhill after all the IT folks, along with most of the others started laughing so hard they couldn't talk for at least 5 minutes.

Re:Hard time reading train wreck stories

[Turminder Xuss]

The five stages of IT projects: 1. Wild Enthusiasm 2. Cold Reality 3. The Hunt for the Guilty 4. Bayoneting the Wounded 5. Promoting the Absent

Re:Hard time reading train wreck stories

[Chris Mattern]

In that case it was Microsoft doing the hosting so good luck in getting anywhere with blaming them, a customer with twenty-five thousand email accounts is ignorable small fry and legal action is pointless.

Having someone to *blame* doesn't necessarily mean having someone to *sue*. It's about keeping your job, not getting legal recompense.

Re:Hard time reading train wreck stories

[Bryansix]

Just because Microsoft hosts your email doesn't mean you can't control the DNS.

Re:Hard time reading train wreck stories

[dbIII]

Microsoft control their own DNS on their own MS Exchange server farm, and a typo there caused the problem apparently.
Nice try trying to blame me for somebody else's mail problems but I refuse to be your strawman - I was trying to get email to the University in question and noticed the fault, and rang them up to find that all they could do is wait from MS to fix their typo, since other stuff under the control of MS couldn't get to it. A local change at the University did not fix their problem. That's what most hosting does, you lose the control you would have if the stuff was local or if it was your own machine or VM in somebody else's datacentre.

Re:Hard time reading train wreck stories

[Bryansix]

Maybe you should look at this then. http://www.microsoft.com/en-us/download/details.aspx?id=18128

Re:Hard time reading train wreck stories

[Anonymous Coward]

It's hard reading IT train wreck stories, especially when the damage is self-inflicted. And yet I saw that same attitude, on both sides of the transaction, acted out over and over.

For me, I find them endlessly amusing.

Re:Hard time reading train wreck stories

[metaforest]

you forgot one:
6. T-shirts for non-participants.

The Moral of the Story

[DingerX]

So, this brokerage was set up as a flag of convenience fifteen years ago and, to all appearances, operates as a loose federation of unchecked agents. One broker is charged with defrauded his clients, assigning all profitable trades to his wife, and all losses to the client. Another gets busted in a massive Ponzi scheme involving retirees and refinancing. Only when they're on the ropes does the SEC come looking at their IT operation, outsourced, from what I can see in the article, via an obvious conflict of interest to a "see-no-evil" boss and a pathological engineer. And the SEC only finds the very tip of the problem.

And that's the only time the SEC fined anyone for IT breeches of customer confidence.

Sleep well, America.

Re:The Moral of the Story

[Anonymous Coward]

I want some breeches of customer confidence! They'll go nicely with my cloak of consumer empowerment and my helm of impulse purchasing.

Should I remind that...

[xded]

You should never ascribe to malice that which is adequately explained by stupidity.

Re:Should I remind that...

[ax_42]

You should never ascribe to malice that which is adequately explained by stupidity.

If you're going to sound profound, at least cite sources: http://en.wikipedia.org/wiki/Hanlon's_razor

Re:Should I remind that...

[Anonymous Coward]

WUT!?

their CIO dropped in on the conference call and said not only are they not gonna renew the contract but he was gonna have us blacklisted with other financial companies

You should ascribe to malice what is said with malice.
Also, sue his ass for threatening to blacklist you. That shit is illegal.

Re:Unions can be a big help in stopping BS like th

[Anonymous Coward]

Are you kidding? Unions are the first to resort to such underhanded behavior. Just look at what happened at American Airlines. Some maintenance worker loosened up a bunch of seats, and bingo within a week the Pilot's union has a new contract after over a year of negotiating. Some coincidence!

jeez, exchange is still used?

[cheekyboy]

Yeah yeah we know it does work, mostly, and is probably written in VBscript or cobol.

But damn, you can afford a EX licence, but cannot afford a high end intel 512G SSD x 2.

Restore in 5mins.

Hardrives, puhhhh.... so 90s, like C64 tapes. Get with the future dude.

Re:jeez, exchange is still used?

[PsychoSlashDot]

Yeah yeah we know it does work, mostly, and is probably written in VBscript or cobol.

But damn, you can afford a EX licence, but cannot afford a high end intel 512G SSD x 2.

Restore in 5mins.

Hardrives, puhhhh.... so 90s, like C64 tapes. Get with the future dude.

Sure. So you restore in minutes but that's when you realize that the information store is - by definition - backed up dirty because it's in use. A moment later you discover that Exchange insists on you running some nice ISINTEG routines to mark the database as clean before it can be mounted. Those routines joyfully take a minor eternity, even on SSD if you have a huge database. Like... 450G. When you're done with ISINTEG, if you're really lucky you can have a bonus round of ESEUTIL followed by ISINTEG again if it turns out there was any minor database structural issues you didn't know about.

High I/O absolutely helps, but don't write this off as if massive database restores are trivial just because someone follows your advice. For businesses that are big enough to accrue huge amounts of data but not big enough to afford redundant servers, TIME is the cost they pay.

Re:HAHA

[dbIII]

However no jail time. Refusing to disclose a password in case it's used by such an incompetent carries jail time, but being deliberately criminally incompetent does not. It's a pretty nasty lesson we are teaching the next generation.

Don't be too hard on them

[dbIII]

MS Exchange is difficult to care for from what I've seen and the competence or otherwise of the people that look after it doesn't seem to spare such dramas from what I've seen. The experienced seem to run several MS Exchange servers (even in small places of 100 users where a 300MHz machine with Sendmail would do the job) that way the blowups and disasters may happen on one server but the mail still gets through on another.
It's a shambolic pile of services and applications loosely stuck together with gum, and there was no reliable way to get usable backup without stopping it (ie. the entire fucking thing to put on a new server and not just a portion of the mailboxes), until volume shadow copy came around - the MS Exchange programmers never supplied what every other MTA provided on first release!
You probably do need to be a hero to keep a single instance of it running.

Re:HAHA

[1s44c]

There's other gold in there too, like the time the guy pulled the cable on a production rack in order to create a catastrophe so he wouldn't have to travel to a business meeting, or his habit of remoting into IT infrastructure (Blackberry and Exchange servers were mentioned) on the weekends to fuck up their configuration, just so he could "magically" fix it on Monday morning.

He was, apparently, eventually fired.

Wha!??

What was this guy? The Harold Shipman of IT?

Re:HAHA

[mlush]

'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

The thing I'm really struggling with is why on Earth would anyone do such a thing

Re:Unions can be a big help in stopping BS like th

[DNS-and-BIND]

What about when the mafia who controls the unions comes around looking to get paid? What about when politicians looking for paybacks for favors granted to the union demand you employ 50 people who will collect paychecks and yet never show up for work?

OOHHH GOD!!

[higuita]

OOHHH GOD!!

WHY, but WHY people still use that exchange garbage!! With so many exchange replacements, so many webmails, so many SAAS alternatives... WHY!?

At least they pay the (heavy) price for it! (money, work hours, never ending troubles)

Re:OOHHH GOD!!

[Anonymous Coward]

The "webmail and alternatives" only satisfy some of the requirements big firms have. Also the supposedly heavy price paid for subscriptions with support is probably offset by the labor costs associated with in-house support of some of your SAAS alternatives.

Just because it's "free" doesn't mean it's low cost.

Re:OOHHH GOD!!

[Anonymous Coward]

Show me a complete exchange replacement that actually works.

Re:OOHHH GOD!!

[Anonymous Coward]

Sure, but first, show me an exchange installation that actually works.

Re:OOHHH GOD!!

[cusco]

Because even stupid people can make it work most of the time, and there are enough non-stupid people with the necessary training for the other times that competition keeps their price low.

Gate's moment of brilliance was really when he decided that Windows and Office didn't have to be 'perfect', it just had to be 'good enough'.

Re:OOHHH GOD!!

[Anonymous Coward]

God here,

its used because its simple, it works, it scales, its reliable (assuming you use reliable dedicated hardware/vm), you can do a ton of side-features with it, and its actually pretty cheap for a business product.

I'm a huge GNU fan, and I've seen nothing that is close in capability for the value.

Re:OOHHH GOD!!

[RevDisk]

Works well with AD, supports collaborative stuff, plays nicely with other MS products.

Don't look at me funny, I use postfix. I used to use qmail, for entertainment's sake, but the qmail dev thinks his product is 100% perfect and won't touch it. It doesn't natively support multi-domain email very well (aftermarket patch for that). So postfix. I screamed, hit my head on my desk and did get postfix installed. Granted, it was interesting experience. Much worse than installing Exchange into any AD environment. But postfix has worked without any real hiccups in years.

I hate maintaining Exchange, but it does pay well and the alternatives aren't there when it comes to MS integration. If they were, I'd make piles of cash replacing Exchange.

Re:OOHHH GOD!!

[higuita]

Just because you use outlook, doesnt mean that there arent any other email clients. Exchange just exists because people have MS Office installed and use outlook. Change the email client and you will see that exchange is expensive, redundant and hard to live with.

Outlook only works well with exchange, exchange only works well with outlook. There is a universe outside! :)

Re:OOHHH GOD!!

[higuita]

You have postpath, it's a exchange drop-in server that is not exchange (only low level exchange tools fail) ... sadly cisco acquired then and turn then in to SAAS... maybe openchange.org will someday reach the same level...

then you have many alternatives... but please forget outlook, as outlook only works well with exchange, exchange only works well with outlook and trying then to work well with other tools usually is a ticket for trouble-land

here are some alternatives:
atmail
kerio connect
clarkconnect
zimbra
axigen
SOGo
hyperoffice
communigate
citadel

all depends of what you need and what you know/have (resources, time knowledge, etc)

For most exchange people, zimbra and communigate are the first ones to try

Re:OOHHH GOD!!

[higuita]

see this post

All tools can today use the AD, at worst case just enable the unix template in the AD...
Of course tools from MS work best with other tools from MS... just like tools from Apple works best with other tools from Apple... that doesnt mean that other cant do that, you just have to define what you need and seek tools with that. And no, that tool to backup exchange mailstores will not play nicely with any exchange alternative... define services needs, not tools... if in the end exchange is the only option, so be it! And good luck! :)

Re:OOHHH GOD!!

[labnet]

I've used exchange in a small company (20 users) for 12 years (since SBS4.5), and can't even remember the last time I had a problem with it. It just works (tm)

Re:OOHHH GOD!!

[Bryansix]

Sure, when was the last time Office 365 went down? I rest my case.

Re:OOHHH GOD!!

[Bryansix]

None of them give you the full functionality of Outlook except Exchange. Now you can argue that you should use Lotus Notes instead but that's a whole different discussion for a different day. The point is there are only a few Enterprise level email solutions out there and Exchange is still one of the best. If you don't want to host it yourself then look to Office 365 or Intermedia.

Re:OOHHH GOD!!

[Cramer]

CommuniGate. Google. Depends on where you draw the line for "complete".

Re:OOHHH GOD!!

[Cramer]

Small office that places very little demand on it, I could see that. But you are a statistical anomaly on the very edge of the bell curve. (or you have very bad memory :-))

I've never seen a medium sized exchange (500-1000 users) system *not* have some sort of outage yearly, if not several times a year. And that's in companies that have multiple people who's entire job is to manage Exchange(tm). I seriously doubt those people are making work for themselves. (read: they don't need to.)

Re:OOHHH GOD!!

[PlusFiveTroll]

Even if each user had metric shittons of mail, Exchange, or any other mail server should handle 20 users without stressing even a weak server. Start getting hundreds of connections at the same time, that's when you see what a server is made out of.

Re:Unions can be a big help in stopping BS like th

[furytrader]

Come to Chicago sometime and you can see how helpful the unions are when it comes to running a business ... right out of Illinois.

RE:Lazy, stupid, or troll?

[Anonymous Coward]

Gonna, nothin... okay;
Madder, intensive porpoises, pwned... lame.

Re:HAHA

[leonardluen]

the summary says:

the act violated SEC rules because the routed information was not being logged.

are they sure he wasn't logging the data?

Jesus H. Christ

[Anonymous Coward]

I know it's hard for you to understand, but Exchange is a little bit more than an MTA.

As well as email and calendaring and resource sharing and telephony integration, Exchange also allows other functions. How about OTA smart phone synchronization and management. How about user management and seamless integration across domains, subdomains, continents...? There's also journaling and regulatory compliance, continuous replication, load balancing, redundancy and offsite automatic fail over. There's a lot more too that I can be bothered with right now.

The point is that anyone that compares Exchange with Sendmail or any other MTA obviously doesn't have an effing clue what Exchange is and is completely unqualified to have the discussion in the first place.

Re:Jesus H. Christ

[Anonymous Coward]

Yeah, he should've used Postfix.

Re:Jesus H. Christ

[dbIII]

You wrote:

I know it's hard for you to understand, but Exchange is a little bit more than an MTA.

after apparently never bothering to read:

It's a shambolic pile of services and applications loosely stuck together

Re:Jesus H. Christ

[Cramer]

True, Exchange(tm) can do a great many things. On the whole, most people don't need most of it. The more gears you try to keep spinning, the more places you have for a wrench to fall. Around here, not a month goes by that something doesn't happen to the 3rd party smart phone integration parts (iOS, Android, and Blackberry are INDEPENDANT connectors.) [Windows phones can natively talk MAPI.]

Re:Jesus H. Christ

[dbIII]

Yes, it was a large collection of 3rd party bits that handled stuff like fax to email that were the worst of the steaming mess the time I had the misfortune to have anything to do with MS Exchange. One braindead portion would not work if you changed the admin password on the MS Exchange server, which was a bit of a problem considering the circumstances the previous admins left under, and the situation where I was only going to be at the site for a few months and would have preferred all admin passwords to be changed by the permanant guy in case I ever ended up working for a direct competitior.
Running several servers for even a small site and automatic failover seemed to be necessary, and that's what I heard from people that actually thought the thing was good and had a lot of experience with it.
Anyway, my opinion is that it's very name is a warning to swap it with a different environment, but I'm used to collections of relatively old and well behaved software that doesn't keep all the mailboxes hidden in a fragile and fucking obfiscated database and keeps calendars etc in separate well behaved areas. Others like it.

Re:Unions can be a big help in stopping BS like th

[Joe_Dragon]

No the NON unions american airlines el salvador maintenance works did it.

Re:HAHA

[cusco]

He did it as a test to make sure that he understood his routing tables, and then forgot to go back and fix it. For a week.

Did they wear suits and ties?

[Anonymous Coward]

I bet every one of them that messed up here never wore suits and ties. I bet they wore polos and regular kacki pants, maybe even tee shirts and jeans. Remeber, this is the business world we are talking about here, you can't trust people not wearing a suit and tie.

Re:HAHA

[tibit]

Protip: the world is full of people who do stupid shit for apparently no rational reason at all. There.

Parent needs some mod points.

[mu51c10rd]

Perhaps one of the greatest comments ever seen regarding I.T. projects...

Re:Unions can be a big help in stopping BS like th

[T.E.D.]

Just look at what happened at American Airlines. Some maintenance worker loosened up a bunch of seats, and bingo within a week the Pilot's union has a new contract after over a year of negotiating. Some coincidence!

No the NON unions american airlines el salvador maintenance works did it.

Exactly. It was only after it happened *twice* that they sent everything to the union shop (right here in Tulsa) to get it fixed right. Then they settled with the union (and *still* shipped some more of their jobs to El Salvador, just not as many as they'd been trying to).

I'd really like to see the AC's story about the union NFL referees. The non-union refs are comically bad for weeks, then blow a game-changing call on Monday Night football, and bingo within a week the Referee's union has a new contract after over a year of negotiating. Some coincidence!

Re:HAHA

[Quirkz]

Seriously, this is the most insightful thing I've read on slashdot.

Re:HAHA

[V for Vendetta]

The thing I'm really struggling with is why on Earth would anyone do such a thing

As per TFA:

[...] and we found out that he'd sent the traffic home to ensure that his routing patterns at work were correct," Saccavino told InformationWeek in a recent interview. But after a week, Saccavino said, he'd forgotten to turn it off.

But given the rest of the story, I'm not sure if that's the only reason.

Re:Unions can be a big help in stopping BS like th

[cusco]

Never worked in a Union shop, have you? The difference is that in a Union shop you will get fired 'For Cause', rather than just because your boss doesn't like redheads or Asians. And in this case there was abundant cause.

Regulators?

[whitroth]

I, too, love that they outsourced their IT - they got what they apparently deserved.

But then there's the part in the article where it doesn't appear that before things came down that they'd *never* been audited.

Oh, that's right, most of this happened between '01 and '08, when Bush & Cheney were in charge, and All Republicans Love Deregulation, and if you can't deregulate, strangle the budget of the regulating agency so they can't do their job.

And before you libertarians here jump on me, tell me what you would have done if *you* had invested with them.

                      mark "that's right, you *ain't* rich, or you wouldn't be spending time reading comments on slashdot"

Re:Unions can be a big help in stopping BS like th

[Lithdren]

Not as big of a coincidence as you might think.

I live in Green Bay, and let me tell you, that night things went a little crazy. People on the radio were openly talking about Boycotts.

Nothing changes the mind of a group like the NFL faster, then the concept of lost profits.

Re:Unions can be a big help in stopping BS like th

[mcgrew]

On the contrary, union workers can be fired easily for what this guy is accused of.

Re:Unions can be a big help in stopping BS like th

[mcgrew]

What about when the mafia who controls the unions comes around looking to get paid?

The Teamsters aren't the only union in the US. They are, afaik, the only ones run by the mafia.

Re:HAHA

[Bryansix]

Actually with our backup solution you can get the Backup VM online in about an hour.

Re:Unions can be a big help in stopping BS like th

[mcgrew]

It's taxes that run businesses out of Illinois, not unions. Most businesses here are nonunion and have no problem... until they start fucking over their workers and the workers organize.

Re:HAHA

[plover]

'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

The thing I'm really struggling with is why on Earth would anyone do such a thing

It sounds like a case of Munchausen syndrome ...

[ puts on sunglasses ]

by proxy!

YEEEAAAAAAHHHHHHHH!!!!

Hard time reading? Train wreck :(

[dbIII]

Amusing - some clown that didn't bother to read a short post trying to bury me in a megabytes of text about service level agreements that mean little if they allow week long outages to occur due to quick to correct typos.
Maybe you should stop turning me into a strawman by pretending it was my problem instead of one that I became aware of when I couldn't get a job offer by email out to one of the students who had lost their email access for a week. Please have the decency to read short posts before making incorrect assumptions that you would not have made if you'd read a few short sentences

Re:Sabotage spoiler

[Anonymous Coward]

You've mischaracterized the situation in San Francsco. Terry Childs, the contract network engineer, never sabotaged anything. He just refused to reveal sensitive Cisco switch admin passwords to a cadre of incompetent bureaucrats on a telephone conference call with several unknown participants who were calling into the meeting from a remote office. He was completely correct, and was even willing to go to jail before compromising the security of the city's new fiber optic network that he had designed and configured. He did give the passwords to mayor Gavin Newsome, after explaining to the mayor how poorly the city's IT infrastructure was being managed. The mayor was the only person he felt he could trust, and he was correct. Even his former supervisor stood by him in this dispute, claiming that the city's management was too incompetent to be trusted with such sensitive information.

That supervisor resigned in disgust several months before Childs was arrested. This was during a widespread IT staff revolt that occurrred after the city hired a new Chief Information Security Officer who was completely unqualified for the position and landed the job only because she was the girlfriend of another powerful city administrator. Indeed, even when the managers had the admin passwords in hand, they still managed to screw things up royally, proving beyond a doubt that Childs was correct.

As slick as Mayor Gavin Newsome was, he still couldn't fix stupid.

Re:Unions can be a big help in stopping BS like th

[Anonymous Coward]

I work in a union shop (state government), and nobody ever gets fired, no matter how poorly they perform. The shop steward always grieves any personnel action taken "for cause" and it ends up going no where. The only staff turnover we ever see is among the few contractors we have on board, who are hired for their skills in areas that the union jobs don't cover (e.g., datacenter virtualization, audio-visual support, high-level .NET programming, database administration, BizTalk support, PeopleSoft support, etc.). There aren't many of those folks left, but there is a revolving door kept open them. The union jobs are secure until the employee dies or retires, whichever comes first. There aren't many young folks on our staff except for the contractors, who can keep their skills up to date by training and studying. There's no incentive for union employees to do that, since they are comfortable in their zone and don't welcome any new challenges.

I stand corrected!

[girlinatrainingbra]

Egads, you're right about much of this. I back-tracked through more of the /. history and found out a lot more about this SF network issue. It seems like it was definitely not sabotage; but perhaps a bit (a little bit?) controlling? Anyway, it was not as clear as I made it out to be. It's sad and bizarre that it was taken to criminal court and that he was in jail for such a long time. I did not read about the internal politics and power struggles before except about the CIO as the problem-child who the net-admin would not give the password to [damn dangling infinitive], and I haven't seen any reference about the "girlfriend of another powerful city administrator" anywhere.

.

The CIO as the source of the problem definitely parallels the Gunn-Allen problem, though, and that is the point I was trying to make, though it did not come across clearly as I had wished. Do you have a pointer for the political problems and the girlfriend of the administrator thing? (or are you very in the know and that's why you had to post anonymously?) (by the way, if you'd responded to my comment, I would've been messaged and I would have replied earlier. I think your response is at the same parallel level as my statement. Anyway, thanks for the comment. And in SF, a politically connected boyfriend could be as likely a problem as a politically connect girlfriend, eh? (sez I as a member of the girl gender) )

Re:HAHA

[mlush]

'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.

That's got to be the funniest thing I've ever read on /. Seriously, it sounds like something from an Onion story.

The thing I'm really struggling with is why on Earth would anyone do such a thing

It sounds like a case of Munchausen syndrome ...

[ puts on sunglasses ]

by proxy!

YEEEAAAAAAHHHHHHHH!!!!

That joke was bad and you should feel bad