Slashdot Mirror
Overwhelming Bureaucracy in the IT Department?
Nedry57 asks: "I am in the somewhat unique position of being a technology worker, who lives outside of the IT department in my company (a very large organization in the US). By far, the biggest challenge I face is getting anything done due to the bureaucracy that exists, within IT. There are certain tasks (i.e. anything that happens in the data centers) that I don't have the access to do. Even a simple task, like installing more memory in a non-production server, can take nine months and massive mountains of paperwork (no exaggeration), thus costing many times more than it should. The lack of agility is maddening, because I know we are missing significant business opportunities. My management is extremely supportive and despite our excellent track record of success in creating robust/secure applications--our work has passed audit numerous times with flying colors--we get no support from IT. Even senior management can't break through the barrier. I am very interested in hearing the experiences Slashdot readers have had in similar situations." How do you get your technology work done, when your IT department is more hindrance than help?
He and others in the IT department tried doggedly to get security noticed, only to be shot down by executive management. To paraphrase the CFO and strip out the gratutious profanity, "We're a meat company. We turn happy cows into happy steaks and happy pigs into happy bacon. We're not freaking NASA. We don't need to worry about our computers like Lockheed Martin does."
Several months later a virus hits the company and the phone system, which includes all sales offices, dies. I rush and get the tools to remove the virus in every hand possible.
Ultimately, as I was leaving the company, they finally hired a security manager. This was only because of Sarbanes-Oxley, and that person was given the role of a paper tiger--no authority to change things to be more secure, but a perfect picture for blame should something go awry.
When I left, I entered another office with other politics, but it is nowhere as bad as it was there.
--Chag
SBC corporate (now AT&T) is exactly like this. I was a contractor building an application in 3 months. IT said that it would take up to 12 months AFTER applying for a server in NEXT YEAR'S budget. That's right, it was going to take 16 months and several layers of approval. The VP of the entire division (only 1 person down from the CEO) couldn't bust through that red tape.
Now THAT was funny... 3 months later I had a working application sitting on a shared server, and I had to go. We had about 1 week's worth of data in there, but that was almost 100,000 rows in most tables.
Behold the glorious bragging rights
Some number of years ago, I found myself in charge of a private infrastructure. We had maybe 50 servers and 400 users exchanging sensitive information completely seperate from the main, public network.
Because of the percived importance of uptime on this network, everything required mountians of paperwork. Installing and removing nodes from the domain required three administrators, setting up a new machine required a month on a private VLAN being monitored by a sniffer, memory and hard drives were obselete before they got to the customer.
Anyone who ever worked around an UPS knows how they die. They give plenty of warning. Having an UPS fail is a rediculous way to lose your backbone infrastructure.
My predicessor had done a wonderful job of installing an UPS for every router and switch in the datacenter. Problem is, both power supplies in the routers and switches were connected to the same UPS. In cases where an UPS was about to fail, he unplugged the UPS from the wall and plugged it into, you guessed it, another UPS.
He didn't do it out of ineptitude; it was done because the only option was to clash heads with the IT overlords. They would require studies about how many UPSs failed and if it failed before the MTBF, they'd want us to try and recover money from the manufacturer. They'd want contractors to come in and examine the UPS to bid on a UPS monitor and replacement contract.
In short, asking the overlords was like asking to be turked by a syphalitic bear.
So, some BOFH, overwhelmed by the prospect of repairing the power system, chose another path. He walked over to a failing UPS and simply turned it off. He was the only one with the access to turn it back on, so he had no reason to worry.
Within two hours, all in-progress meetings were cancled. The Supreme Overlords demanded from on high that this lowly tech was to get a blank check and a blank trouble ticket (approved by the Supreme Overlords) to do whatever he needed to do to prevent that from ever happening agian.
Electricians installed two seperate power feeds into every rack.
Each power supply got a seperate UPS.
Old equipment was updated.
Everything was strawberry fields and unicorn giggles after that for the infrastructure department.
Now, to answer your question: You have something that someone wants. Hold it hostage till you get what you need.
I'd rather you do it wrong, than for me to have to do it at all.
Bolony.
The cost savings are barely %15 at the most and the Indian management companies take most of the cost savings away.
You need to spec requirements for any programming projects and you can't outsource business processing that far away. If anything efficiency eats in and costs actually go up.
There are a few companies that are %100 based here in the US where manufactoring, operations, and management are all in one location. Outsourcing to China will actually cost more because work wont flow seeminglessly or as easily with everything apart.
I wonder if this guy works for a government contractor or has the military as a customer? Such companies are required to do tons of checks and ballances and security.
http://saveie6.com/
The IT department exists to make sure they have regular, gainful employment. They do NOT exist to make your job easier, or anyone's for that matter, who does not have direct or closelly indirect firing power over them. There are mouths to feed, mortgages to pay, colleges funds to fund, retirement to dream about.
Cynical? Yes, but also very true. The above is the root of the issue. I'll put it in the terms that IT would:
ITs job is to keep the servers running, smoothly, with as little interruption to daily work as possible. As with any complex undertaking, different users have different priorities. CxOs come first. Period. Internal needs come next (see: "servers running, smoothly," above). High profile departments are next - marketing, sales, accounting. The last one is mostly because it comes under a CxO (F - you can choose what it stands for) who is intimitely involved with the month-to-month operation, and through which everyone gets their pay checks (including previously mentioned CxOs). Development is pretty far down, as you can see. You must understand - you don't bring cash into the organization (sales), nor do your efforts directly affect the price of company stock (marketing), both of which are of top importance to the CxOs.
That does not mean that you are not essential. But you are essential in a way that is ongoing - like the janitorial staff. If they lose development, things will slowly start to degrade, but it will be a while before there is a crisis. Either way, its an expensive mess to clean up, but if you throw some cash at it, you can bring things back to livable.
Now, lets look at the flip side. If IT goes down for a day, there will be hell to pay, and heads may roll. Every IT person knows this. Anyone who has dealt with complex modern systems knows that it's a house of cards. There are so many things that can go wrong. One failure, if not just costing your job, is certainly going to make for a long night getting things back in order. That would be uncompensated overtime, remember. Also, ten years without a single failure will not make you a hero, like landing a new sales client, or scoring a great marketing campaign which lifts the stock price or sales. It will make the company think you're reliable, but boring. Bonus aren't given out for boring. One failure, on the other hand, makes you a villain.
Now, if you've made it this far, how much value is there - for the IT professional - in helping you get your job done faster. In case you've skimmed, I'll tell you: none. It's like playing russian roulette for fun. Unless you just happen to like the life-or-death thrill, or have nothing to live for, it's a fools game.
I wish I had better news for you, but if you have a large corporation, than you have an ingrained corporate culture, and IT subculture. And they don't drift your way.
Oh, I've never been in IT. They piss me off 'cause I'm an engineer and just want to get shit done, and they want to worry about making sure the CEO's internet never goes down. I've learned over the years that, in effect, that is their job. I've stopped fighting them and learned to either (a) work with them or (b) work around them. The latter is done carefully to avoid stepping on toes. Just as they are under the thumb of uper management, they like to exert their power where they can. That would be against you and me. You don't tunnel under a mountain if there's a reasonable way to pass around it.
Is it just my observation, or are there way too many stupid people in the world?
I work at a top 25 law firm. The CFO and a few of his traveling crew from one of the largest cell phone companies in the US was using a few of our vacant offices. We recieved a request that they needed network connectivity and a network printer if possible. We had him up and running on our public vlan with internet access and a laserjet printer in about 10 minutes. He commented that at his facility, it would take about 6 months for something like that to happen.
HA! This is so true...and when it happens, it makes me sick.
I work at a large university. My start in IT began in a non-IT department, and I had to work with the IT people all the time. To them it was a game to try to stop any progress I wanted to make.
They would make me wait months just to add a column to a table in a database that only I used.
They took 2 years to 'investigate' moving from a flat table database (FoxPro) to a relational database (Visual FoxPro) but never migrated anything on the production server, because they were worried about incompatibilities. (FoxPro/Visual FoxPro were the only options they gave me)
I could list dozens of things- but their prevailing attitude was that I was an outsider, and only the IT group should be doing any IT work. I wouldn't have even started doing the work if they had been effective.
Well, now I've moved up, and I head a different programming department. The lessons I learned at my previous position have been serving me well. A little too well in fact- other people who have to deal that those other IT people are coming to me just to get a little server space...even from the other department.
I don't know, but I see IT (especially at a University) as a group that should facilitate others in doing their work- not hinder them.
Okay, so I'm bitching, but this stuff happens. And the sys admins get away with it because their boss doesn't understand what the job entails.
No reason to lie.
Most of the time O/S contracts are not negotiated by tech savvy people which results in ridiculous clauses.
The contract I'm working on at the moment only allows us to delay releases a certain number of times in a year and allows us a certain number of outages.
Fair enough you may think...
Now, if we're close to the limit on delayed releases but way ahead of the curve on actual outages what do you think we're going to do when we have to call go/no-go on a release with only a 50-50 chance of being successful? If we pull it we definitely get hit on the service level agreement; if we put it in we've got a 50% chance of taking no hit and a 50% chance of an outage which we can absorb easily. Is this the best thing for the customer? No. Is it the best thing to do pragmatically to protect the profits of the outsourcer? Yes.
Another outsourcer at my company is only contracted to create 30 (IIRC) user IDs per month. If you're new hire 31+ you're out of luck until the first of the next month & the company normally hires in big blocks (when the graduates become available). Somebody averaged the number of new users over 12 months without negotiating in the flexibility to overspend one month & underspend others. It can be created of course but that means big bucks... That outsourcer had used up all of their projected 5 years budget within the first two years with all of the incurred excess charges for stuff like that. Mind you they were SO incompetent that the failures in other areas of SLA incurred penalty clauses to partially counteract that...
I agree that entrenched IT departments can be really bad to have to deal with but they can be fixed if senior management has the will to do something - maybe the CEO needs to be told there's a problem instead of the usual "everything's fine".
If you have a LARGE IT department and you believe outsourcing is the answer - you probably asked the wrong question. Small-medium companies with limited and well defined requirements can and should outsource. I do not believe large IT departments can be economically outsourced because the increase in management overhead that is incurred more than outweighs any savings that may be made - you end up paying for the outsourcers managers while you have to keep your managers to liase with their managers... If you write a cast iron contract the outsourcer will have already charged you a shedload of money to negotiate said contract and you will have also spent a lot of money on your peoples time negotiating it. If you don't have a cast iron contract then you can open wide & say ARGH!!! because the outsourcer will ream you for every excess charge they can before you go bankrupt.
What are you listening to? (http://megamanic.blogetery.com/)
Wait, dude, this is slashdot, I don't think you get it. You're being all rational and balanced. I admit, I was trolling a bit. You were supposed to respond with irrational vehemence. :)
The right answer is somewhere in the middle, not a bureacracy expanding to meet the needs of the expanding bureacracy, and not departmental IT Lords all deploying their own solutions, Linux here, Windows there. But I so rarely hear about anyone finding that middle ground. I've seen a balance at big tech companies, but a balance of centralized and departmental IT expending 75% of their energy in a tug-of-war. The departments and divisions of a tech company can sometimes effectively fight the bureacracy because there's geeks in all corners who know what they're talking about. At a software company especially, the product teams rule, they know it, and they can fight about IT issues on even footing with the IT bureaucracy. In most other industries, the key departments don't have that advantage, so at the end of the day the IT folks make the IT choices, always making noises about collecting and meeting business requirements, but free to say "no" without much effective pushback.
My basic point was about human nature. Even if you create that balance, with a central IT plus dedicated IT staff across the organization, eventually the centralized guys win because their chief sits at the table with the other C*O's and exerts more pull, making effective noises about standardization lowering costs. It's simple corporate politics. If that CIO sees the big picture and has some humility, s/he might end up leading an organization that does the right things. More likely, even with that CIO, the IT middle management underneath will still play politics and make arbitrary rules and decisions that benefit themselves and disempower everyone else.
On another note, I never meant to suggest that a NAS from Best Buy was a good choice for any office needs. It's just that 6+ month turnaround on upgrades or new solutions is what drives people to route around that crap and starting using things like that NAS, or worse, Microsoft Access. I guess it's kind of a similar phenomenon to the adoption of the PC and M$ software in big businesses in the first place, to route around the mainframe cult.