Slashdot Mirror
Overwhelming Bureaucracy in the IT Department?
Nedry57 asks: "I am in the somewhat unique position of being a technology worker, who lives outside of the IT department in my company (a very large organization in the US). By far, the biggest challenge I face is getting anything done due to the bureaucracy that exists, within IT. There are certain tasks (i.e. anything that happens in the data centers) that I don't have the access to do. Even a simple task, like installing more memory in a non-production server, can take nine months and massive mountains of paperwork (no exaggeration), thus costing many times more than it should. The lack of agility is maddening, because I know we are missing significant business opportunities. My management is extremely supportive and despite our excellent track record of success in creating robust/secure applications--our work has passed audit numerous times with flying colors--we get no support from IT. Even senior management can't break through the barrier. I am very interested in hearing the experiences Slashdot readers have had in similar situations." How do you get your technology work done, when your IT department is more hindrance than help?
This is not an IT-specific problem: all functional areas in large organizations are vulnerable to this sort of bureaucratic barbed wire.
Even a simple task, like installing more memory in a non-production server, can take nine months and massive mountains of paperwork (no exaggeration), thus costing many times more than it should. The lack of agility is maddening, because I know we are missing significant business opportunities.
If you know that there are real costs associated with the lack of agility, you should a) document in detail the actual losses, b) present these figures calmly and respectfully, and c) gauge the reaction from senior management.
I want to drag this out as long as possible. Bring me my protractor.
Progress and stability are often conflicting goals. IT departments generally prefer stability, and that's why your deployments have probably been so stable and passed so many audits. Developers, of course, are charged with driving progress.
The real answer if you need flexibility with regards to "non-production stuff" is to not let IT have anything to do with it at all. Create a separate sub-net if you have to to keep the non-production machines off the IT network, and a firewall between your network and theirs to prevent any viruses, or other effects, from leaking from your net to theirs (this may require having to VPN through it just to work with these machines, c'est la vie). Keep the machines in a different room than the official server room. Maintain them all 100% yourself. Then do what you need to. Anything less and you're asking IT to aid in your development, a task they're probably not equipped to do while maintaining stability.
It's not uncommon for companies to have a "developer", "staging", and "live" system setup that are all completely independent, with some established mechanism and metrics to push products from one level to the next.
E pluribus unum
I agree, let market economics do its work. Any outsourcing partner will be more than happy to upgrade your server in a matter of days. Of course outsourcing does land you with a whole new set of interesting problems (cost control!) but the net effect is positive on the whole. Flame me if you will, but there is a reason outsourcing is so popular with managers... most of the time you get a more responsive IT department for less money.
This sig is just as redundant as the rest of this posting
"Our SAMBA connection is broken. Something changed over the weekend."
"Nothing changed over the weekend."
"You sure about it? Why does the AD server report it's running Server 2003 now?"
"Oh that? We tried to implement Windows Server 2003 to replace our AD server, but we backed it out."
*boggle*
That conversation was with our IT dept. In any controlled environment, things should be thought out, documented and multiple sanity checks performed. Even a dev system can impact a production system if they run on the same segment.
Now, having said that, our IT dept tends to mindlessly enforce rules without thinking about them and getting them to wake up to new technologies (e.g., SOAP, web apps) is like trying to bring around a corpse with smelling salts.
A good IT department should make sure things happen in a controlled and documented way, but should also make it as painless as possible to follow the rules. They should be proactive so if you come to them with something new you want to implement. Not only will they know what you're talking about, but have already prepared a white paper of preferred architecture for performance & security.
A really good IT department brings something to the table.
The bitter lessons of a veteran coder: http://bitterprogrammer.blogspot.com
First off, 9 months seems excessive. Very little should take longer than a business quarter.
..don't assume the issue is on the IT side I guess is the gist of it.
However, in my experience every person outside of IT and security groups has this mindset that IT groups hinder them for no real reason.
I do not doubt there is bureacracy that slows every company's process. However, the fact that you want a change made to one system now doesn't change that these IT people are responsible for the effects any change might have on an entire organization. I don't know how many times I hear "But all I want is X". And that person requesting 'X' doesn't realize that 'X' has these 3 possible security issues associated with it. Maybe it won't effect his server even if it is exploited, but that risk has to be evaluated, approved and lord knows what else.
The fact is, every change *must* go through a certain amount of bureacracy to make sure all that it could effect have taken the appropriate level of responsibility.
My best advice is work through your own internal processes to see if turnaround time can be expedited. Maybe all they need is a motivated developer type with your skills to assist in making their change control system better. Or maybe there are things you don't see. Don't assume IT folk are just pushing your stuff back because they don't like you (though that could be a factor). If you can get a 'champion' type in your IT group that can help you get your stuff moved through the most efficiently.
But in the end, it is not up to you to decide what priority your request is given over someone else's. Even a simple request should be evaluated properly and must be given priority that is likely outside the IT drone's choice... Maybe your manager/director type needs to champion your projects to get them pushed through with greater priority .
Oh, and Bill said he didn't wanna give you your ram because you ate his pudding cup.
Sure -- if I can read between the lines of what you seem to be saying -- the chicks might not care if you're good at your work or not. But some of those mere "other men" you mention might also happen to sign your paychecks.
The guy was complaining that his company is missing significant business opportunities. Translation: The company is missing significant business opportunities that he could have been instrumental in acting upon. But he can't, because of IT bureaucracy.
OK, so it's not his fault -- but do you think that's going to matter next time he goes in for a raise or a promotion? They'll want to see all the forward-thinking plans he's executed on, and he's going to have nothing, because trying to do anything is like wading through mud.
Even worse, what happens when it's time to a round of layoffs? What justification will he have to keep his job then?
Maybe it's easy for you to just sit there and be grateful you have a job. If it is, it's probably because you've only had one or two entry-level jobs. For people who have had a job for a number of years, however, just having a job no longer seems like Goal #1. Those people start to have other ambitions -- like buying a house, for instance, or a new car, or providing for their families. Maybe you've put yourself through college. Have you put anybody else through college lately? Dads sometimes like to do those kinds of things. They're hard to do when you've spent the last five or ten years sitting at the same desk in the basement, just spinning your wheels.
Breakfast served all day!
OK, by this point I have read TFA at least a few times and I still didn't find what was the good fight this guy fought. I mean he doesn't list any steps that he has taken to fight the IT department. He and his management are unhappy with the way IT department works. So just for argument sake, can I assume that you are making assumptions that may not be valid and drawing conclusions that are plain wrong?
I am not denying that this situations don't exist, but most people just whine about it, they don't do anything about it. For e.g. has this guy filed a formal written complaint to the upper management stating that the IT department is not co-operating? Has he tried forging some good rapport with the IT department? The only time any one remembers the IT department is when stuff don't work. Sometimes acknowledging that they are part of the company and their success may lead them to co-operate more. To support my argument read what the author states:
The lack of agility is maddening, because I know we are missing significant business opportunities. My management is extremely supportive and despite our excellent track record of success in creating robust/secure applications--our work has passed audit numerous times with flying colors--we get no support from IT.So apparently according to him all the bad things that are happening in the company is due to the incompetence of IT and all the good things are happening because of his development team. Gimme a break!! That attitude (treating IT department like they are 3rd rate employees, a burden) is not going to get him or any body favors.
Suggestion to author: Try toning down your ego, treat IT department with respect, give them credit and appreciate their work. They are the ones who save your ass when you type "rm -rf /". And ocassionaly buy them beer and lunch and see those 9 months turn to 9 seconds!
Most of the Pharma data centers i've dealt with in the last 5 years are locked down VERY tight. They have to deal with 21 cfr part 11, Hippa, SOX and a list of others issues and as such sound EXACTLY like the situation you describe. Example, in order to upgrade from MSSQL 7.0 to MSSQL2000 on a Pharma house manufacturing server requires changes to the following: Changes to the original Validation Plan Detailed Design Specs Functional Requirements Specs IP's - Installation protocols OP's - Operational protocols QP's - Qualification protocols ALL of the above operations require testing-testing-testing, a multitude of meetings and of course approvals from God all along the way. Then QA-IT-Engineering has to oversee everything. Its a very cumbersome, expensive process. This is for your-our own good. I have seen manufacturing data corrupted if this process is not followed exactly. Remember this the next time you think about the consistancy of those pills you take.
*--- Sometimes a majority only means that all the fools are on the same side. ---*
Interesting comments...
I've been in IT for close to twenty years in a couple small startups to some multi-nationals and in my own consulting business. One thing that lots of IT folks lose sight of is that IT is first a support organization within the larger organization. If the larger organization is sufficiently forward thinking, then they can (arrg, PHB-speak) *leverage* IT to be more competitive. But IT folks still have to make sure the website is up, the file server is accessible, users can login, etc., *before* you start thinking about the add-ons.
If the business doesn't want to spend money on the servers, then document what the consequences and benefits are for their decision. Don't just write that they'll have slower machines, but play Devil's Advocate and write up the business case for not adding memory.
Or, figure some way to optimize your resources so that less memory is required. This can be as simple as turning off services, or as complex as setting memory and processor caps within the virtual partition. And if you've tried all these and you're just short of memory, let them know.
In my consulting business my first goal is to keep my customers' infrastructure running. Next is to save them money versus some other consultant. Sometimes they need to spend money up front to save more down the road. Let them know if this is the case.
Here's what you do: take a simple task like adding RAM to a non-production server, and go through the entire, exhausting process in letter-perfect fashion, meeting every paperwork, audit, and permission requirement. Along the way, document every minute you spend on the process, showing exactly what you're doing, how long it took in minutes, and what requirement you were meeting. At the end, create a spreadsheet showing in careful detail that adding a $500 SIMM actually cost the company $5,000 in processes.
That spreadsheet becomes the club with which your managers and directors can beat the IT department because they're effectively offloading cost onto you at a rate of 1,000%.
Anyone who loves or hates any language, platform, or manufacturer, doesn't know what they're talking about.
You have quite a defeatist attitude. Contrary to Slashdot common belief, responding to challenges by quitting your job doesn't solve any problems, unless your challenge is to find a job where you don't have to think or do any real work. (In that case, you should probably go join this company's bureaucratic IT group -- they seem to have that area covered.)
In any case, the best way to get interdepartmental problems fixed is by providing rewards to both sides for working together. Short of that, you can start your own IT group or work with an outside company to get your solutions hosted. Your IT group should be a resource for you -- if they're not, you should be able to use other resources instead.
One of the managers I'm currently reporting to used to run into a similar problem at his last company. He's a 2nd level manager, and he decided that he would pay the salaries of a few people in the IT group in exchange for them specifically working on projects for his team. It worked great, and they were able to push out new releases every couple months. Before he started the arrangement, releases were taking 9 months.
In conclusion, you should give financial incentives to the other team to reach your goals: Whether it's through paying their salaries, or taking away their work by going with someone else. Unless they have an incentive to work with you, they probably won't.
my blog
There are certain tasks (i.e. anything that happens in the data centers) that I don't have the access to do. Even a simple task, like installing more memory in a non-production server, can take nine months and massive mountains of paperwork (no exaggeration)
So, let me get this straight...
User is frustrated because request to make standard servers non-standard with a custom request in a datacenter requires paperwork and time. User is upset because IT has formal procedures for change control, service level agreements, and standard hardware configs. User doesn't get ram upgrade and posts rant to Slashdot.
User is technical, probably dual boots their desktop to non-supported OS, probably hacks computer stuff at home, probably very smart and capable of supporting five or six computers by him/herself.
IT department probably supports 1000+ machines, and that number has doubled in the last year or so while staffing has been cut.
IT department probably has 200 servers per admin and only maintans this ratio by with consistent server deployments that maintain standard configurations.
A good IT organization understands the company's business.
A good technology worker needs to learn to work with IT to get what they need. You would probably be able to request and justify 10 servers and get them in the time it takes to get a one-off upgrade
The lack of agility is maddening, because I know we are missing significant business opportunities.
Lack of planning on your part does not create an emergency on my part.
Learn how your IT organization works. Work with it.
we get no support from IT. Even senior management can't break through the barrier
Perhaps you don't see the big picture. Perhaps you don't see the corporate IT budget and where you/your team/your project is on the priority list for that budget.
I'm sure there are all sorts of IT departments, but the *good* ones understand the core business and know what's important to the company's bottom line. If your IT department doesn't understand that, then I'm afraid you're going to have to become the IT liason and teach them. Provide them with your requirements well in advance so that they can plan proper deployments. Work together so that you can understand the pain points of IT, and they can understand your hardware/support requirements and the *value* that this will provide to the company.
...or just fire all of the upper and middle management in the narcissistic IT bureaucracy (you might have to barge in on them while they're jerking off to their org charts), and reassign all of the actual skilled IT staff and direct managers to the divisions of the company that they're supposed to be serving. Any arguments about efficiencies of scale are bullshit territory-marking, you can replicate much of that by centralizing procurement and licensing (but not budgets or purchasing authority!). Even if you lose a bit of efficiency, you more than make it up overall by greatly empowering divisions and departments. Costs plummet and productivity skyrockets when functional areas of the business get (only) the information systems they need, instead of forcing enterprise-wide adoption of the same adequate-for-everyone-but-powerful-for-no-one systems, or adding the same immense operational costs to every server when 90% of them need little security and no redundancy. If my creative team needs some more file-sharing space, is the business better served by me going out to Best Buy (ick) and getting a $400 NAS that I can hook up in 15 minutes (and takes my departmental IT guy 5 minutes to include in backups), or waiting 9 months for a $30K/year file server to be deployed to some server room in another time zone?
If you're building an assembly line, do you give everyone a hammer just because it's cheaper than buying different kinds of tools? Most Fortune 500 CIOs would.
When corporate information systems need to be integrated across business units or divisions, then build a development team for that, and have it report to the COO or CFO or someone else who can lean on upper management, rather than just making one centralized self-centered priesthood that controls everyone's systems top to bottom. I'm baffled that anyone can imagine how that could ever work well. That delusion requires a deep ignorance of human nature.
In a well-led enterprise, only a few of the business functions are really important, because they're central to the strategy of the business. Internal IT is never one of those functions. Yes, everyone depends on it, but IT is not really an "it". All employees have similar requirements for air conditioning and paychecks and parking lots and health plans, but IT requirements vary tremendously. Meeting those requirements is hard, and getting hard things done in a corporation requires incentive and accountability. Centralized IT has neither of those, so they say "no" instead of "yes".
Beside which, what better way of making things better for those who stay? If management loses good people because of some problem, they just might address that problem.
Suggestion to author: Try toning down your ego, treat IT department with respect, give them credit and appreciate their work. They are the ones who save your ass when you type "rm -rf /". And ocassionaly buy them beer and lunch and see those 9 months turn to 9 seconds!
A professional turns around a job in the same amount of time, regardless of his opinion of the other person. Sounds like you're saying the IT department there at best isn't very professional.
If the IT department is having a problem with the author, then they should be bringing it up with his supervisor. I have asshat coworkers as well. I bitch about them mightily, but I don't refuse to do my job just to spite them. Then I'm in the wrong and have no room to complain.
There are some exceptions, but they'll charge you more money for the flexibility. That's the other way outsourcers make you money - precisely defining the scope of work and charging higher prices for anything outside of it. Sometimes that's a Great Thing - outsiders who want to charge money are often much more willing to do what you want than insiders whose reward structure is that they're a Cost Center incentivized to cut costs. But the kinds of bean-counters who outsourced your IT department on you are usually going to prevent you from getting the extra-value services if they can.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Perhaps you could consider alternative experiences being possible, despite your own, before using such definitive language like "obviously you haven't," etc, etc. Obviously I have, and the experience you describe is not even remotely the one I've had. I've experienced outsourced development as less productive than on-site team members, but overall an eager, capable, and invaluable source of talent when used as a supplementary source.
I think I'll stop here.
Of course, the cost per GB of many managed SANs ends up running into the $100-200 range (per GB!). This is what leads to 50MB quotas on employee home directories. No, a well-managed SAN doesn't need to cost that much, but that is how much they cost when you buy one from the outfit that will take the CIO out to some nice dinners...
I've found there is a cycle:
1. Company has ancient central IT group populated by dinosaurs and BOFHs. Centrally run servers are underpowered and undermanned.
2. Individual business areas start putting together their own rogue IT groups, and servers spring up all over the company in every closet. The new groups are agile, but not as secure, and the process is not 100% efficient.
3. CIO centralizes the IT groups so that infrastructre is centralized and well-maintained. Costs plummet and service actually increases since the datacenter can be 24x7 monitored unlike the closet down the hall.
4. CIO discovers he can cut his budget 10% without much loss of quality. CIO gets bonus.
5. Repeat step 4 20 times.
6. Goto step 1...
If people stopped at step 3 with the right combination of central infrastructure and business-focused developers/support then you'd probably hit the sweet spot in terms of the best services and cost.
It drives me nuts to see companies that make billions of dollars a year shaving $1000 costs by undersizing server hardware, insufficiently supporting core applications, etc. IT is a force multiplier - every dollar spent in the IT group can enable all kinds of money-making in the business areas - as long as the IT spending goes to the right places. Nickeling and diming IT is like saving money by getting rid of phones at every desk...