Overwhelming Bureaucracy in the IT Department?

Nedry57 asks: "I am in the somewhat unique position of being a technology worker, who lives outside of the IT department in my company (a very large organization in the US). By far, the biggest challenge I face is getting anything done due to the bureaucracy that exists, within IT. There are certain tasks (i.e. anything that happens in the data centers) that I don't have the access to do. Even a simple task, like installing more memory in a non-production server, can take nine months and massive mountains of paperwork (no exaggeration), thus costing many times more than it should. The lack of agility is maddening, because I know we are missing significant business opportunities. My management is extremely supportive and despite our excellent track record of success in creating robust/secure applications--our work has passed audit numerous times with flying colors--we get no support from IT. Even senior management can't break through the barrier. I am very interested in hearing the experiences Slashdot readers have had in similar situations." How do you get your technology work done, when your IT department is more hindrance than help?

Comments

IT

[Anonymous Coward]

You don't. You fire them and outsource their jobs to India.

Re:IT

[Fjan11]

I agree, let market economics do its work. Any outsourcing partner will be more than happy to upgrade your server in a matter of days. Of course outsourcing does land you with a whole new set of interesting problems (cost control!) but the net effect is positive on the whole. Flame me if you will, but there is a reason outsourcing is so popular with managers... most of the time you get a more responsive IT department for less money.

Re:IT

[Billly Gates]

Bolony.

The cost savings are barely %15 at the most and the Indian management companies take most of the cost savings away.

You need to spec requirements for any programming projects and you can't outsource business processing that far away. If anything efficiency eats in and costs actually go up.

There are a few companies that are %100 based here in the US where manufactoring, operations, and management are all in one location. Outsourcing to China will actually cost more because work wont flow seeminglessly or as easily with everything apart.

I wonder if this guy works for a government contractor or has the military as a customer? Such companies are required to do tons of checks and ballances and security.

Re:IT

[Jelloman]

...or just fire all of the upper and middle management in the narcissistic IT bureaucracy (you might have to barge in on them while they're jerking off to their org charts), and reassign all of the actual skilled IT staff and direct managers to the divisions of the company that they're supposed to be serving. Any arguments about efficiencies of scale are bullshit territory-marking, you can replicate much of that by centralizing procurement and licensing (but not budgets or purchasing authority!). Even if you lose a bit of efficiency, you more than make it up overall by greatly empowering divisions and departments. Costs plummet and productivity skyrockets when functional areas of the business get (only) the information systems they need, instead of forcing enterprise-wide adoption of the same adequate-for-everyone-but-powerful-for-no-one systems, or adding the same immense operational costs to every server when 90% of them need little security and no redundancy. If my creative team needs some more file-sharing space, is the business better served by me going out to Best Buy (ick) and getting a $400 NAS that I can hook up in 15 minutes (and takes my departmental IT guy 5 minutes to include in backups), or waiting 9 months for a $30K/year file server to be deployed to some server room in another time zone?

If you're building an assembly line, do you give everyone a hammer just because it's cheaper than buying different kinds of tools? Most Fortune 500 CIOs would.

When corporate information systems need to be integrated across business units or divisions, then build a development team for that, and have it report to the COO or CFO or someone else who can lean on upper management, rather than just making one centralized self-centered priesthood that controls everyone's systems top to bottom. I'm baffled that anyone can imagine how that could ever work well. That delusion requires a deep ignorance of human nature.

In a well-led enterprise, only a few of the business functions are really important, because they're central to the strategy of the business. Internal IT is never one of those functions. Yes, everyone depends on it, but IT is not really an "it". All employees have similar requirements for air conditioning and paychecks and parking lots and health plans, but IT requirements vary tremendously. Meeting those requirements is hard, and getting hard things done in a corporation requires incentive and accountability. Centralized IT has neither of those, so they say "no" instead of "yes".

Re:IT

[Rich0]

Of course, the cost per GB of many managed SANs ends up running into the $100-200 range (per GB!). This is what leads to 50MB quotas on employee home directories. No, a well-managed SAN doesn't need to cost that much, but that is how much they cost when you buy one from the outfit that will take the CIO out to some nice dinners...

I've found there is a cycle:

1. Company has ancient central IT group populated by dinosaurs and BOFHs. Centrally run servers are underpowered and undermanned.

2. Individual business areas start putting together their own rogue IT groups, and servers spring up all over the company in every closet. The new groups are agile, but not as secure, and the process is not 100% efficient.

3. CIO centralizes the IT groups so that infrastructre is centralized and well-maintained. Costs plummet and service actually increases since the datacenter can be 24x7 monitored unlike the closet down the hall.

4. CIO discovers he can cut his budget 10% without much loss of quality. CIO gets bonus.

5. Repeat step 4 20 times.

6. Goto step 1...

If people stopped at step 3 with the right combination of central infrastructure and business-focused developers/support then you'd probably hit the sweet spot in terms of the best services and cost.

It drives me nuts to see companies that make billions of dollars a year shaving $1000 costs by undersizing server hardware, insufficiently supporting core applications, etc. IT is a force multiplier - every dollar spent in the IT group can enable all kinds of money-making in the business areas - as long as the IT spending goes to the right places. Nickeling and diming IT is like saving money by getting rid of phones at every desk...

Not an IT-specific problem

[sczimme]

This is not an IT-specific problem: all functional areas in large organizations are vulnerable to this sort of bureaucratic barbed wire.

Even a simple task, like installing more memory in a non-production server, can take nine months and massive mountains of paperwork (no exaggeration), thus costing many times more than it should. The lack of agility is maddening, because I know we are missing significant business opportunities.

If you know that there are real costs associated with the lack of agility, you should a) document in detail the actual losses, b) present these figures calmly and respectfully, and c) gauge the reaction from senior management.

Conflicting Goals

[samkass]

Progress and stability are often conflicting goals. IT departments generally prefer stability, and that's why your deployments have probably been so stable and passed so many audits. Developers, of course, are charged with driving progress.

The real answer if you need flexibility with regards to "non-production stuff" is to not let IT have anything to do with it at all. Create a separate sub-net if you have to to keep the non-production machines off the IT network, and a firewall between your network and theirs to prevent any viruses, or other effects, from leaking from your net to theirs (this may require having to VPN through it just to work with these machines, c'est la vie). Keep the machines in a different room than the official server room. Maintain them all 100% yourself. Then do what you need to. Anything less and you're asking IT to aid in your development, a task they're probably not equipped to do while maintaining stability.

It's not uncommon for companies to have a "developer", "staging", and "live" system setup that are all completely independent, with some established mechanism and metrics to push products from one level to the next.

Re:Conflicting Goals

[Xzzy]

Create a separate sub-net if you have to to keep the non-production machines off the IT network, and a firewall between your network and theirs to prevent any viruses, or other effects, from leaking from your net to theirs

Just take special care to educate everyone using the private network that it's not supported by the IT department, and questions regarding such are likely to be met with quite a bit of hostility. I work on the other side of the fence from the story submitter, and the general feeling is that even the technologiclly minded developers don't know diddly about maintaining a stable server. People are generally encouraged to set up their own work environment, but as soon as root access is given out it's made clear that it is no longer our (that is, IT's) problem.

More importantly, after a couple years of running a private network, never ever consider passing off the burden of maintaining the rickety development system that is suddenly 24x7 critical to IT. Those kinds of moves are exactly the kind that destroy IT's willingness to accomodate user requests.

you need more meetings

[juan2074]

Management can keep holding long meetings to find out why work is not getting done.

We guard you while you sleep.

[Anonymous Coward]

Look, the people you are after are the people you depend on. We install your memory, we code your apps. We run the internets, we guard you while you sleep. Do not... fuck with us.

Dear IT Professional:

[path_man]

Dear IT Professional:

Please don't change anything about the way your IT organization does business. We love the way you and your team fail to communicate; the way mindless mandates from on-high drive pointless initatives; the way the latest technology trend shifts focus from project to project like the attention span of a two-year-old.

Especially don't pay any attention to streamlining the use of hardware and software investments that you've already made. You and your team need MORE MORE MORE to get this project wrapped up on time. Have you upgraded to the newest rev of our software? Can't you just taste the new-and-improved speed of our lastest hardware?

In summary, we love the way your IT organization is today, and wouldn't change a single thing.

Yours Truly, Your software & hardware vendors

"We're Not Freaking NASA"

[Chagatai]

I worked for a meat producer, with a staff of 60 IT folks for a company of 20,000. At the time, I was a real security nut and wanted to improve the company as much as possible. I was there for about a month when I spoke with one of the IT directors about the company's security policy. His response? "There is no security policy."

He and others in the IT department tried doggedly to get security noticed, only to be shot down by executive management. To paraphrase the CFO and strip out the gratutious profanity, "We're a meat company. We turn happy cows into happy steaks and happy pigs into happy bacon. We're not freaking NASA. We don't need to worry about our computers like Lockheed Martin does."

Several months later a virus hits the company and the phone system, which includes all sales offices, dies. I rush and get the tools to remove the virus in every hand possible.

Ultimately, as I was leaving the company, they finally hired a security manager. This was only because of Sarbanes-Oxley, and that person was given the role of a paper tiger--no authority to change things to be more secure, but a perfect picture for blame should something go awry.

When I left, I entered another office with other politics, but it is nowhere as bad as it was there.

Re:deal with it

[PCM2]

be grateful you have a job. it's obvious there is nothing you can do about it, so why are you sweating it? go with the flow and live a less-stressed existence. it's not worth creating ripples. the only people who judge you for your work aptitude are you and other men; no one else cares.

Well all right! Way to spend your life being a doormat.

Sure -- if I can read between the lines of what you seem to be saying -- the chicks might not care if you're good at your work or not. But some of those mere "other men" you mention might also happen to sign your paychecks.

The guy was complaining that his company is missing significant business opportunities. Translation: The company is missing significant business opportunities that he could have been instrumental in acting upon. But he can't, because of IT bureaucracy.

OK, so it's not his fault -- but do you think that's going to matter next time he goes in for a raise or a promotion? They'll want to see all the forward-thinking plans he's executed on, and he's going to have nothing, because trying to do anything is like wading through mud.

Even worse, what happens when it's time to a round of layoffs? What justification will he have to keep his job then?

Maybe it's easy for you to just sit there and be grateful you have a job. If it is, it's probably because you've only had one or two entry-level jobs. For people who have had a job for a number of years, however, just having a job no longer seems like Goal #1. Those people start to have other ambitions -- like buying a house, for instance, or a new car, or providing for their families. Maybe you've put yourself through college. Have you put anybody else through college lately? Dads sometimes like to do those kinds of things. They're hard to do when you've spent the last five or ten years sitting at the same desk in the basement, just spinning your wheels.

Make It Happen

[Bios_Hakr]

Some number of years ago, I found myself in charge of a private infrastructure. We had maybe 50 servers and 400 users exchanging sensitive information completely seperate from the main, public network.

Because of the percived importance of uptime on this network, everything required mountians of paperwork. Installing and removing nodes from the domain required three administrators, setting up a new machine required a month on a private VLAN being monitored by a sniffer, memory and hard drives were obselete before they got to the customer.

Anyone who ever worked around an UPS knows how they die. They give plenty of warning. Having an UPS fail is a rediculous way to lose your backbone infrastructure.

My predicessor had done a wonderful job of installing an UPS for every router and switch in the datacenter. Problem is, both power supplies in the routers and switches were connected to the same UPS. In cases where an UPS was about to fail, he unplugged the UPS from the wall and plugged it into, you guessed it, another UPS.

He didn't do it out of ineptitude; it was done because the only option was to clash heads with the IT overlords. They would require studies about how many UPSs failed and if it failed before the MTBF, they'd want us to try and recover money from the manufacturer. They'd want contractors to come in and examine the UPS to bid on a UPS monitor and replacement contract.

In short, asking the overlords was like asking to be turked by a syphalitic bear.

So, some BOFH, overwhelmed by the prospect of repairing the power system, chose another path. He walked over to a failing UPS and simply turned it off. He was the only one with the access to turn it back on, so he had no reason to worry.

Within two hours, all in-progress meetings were cancled. The Supreme Overlords demanded from on high that this lowly tech was to get a blank check and a blank trouble ticket (approved by the Supreme Overlords) to do whatever he needed to do to prevent that from ever happening agian.

Electricians installed two seperate power feeds into every rack.

Each power supply got a seperate UPS.

Old equipment was updated.

Everything was strawberry fields and unicorn giggles after that for the infrastructure department.

Now, to answer your question: You have something that someone wants. Hold it hostage till you get what you need.

You've missed the entire point

[Overzeetop]

The IT department exists to make sure they have regular, gainful employment. They do NOT exist to make your job easier, or anyone's for that matter, who does not have direct or closelly indirect firing power over them. There are mouths to feed, mortgages to pay, colleges funds to fund, retirement to dream about.

Cynical? Yes, but also very true. The above is the root of the issue. I'll put it in the terms that IT would:

ITs job is to keep the servers running, smoothly, with as little interruption to daily work as possible. As with any complex undertaking, different users have different priorities. CxOs come first. Period. Internal needs come next (see: "servers running, smoothly," above). High profile departments are next - marketing, sales, accounting. The last one is mostly because it comes under a CxO (F - you can choose what it stands for) who is intimitely involved with the month-to-month operation, and through which everyone gets their pay checks (including previously mentioned CxOs). Development is pretty far down, as you can see. You must understand - you don't bring cash into the organization (sales), nor do your efforts directly affect the price of company stock (marketing), both of which are of top importance to the CxOs.

That does not mean that you are not essential. But you are essential in a way that is ongoing - like the janitorial staff. If they lose development, things will slowly start to degrade, but it will be a while before there is a crisis. Either way, its an expensive mess to clean up, but if you throw some cash at it, you can bring things back to livable.

Now, lets look at the flip side. If IT goes down for a day, there will be hell to pay, and heads may roll. Every IT person knows this. Anyone who has dealt with complex modern systems knows that it's a house of cards. There are so many things that can go wrong. One failure, if not just costing your job, is certainly going to make for a long night getting things back in order. That would be uncompensated overtime, remember. Also, ten years without a single failure will not make you a hero, like landing a new sales client, or scoring a great marketing campaign which lifts the stock price or sales. It will make the company think you're reliable, but boring. Bonus aren't given out for boring. One failure, on the other hand, makes you a villain.

Now, if you've made it this far, how much value is there - for the IT professional - in helping you get your job done faster. In case you've skimmed, I'll tell you: none. It's like playing russian roulette for fun. Unless you just happen to like the life-or-death thrill, or have nothing to live for, it's a fools game.

I wish I had better news for you, but if you have a large corporation, than you have an ingrained corporate culture, and IT subculture. And they don't drift your way.

Oh, I've never been in IT. They piss me off 'cause I'm an engineer and just want to get shit done, and they want to worry about making sure the CEO's internet never goes down. I've learned over the years that, in effect, that is their job. I've stopped fighting them and learned to either (a) work with them or (b) work around them. The latter is done carefully to avoid stepping on toes. Just as they are under the thumb of uper management, they like to exert their power where they can. That would be against you and me. You don't tunnel under a mountain if there's a reasonable way to pass around it.

Dealing with IT.

[Rufty]

I used to work in a *very* bureaucrat infested research lab. This is the place that firewalled "new" ssh but let the "known protocol" telnet, out... Friend of mine was running very numerically intensive spectral analysis/matching on samples. Bung in sample. Get data. Process for about 8hr. So, do last thing of the day and you've got the results next morning. Until, in the interests of a uniform computing experience *all* boxes were required to have the same basic setup and were bolted down tight. This included everything. Including the screensaver that seamlessly blended from slide to slide of the company's publicity shots. Bingo! 100% CPU when the screensaver kicks in and the analysis runs can no longer work unattended. Bummer! OK so my friend takes an old mouse, a clamp stand, a magnetic stirrer and flea, and some epoxy. Glue magnetic flea to mouse ball. Clamp mouse over stirrer. Stirrer agitates mouse. Screensaver never gets to run. Once again work can happen!

Re:Recognize those things you cannot change....

[Zoop]

Suggestion to author: Try toning down your ego, treat IT department with respect, give them credit and appreciate their work. They are the ones who save your ass when you type "rm -rf /". And ocassionaly buy them beer and lunch and see those 9 months turn to 9 seconds!

A professional turns around a job in the same amount of time, regardless of his opinion of the other person. Sounds like you're saying the IT department there at best isn't very professional.

If the IT department is having a problem with the author, then they should be bringing it up with his supervisor. I have asshat coworkers as well. I bitch about them mightily, but I don't refuse to do my job just to spite them. Then I'm in the wrong and have no room to complain.