Half of All Data Centers Understaffed

alphadogg writes "Fifty percent of IT executives say their data centers are understaffed, and companies are still looking for more ways to cut costs, according to Symantec's latest 'State of the Data Center' report. Sixteen percent of survey respondents said their data centers are extremely understaffed, and another 34% called their data centers somewhat understaffed. At the same time, data centers are becoming more complex and harder to manage, with more applications, data and increasingly demanding service-level agreements. 'Data center complexity has led to a lot of staffing challenges,' says Sean Derrington, director of storage management and high availability at Symantec."

Comments

Th e other half

[ascari]

And the other half runs Linux!

Re:Th e other half

[Penguinisto]

Laugh all you want, but there's a kernel of truth in that. All the *nix servers in my care mostly run on autopilot, and I pop in only once in awhile to check up on them, change/enter something in BIND, occasionally patch the ESX machinery, or put in the occasional patch that yum or ports can't get out of a repo (e.g. our custom help desk site software).

OTOH, a huge chunk of time is spent in Exchange and SharePoint - mostly chasing down errant mails, or fixing bugs and glitches. To be fair, those two bits are customer-facing, thus more open to calls - but even still, so is our help desk site (which runs on Linux), and I rarely have to bother with that on the back-end. Also, I've run pure *nix email setups before, and it never ate as much time percentage-wise as Exchange does now - even when chasing bounces.

On average, the 'doze servers eat about 95% of my time, but they comprise only 60% of the population.

Nota Bene: One thing I've found to be awesome - get up a script that sends a copy of your Exchange logs to another box... that way you're not fighting store.exe for RAM when you want to parse through them, and you can use a real text editor (vi or EMACS - you pick) to read them.

Re:Th e other half

[ka8zrt]

-1 or more about not thinking this through though. (and not funny at all)

As someone who has until recently done research in data centers and their operations, and personally dealt with the *NIX side of *NIX vs. NT years ago, I know the reality as opposed to the half-thought-out dreams some have. Yes, *NIX makes it much simpler to manage a machine, and increase the (servers/admin) ratio, among others, but it is not a solution which scales to where one person can administer 10K servers. As you add servers and applications, that ratio will reach a limit where you have to add yet another admin (operational, network, hardware, etc.). And should that site not be willing to do so, you end up with one of those "understaffed" data centers. Where that point is reached depends on a multitude of factors, including the behaviour of those using the data center (stupid developers, hands on users or workload characteristics cause that point to be reached sooner), the applications (a bunch of database servers will likely reach it before an equivalent amount of web servers), the amount of storage on those servers, and even the individual admins and how they are organized themselves. Throw in things like buying the cheapest hardware, or buying bleeding edge hardware (say 1.5TB drives when they first come out, or 10Gb ethernet cards), and it gets even worse as you try to deal with first generation drives failing or buggy drivers.

Can two people administer 500+ servers with 1.5PB of storage? I know personally that it is possible. But to do it and keep everyone 100% happy? No. And that precludes things like having people who are hard to satisfy, having to backup all that data, running it in a non-university production environment, etc. When I left CompuServe in 1997, the numbers were far different, with IIRC 25-30 operators of varying skill levels, about 10 of us in admin positions (who were called upon by the operators when they could not handle something), and around half a dozen or so network and hardware folks. Total number of servers? Around 1200 running BSD/OS, and around another 1000 running either our proprietary OS on systems which came out of the DecSystem 20 designs, or systems running a specialized NT 3.51 load, and perhaps a total data storage of around 1.5TB. And things were simplified by things such as having dozens of machines which were identical handling application X. Of course, we also had 3 data centers, and did backups of at least one of each machine in a given group. And then there is the fact that some applications required the developers to administer the application itself.

And looking forward... There were no regular 12 hour shifts at either of these. Yes, I was on call darn near 24*365 (I got vacation time off at my latest employer, but at CSI, I was on call even during vacation, and averaged 80hrs/week at the end). But when the fecal material hit the fan, and we had unusual problems like a computer room flooding or a critical server failing... it was possible to have to put in a 24 hour shift. Such is the life of a senior systems engineer in an operations group, which is one reason I try to avoid positions like these.

In other news...

[HBI]

50% of all datacenter operators lie about their staffing levels.

Re:In other news...

[halcyon1234]

50% of all datacenter operators lie about their staffing levels.

The other 50% didn't return calls in time to be included in the survey.

12 hour shiths are not the ansaser

[Joe The Dragon]

12 hour shifts are not the answer as well makeing people work every weekend holiday night while the boss / PHB never does any of that.

Would this be a good time for a union?

[starbugs]

> 50 % understaffed, 16 % seriously.
So how many of you have to answer your blackberries after work?
Is this not the kind of situation that a Union would prevent?

(just an honest question btw, I'm not trying to troll)

Re:Would this be a good time for a union?

[gclef]

That's going to be very dependent on the union. (Devil's always in the details.) Many IT folks still have the free-wheeling "just get out of my way & I'll get this fixed" attitude, and in those cases union interference in their work will not be welcomed.

Basically, a collective bargaining agreement is one thing...having someone outside the organization set the bounds of your job (and set limits on how you can be promoted, or which incompetent f-up can be fired) is quite another. I won't say a union is impossible, but it probably wouldn't be one of the big names.

Re:Would this be a good time for a union?

[Anonymous Coward]

however considering everything else that has been done for union's sake lately (see: destruction of US auto industry) I would suggest you take the unionization decision VERY seriously.

Hahaha. As much as I dislike unions, the destruction of US auto industry was caused by complacent & incompetent US auto industry management.

The US auto industry kept designing & building cars at a price point that few people wanted to buy. Simply put, foreign car companies (on average) made better, more reliable cars.

The union wanted better salary & benefits for their members (entirely understandable, we all want to make more money). But if management agreed to ridiculous levels of compensation, to the point where the business is no longer viable, then that is the fault of management for making stupid decisions.

Should this be surprising?

[Anonymous Coward]

Does this really surprise anyone?

Many data centers these days are no longer run by engineers or technologists, who have at least some idea regarding the technical aspects of the operation. Rather, many of them are run by people who received their higher education in finance, commerce, accounting, "business" or (perhaps worst of all) even marketing.

Of course, such people have a very hard time seeing beyond the numbers, since they usually have absolutely no understanding of technology, nor what it takes to truly run an effective data center. They insist that the current number of staff are sufficient, even when they clearly aren't, and even when they could easily afford to hire more employees.

I think this just reflects a greater problem of the American corporate society as a whole. People with actual technical knowledge in a specific field get pushed out in favor of people with meaningless MBAs (but all of the right "connections"). So it's no wonder American productivity and competitiveness is grinding to a halt.

Other areas of the world, namely Asia, India and Eastern Europe, realize that it isn't the accountants and financiers who provide productivity, but rather the engineers, scientists and technologists. That's why they can build better cars at a far lower cost than their American competitors can, for example. That's why Korea and Japan have broadband networks that put to complete shame anything in America.

Re:Should this be surprising?

[vlm]

I don't think that it's a big deal that people with diverse backgrounds get into IT. Either they are competent or they are not, and there's no reason someone in finance can't become competent in IT and switch careers.

No problem, but put them at the end of the very long line of folks whom already know what they're doing.

Re:Should this be surprising?

[vlm]

Ahh I was somehow under the false impression that they were able to make cheaper cars due to lower wages, less environmental regulations, and the lack of labor unions.

Actually it only takes about $2K of labor to build all cars and trucks. Some robot factories cost less, some cost more.

Most of the revenue goes to executive bonuses.

I'll buy American made, Japanese managed, cars. But I won't buy Mexico made, American managed, cars.

Re:Should this be surprising?

[Fastfwd]

Actually it only takes about $2K of labor to build all cars and trucks

That's probably true of most things/services. There is an amazing amount of "friction"(ie: added cost) from all levels of management, marketing, etc. Some of it is necessary, a lot of it is not. It's strange that the people you are 100% sure you need(engineers/builders) are often at the bottom of the salary food chain.

What is "understaffed"

[vlm]

I believe understaffed means no one (in the US) is replying to the following ad:

Want to hire data center cat5 cable install tech, mandatory 60 hr week overtime, weekend 2nd 3rd shift and holidays required, require CCIE, MBA, at least masters level degree (prefer phd), minimum ten years experience with "windows server 2008R2" yearly salary $25K/yr no benefits.

Golly, we got us a shortage, best open the H1B floodgates!!!!!!!!!!!!!!!!!!

Data Centers

[electricbern]

1. Lay off staff 2. Hire Oompa-Loompas 3. Profit!

Re:Whatever.

[Captain Splendid]

In the meantime, get on your knees every morning and thank your personal god that you have a job.

It's attitudes like that why wages stagnate. Gonna get flamebait for this, but what happened to the yankee spirit? The Founders would puke at the current complacency.

Don't forget Western Europe

[brucmack]

I work for a Swedish company that understands the value of IT and invests resources in it accordingly. Based on my experiences with other Western European countries, this isn't abnormal.

The difference in work culture between here and the US is astounding. While it seems most American companies see IT as the place to save costs, the companies I've dealt with here recognize that our IT systems contribute directly to our competitiveness in the global market, and invest accordingly.

Get Your History Right

[mpapet]

Unions were invented to protect unsuspecting workers from manipulative business owners

No. It was really much simpler than that. People were tired of working for peanuts. Lots of people were tired of working for peanuts. Lots of those people were plenty smart. How else do you think they got organized?

Before unions, the institution of the 5-day work week was another long, hard-fought, pitched political battle that business was *sure* would absolutely end the U.S. economy. When Ford doubled pay and shrank working hours, the rest of American industry would not follow because from a capitalist's perspective, you are blowing your labor costs out of site! History suggests it seemed to have worked for Ford.

You don't get to blame organized labor for all of the auto industry's ills. Maybe you recall the Pontiac Aztek as possibly the apex of bad auto product? The labor that allocated resources for that project and a long history of uninspiring ones before that, weren't part of a Union. What's the managerial ratio at those companies 'burdened' by Union labor? What are the managerial labor costs at those firms 'burdened' by Union labor? I think you will find them both expensive and inefficient non-union workforces.

http://en.wikipedia.org/wiki/Working_time

http://en.wikipedia.org/wiki/Eight-hour_day

It's time to bury that notion that Unions cripple an economy. It's used primarily to reinforce the ridiculous American ideal of 'rugged individualism triumphs over all" and concentrates power and resources to the least efficient few.