Slashdot Mirror
Half of All Data Centers Understaffed
alphadogg writes "Fifty percent of IT executives say their data centers are understaffed, and companies are still looking for more ways to cut costs, according to Symantec's latest 'State of the Data Center' report. Sixteen percent of survey respondents said their data centers are extremely understaffed, and another 34% called their data centers somewhat understaffed. At the same time, data centers are becoming more complex and harder to manage, with more applications, data and increasingly demanding service-level agreements. 'Data center complexity has led to a lot of staffing challenges,' says Sean Derrington, director of storage management and high availability at Symantec."
12 hour shifts are not the answer as well makeing people work every weekend holiday night while the boss / PHB never does any of that.
> 50 % understaffed, 16 % seriously.
So how many of you have to answer your blackberries after work?
Is this not the kind of situation that a Union would prevent?
(just an honest question btw, I'm not trying to troll)
Does this really surprise anyone?
Many data centers these days are no longer run by engineers or technologists, who have at least some idea regarding the technical aspects of the operation. Rather, many of them are run by people who received their higher education in finance, commerce, accounting, "business" or (perhaps worst of all) even marketing.
Of course, such people have a very hard time seeing beyond the numbers, since they usually have absolutely no understanding of technology, nor what it takes to truly run an effective data center. They insist that the current number of staff are sufficient, even when they clearly aren't, and even when they could easily afford to hire more employees.
I think this just reflects a greater problem of the American corporate society as a whole. People with actual technical knowledge in a specific field get pushed out in favor of people with meaningless MBAs (but all of the right "connections"). So it's no wonder American productivity and competitiveness is grinding to a halt.
Other areas of the world, namely Asia, India and Eastern Europe, realize that it isn't the accountants and financiers who provide productivity, but rather the engineers, scientists and technologists. That's why they can build better cars at a far lower cost than their American competitors can, for example. That's why Korea and Japan have broadband networks that put to complete shame anything in America.
I believe understaffed means no one (in the US) is replying to the following ad:
Want to hire data center cat5 cable install tech, mandatory 60 hr week overtime, weekend 2nd 3rd shift and holidays required, require CCIE, MBA, at least masters level degree (prefer phd), minimum ten years experience with "windows server 2008R2" yearly salary $25K/yr no benefits.
Golly, we got us a shortage, best open the H1B floodgates!!!!!!!!!!!!!!!!!!
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
If you want to get a true picture of life in a data centre look at what the management actually do, what they spend money on and what they produce. If you rely on the answers they give you'll end up broke very quickly. The only way to tell if datacentres really are understaffed is if they start hiring more people: any other action just shows the lie in their responses.
When managers say they need more staff, they generally mean they need more cheap staff (often to replace the expensive staff they already have). They could always fill any critical needs very quickly by offering more financial incentives (the only ones that really mean anything), but this almost never happens. Somehow they manage to bumble on with their "staff shortages" and still meet their targets.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
Not really surprising; they fall into the range of companies who tend to have enough money to invest in new tech but lack the corporate clusterfuck that stops them from achieving any kind of change.
In the meantime, get on your knees every morning and thank your personal god that you have a job.
It's attitudes like that why wages stagnate. Gonna get flamebait for this, but what happened to the yankee spirit? The Founders would puke at the current complacency.
Linux, you magnificent bastard, I read the fucking manual!
They are surveying enterprises of certain size, and asking someone (who? the manager?) if their perception is if they are understaffed or not. Similar sized networks could be seen as under or overstaffed depending of how much troubles they have, how much busy they feel, a quiet datacenter with half of the usual staff could be seen as overstaffed if no troubles or most of the common trobles are solved automatically, compared with a chaotic one with lots of troubles. Where i work in a year we passed from a perception of understaffed situation, where troubles jump at every moment, to an almost overstaffed one, same datacenter size, almost half of the people, but better architecture.
The problem is that staffing levels are very often highly subjective. For most concerns, the complaint of being "under staffed" only indicates that the current staff feels overworked, a condition almost universal in all sectors of a healthy, i.e., growing, organization. For the ISO 9000-ish (or ITIL?) crowd, under staffed might mean that some formal document published a desired level at some specific point in time, the best against a workload study, and industry rules of thumb. But, since every such study measures a specific point in time, they become out of date, often obsolete by the time full staffing achieved. So, "fully staffed" is ever elusive, and this applies to every sector. We're all Bozos on this bus. In fact, any staff that's manned to the point that they're not feeling some pain risks being seen as over staffed, and a target for reallocation or cuts. Sorry to put a damper on any delicious feelings of workforce martyrdom. People also get mad at me when I point out that, by definition, nearly half of the population ranks below mean intelligence.
I can't believe your were modded up.
He was modded up because he's absolutely right.
America was not built with "outsourcing" in mind.
When the going get's tough, you get off your ass and get tougher!
Server OS is not the only thing in the datacenter that needs staffing. Facilities work (cabling, power, cooling, etc), SAN, Network infrastructure, and that's without even getting into the middleware or applications themselves.
Even if your base servers administered themselves, it still takes quite a staff to actually do something with those servers.
Like if producing gas guzzlers that are inefficient and brake easily is the fault of the unions.
I thought that the geniuses commanding those huge bonuses, golden hand shakes and parachutes were the ones dictating corporate policy.
But hey, whatever rocks your boat matey.
IANAL but write like a drunk one.
Very, Very few Center Managers actually performed any kind of statistical process control analysis for quality in the datacenters I worked for which were huge and did work for .gov, finance and top 500 and they barely did it. They eventually fired the poor guy as he kept proving management wrong. We had long conversations that helped me understand technology for what it was: "La Technique: L'enjeu du siècle" was an eye opener.
Very few managers understood what project management & change windows were in a datacenter and usually managed to a staffing model which could only break in times of heavy load, inducing a bigger emergency later on.
Really management held the mentality of "the Maytag repair man" is who they need to hire by the business plan but the reality was a team of MacGuyver's were needed for the workload over several clients.
Even educating them on vendor patch cycles and technology refresh could not break them out of scarcity = profit mentality.
The admin team experienced over and over that that next month is the month where Microsoft is going to pound us with new critical patches. Despite explaining this, management also put the work of putting unrelated project X into motion or finishing that same next month.
Extensive studies about patch cycles or changes (failed, back out, succeeded) take a long view - an X bar control chart shows spikes and abnormals. It then takes some analysis to then determine staffing levels that can handle the work on average. It takes a huge business insight to understand why something fell outside the norm and how to handle it when it comes again. Really I don't think the technology today can be managed to wholly eliminate outlying events like traditional manufacturing processes. The now typical 3 year tech cycle prevents such work.
It is true statistical process control can be made to lie but it is better than uneducated guessing.
To me datacenters are huge machines you can walk inside of, really no different than megawatt power generation or pharmaceutical manufacturing and ought to be better managed. I say this in that when they go offline or do not function well an aspect of human safety and productivity is jeopardized.
My favorite manager who I worked for briefly before he retired said, "Cowboy managers and admins have no place in my shop. I want science put back into Computer Science, don't snow me with new technology."
I agree with him.
I agree with the statement but not the reason.
Doctors, Nurses, Medical Staff, Police, Fire/rescue often work 12 hour shifts and holiday.
However those professions realize and have by experience been bitten by the consequences which aid them in helping the professional know their limits and the limits of their peers.
First these professionals make mistakes during the day. More so when overtired, Even more so when out of their normal sleep pattern. Technology professionals somehow ignore this and think they are superhuman and often promote this.
"Oh I stayed up all night to fix your server!" Pat me on the back! While probably true, I don't want to hear that sentimentality from my admin. It meant something went horribly wrong and I don't want it to ever happen again.
Doctors, etc as cited know that they would perform in a diminished capacity the next day and not schedule surgery and/or the hospital management would know to give them a resting day as the liability of mistake be too great. Safety services know that some other station has to possibly cover a crew that just came off a fire/rescue and be very wary to send the same crew back in. Technology companies ignore this to their own embarrassment which is justly earned.
Second doing business changes (minor or major) on weekends or holiday nights is _bad business_ in that it demonstrates the fragility and unreliability to which they do not admit to customers. Why not do the same operation during normal hours?
Would anyone take their business' truck to the car mechanic for an Oil change and accept, "well we have to do it between 3am and 4am so as not to impact your business."
But it's an OIL change, it happens frequently, everyone ought to expect it to happen! This is exactly the same to me as a minor patch, price lists, firewall rules, and application rules for business policy. Such ones are expected, frequent and shouldn't have to be done like as they are now at a forsaken hour in the morning.
The more complex example is "Oh the engine overhaul is going to be b/w 3 and 4am" - I would say give me another truck that does the same thing and I'll be back after you fix my truck during the day when your awake. The analog is the system upgrade. Providers go into fits -" but but your system was so tweaked, We can't simply move it to another CPU", etc... Blah. Its because most centers don't know how to offer a real solution.
IT Professionals ought to advance the profession and figure out why they are working 12 hour shifts and holidays and then systematically eliminate these events as much as possible till only having to do so when a human life or safety systems is jeopardized.
Why IT professionals are not publicly beating up IT vendors for poorly written OS, barely redundant equipment, poorly designed apps, etc, is beyond the scope here.
Who is going to be the next Ralpf Nader, who will write "Unsafe at Any Speed" for the IT industry/Computing Science.
Sorry, while Unions may serve a role in blue-collar jobs, I don't agree that they have much of a place it IT jobs. Especially when it comes to protectionism and "seniority". I work at a public university and our IT is unionized. It is basically impossible to fire the old IBM mainframer who refuses to learn anything new because he is retiring in 5-10 years and pines for the old days. So in these difficult budget times, we have to lay off the young, productive staff and are stuck with the useless ones. As is probably typical in the unionized government sector, if you could pick and choose the 20% to lay off, you would save a ton of money and productivity would probably go UP! As it is, you get rid of the modern, young talent and productivity actually goes down far more than the 20% you are laying off.
I see your point, but there's a flip side to that argument. Just wait until you get older and have to find work.
The classical career arc that was in place from the late 40s until maybe the mid to late 90s went like this for college graduates:
- Graduate high school
- Graduate college
- Get a company to hire you in some kind of traineeship
- Work your entire 40-year career for the same or related companies, with a series of progressively responsible positions that were designed to meet your need for more income and responsibility
- Make progressively more money in a safe job, and not be afraid to do things like buy a house or car.
- Get a gold watch and retire with full benefits and heartfelt thanks from a company who was happy to have the institutional knowledge preserved for that long.
For non-college graduates, you could get a job in a factory, work your butt off for 40 years and still come out OK due to union negotiation of wages and benefits. A bachelors' degree wasn't a mandatory admission ticket to a job, so fewer people were burdened by student loans for degrees they didn't even need.
During this time, there was a huge middle class that was able to spend more money on goods and services. People weren't constantly in fear of layoffs or downsizing. And yes, it came with a price of higher wages and a stronger labor movement.
Today, the pendulum is back near the other extreme. In IT it's worse, because there's no career path for most people above a certain age. Your example of the mainframe guy is valid...but too many employers think that everyone over 40ish is unwilling to learn anything. I know incredibly talented colleagues have had to get lucky and find companies who value experience over the willingness to work 80-hour weeks to compensate for sloppy processes or bad work. On top of that, inflation of education requirements means that more people are being force-fed through college. That's good for the university system, but bad for the economy. To make an economy flow smoothly, you need jobs tailored for different intelligence levels. A secretary doesn't need a BA in Communications to answer the phone. A factory worker doesn't need an engineering degree.
I actually like the model of state employment, which follows the classic model. You may not be able to be as much of a rock star in front of your colleagues, but nothing beats job security and safety. The problem, as you point out, is that people do tend to abuse the system as they progress. However, having a steady stream of consumers who aren't afraid to spend real (non-credit) money is what actually drives business.