Slashdot Mirror
Can Maintenance Make Data Centers Less Reliable?
miller60 writes "Is preventive maintenance on data center equipment not really that preventive after all? With human error cited as a leading cause of downtime, a vigorous maintenance schedule can actually make a data center less reliable, according to some industry experts.'The most common threat to reliability is excessive maintenance,' said Steve Fairfax of 'science risk' consultant MTechnology. 'We get the perception that lots of testing improves component reliability. It does not.' In some cases, poorly documented maintenance can lead to conflicts with automated systems, he warned. Other speakers at the recent 7x24 Exchange conference urged data center operators to focus on understanding their own facilities, and then evaluating which maintenance programs are essential, including offerings from equipment vendors."
Maybe there's a sweet spot between "no testing at all" and "replacing everything every three months"? In my experience, there is a lot of work to do in most places to make sure that proper testing is done, or at least that emergency procedures are known and people are well trained in them. Very often documentation is lacking and the onsite support staff have no clue where that circuit breaker is. That is the most common scenario in my experience, not overzealous maintenance.
Semantics is the gravity of abstraction
I believe the article is referring to major hardware replacements, stress testing, etc. But there is other preventative or even detective work that needs to be done in data centers large and small that have nothing to do with equipment. You can't just blithely assume that things are always going to work as they are supposed to work. One time, we discovered that the camera server for one of our clients had stopped recording for no good reason, and upon closer inspection discovered that the hard drive failed and we had no alert system in place since it wasn't a "real" server but just a heavy duty XP machine. After that blunder, I was asked to check on all the cameras servers once a week and make sure I could actually open up and view recordings from days past. This is a preventative action, but not really a maintenance one.
Occasionally living proof of the Ballmer peak.
Sometimes I get the feeling that security updates can in most cases cause more problems than the issues themselves.
I can think of many occasions that a security update has broken a server/router/etc. Obviously the lack of a security update can lead to a bigger headache in the future. But the typical user doesn't understand and has the attitude "IT broke the server again".
If a virus or hacker causes an issue the attitude is "I hope they fix that soon. I hate viruses/hackers" (obviously this is a huge generalization).
===
"Is preventive maintenance on data center equipment not really that preventive after all? With human error cited as a leading cause of downtime, a vigorous maintenance schedule can actually make a data center less reliable, according to some industry experts.'
===
It isn't just human error: the very act of performing intrusive tasks under the theory of "preventative maintenance" can greatly reduce reliability of systems built of reasonably reliable components. This was studied extensively by the US airlines, US FAA, and later the USAF in the 1970s when the concept of reliability centered maintenance was developed for turbine engines and eventually full airliners. Look up the classic report by Nowland & Heap. Very much counter-intuitive if one has been trained to believe in the classics of "preventative teardowns" and fully known failure probability distribution functions, but matches up well to what experience field mechanics have been saying since the days of the pyramid construction.
sPh
Of course, today there is a huge "RCM" consulting industry, 7-step programs, etc that bears little resemblance to the original research and theories; don't confuse that with the core work.
That being said, it was because their procedures were shit, not because they were doing maintenance.
After that blunder, I was asked to check on all the cameras servers once a week and make sure I could actually open up and view recordings from days past. This is a preventative action, but not really a maintenance one.
No, it's not preventative. It does nothing to prevent the problem. It detects the problem earlier (before, say, a business user does). That's monitoring. It's proactive, not reactive - perhaps that's what you mean?
The guy at the garage always recommends I do an $80 transmission flush.
===
No, it's not preventative. It does nothing to prevent the problem. It detects the problem earlier (before, say, a business user does). That's monitoring. It's proactive, not reactive - perhaps that's what you mean?
===
It is deeply unclear whether what is traditionally termed "preventative maintenance" (intrusive work involving disassembling, eyeballing, software probing, etc) actually improves reliability over conditioning monitoring tests followed by break-fix work as described by the parent post. More PM, more procedures, more teardowns, and so forth are the standard prescription for improving reliability but there is metric tons of evidence the universe just doesn't work that way.
sPh
asking why everyone hates the IT department
Seriously...I sometimes think the average IQ is dropping on a daily basis (and, yes, I get the irony)...Both with what I read, and my own experiences working in IT, I become more and more convinced that society will eventually collapse under the weight of bad advice from consultants (and, no, I don't own a fallout shelter)...and I spend more and more time thinking about ways that I can profit off of the stupidity of leadership.
In days of old, running "big iron" from Control Data and Cray, the worst days of system instability were those following "preventive maintenance". Plus ca change....
From TFS:
"... poorly documented maintenance can lead to conflicts with automated systems ..."
That doesn't mean maintenance makes datacenters less reliable. It means cluelessness makes datacenters less reliable.
Sheesh.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
There's something to be said for this. Back when Tandem was the gold standard of uptime (they ran 10 years between crashes, and had a plan to get to 50), they reported that about half of failures were maintenance-induced. That's also military experience.
The future of data centers may be "no user serviceable parts inside". The unit of replacement may be the shipping container. When 10% or so of units have failed, the entire container is replaced. Inktomi ran that way at one time.
You need the ability to cut power off of units remotely, very good inlet air filters to prevent dust buildup, and power supplies which meet all UL requirements for not catching fire when they fail. Once you have that, why should a homogeneous cluster ever need to be entered during its life?
Any maintenance done wrong or in excess can be more damaging to a system than no maintenance.
If it isn't broken, don't fix it.
I have done data center hardware maintenance, IMAC, troubleshooting, repairs, etc for the past 15 years. From my experience, the biggest problem has been with Sys Admins who do not know their hardware, do not follow procedure, quick to point fingers at hardware and to root problem, and and over-all belief that firmware upgrades are the "Holy Grail" to prevent and fix all problems.
With most systems redundant in power, storage, etc, everything does not always need to be power cycled. That causes fans and other components to fail when coming back up....yes, they were probably failing but if a repair can be done on the fly, why power cycle all the time?
It really gets fun when someone does not have their system alerts set up properly and someone takes down a server with updates to the OS/App side and then discover the problems and they all complain about how they are losing thousands of dollars in down time. If the system is that critical, why not design a fail over system, it would be cheaper in the long run....
Also, most hardware service outfits have several spares available for 7/24 repairs but when a Sys Admin or business unit decides to down several systems in a weekend that have been running for months/years and upon power up there are several drive failures, fan failures, PPM failures, power supply failures, cache battery failures, etc, most vendors including OEMs cannot have all parts available at a moments notice since we spare according to failure rates an part pricing. A pro-active heads up project with a list of hardware that is going to be worked on, would be nice. We could stand by with parts available for repairs and even on site "eyes and hands" if the systems crap out and need on site assistance.
There are several flaws in data center management from the single file and print server in a small office to large data centers with huge support groups like banks and government centers. The same problems exist on all levels...some are just better prepared for the "worst" where others think we are magician/mind readers....
As "Bones" would have said if he were in IT..."Dammit Jim, I am a technician....not a magician!"
... is actually quite simple: You keep your hands off the systems. Period.
In detail, you plan, install and _test_ your setup before it enters production. You make sure that you can survive whatever you throw at it wrt. errors and incidents. You then figure out how much downtime you are allowed to have according to SLA. You then divide this number into equal sized maintaince windows together with the customer. And then you adhere to these windows! No manager should ever be allowed to demand downtime out of band. Period. In between you basically minimize your involvement with the systems and plan your activities for the next scheduled closing window.
And you ofcourse only deploy stable, true and tested versions of software and operating systems. And even though your OS supports online capacity expansion on the fly, you really shouldn't use the capability unless you absolutely have to. Instead you plan ahead in your capacity management procedure and add capacity in the closing windows. And you do not test and rehearse failures! It only introduces risks ... besides that you have already tested and documented them. And as you haven't changed the configuration, there is no need to test again.
So in essence. Common sense will easily yield 99.9%. Carefull planning and execution will yield 99.99%. The really hard part is 99.999%... /zensonic
Thomas S. Iversen
Obviously not doing maintenance is much worse than the risk incurred by doing maintenance. However in the 2 years of using the datacenter my company relies on, I can say the only 3 major outages have been due to non-routine maintenance. Once was during a power upgrade, the datacenter supposedly has redundant power company connections, but the plug was somehow pulled during the upgrade anyhow. Another was during network maintenance, our dual redundant internet connections turned out not to be so redundant when half the system went offline. And finally during AC upgrades that caused our entire server cluster to overheat and shut down.
So, the story here isn't "maintenance causes unplanned downtime therefore doing less maintenance causes less downtime". Its more like "unusual maintenance is more likely to get messed up". One would think you would be more careful when performing unusual procedures, but that is still when things get screwed up most often.
I'm a good cook. I'm a fantastic eater. - Steven Brust
Check your transfer switch ratings. I guarantee it will be spec'd much lower than you think. The electricians think it'll only be switched a couple times in its life. The diesel service provider thinks you're running it twice a week. Whoops. If you run it once a week, it'll only survive a couple years, then you'll get a facility wide multi-hour outage. I've personally seen it over and over again over the past two decades. The best part is "we have a procedure" so it'll only be run during maint hours and the desk jockeys 200 miles away will run it rain or shine, so its guaranteed that the xfer switch destroys itself at 2 am during a blizzard and it'll take half a day to repair.
Very few xfer switches are more reliable than commercial utility power. Installing a UPS actually lowers reliability in almost all professional situations.
My favorite power outage was caused by a gas leak a couple blocks away, where the utility co shut down the AC and then threatened to take an axe to the gen/UPS if not also shut off. This was not in the official written report, just word of mouth.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Planned obsolescence has been promoted in all aspects of life since post WW2 and now it is hard to imagine the world without it. That line of thinking has been creeping into everything even in areas where it doesn't seem to apply.
Does this play a factor on the perception of preventative maintenance or its frequent application? I think it probably does in at least a couple ways, don't you?
Democracy Now! - uncensored, anti-establishment news
Internal monitoring of components is a lot better now than it used to be. We used to go around and check all of the supply fans once a month because it was the highest failure rate component on the desktops we were using and there was no indication until the machine started crashing from overheating.
I read through the entire article, and saw zero data to support his assertion. I'm sure he has the data, but the article didn't reference a single piece of it. Without any data to support the theory all we have is a fluff opinion piece. Shame on Data Center Knowledge for writing an article about a scientific investigation, and not presenting a single piece of scientific evidence.
AccountKiller
===
Obviously not doing maintenance is much worse than the risk incurred by doing maintenance.
===
That's far from obvious, actually, and is demonstrably wrong for many types of systems and installations.
sPh
The purpose was partly to stop qualifying being its own arms race, with cars in completely different specification than for the race, and partly to reduce costs and the number of travelling staff. At the same time, "T Cars" --- a third car, available as a spare --- were banned, so that if a driver destroys a car in practice the team either have to rebuild it or not race. They're allowed to travel with a spare monocoque, but it cannot be built-up and it does not get pit space.
There were endless howlings from the teams, claiming that without a complete strip-down after qualifying, with a large crew working overnight to check everything on the car, reliability would go through the floor and races would finish with only a handful of stragglers fighting a durability battle (our US viewers may find this ironic in light of a certain US Grand Prix, of course).
The same argument was advanced, mutatis mutandis, over limitations on engines and gearboxes, limitations on the number of gear clusters available, limitations on certain forms of telemetry and a wide variety of "the cars can't just be left to run themselves, you know" interventions.
In fact, reliability is now far greater than ten years ago. It's not uncommon for there to be no mechanical retirements, certainly not from the longer-standing teams, and the days of engines imploding on the track are long gone. A front-running driver will probably only have one, if even that, mechanical DNF per season. The teams deliver a functioning car when the pit lane opens at 1pm Saturday, and that car then runs twenty or thirty laps in qualifying and sixty or seventy in the race, a total of perhaps 250 miles, without much maintenance work beyond tyres, fluids and batteries (section 34.1 on page 18 of the sporting regulations).
So again, we see that "preventative maintenance" turns out to really be "provocative maintenance", and leaving working machines alone is the best medicine for them.
Sounds like the "If it's working don't touch" maintenance policy.
...is the one farthest from the nearest engineer."
Consider The Pioneer and Voyager spacecraft and the Mars landers.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
vigorous maintenance
excessive maintenance
poorly documented maintenance
Those are all qualified as out of the ordinary. Anything in excess (on either side of the scale, whether it is too much or not enough) is a problem. Of course maintenance must be performed, but I guess some data centers have a strange idea of best practices, or they do not follow them.
Twinstiq, game news
battery maintenance / changing out old battery's is need as a dieing battery can fail to work when need or at the worst they can have a explosion.
Commercial Aviation has learnt this lesson a long time ago, after analysis showed that maintenance induced failures exceeded the potential failures avoided.
Now most parts for airframes and engines are on IRAN schedules - Inspect and Repair/Replace as needed. Only a few filters and things are replaced on a fixed schedule.
Some times software / os needed at least a soft reboot from time to time to clean up stuck software and remove memory leaks.
Now some stuff like firmware updates may need a hard reboot.
As for power cycleing some times you need to do it to get back into a crashed system.
Precisely my thought.
Maintenance, like anything else you do in a datacenter or wherever you work, must be done correctly. If maintenance reduces the reliability of the maintained entity, then per definition, it was not correctly performed.
Doing something correctly requires knowledge, planning and training. Just like everything else.
One of the Asian automotive manufacturers, Toyota, Honda, or Nissan, sent around a crew in their assembly plants to retorque any loose nut/bolt/screw they found. They saw a dramatic reduction in plant down-time. Ben Franklin 'a stitch in time' comes to mind. Computers will fail eventually. If the system is so unstable that 'maintenance' tips it over the edge then you were about to have a severe incident anyway (and probably not the team available to fix it, like the maintenance crew that are still right there). Make prudent backups before any major maintenance, plan for some random glitch getting things back on line, and you're covered. It's hard to record the 'things gone right" from maintenance .. but easy to record the number of issues of problems after maintenance.
http://www.theatlantic.com/magazine/archive/1998/03/the-lessons-of-valujet-592/6534/
William Langewiche, Atlantic Monthly, "The Lessons of ValuJet 592". It was basically done in because it was transporting safety equipment itself, which was vulnerable to a hard-to-predict failure. The more complex we make air travel, with its multiple checks and layers of protection, the more opportunities for failure. Adding another check to avoid 592, as they did, creates yet another opportunity.
It is, as they say, a Hard Problem. Yet, still: the US recently celebrated 10 whole years without a major airliner loss, despite a phenomenal amount of air travel. Things are getting better. Hard != Insoluable.
Lets face it... if 'we' administrators REALLY did our job and REALLY knew our stuff, we wouldn't have to patch nearly as often as we do, because our systems would be hardened and properly protected leaving many patches unneeded unless for the services our systems were providing required them.
And in those cases, our documentation would allow us to REALLY test exactly what we needed to and script the installs specifically for the systems that needed the patching.
Unfortunately, we DON'T REALLY know our systems the way we need to in order to PROPERLY secure them and maintain them. I see all too frequently the practice of 'getting it working' versus 'proper implementation' because we don't have the time or resources.
THIS unfortunately is what leads us to the regular patch cycles that we have, because when we don't really know and understand our systems and defend them independently by proper hardening, we must rely on patching everything, because in the end, everything is exposed. (Mind you, that's Windows, Linux, mobile, etc. etc.)
BUT, I don't want to leave it all to us administrators, lets face it, VENDORS and MANUFACTURERS have a hand in this as well. Nine times out of ten they don't even know their products well enough to tell us administrators what to do, or how to properly implement their solutions. They are so quick to market, their products stink. A proper and secure implementation with a product that is not written properly might as well be a waste of time. They are LAZY too. Their products don't work as advertised and they are so focused on sales, they don't even care that they aren't installed properly. When I have to work with top level support to get a security product to function as advertised because it 'isn't normally installed that way'; however, THAT IS they way it SHOULD BE installed to be properly secured, then there is a problem with the product, PERIOD.
So, to my brethren in IT, let's make a statement to really and truly understand our environments and NOT let our product manufacturers or in-house developers BE LAZY and make our systems unreliable.
Having been involved in Technical Ops of both large and small companies for many years, I have seen DR exercises and design that have run the gambit. I tend to think The key thing I have found to the success of any organization, exercise, or philosophy, is the underlying process that drives execution. The larger the team/org, the more change points, which in turn leads to more variables between tests. This creates complexity, as a test that ran fine a few months ago may not run the same today. However, ensuring change does not overrun process in understanding and applying the change into the greater design is a key to ensuring each test improves upon the last, until such time this is a finite process.
For example, when working for one of the big 401k's, the first DR exercise evaluated the data center completely being leveled and re-locating both technical services as well as the ~300 on site employees to another location. Long story short, the first exercise of this was scheduled for 2 days, and while it worked, we identified dozens of issues. We scheduled the next test 6 months later and addressed what we believed were all of the issues; on next test, we ran into perhaps ~10 issues. The next test we scheduled 3 months ahead and ran into ~2 issues. All awhile, things continue to change and innovation is occurring, change process control is ensuring that new things are being factored into the continual DR process/exercise. For a small telecom I worked for, the same type of testing was accomplished with ~2-3 week turn around time (smaller team, less change points, more dynamic response), but with same underlying principles.
Documentation of such things is critical, and employee turnover is often one of the greatest risk points. Having a diversified staff with overlapping knowledge should minimize the later risk to some degree, and if implemented fully, risk should be diminished.
So how does all this tie back into maint? Well, it is anticipated that if any system runs long enough, their will be opportunity for failure. It is preparation for when such failure occurs, one can balance the capability of providing a measured window of downtime (if any) and provide some degree of predictability (i.e. I test once a quarter). The counter to this can certainly be overzealous maint, so certainly their is a point to being reasonable. For example, what many of go through with our cars - the dealer wants us to come in every 3k miles for an oil change, whereas realistically most mfr's and my own experience dictates that ~5k (if not longer depending on circumstance) is much more cost effective. Either way, this is providing some degree of confidence that this should prolong engine life.
tell that to my spiders. they are trying to "maintain" their webs in my servers for some reasons (hint:very bad)!
well some windows updates still need reboots it's less then it was in the past but still more then with linux.
Also a lot of NON OS software updates / installers at least say they need a reboot.
If "maintenance" means doing a forklift upgrade of all the computer and networking equipment every year or two then of course your reliability is going to suck, especially the human error factor with all of that new, unfamiliar equipment.
On the other side of things if someone thinks that never changing the oil in the generator is going to make it more reliable then they're in for a surprise. When I think about datacenter "maintenance" I think: changing the CRAC air filters, cleaning any outdoor coils, changing the oil on the generator, loading the generator, replacing old lead acid batteries, checking building integrity, making sure birds aren't nesting anywhere stupid, and so forth. Physical plant won't last forever.
this is my sig
See especially the sections "How equipment fails" and "Operating Context and Functions"
Although everyone makes mistakes, some people make hundreds of times more errors than others. Whether that's due to inherent lack of ability, poor training, lacking oversight, laziness, time pressures or just a slapdash attitude varies with each person. One place I was involved with (as an external consultant) made over 12,000 changes to their production systems every year. It turned out that well over half of those were backing out earlier changes, correcting mistakes/bugs from earlier "fixes" or other activities (a lot that resulted in downtime, and far too much of it unscheduled or emergency downtime) that should not have happened and could have been prevented.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
Talk like that will drive the economy into a (deeper?) recession.
If you have rushed, underqualified people do the maintenance, then sure, it decreases reliability. If you have careful, non-rushed and competent people doing it, I doubt very much that the same is true. These people tend to be a bit more expensive, but cutting cost in the wrong places is a traditional occupation of managers in IT.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Age old proverb, true today as it was a century ago.
All I want are systems with interchangeable fan/air inlet filters on the outside of the case that do not require a tool to remove and replace - let alone a power cycle. Is that so much to ask?
It's funny...I have cases where that sort of filter exists for a bottom-mounted power supply, but the case's own fans? Have to take 'em apart to properly clean the filters. And please don't say "Just lug a vacuum cleaner around." - they rarely do a good job if they actually are "luggable"; they (in a rare phrase) don't suck hard enough.
It is my experience that dirt and heat are the single greatest enemies of any electronic device, and will be as long as superconductivity without cooling is infeasible.
Orwell: "In a Time of Universal Deceit, telling the Truth is a Revolutionary Act"
... a very long time ago, we had a saying about this.
"It's called preventive maintenance because it prevents the computer from working".
I am sure that there are many other solipsists out there.
Is that Documentation requires time, and thus money - something Management just can't be bothered to allocate resources for.
This article misses the Real Problem by miles.
FAIL.
It's planned to survive for however long it's estimated to be used by the consumer. That's because the one determining factor is technological advancement. Progress is a moving target. It's reflected in the market place. Once we hit a brick wall in progress, the focus will return to reliability and planned long-life. Similar to how my old 1950s Sunbeam toaster will be passed on from generation to generation like a family heirloom.
There, fixed the headline for you.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
One bad automated update can lead to your system hosed or obscure reliability problems, perhaps not showing up for a while and the worst ones again leaving you with little option but to rebuild a system.
So I turn off auto update on everything I can, and manually update periodically. I consider the security risk smaller this way. I get it stable, and let it run that way for a few months at least. Then update security fixes etc.
These posts express my own personal views, not those of my employer
https://en.wikipedia.org/wiki/Preventive_Maintenance_Checks_and_Services
Or to use a more common example, think about changing the oil in your car every (time interval) or (distance interval). Will it stop failures? Maybe. Maybe not.
On the other hand, every time you "work" on a system you introduce entropy.
As long as you remove more entropy than you introduce, you should have a more reliable system (than if you hadn't worked on it at all). But that gets into the training/knowledge of the person performing the PM.
You only need a server for each item that is different. So if you standardize on hardware / OS then you only need 1 server to test hardware drivers and OS updates and so forth.
Beyond that, you really should have a test database system and a test app system. You never want to deploy updates into a production environment without going through a test system first (which is NOT the same as a development environment).
Virtual systems can help a lot with the server requirements. But you still need to understand the hardware / virtual / OS / app differences and plan accordingly.
I work for a global pharma company with several key data centers (DC) around the world, and they are interconnected with each other and 100's of client sites. The problem is that this interconnection infrastructure and the internal infrastructure at each site are in a constant state of flux as new requirements are met (technical, business, regulatory, etc), and old ones go away. Thus the apps and infrastructure and their interdependencies are always several steps ahead of keeping documentation up to date despite rigorous requirements (not only per good IT practice, but also per medicinal regulatory requirements).
There have been "unintended consequences" every time there has been such maintenance, from upgrading a few switches to full DC power recycles to check UPS batteries/generators, and other parts and pieces - "oh, the US network to the rest of the globe goes through THOSE switches now? ... They use WHICH LDAP server?" The complexity is beyond any of most ambitious and determined efforts of a lot of very smart people working their tails off trying to make it all go right using all the IT buzzword-compliant techniques - ISO 9000, ITIL, LEAN, Six Sigma, etc. That is exacerbated by personnel turnover (or just plain too much laying off per the bottom-liners) throwing new support people into unfamiliar circumstances as someone mentioned above.
We have to do this stuff, but we have not figured out how to keep up with all the moving parts as they keep changing...
YMMV
FTFY
The Admin and the Engineer
My super-cool Linux box at home, which I work on all the time, is much less reliable than my work desktop running XP, which I never touch except to do my job. The most reliable, of course, is the headless Linux server that sits under my desk at home and never gets touched. In fact, I have this separate server precisely because I know that I will mess up my desktop by trying to fix/maintain it all the time, and I intend never to touch the server unless something goes wrong.
http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/100000/20000/5000/600/125621/125621.strip.zoom.gif
This post contains no rudeness or derision of any kind. All arguments are friendly. Terms and exclusions may apply.
I remember they found that reducing the bank cleaning frequency increased to reliability of strowger exchanges (old school mechanical phone exchanges)
Whenever I hear this meme touted (and I've heard it a *lot* over the last 40 years) I immediately think -- someone wants to shave a few maintenance dollars, trading short-term gain for long-term pain.
Your money, your choice.
Do not mock my vision of impractical footwear
Having been in telecom for over a decade, I can attest to the fact that network reliability is closely related to the amount of change activity taking place in the network.
Just notice that there is some value in forcing your hardware to fail in a time your downtime will be cheaper. Also, if you are smart, you'll induce redundant components to fail in different times, so downtime will cost just the maintaince price.
Of course, places tha fail to see that they are antecipating failures by doing maintence won't ever plan for that.
Rethinking email
according to some industry experts.'The most common threat to reliability is excessive maintenance,' said Steve Fairfax of 'science risk' consultant MTechnology. 'We get the perception that lots of testing improves component reliability. It does not.' In some cases, poorly documented maintenance can lead to conflicts with automated systems, he warned. Other speakers at the recent 7x24 Exchange conference urged data center operators to focus on understanding their own facilities, and then evaluating which maintenance programs are essential, including offerings from equipment vendors
Well, yeah, now that you identified the weasel words and marketing speak, it sounds a lot less worse. In other news: government says it needs to expand itself, banks say you should put your money with them and Coca-Cola says they deliver a better product than Pepsi.
I clicked through to the website - I get such magazines for 'free' every month that praise vendor after vendor for their proprietary products to help manage some or other problem that is simply fixed by any decent sysadmin. The whole article is just fluff about common sense - yes, if you PM a component it's not going to be available in a double setup, that's why if you really need availability the mantra that a system is not truly redundant unless you have 3 independent systems - this has been long known by those that build high availability clusters (as in really high availability) but due to cost "savings" by upper levels it's often down to two systems or one real and one virtual system.
Custom electronics and digital signage for your business: www.evcircuits.com
There is also that update that fails and is stuck at an arbitrary percent regardless of everything you throw at it fails, short of lobotomizing the registry. With these kinds of experiences it's no wonder anyone would feel like they are sticking a firecracker in their baby for [kicks] and giggles.
Stop "That"!
Don't touch "It"!
You will go blind.
It's because Microsoft doesn't support replacing in-use DLL's. The primary reason for that is DLL's don't implement binary-compatible interfaces - an in-use DLL may have a different ABI than that of the new one.
The "reboot" requirement comes from this - to allow replacing of in-use DLL's, the system has a special registry entry that allows listing files in use to be moved by the kernel on reboot.
It affects Linux less as the main libraries are typically not only binary compatible, but binary compatible with previous versions that may have incompatible interfaces. Plus IPC tends to happen across sockets with well-defined interfaces, unlike that for random DLL's that interact through things like COM.
Anyhow, it's always good practice to after doing an update, rebooting the system twice. The first one applies afl the updates and makes sure things work. The second is to ensure things come up *again* in a normal reboot, and not one that did stuff like update DLLs and such.
The power systems comparison is in the old days (through about '92), we would frequently do building power-down maintenance at data centers. It started out at 3-year intervals, and was stretched to 5. Hundreds of electricians and several hundred IT staff would participate. Thermal shock killed 5% of the equipment on power-up, but everybody was standing by with spares. Everything would be cleaned, torqued, repaired, and tested. The DR systems would actually be put into action.
It took about 6 months of preparation, and cost roughly $4MM per site (200,000 square feet of raised floor).
Every few years there was a 'gotcha' moment, one year someone was killed, another the facility dropped load after restart due to human error. This all despite proper procedures and preparation. Today, with better UPSs, conical washers, and 2n designs,we eliminate the issues and get by with token IR scans.
Flooded batteries require maintenance. VRLA batteries require replacement. In both cases, a battery monitoring system measuring cell impedance goes a long way, but periodic discharge will tell more.
What is the definition of "failure". Steve Fairfax said, “if you buy a 2N data center, you’ll have twice as many component failures as a 1N data center. But you’ll be more reliable.”
More redundancy = higher reliability.
Rebooting does not require power cycling.
The longer a device runs, the higher the likelihood that some counter will overflow/wrap and cause odd/fatal behavior - the Windows 49 day bug is one example.
Projects tend to be late and over budget.
Call it maintainence to bypass oversight & embarassment.
At what point does 'mainainence' become: 'keep the paint - gut everything else'?
http://developergeeks.com/article/60/software-reliability-engineering
Until we started building our servers as VMs, I always thought - leave well enough alone. Patch when necessary, but don't mess with success.
Decoupling servers from hardware has been a huge help to our testing, backup, patching, and recovery processes.
Server borked? Restore the most recent snapshot on another machine - or better still have the hypervisor do it for you.
We even have our servers snapshotted before and after patching - just in case.
-ted
I used to work for a company that managed systems remotely. Everything from network devices, servers, etc. The sites all manage their own workstations and internal network devices, but we took care of the Enterprise. So... One day, while looking at my logs I notice an alert pop up for one site that indicated the network devices all went down. This site in particular is a small site that does not generally have people needing outside connectivity, but they pay to have uptime and are therefore just as important as any large site. Due to the polling process of our alert system, this means that there can be a 5 minute window of error on the alert. It could have happened anytime from the second the poll went out, to 5 minutes ago when the last poll went out. I investigate by attempting to log into the device, and am able to log in without any problems. I check the logs and see...sure enough, power failure. So, I call the site admin to make sure everything is alright, and the admin tells me...No power failure at the site. He checks the network room and finds nothing out of the ordinary. Well, this warrants further investigation just to see if there is any issue with the equipment...and because I am bored enough to check on anything at this point. So after looking at logs for the last two months, I find a very interesting fact. Power Failure tickets auto-gen at the same time each Monday and Wednesday for the last two months. Long story short, we find that the cleaning lady doesn't have a free power outlet to plug in the vacuum, so is unplugging the equipment, does her work in the small area, and then plugs the equipment back in. Since the cleaning lady has room clearance, and everyone in the building knows her, there simply was never a question when she went into the small room to clean. Now, I know its a different kind of maintenance than the OP is discussing, but I thought I would share a funny little story for your Monday. Hope you enjoy.
You must not deal with any Oracle database servers. They leak like a sieve.
I deal with four of them that run pretty much 24x7x365. Running on Window 2003 Server 32-bit no less. Only one of them undergoes weekly reboots to deal with a problem, that ironically is a Microsoft leak, and has nothing to do with Oracle. The other 3 boxes run until something forces a reboot, generally about once a year.
These Oracle servers run critical roles a 911 public safety emergency dispatch center too.
I also run Oracle on Linux and Oracle on AIX and those are even a lot more stable than the Oracle/Windows boxen. The Oracle/AIX server (running a legacy utility billing system for a municipal govt) once ran nonstop for almost three years uninterrupted before it was shutdown due to a new UPS/generator system being wired into the data center.
If you cannot run an Oracle database server stably, you're doing something either very wrong... or very unusual.
I've been an Oracle admin for over 15 years and although Oracle is sophisticated and complex, it's not insurmountable.
heck, i discovered a while ago, that every hour my car spent at the mechanics needed an additional half hour on my part to fix whatever the mechanic screwed up in his work; rerouting the wires which had been left dangling on the exhaust manifold, replacing missing fasteners, etc.
I can't remember where I heard this, but the more I think about it, the truer it becomes.
Hardware is like an erect penis - it will stay up unless you fuck with it.
Corollary 1) Of course, paying it a bit of attention every once it a while will be necessary
Corollary 2) Nothing lasts forever, no matter how good it is at the start.
Perhaps he needed to reach a minimum pagecount?