Slashdot Mirror


The Decline and Fall of System Administration

snydeq writes "Deep End's Paul Venezia questions whether server virtualization technologies are contributing to the decline of real server administration skills, as more and more sysadmins argue in favor of re-imaging as a solution to Unix server woes. 'This has always been the (many times undeserved) joke about clueless Windows admins: They have a small arsenal of possible fixes, and once they've exhausted the supply, they punt and rebuild the server from scratch rather than dig deeper. On the Unix side of the house, that concept has been met with derision since the dawn of time, but as Linux has moved into the mainstream — and the number of marginal Linux admins has grown — those ideas are suddenly somehow rational.'"

14 of 500 comments (clear)

  1. From personal experience by Xacid · · Score: 5, Insightful

    "they punt and rebuild the server from scratch rather than dig deeper."

    From personal experience this is normally due to management jumping down our throats to simply "get it done" which unfortunately runs counter to our inquisitive desires to actually solve the problem.

    I suspect it's the end result of pressure to get more bang for their bucks in a tight economy, but that's pure speculation. It really could be a trend of the times.

    1. Re:From personal experience by Nerdfest · · Score: 4, Insightful

      As I've said below, there is a benefit ... you can actually investigate and fix the problem rather than the symptom. The bonus with VMs though is that you can frequently do both. You can create a copy of the VM tio dig into, and create a new fresh instance for production to get them working again.

    2. Re:From personal experience by Darth_brooks · · Score: 4, Insightful

      ....and his was the right answer. With XP, you're almost certainly talking about a client machine. Why bother dicking with it? It's a hundred dollar OS on a four hundred dollar piece of hardware. Wipe, reload, move on to big boy problems. Even if you're talking about a problem that ends up affecting a number of users, and it happens to be a client side problem, you're farther ahead to nuke and reload.

      In my last position I was the only end user support guy for 150 to 200 people. If I sat around and fucked with every little nuance of XP and it's associated ills, I'd have ended up even farther behind than I was when I left. I wrote up a quick backup script that grabbed anything the user didn't (against company policy) store on the network drive, grabbed their local e-mail (Notes), then nuked the machine and reloaded. I could take a user who was dead in the water and have them back up and running in 15-20 minutes. If they had a lot of data to restore, maybe 35-45. Spending an hour 'troubleshooting' was a waste of company time, and my time.

      --
      There are some people that if they don't know, you can't tell 'em.
    3. Re:From personal experience by Tom · · Score: 4, Insightful

      In an environment of thousands of servers (or even dozens), deep-diving into a problem [generally] is a waste of time. While it is interesting intellectually, there is no other benefit.

      Except, of course, finding what the heck was wrong in the first place and fixing it, preventing future outtages.

      Sometimes, rebuilding is faster than fixing, and in some contexts, it makes sense. Even then, the original machine should still be examined and the "root cause" (if you need a management buzzwod) identified. At the very least, a reasonable amount of time should be given towards the attempt. It's true that it is pointless to dig around for days and days - but that is not a reason to not at least start looking, as it might turn out you only need a few hours. And more often than not, finding the real problem tells you something that helps you
      a) fix other bugs,
      b) avoid the same problem on the next server,
      c) avoid a repeat performance,
      d) makes you realize what you thought was a random server crash was really a break-in / hardware failure / systematic problem and other, additional steps need to be taken.
      All of the above have happened before, you would by far not be the first.

      A proper incident management process does allocate resources towards follow-up examination. The right thing to do is not suppress it with generic blabla about wasted time, but to set the proper amount of resources for your organisation. Maybe it's half an hour and no money, so some sysadmin can check the logs and do a quick check-up. Maybe it's a full-out forensics analysis. That depends on your needs, your resources, your environment and context.

      --
      Assorted stuff I do sometimes: Lemuria.org
    4. Re:From personal experience by causality · · Score: 5, Insightful

      Reminds me of our Windoze guy in my previous job. I had run into some kind of problem with XP at home and spent a good amount of time banging my head against it. Finally I told him what was going on and asked what he recommended. "Reinstall XP" was the answer. I thought he was joking, nope he was serious. :P

      Not only was he completely serious, he probably can't understand why you might have thought he was joking.

      The idea that it's a black box and you shouldn't expect to understand how or why something happened is definitely one of the more subtle costs of Microsoft systems. It lends credibility to the (false) notion, so common among average users, that you're either a completely unskilled newbie or a serious expert who can discern the inner workings of the mysterious black box. It discourages middle ground for intermediary skill levels, the kind of thing that would otherwise occur naturally as users gain experience over time.

      Most of all, it's supports the falsehood that it's unreasonable to expect the most basic competence from non-experts.

      --
      It is a miracle that curiosity survives formal education. - Einstein
  2. Re:Gee, ya think? by rhsanborn · · Score: 5, Insightful

    There are a lot of cases where pressing the button means that the problem will go away...for a few weeks. It will work right until you hit the same conditions that caused the problem in the first place. Suddenly, your using the refresh to cover up either a poor implementation, or a standing bug, and it isn't going to go away until you call that guy in suspenders.

  3. Re:Clone my car! by shawb · · Score: 5, Insightful

    The real solution? Reimage the production server to just get it working, then you dig around on the dev server until you find out what's actually going on.

    --
    I'll never make that mistake again, reading the experts' opinions. - Feynman
  4. Re:Clone my car! by Ephemeriis · · Score: 5, Insightful

    The real solution? Reimage the production server to just get it working, then you dig around on the dev server until you find out what's actually going on.

    Exactly.

    If the machine is in production it needs to be working. You don't have time to dig around and find the root cause. You need it to work. Now. If you've got a virtualized environment it is trivial to bring up a new VM, throw an image at it, and migrate the data.

    Then you take your old, malfunctioning VM into a development environment and dig for the root cause, so that you don't see the same problem crop up on your new production machine.

    --
    "Work is the curse of the drinking classes." -Oscar Wilde
  5. Re:Clone my car! by Isca · · Score: 4, Insightful

    That's assuming your new tool that's vitally important actually has a man page. Very little is documented as well as it was 10 years ago.

  6. endless cycle by roc97007 · · Score: 4, Insightful

    I'm not sure I buy everything in TFA, but have to admit to a certain extent this phenomenon is real. I've noticed, however a tendency to regenerate an instance, and when it doesn't work regen it again, and again and again because the purposely overextended and/or undertrained admin doesn't have time to figure out that the problem is in his template or due to something external like a dup ip. Come to think of it, this type of endless cycle seems to be fairly common in the Windows world. I guess we've caught up.

    Sometimes the user has to diagnose the problem themselves, which is a win for the IT manager because the time didn't come out of the IT budget.

    I'm hoping that at some point these practices will be recognized as the false economics they are. But I'm not holding my breath.

    --
    Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
  7. Re:Hyperviser by __aamnbm3774 · · Score: 5, Insightful

    This whole argument is retarded. I always pick the most appropriate response to the problem at hand. If your server is hosed and not booting, I don't have time to mess around with some Knoppix DVD, trying to figure out exactly where in the boot process it is dying. Especially if you have nightly backups! Sometimes a clean sweep and restore is perfectly acceptable and reasonable. Why even sacrifice downtime trying to troubleshoot an issue that could be resolved within minutes?!

    Now, if it happens again the following night, you do have a deeper problem and should investigate it further, because constantly restoring the machine is now the inefficient part in the process.

    It's like we've lost common sense in favor of our technical ego.

  8. Re:Hyperviser by jc42 · · Score: 4, Insightful

    ... documentation for a GUI, if it exists at all, is often useless..,

    How true. There popular explanation of the difference between a CLI and a GUI is that CLIs are so complicated that you need a manual to use it, whereas GUIs are so simple and intuitively obvious that no manual is needed.

    Of course, the reality is that this attitude allows vendors to supply GUIs without any "unnecessary" manuals, but make the nested tree of windows and menus so deep and complex that nobody can ever remember where everything has been hidden, and there are no good tools to help you find something that you know is in there somewhere.

    Meanwhile, the people who build the CLI know that nobody can ever remember it all, so they include tools for finding your way around. They also tend to make the defaults for the commands fit the most common cases, so you don't have to use the manuals all that often. And most tools have a -help option (though they can't quite agree on how to spell it), to provide quick reminders. And the CLI includes a current directory, search paths and aliasing, so you don't have to remember full paths to everything.

    One of the ongoing frustrations with every GUI is constantly seeing a new window pop up, which is positioned back at the root directories, and I have to laboriously poke at things to get down to the directory that I'm working in. Then, when I do what the window was opened for, it closes, all that navigation is lost, and I have to do it all over again the next time I want to access a file in the same directory.

    GUIs may have some aesthetic appeal (aka "pretty pictures"), but they remain the slowest, clumsiest way to use a computer that we've yet developed. But I trust that people are working on finding ways to make it even clumsier and slower. This seems to be happening with the "cloud" approach, for example.

    --
    Those who do study history are doomed to stand helplessly by while everyone else repeats it.
  9. Production enviroments by DerekLyons · · Score: 5, Insightful

    Some of the comments here remind me of a post on a woodworking board a few months back. Essentially, the poster was lamenting because he had to fire a guy because he couldn't afford to keep him... Not because of the economy, but because the guy was an absolutely inflexible perfectionist. He'd spend $300 worth of time on what should have been a $60 job... The guy was a hell of a woodworker, at home in his own shop, but just couldn't adapt to a production environment.

    This isn't about Windows vs. Unix. This is about admins not understanding their job is to get production rolling again, not to satisfy their obsessive need to understand every problem or their need to satisfy their ego. ("I'm a UNIX admin dammit, I refuse to use habits that make me look like a Windows admin" or it's equivalent is a refrain modded up again and again here on Slashdot.) If a reboot or a re-imaging fixes the problem, that's the right solution. If it doesn't, *then* you dig deeper.

  10. Re:Hyperviser by drsmithy · · Score: 5, Insightful

    For example, Linux is extremely easy to use -- if you understand it. Windows is a hell of a lot easier to learn but knowing all about it won't make it much easier to use.

    That, is entirely a matter of opinion.

    Your comment there describes what is easy to learn.

    No, it doesn't. Your comment assumes that an interface should *have* to be learnt, to be easy to use.

    The CLI appeals to people who are willing to learn, who like learning new things and consider it worthwhile.

    No, the CLI appeals primarily to people who like to focus on memorising semantic minutiae and believe that doing so is, in and of itself, a productive endeavour.

    The terminal is for non-trivial tasks.

    The implication that GUIs are only used for "trivial" tasks is ridiculous on its face.

    The average Windows user who views learning as an unreasonable burden that should never be expected of anyone who wants to use a complex machine ... they avoid the up-front investment of learning to understand the system. Instead, they can jump in and start using the system right now. But they continuously pay for it over time in the form of enjoying few or none of those advantages.

    There is nothing unique to Windows, or even computers, about this. Do you know the intricacies of how your car works ? How about your blender or oven ? Could you fabricate a new bed or sofa from raw materials, and without modern tools ? Do you grow your own produce ? Could you butcher a cow or chicken ? Could you set a complex fracture or create your own painkillers ? Can you brew your own beer ?

    It's like the difference between people who live within their means and use plastic only as a form of payment, saving up until they can actually afford something before they purchase it, versus those who live all the time on credit. The person living on credit gets the stuff they want right now but ultimately pays quite a bit more for it and can quickly find themselves in over their head. The discipline and delayed gratification that the latter is trying so hard to avoid is something that the former considers to be virtues worth cultivating.

    No, it's nothing like that at all. One is an example of financial irresponsibility and the other is simply realising that you do not need a deep and intricate understanding of a given thing to use or take advantage of the services or benefits it provides.