Ask Slashdot: System Administrator Vs Change Advisory Board

thundergeek (808819) writes "I am the sole sysadmin for nearly 50 servers (win/linux) across several contracts. Now a Change Advisory Board (CAB) is wanting to manage every patch that will be installed on the OS and approve/disapprove for testing on the development network. Once tested and verified, all changes will then need to be approved for production. Windows servers aren't always the best for informing admin exactly what is being 'patched' on the OS, and the frequency of updates will make my efficiency take a nose dive. Now I'll have to track each KB, RHSA, directives and any other 3rd party updates, submit a lengthy report outlining each patch being applied, and then sit back and wait for approval. What should I use/do to track what I will be installing? Is there already a product out there that will make my life a little less stressful on the admin side? Does anyone else have to go toe-to-toe with a CAB? How do you handle your patch approval process?"

Comments

Nonsense

[ruir]

They want bureaucracy, they make the paperwork. Tell them to track windows and distro security pages, the changes are there. I would be toasted with that kind of tape, I updated my servers in a pinch immediately after the first news of heartbleed at 3 in the morning. 0300AM right. How about dusting your resume and changing jobs? Let them play the shuffling reports game alone.

Re:Nonsense

[N1AK]

Any remotely well organised IT department will have processes for handling both emergency deployments and retrospective approval. I'm not going to be cheerleader for the concept of CAB but if you're going to make a case against it then at least make a reasonable one because hiding behind obvious nonsense like this will just make you look stupid and change averse to your employer.

Re:Nonsense

[Anonymous Coward]

Any decent change control process should have an emergency change category and process.

Re:Nonsense

[Anonymous Coward]

Somehow reminds me of that joke where initially there's just one worker, then layers and layers of staff are added to manage that worker, then finally the worker is fired for underperforming.

Can't find it on Google or Bing though for some reason.

Re:Nonsense

[sg_oneill]

Back when I worked as a web administrator at my local university back in the early 2000s, the admin make-work types decided to bash out a web policy , mostly to keep standards up and guard against legal liability (Admittedly we had students setting up websites on chemistry lab pcs turned webserver with novel meth recipes and all sorts of shenanigans before that). All good and fine, I asked to be on the committee as an advisor, and so I was.

Then the whole thing went off the rails, every page needed to be approved by a department head, 10,000+ pages of previously existing data had to be retrofitted with full dublin core metadata descriptions, and so on and so on for about 400 pages of rules and policy that despite my best efforts I could not stop. These people had no fucking idea.

The crown was an insane rule that every new hyperlink had to be aproved not just by a department head but by the vice chancellor himself.

And so thats what I did, and I made sure it was done good and proper. I wrote a perl script that took all new pages on the webserver network (about 50-100 new pages a day) and then whenever a hyperlink appeared it spat out a 1 page document for approval *per link* requiring the vice chancellor and a lawyer to co-sign off on. All with witnesses. All in all about 400 pages a day of paperwork for the vice chancellor and a lawyer.

The policy lasted 3 days before I was dragged into the admin building to be ordered to stop producing the reports. I went in with my union rep. I said "Sorry , no , thats the official policy as passed by the university senate and the website will need to be shut down if this isn't done.". Since the next senate meeting was two weeks away, I made sure every god damn day that stack of paperwork was done by the vice chancellor for a glorious fortnight before the senate could revoke the whole damn policy.

It was a magical and golden time to be a union protected government (Universities are mostly run by the state in australia) employee.

For some reason later that year I was passed over for a promotion though. I wonder why, lol.

Re:Nonsense

[Opportunist]

This. Ask them if they have taken care of things like this. The answer to this alone will tell you whether there is some kind of deep consideration behind it or whether some PHB had a consultant toss the cool buzzword "change advisory board" in his direction.

If it's the latter, run. Run like the wind.

Re:Nonsense

[JosKarith]

You should have discussed being passed over for promotion with your union rep - you'd have had a pretty strong case.

Re:Nonsense

[mikelieman]

Pretty much this. Change Management is a process. I wonder if they even have any systems in place to manage this. Tracking migrations really benefits from a good system behind it. Maximo is however, not that system.

Re:Nonsense

[timepilot]

Dr. Seuss: “Oh, the jobs people work at! Out west near Hawtch-Hawtch there's a Hawtch-Hawtcher bee watcher, his job is to watch. Is to keep both his eyes on the lazy town bee, a bee that is watched will work harder you see. So he watched and he watched, but in spite of his watch that bee didn't work any harder not mawtch. So then somebody said "Our old bee-watching man just isn't bee watching as hard as he can, he ought to be watched by another Hawtch-Hawtcher! The thing that we need is a bee-watcher-watcher!". Well, the bee-watcher-watcher watched the bee-watcher. He didn't watch well so another Hawtch-Hawtcher had to come in as a watch-watcher-watcher! And now all the Hawtchers who live in Hawtch-Hawtch are watching on watch watcher watchering watch, watch watching the watcher who's watching that bee. You're not a Hawtch-Watcher you're lucky you see!”

Re:Nonsense

[Antique Geekmeister]

> Any remotely well organised IT department will have processes for handling both emergency deployments and retrospective approval

Not when the architect is offline and is needed for every significant change. If there is going to _be_ a policy, a manager needs to be ready to enforce it, or it's going to be everyone making up their own undocumented and impossible to synchronize policies.

Re:Nonsense

[fatp]

In the version I heard, the worker was fired because the department is over-staffing.

Re:Nonsense

[RabidReindeer]

They want bureaucracy, they make the paperwork. Tell them to track windows and distro security pages, the changes are there. I would be toasted with that kind of tape, I updated my servers in a pinch immediately after the first news of heartbleed at 3 in the morning. 0300AM right. How about dusting your resume and changing jobs? Let them play the shuffling reports game alone.

I've served on a change control board. Every application and system update was supposed to be bundled to make the sysadmin's job easier, include a document that outlined the nature of the change and why it was needed, the instructions on how to apply the change, and the instructions on how to recover if it didn't work.

Change committee met once a week, approved/scheduled, deferred, or rejected changes. In case of emergency, the CIO or designated proxy could approve an out-of-band change request.

We didn't attempt to micro-manage changes, just understand the business risks and rewards. Obviously, the more details you could capture the better prepared you were to understand the consequences and the ways you could recover. But when Microsoft hands you a CAB that includes patches for SSL, IE, 6 GDI bugs and Windows notepad, that's their problem, not yours.

The one thing that we didn't do (obviously!) was allow automated Windws updates. Then again, considering the damaged that some Windows Updates have done to desktop machines, I didn't even allow that on my desktop machine.

Re:Nonsense

[OzPeter]

So... the business made a stupid decision, and when they realised the error of their ways, rather than trying to reach agreement on the best way forward, you delighted in rubbing their noses in it, using processes designed to protect you to hurt your employing organization instead.

If he had said .. "OK .. sure I'll stop sending you those 400 pages of paper per day", then the policy would still have been left in place, and sometime win the future his employer could have used his inability to follow policy as an excuse to ream him over. Yes its CYA, but some employers are not above using any tool at their disposal to justify their actions.

Only by being a genuine PITA does the stupid police get removed, rather than ignored until convenient.

Re:Nonsense

[mysidia]

like this will just make you look stupid and change averse to your employer.

No... it's obviously just aversity to excessive, unnecessary and crippling micromanagement. It's obviously some idiots in suits who are change averse and feel they need to justify their existence by "approving" or "disapproving" of each and every required security update or patch or system admin action.

Which involves real costs. With this kind of bullshit, they need to hire additional system admins for systems to approach proper management just to deal with the reduced time efficiency and increased waste caused by bureaucracy.

Re:Nonsense

[rjune]

You should have asked for them to put that in writing. In fact, you should have made a written request for them to give you a written directive to stop producing the reports. Failing to generate the required reports would have given them grounds to fire you. What a glorious fortnight of rubbing their noses in their "Official Policy"!

Re:Nonsense

[Threni]

> The one thing that we didn't do (obviously!) was allow automated Windws updates.
> Then again, considering the damaged that some Windows Updates have done to
> desktop machines, I didn't even allow that on my desktop machine.

You have to perform OS updates in some industries. You might disable automatic updates, but that doesn't prevent damage, just that you'll be kicking it off manually.

Re:Nonsense

[Registered Coward v2]

So... the business made a stupid decision, and when they realised the error of their ways, rather than trying to reach agreement on the best way forward, you delighted in rubbing their noses in it, using processes designed to protect you to hurt your employing organization instead.

If he had said .. "OK .. sure I'll stop sending you those 400 pages of paper per day", then the policy would still have been left in place, and sometime win the future his employer could have used his inability to follow policy as an excuse to ream him over. Yes its CYA, but some employers are not above using any tool at their disposal to justify their actions.

Only by being a genuine PITA does the stupid police get removed, rather than ignored until convenient.

While what you say is true, it is not necessarily the best way to deal with stupid policies. In the end, the PITA gets remembered long after the stupid policy is forgotten; all people know is someone was a PITA and eventually will be made to pay for it. You could, for example, send all links on one page and have that be signed. After a few were signed its's time to broach the "is there a better way to do this" argument. Malicious compliance with rules generally results in pissing people off and payback at some future point; it can even be fun watching the PITA get a taste of their own when the opportunity arises.

Re:Nonsense

[MachineShedFred]

It should also have a method of implementing a "standard change" - meaning a change that repeatedly happens, and has a good track record of being successful with minimal fallout. Implement a WSUS test server for having a patch test environment, and point some less critical systems at it. Should anything go wrong there, you'll then know what patches to NOT apply automatically to the critical infrastructure.

Change management and review is there to help you, not be a pain in the ass. Remember - if they sign off on your documented change, they share the responsibility should something go sideways.

Re:Nonsense

[GlennC]

+1....well played, sir!

Re:Nonsense

[plopez]

I wish I had a union rep.

Re:Nonsense

[plopez]

Which I believe required Senate approval...

Re:Nonsense

[Ash Vince]

They want bureaucracy, they make the paperwork. Tell them to track windows and distro security pages, the changes are there.

Yep. They're the "experts". Just tell them the Microsoft KB number, that's all the information they need.

Yup, follow this advice and come across like an unhelpful douchbag.

Or, bend over backwards to help them. Provide them with a break down on every single patch (a few line summary with a link to the KB article for the full details), then give each patch a priority based on its impact and come up with different deployment routes for each one, then explain to your manager who allocates your time why patch management for the CAB board just became a full time job.

Also, if they ever reject any and you end up a dependency hell where you cannot install a critical patch because of a low impact one you rejected (you do test each patch deployment run on a dummy server don't you?) then explain why the process failed, politely, without saying thing like "I told you this was dumb years ago!".

Alternatively, if the system runs for a few months and every single patch sent to the CAB board has been approved then you can clearly demonstrate the do not really add anything and start making rational arguments to abandon the process from a sound basis while demonstrating you are an excellent team player who easily adapts.

But if you would rather come across like a non-team player who hates any interference in your system admin fiefdom then, just go with the douche bag option and watch your job get outsourced in 6 months.

In my experience the world of work is full of crap like this, times when processes that are overly bureaucratic are forced on us techies even though we clearly see them as a waste of our time. Unfortunately this is generally just stuff we have to lap up as part of our job, if you can, you generally end up earning more and with the greater long term job security that working as part of a larger company provides.

An excellent book on this sort of business related stuff is called "Who moved my cheese" and the gist of it is that you want to come across as and "enabler" rather than a "blocker". That often means trying your best to make what is clearly a stupid idea a success.

Re:Nonsense

[MrNemesis]

So refusing to comply with an order that's in direct violation of your contract is acting like an arsehole now? And you're happy for the rule you're being routinely forced to violate in the course of your professional duties to be left on the books to trip up the next person who doesn't have the guts to stand up and say "no, I won't shoot myself in the foot"? Will HR even remember you have a signed waiver before marching you out of the building for knowingly violating company policy?

Sorry, but no. If you're stuck in a Kafkaesque situation like the GP was, the only professional thing you can do is give them exactly what they want. Especially when you've explained to them why giving them exactly what they want will be bad and they've given a written response that amounts to "we don't care, do it anyway".

If you act like something that badly needs fixing doesn't need fixing and you're happy to see people and companies ruined over it, but all means keep thrusting ones cranium into the pulverised silicon dioxide. Some people might say you're acting like an arsehole however.

Re:Nonsense

[Anonymous Coward]

This. We have a CAB. We got patches to be considered a routine process with specific scheduling - DEV on day X of the month, QA on day Y of the month, and Production split between days A and B. Submissions to CAB for automatic patches are FYI and made in batch.

Re:Nonsense

[Anonymous Coward]

The point of doing it manually is that you choose the time when you do the updates and what updates you want to install as opposed to automatically just installing every update Microsoft thinks is critical overnight on patch tuesday. This allows you to wait a few days to understand exactly what these updates are going to break for everyone else before you go and break your own shit.

We have a webserver which has had a critical update for Office waiting for almost two years now. We don't install this update because it will break SharePoint content deployment and then everything is fucked. See? We prevented damage.

Re:Nonsense

[Lonewolf666]

You can delay roll-out of the update while it goes through testing. Now I've never worked in a company that did this consequently, to the best of my knowledge. But it is an option.

Re:Nonsense

[PrimaryConsult]

How is it his job to come up with the better solution? It was a legal/paperwork issue, not a technical one. If the Vice Chancellor and lawyer did not want to sign all that paperwork, they were the ones who needed to offer up the alternatives.

Re:Nonsense

[RabidReindeer]

> The one thing that we didn't do (obviously!) was allow automated Windws updates.
> Then again, considering the damaged that some Windows Updates have done to
> desktop machines, I didn't even allow that on my desktop machine.

You have to perform OS updates in some industries. You might disable automatic updates, but that doesn't prevent damage, just that you'll be kicking it off manually.

Not always. Sometimes it's a matter of letting other people be your guinea pigs. Microsoft has on several occasions had to follow up an update with a corrected update, so in cases like that, you just ignore the bad update entirely and skip ahead to the better one.

The other thing deferred updates allow is the ability to do small-scale in-house testing before letting it loose on the entire infrastructure.

Re:Nonsense

[Ash Vince]

The crown was an insane rule that every new hyperlink had to be aproved not just by a department head but by the vice chancellor himself.

At that point you should have just emailed everyone on the committee, and copied in the vice-chancellor with some stats on exactly how many approvals this would generate on a daily basis. Include the actual statistics for the previous 7 days so if this was generating 50 pages per day you had some clear number to back this up while still in the planning stage. That was clearly why you were put on the committee, to stop a bunch of know nothings from coming up with a stupid policy, you failed.

The way to succeed as a techie is not about being technically brilliant any more, it is about how you can talk people round to your way of thinking and use evidence to back up your points of view.

Re:Nonsense

[nabsltd]

You could, for example, send all links on one page and have that be signed.

Since we don't know exact details, it's possible that the official wording was something like "each link has to be individually approved".

Even if that wasn't the case, with 400 or so new links per day, that would be 5-8 pages depending on font size, margins, etc. Sure, the powers that be could just rubber-stamp the process without actually reading and investigating each link, but then what happens when one of links is an issue (points to porn, material copyrighted by big media, etc.)? Before, they could just write it off as a mistake by some low-level web coder. With the signature of the vice-chancellor on it, it pretty much becomes officially endorsed by the university.

Re:Nonsense

[plover]

"Yes men", or "enablers" as you call them, are the root cause of the out-of-control bureaucracy problem.

Management wants desperately to have processes that can be followed by minimum wage desperate people. They want to believe that they're incredibly smart and insightful, and that their knowledge of their business is so absolute and perfect that a process is easily applied to every situation. The enablers then create the real problems by saying "yes, we'll make this work," when they are actually lying. The problems then get worse, because the enablers also feel that every status report must be green, otherwise their process is not as perfect as they said it was.

The transparent lies breed more processes to control the mistakes that never seem to get fixed, despite the change review board processes. Nobody ever questions the process, because they'd be seen as a blocker. Change becomes impossible, because the middle-management process owners who have been lying "yes" will lose their jobs if their process is removed.

The process explosion reaches critical mass as each failed process begets two more to control it. The business goes into a death spiral of bloat and inefficiency.

Everything is painful. The smart middle managers flee early, leaving only the enablers behind, and they refuse to see or acknowledge any problems. The loyal people have their jobs turned into paperwork and outsourcing. With luck, the board will recognize the out of control costs, and bring in a lean outsider. An organization this bad off may need to fire 50%-90% of the people. Or the board may go the wrong direction, and outsource the whole damn mess, further eroding their ability to change.

This philosophy has caused more damage to business and productivity than any other idea, ever.

Avoiding this is simple, in theory. Tell the truth, and don't be afraid to change your mind if shown evidence that you're wrong.

Re:Nonsense

[mlts]

Another thing that might happen is that change management gets selectively enforced. One set of machines would be scrutinized where every change, even an addition of a drive to an array, would require a meeting and people signing off on the change, while the machines running a different OS would be able to be taken down, reinstalled, or otherwise modified at will without any paperwork needing to be done. (And vice versa.) Even SANs need to be documented because if someone puts both paths of a production box's MPIO links on the same drive controller, then reboots the controller, there will be Hell to pay.

Change management needs to be even across the board, be it SAN configurations, Windows, UNIX, router configurations, ASA rules, phone switch configs, VMWare configurations, and so on. If one group starts getting a free pass, then the whole system ends up being pointless come an outage that ends up being traced to undocumented changes in a part of the company that has gotten carte blanche.

Change management in even a SMB requires someone dedicated to the task of dealing with documenting changes. It requires a dedicated server, change management software, and someone who will maintain/backup/archive that. That server will be a PITA... until an outage happens and the fingers start pointing. Then, it can save a person their job.

Ideally, the change management software should allow people to put in their own changes. Say an admin is changing passwords or moving files from one filesystem to another. Might as well have a tool where items like that can be documented. Same with calls to a vendor for support, so later on, if something breaks, a simple search might come up with historical data.

All and all, a change management system is a good thing. However, it needs to be universally enforced with various grades of policies (emergency fixes can go on without approval, for example) for it to be of any good.

Re:Nonsense

[mlts]

There is a balance. If you bend over too much and let them "do your job", there may be grave issues several months down the road.

The problem with that is the "it happened on your watch" statement that will be uttered come any calamities in the future. The patch they rejected that causes an outage later on won't fall on their heads. It will fall on the sysadmin's head. Even though it won't be the sysadmin's fault, they will get fired because management has to appear to do something, and the sysadmin was in the driver's seat.

One can't be a complete douchebag, but one can't just cede control over completely. If push comes to shove, it is better to get laid off because a H-1B is taking over than be fired for cause.

Re:Nonsense

[Registered Coward v2]

How is it his job to come up with the better solution? It was a legal/paperwork issue, not a technical one. If the Vice Chancellor and lawyer did not want to sign all that paperwork, they were the ones who needed to offer up the alternatives.

Perhaps because offering a solution to a problem rathe than just being a PITA in hope sit will go away is the professional thing to do? When someone just brings up issues without offering solutions it quickly becomes apparent they are not capable of the level of thinking needed to advance. They may be good at what they do but that's all they'll ever do; that may not be fair but it's the reality of most workplaces.

Re:Nonsense

[Registered Coward v2]

You could, for example, send all links on one page and have that be signed.

Since we don't know exact details, it's possible that the official wording was something like "each link has to be individually approved".

Even if that wasn't the case, with 400 or so new links per day, that would be 5-8 pages depending on font size, margins, etc. Sure, the powers that be could just rubber-stamp the process without actually reading and investigating each link, but then what happens when one of links is an issue (points to porn, material copyrighted by big media, etc.)? Before, they could just write it off as a mistake by some low-level web coder. With the signature of the vice-chancellor on it, it pretty much becomes officially endorsed by the university.

True, but who is really going to investigate each and every link? That would be a full time job. My guess is that someone though "if chancellor approval is needed people will be careful about where they link..." when in reality nobody worries about that and now have official OK for the link as a CYA. Which is why I think providing a list of the links requiring approval and asking "do you really want to do this?" and suggesting letting Departments approve them on their own, holding them accountable for what they approve, and running spot checks for obvious problems is a better solution then becoming a PITA.

Re:Nonsense

[rnturn]

Having patches approved by a CAB should not be a big deal. A brief write-up of the patches to be applied -- or an attachment listing the patches, reasons for applying them, etc -- was all that was required. Every CAB I've ever worked with has a procedure for an emergency like applying a patch for something like Heartbleed. All it usually took was a phone call to certain people and getting a verbal authorization. (You filled out the standard change request forms after the fact.) Working with a CAB is no big deal. Really.

But speaking of pointless paperwork... We had someone in a QA role stand up in front of the IT group and tell us that they wanted a screen shot of every single patch installation for every single server the patch was installed on. (And the rest of the QA team nodded their heads in unison like robots.) When it was pointed out that the length of time required for making a separate screen shot -- signed and dated by hand to boot -- for each of the patches in your typical Microsoft service pack times hundreds and hundreds of servers and that such a process would be prohibitive (to say the least) they eventually backed off. If that initial request wasn't bad enough, they actually wanted the process to be: Install the first patch, take the screen shot, print it, label it, sign and date it. Only after those steps were completed would you move onto the next patch or server. If their plan had been implemented the company would have had to build a new building just to house the printed screenshots.

Re:Nonsense

[Anon-Admin]

I have worked for those companies. lol

27 page CAB form and full CAM meeting. Just to edit the /etc/login.defs and change PASS_MIN_DAYS from a 0 to a 7.

I still laugh about it to this day. A single character change and 27 pages of paperwork.

Re:Nonsense

[Anonymous Coward]

You missed the point where he was in a union shop. The reason for this kind of policy is legal protection, but unfortunately those who prefer to make everyone else work tend to miss the mark in brilliant ways. This was a horrible policy to begin with, and if the VC hadn't been signing off on it then someone else would be 'responsible' to continue tracking hyperlinks. Poo rolls downhill, which is why union protection is important in these areas.

Re:Nonsense

[Minwee]

Somehow reminds me of that joke where initially there's just one worker, then layers and layers of staff are added to manage that worker, then finally the worker is fired for underperforming. Can't find it on Google or Bing though for some reason.

That's because the worker was really an ant..

Re:Nonsense

[Anonymous Coward]

I worked for a Federal Government Prime Contractor. The Oracle DBA wanted to approve every patch on the Solaris servers that I was sysadmin-ing. Not the Windows servers or the Red Hat servers, just the Solaris servers. At that time Sun was not owned by Oracle. I would download the patch descriptions (both security and OS) and give it to them to approve or disapprove--all or some of the patches. Then I would sit back and wait. This took the patching responsibility off my plate and put it onto their plate. With Government Contractors, the Blame Game is a big deal and they gave me a "free pass" on responsibility. They choked on the paperwork. Eventually, they asked an outside contractor to approve the patches for them--which was quite funny since their "experts" agreed with me.
The Oracle DBAs were replaced by a CAB (which at least one Oracle DBA sat on) and the process continued. The job got easier and easier and I was given raises each year and stayed on with the Contractor.

So, the CAB relieves you of responsibility. Give them the paperwork that you get without lengthy description. (Unless, of course, you like doing the paperwork rigamarole OR doing the paperwork gets you out from under a more disagreeable task. "I can't do that. I have CAB requests to write up and they meet tomorrow.") So, if someone exploits a "hole" before the patch is approved and applied, it's not your fault--how can it be; you followed the procedure (and now you get honoray ISO 900x certs).

Every problem is a blessing in disguise.
You see the CAB procedure as a liability; "they" see it as a "valuable assest". Change your viewpoint and get on the bandwagon (at least at work). It can make your work life easier.

Remember we sysadmins are control freaks. The company has just given you "free" control freak remedy counseling--be thankful.

Re:Nonsense

[Slashdot Parent]

Yup, follow this advice and come across like an unhelpful douchbag.

Or, bend over backwards to help them. Provide them with a break down on every single patch (a few line summary with a link to the KB article for the full details), then give each patch a priority based on its impact and come up with different deployment routes for each one, then explain to your manager who allocates your time why patch management for the CAB board just became a full time job.

So you advocate becoming a passive-aggressive douchebag instead? If you don't have enough time for a new job responsibility, then you need to say something right away, not burn up a bunch of time of menial tasks and wait until your boss notices the productivity drop.

Alternatively, if the system runs for a few months and every single patch sent to the CAB board has been approved then you can clearly demonstrate the do not really add anything and start making rational arguments to abandon the process from a sound basis while demonstrating you are an excellent team player who easily adapts.

You know, it's interesting. My current client has a CAB process, and they do a good job of balancing process and agility.

Anyway, they don't use it to micromanage every stupid little patch. They recognize that the Linux admins are trained professionals and know which patches to apply. Same with the DBAs and same with the Network admins and same with the storage admins. Nobody questions them down to the individual patch level unless there is a known conflict with a specific patch, which basically never happens.

What they use it for is for auditing and coordination. Multiple groups never patch during the same maintenance window without a good reason (e.g. Heartbleed. Network and Linux both patched at the same time to get the patch out ASAP. But this was a rare event.). In other words, let's say the DBAs and the Linux admins both have patches that need to be applied. Well, they never patch during the same window, so if something breaks, we know that the problem is isolated to the database patch vs. the Linux patch. Also, if we get a report that something stopped working right on, say, 3/8/2014, then we know that the issue was with a patch to their main web proxy server.

CAB type groups can be a valuable, but it's important to find the sweet spot between micromanagement and chaos. It's good to have all of the various groups know what each other are doing and can advise each other regarding potential issues. It's not like when the DBAs do patching that everybody discusses the Oracle changelogs in great detail for half a day or anything.

Re:Nonsense

[godefroi]

The OP doesn't see the procedure as a liability. That's a lie. The OP sees the CAB as a threat to his authority. He sees the servers as "his", and this will remove some of the power he has over them.

The OP needs to understand that the servers belong to the company, and that the company gets to decide how patches are approved and administered. If the job changes into something he's no longer interested in doing, then he needs to move on.

Being an asshole administrator might seem like fun, but usually, in "normal" businesses, it gets you fired.

Re:Nonsense

[squiggleslash]

I'm not trying to be mean, but I don't think he has any case for promotion under those circumstances.

Yes, I'm aware it looks like the committee was staffed with "idiots", that is, people whose expertise was necessary for the committee to function but wasn't technical. His job was to provide the technical expertise, and to make the committee aware of the technical implications of what they were deciding upon.

He failed. Maybe it was because they really were idiots. More likely, he didn't have the political, persuasive, and perhaps even conversational skills necessary to persuade a group of non-technical people what the implications were of what they were asking for.

Either way, the committee made recommendations his job was to prevent.

Now the purpose of a promotion is to put you in a position where your political skills can be used more directly to steer the direction of an organization. If someone has poor political skills, they're going to botch that job, and their organization will be hampered, not helped, by their promotion.

As nerds we tend to be a little technocratic in our viewpoint and think that organizational structures work with the most knowledgable person at the top. They don't. What matters is that as people rise within an organization, their skills tend towards listening, delegating, and communicating difficult ideas. We're seen at least one case recently where geeks went in a rage because someone with zero skills in those areas got promoted, and then kicked out, because a particular incident that required their skills to be top notch was completely botched. The tech community refused to believe that and decided it was because the person had disagreeable opinions instead.

But that's the way the world works. And promotions need to be given to people suited for particular roles in an organization, not as rewards because you were vindicated after the fact, rather than able to convince people to stop a disaster from occuring to begin with.

Re:Nonsense

[gl4ss]

Then you get a signed exemption to the policy from the vice chancellor for this specific instance.

had he the power and will to issue such exemption I'm pretty sure the vice chancellor would have done so...

Re:Nonsense

[jeffmeden]

I have worked for those companies. lol

27 page CAB form and full CAM meeting. Just to edit the /etc/login.defs and change PASS_MIN_DAYS from a 0 to a 7.

I still laugh about it to this day. A single character change and 27 pages of paperwork.

Yeah, show them you know your shit, go ahead and make single character changes all over the place to prove them that their system is pointless. Start with /etc/hosts and then work your way down to /etc/fstab.

With tongue removed from cheek, there is a reason CABs exist in the first place, namely that what one person sees as a no-brainer one character change actually impacts availability to a large number of people because they didnt realize what the true scope of the change was.

Re:Nonsense

[Anonymous Coward]

I had an enabler installed as manager above me, the only IT guy and developer for the e-commerce company. The promises got bigger, the progress much slower and deadlines pushed back for the entire 4 month I tolerated it. I tried to do the work but suddenly the company seemed only interested in meetings and my only point of contact was the manager who allowed the director massive interference over projects, deadlines and even little tasks. It took 3 bodies to replace my role and the primary project was delivered 2 month later.

The most recent mumblings dont sound too promising for the company, but then none of the original staff seemed happy to be there nor the (negative) change in attitude towards the workers.

Re:Nonsense

[nblender]

you're right. It's funny, reading this. I can see the douchebag admin that I was over 20 years ago... I did this sort of 'rub my nuts in your face' kind of thing all the time...Then I learned to embrace the beauracracy creatively and watch it passively... Things in my career got better after that.

We used to have to fill out timesheets once a week and hand them to our department admin assistant. I wrote a script that generated my timesheet for me using cron with input from a config file that defined charge codes and percentages... (20% of my time was to be spent on FOO, 35% on BAR, 40% on BAZ, etc)... The timesheets had random hours that over time averaged out to the correct percentages. Another config file denoted the statutory holidays. Every monday morning my timesheet appeared on the printer next to the admin assistant and she would promptly type it into the management system. I put the script in a central repository of tools used by engineering staff... Soon others discovered the script and started using it. Inside of a year, there were over 100 engineering staff using the script. I eventuallly caught shit and was ordered to remove my script and was seen as a trouble maker...

later I discovered, I get paid the same if I follow the beauracracy or fight it... If I follow it, it amuses me and I am less stressed. Being less stressed, and less of a douchebag has led to me having a more lucrative career. Now I'm an embedded firmware developer... I still have beauracracy; it's just different... I give my opinion once, when asked, and then I do it the way they want... Usually, later, I get paid again to do it my way when their way doesn't work. The experienced project managers now don't interfere and trust me to just get it done and don't micromanage me...

Re:Nonsense

[Dishevel]

Or you could hold the CAB meetings on a fire escape.

Re:Nonsense

[fair_n_hite_451]

Go directly to your Change Manager (not the CAB, but the person who is invested in the day-to-day management and running of the process).
Utter the following sentence "I wish to get regular patching of Windows Servers defined as a standard change, what do I need to do?"
Follow those instructions.

It might seem like more beauracracy for the first run through, but it'll be smooth and seamless after that. As a Change Manager for years, I can tell you that the people in that position are far more worried about project managers trying to push through craptastic updates at the last second than they are with competant domain support people. They'd love to get you off their radar just as much as you want to be off the CAB's radar.

Re:Nonsense

[bill_mcgonigle]

and with the greater long term job security that working as part of a larger company provides

Aye, there's the rub. It works out until it doesn't. Wouldn't this guy be ripped if the put up with two years of this crap to just get outsourced anyway?

Because that's what they're saying here. They don't trust him to do his job. Maybe that's fair, maybe it's not, but it's something a professional in his line of work can handle and they're saying "no". They wouldn't ask a surgeon to file paperwork on each cut he intended to make, because they feel the surgeon is competent to make the best decisions in the time alotted. Him, clearly not (I'm assuming this is standard work, not 10-9's / life safety).

So, they're going to fire this guy anyway at some point. He might as well find employment with an outsourcing company that gets paid by the value and minimizes their time expense, which it sounds like the environment he's more comfortable being in.

You can live to work or work to live - it's not worth being in a sucky job when there are so many opportunities to get or create a different means of employment.

Re:Nonsense

[LordLimecat]

Avoiding this is simple, in theory. Tell the truth, and don't be afraid to change your mind if shown evidence that you're wrong.

Hes not saying "dont do that", hes saying "dont be an obnoxious obstacle when this stuff comes up." Tell them theyre doing it wrong, if they insist, fulfill the request to the best of your ability, and make sure you have records of where you told them they were doing it wrong.

If you just constantly obstruct, theyll get rid of you and find someone else to implement their dumb idea, and chances are that person will make it worse.

Re:Nonsense

[Just Some Guy]

So... the business made a stupid decision, and when they realised the error of their ways, rather than trying to reach agreement on the best way forward, you delighted in rubbing their noses in it, using processes designed to protect you to hurt your employing organization instead.

One of the most important pieces of career advice I've received is to make sure that people who cause pain feel the pain. It is not my job to be a whipping boy who suffers for every bad decision I tried to warn someone about. If management insists that I do something really goofy, then they should not be spared from the consequences of their plans. Insulating them only enables them to keep making bad choices and inflicting them on codependent organizations.

You say "rubbing their nose in it". I say "making sure decision makers understand the results of those decisions".

Re:Nonsense

[bluefoxlucid]

Any remotely well organised IT department will have processes for handling both emergency deployments and retrospective approval. I'm not going to be cheerleader for the concept of CAB but if you're going to make a case against it then at least make a reasonable one because hiding behind obvious nonsense like this will just make you look stupid and change averse to your employer.

At least someone gets it.

OP should probably pick up a book on project management and familiarize himself with change control management concepts, and the need thereof. In large IT shops, change controlling patches is important: I've been in shops where critical patches were held back because the 2 week testing round showed that they caused critical services to fail, which was unacceptable. We worked out workarounds, fixed our own software, or got the vendor on the phone as necessary and suspended those patches for those systems in particular until we could get things straight; but if we'd just fired patches off as they came, we would have been in a WORLD of hurt!

My current employer has no change control management for anything, and every time something breaks there's just arguments and shouting. I have to weed through a whole lot of shit, de-tangle lossy memories, and eventually form a picture of when this was first noticed, how frequently it occurs, what changed around that time, what else could be broke in the same way but is not, etc. Kepner-Tregoe problem analysis. If we had a CCM, I could just pull up the changes at that time and avoid a whole lot of guesses and conjecture, solve problems quicker, and have fewer instances of problems never getting solved because everyone would rather sit around and cry like babies while throwing wooden blocks at each others' heads.

And they wonder why I'm suddenly studying for project management...

Re:Nonsense

[bluefoxlucid]

There should be proper operational management and risk management at every level. If you're elevating everything to the same people in full, uh. Yeah, that's broken.

Patching is an operations issue. If the patch process ever changes--if you find a patch that breaks your stuff and you need to take an unprecidented action--that's some kind of project. Decisions are made. Usually it's a very minor project that ends in a decision to exercise a known process (upgrade, contact vendor for fix, etc.), and doesn't require spinning up a whole waterfall methodology and going through all these motions of scheduling and such. Part of project management is knowing when to be more or less formal.

Re:Nonsense

[Richy_T]

Most likely he wasn't even consulted. There's a strange kind of mentality in many managers that totally ignores domain knowledge that their underlings has and produces requirements based entirely on whatever happens to be the "ooh shiny" at the moment or sometimes not even that.

Re:Nonsense

[MrNemesis]

This. Absolutely, 100% this.

As I've alluded to in my other posts, as soon as I graduated from cowboy sysadmin to a "proper" sysadmin that files change requests and writes project documentation, I've come to love change managers for precisely the reasons above. Change managers are under continual bombardment from non-technical project managers and developers that might well have deep, deep insight into a certain area but can't see past the end of their nose. A good change manager will often trot up to us sysadmins and say "So-and-so has submitted this change but doesn't think it needs approvals from you guys, can you take a gander?" to be met with either a "yeah that's fine" or a "Holy crappingon what-the-fuck in a god-buggered handbasket NO!". Good sysadmins in a constructive environment see a bigger picture than the project managers and the developers and, as far as CAB is concerned, submit better change requests as a result - because risk analysis is such an innate part of our job that most of us don't even realise we're doing it. But change managers see a bigger picture still because they're exposed to the sysadmins, network admins, security admins, user admins, mail admins, storage admins, admin admins, admin users, sysadmin networks, bread, eggs and breaded eggs.

Change managers exist to protect the business. Sysadmins exist to run the business' IT. Change managers realise that sysadmins are often asked to do dangerous or even outright impossible things by powerful people with only an inkling of what consequences such an action might have; it's a change manager's job to communicate with and understand the sysadmin (and everyone else) in such repsects, just as it's the sysadmins' responsibility to communicate to the business why change X is crucial or dangerous. In a properly functioning IT dept, sysadmins and change managers protect both each other and the business from stupidity, mis-co-ordination and lack of oversight. As a sysadmin, change managers are almost always on your side - either pushing for that change that's so essential, or holding you back where there's a risk. They're a highly valuable ally. When something goes to shit, they're the first people to step in and say "no, the sysadmins had nothing to do with this incident".

I'm MrNemesis and in the last three years I've learned to love my change managers.

Re:Nonsense

[clodney]

like this will just make you look stupid and change averse to your employer.

No... it's obviously just aversity to excessive, unnecessary and crippling micromanagement. It's obviously some idiots in suits who are change averse and feel they need to justify their existence by "approving" or "disapproving" of each and every required security update or patch or system admin action.

In my experience, managers are lazy. If patches are going smoothly, with no unanticipated downtime and no obvious problems, it is unlikely that somebody will be pushing to implement a CAB, knowing it means more paperwork and more meetings. I suspect that there was some high profile downtime, or an unannounced outage that led to this.

Re:Nonsense

[suutar]

possibly, but that's not how I read the question. I read it as "they're adding this responsibility, which looks like it's going to eat all my time. Any tips on how to make it take less time?" which seems perfectly reasonable.

Re:Nonsense

[phorm]

Yup, this is what I've seen. Changes tend to come in three types

a) Routine: A known change that occurs regularly and doesn't deviate much from a standard pattern. Stuff like swapping tapes, etc. It still gets logged and can be reviewed, but doesn't need approval.
b) Standard: Goes through an approval process before being implemented. These are basically known changes that can be planned ahead of time, but they don't necessarily have a standard set of steps
c) Emergency: Basically a form of (b) that needs to get done FAST. They can be reviewed by the CAB, but don't go through the normal approval process (they may need sign-off from one uber-admin, or none depending). This is intended for stuff that can't be pre-planned and is high-impact/danger if left unhandled. For example, patching may be a standard thing, but patching openSSL due to heartbleed would probably have been an emergency in many cases.

Re:Nonsense

[david_thornley]

This wasn't a technical problem. This was an organizational problem. He was complying with a policy he'd argued against and couldn't get changed. Had he not complied with the policy, he would be leaving himself open to disciplinary action, if, say, a vice-chancellor had gotten annoyed at him, or somebody involved making the policy had resented him for being correct.

Very simply, the only solution he could offer was getting the University Senate to change the rules. Alternatively, he could have delayed any new pages until the policy could be changed, but that apparently wasn't satisfactory.

Organizations that get too deep into mandatory policies get into problems, and there's no way for a sysadmin to stop it.

Re:Nonsense

[Kremmy]

As an advisor on the committee which put forward the policy, he probably actually did provide the alternative. But his alternative was given pass for the pain in the ass. Scapegoating the guy who warned against the problem? That's why these places have this kind of problem to begin with.

Re:Nonsense

[GPS Pilot]

Did the Vice Chancellor know you were on his side -- i.e. that you thought this policy was ridiculous? I would have said "Sir, I will be your best ally in getting this policy repealed, but in the meantime we have to follow it." Or could he have gotten the impression that you were just gleefully implementing the policy like some tool? The former would set anyone up for promotion more than the latter.

Re:Nonsense

[plover]

Hes not saying "dont do that", hes saying "dont be an obnoxious obstacle when this stuff comes up." Tell them theyre doing it wrong, if they insist, fulfill the request to the best of your ability, and make sure you have records of where you told them they were doing it wrong.

That would be fine if it were true, and if it were the end of it. But it's not. The enablers take over. If the bad ideas aren't stopped early by facts, their owners proceed down whatever path they've concocted, and the further they get without objection the more convinced they are that it's the correct path. An enabler will not tell them they're on the wrong path; or they'll say it once, but never correct them again for fear of losing their job (only a blocker says "you're still on the wrong path".) Without honest feedback about the mistakes being made, you can go a long way before realizing that you've led yourself astray.

One big problem is the belief that all problems can be stopped by governance processes. Therefore, all these processes are designed to be a form of change prevention. The idea is that by preventing incorrect changes, you avoid risk. But a process cannot distinguish between an incorrect change and a valuable change until after it has executed, so it must slow them all down equally. A process also handles the unknown poorly - it is designed to handle only certain changes, and everything else is awkward or not streamlined.

Change approval processes also encourage lies. When someone has to get a change through a process, they will tick whichever checkboxes will get them through the process with the least amount of effort, struggle, or paperwork; they will not voluntarily tick the box that ensures a microscopic review of their change, even when it may be appropriate.

Worse than all of the above, governance processes are hugely inefficient in that they're after the fact: create a large pile of changes, try to deploy it, then wait around days, or weeks to learn only then that the changes aren't approved. The feedback from governance is so late that the developer has long moved on to other tasks. Stakeholders get their changes in months instead of minutes.

Another sign the process is off the rails is if the disapproval is issued due to failure to follow the process, not with problems in the task being attempted. Too many failing processes leads further around the vicious cycle of process 'improvement', that then creates a process to follow the process, inserting delays into the delays. (Yo, dawg, I heard you like process, so I put process in your process...)

If you ever want to read a story about how bad process can get in the real world, read Red Plenty by Francis Spufford. He tells an interesting tale of just how far the Soviet Union's bureaucracy went, including goofiness such as one process that valued a machine by weight. The more modern machine that doubled production weighed less than the older machine it replaced, therefore the older machine was more valuable, and the budget rules that ensured progress did not permit replacing a more expensive machine with a cheaper machine.

Instead of after-the fact governance process, strive for continual, automated testing, starting with Test Driven Development. Have a repeatable method for delivering products that have quality built in from their very design. Once you've established the trust, you can minimize the processes. Something else valuable is a fail-forward philosophy: if you acknowledge that bugs will happen no matter what ("Failure is always an option"), you can often survive by putting in place the ability to recover from defects within minutes by being able to push out new patches. So instead of trying to avoid all risk by using a big process, you can get away with minimal process by accepting a little risk. This is a great approach because everything moves fast, especially the delivery of benefits.

Re:Nonsense

[Kernel Kurtz]

Agreed. I work with networks, not servers, but both groups in my org have an overbearing CAB policy.

For security issues or imminent catastrophes absolutely we grudgingly jump through the hoops.

For functionality enhancements or proactive hardware or configuration changes, it's not worth the aggravation. Far more efficient for the organization as whole to just let it fail and then fix it without hoops. Hopefully during regular business hours.

Re:Nonsense

[godefroi]

If that's true, then it's great for him. It means he'll be able to hire more subordinates, thereby climbing the corporate ladder.

Re:Nonsense

[dublin]

Wow, there's a LOT of negativity and assuming that the CAB is only a bureaucratic, bad thing. (It may be, but hear me out - it shouldn't be, and if it is, you can help change that...)

I think part of the problem here could be that the OP is assuming that no good can come from this.

If the CAB is doing its job, then it should be *helping* to determine which patches to apply, why, and when, based on taking into account the hardware, software, networking, and application environments and the "risk" a patch represents to each. That kind of support is a real net plus to a sysadmin. Note that it's implicit that the CAB is either doing or facilitiating this extra work, not just dumping it on the admin. (In that case, it's not really a board, but the worst sort of bureaucratic assemblage holding authority but no responsibility by dictating policy to be implemented by others who have responsibility without authority.)

Yes, this *is* a lot of work, and it *may* be justified, especially if there's been a history of being bitten by patches that were more of less blindly applied simply because a vendor or package owner/author posted them.

As with all process issues, the important thing to understand is "*WHY* are we doing this?" That questions is frequently answered the best by answering other related questions, including, "Is this the best way?", and "How else could we achieve the same goal?" , and perhaps even more important in winnowing down the answers from that one - "What could we do that's 'close enough' in benefits, but way easier to implement and support?"

Asking the right questions is *really* important!

Re:Nonsense

[sg_oneill]

Yes, I'm aware it looks like the committee was staffed with "idiots", that is, people whose expertise was necessary for the committee to function but wasn't technical. His job was to provide the technical expertise, and to make the committee aware of the technical implications of what they were deciding upon.

Oh I can assure you my opinion was made very loudly heard before hand. The problem is a mere techies opinion counts for nothing when the university counsel is saying that links must be approved at the highest level because of legal liability. (Ironically these days the opinion on this sort of thing tends to be that its better for the higher levels NOT to know, because of innocent-dissemination safe-harbor laws)

And once that policy was passed by the university senate, it had the force of government regulation, so I made sure the law was followed to the letter.

Re:Nonsense

[Ash Vince]

"Yes men", or "enablers" as you call them, are the root cause of the out-of-control bureaucracy problem.

The problem is that according to this particular management theory the opposite of an "enabler" is a "blocker". You do not want to come across as one of them if you value your job and future employment prospects for large organisations that subscribe to such things.

Patching....

[Anonymous Coward]

What we normally do is get a blanket approval if its coming from the OS provider with an understanding that patching will be done on a specific schedule.

IE. If all the patches come from Redhat there is no approval its necessary to keep them up to date for security purposes. The same is true for patches pushed out from Microsoft.

Then your only dealing with 3rd party applications. Even those the more common ones we get added to the blanket approval, ie. Adobe. This way you are only telling them you are bringing them into line with the latest set of patches provided by the OS vendor without having to list all the packages that are being updated. Then they only have to ask you if a program has or does not have a certain bug.

Re:Patching....

[rioki]

I totally agree with the above. These change review rigmarole is often done for reasons of security and operational stability. This is a laudable goal, but often the added red tape make the entire system more vulnerable when they want to decide which security fixes get applies. You need to hammer it home that each second between the time the security fix is published and the time the fix is applied the systems are vulnerable. This is because, once the security fix is published, every hacker knows about the issue too. If you have something worthwhile to protect, which is probably the reason why a change review board was established, you do not want add more time to that time window. If they need red tape, you should get a blanket agreement that you apply security fixes from vendors for critical software (OS, databases, etc.) ASAP and they get a notification of when and what patch was installed.

Re:Patching....

[N1AK]

If you have something worthwhile to protect, which is probably the reason why a change review board was established, you do not want add more time to that time window.

No, CABs often get implemented because someone is worried about the damage a borked patch/update could do and doesn't have confidence that it could be reliably fixed quickly. Most of the 'admin' in a change request is things like a process plan (which surely you already know if you're deploying an update to a critical live system) and a rollback process (which again, surely you should be considering before risking fubaring the system).

What I will say is that you should ensure that the CAB members are aware of the need to be able to handle emergency requests (meet, agree and deploy in hours) and should have some process to handle retrospective requests if a business critical update comes out and you can't wait for CAB approval. Normally the requirements for retrospective requests is that it's genuinely critical and that you send a completed request before the update. It might sound odd, but the idea is that they can use that to see if you had properly thought through the process and not just gone Rambo on it.

Re:Patching....

[ixl]

In addition to the above two comments, if the policy changes the CAB is instituting impair sysadmin efficiency (and it sounds like they do), then the CAB should be held accountable for the effects of those changes. This means that they should have to find additional funding for additional sysadmins for these servers.

Re:Patching....

[Zocalo]

Blanket approvals and template documents that you can cut and paste notifications into are the way to go, especially when it's on a schedule like MS, Adobe & Oracle. If they push back, suggest a documented process (this is ITIL, right? You can avoid the need for a CAB if it's an approved and documented procedure) where you push the patches to a few test systems on Tuesday (in the case of MS) then deploy to the rest later in the week - whatever they are happy with - if there are no issues. Depending on your timezone Tuesday PM or Wednesday AM are good slots for weekly CABs to pick up this; push to the test servers on the day, then the rest at the end of the week. For *nix, i've done updates this way for anything that didn't require a reboot so only stuff like Kernel updates and major low-level libraries needed to get approval via a CAB.

For everything else, it's your call. Either the patch waits for the next regular CAB or you play the game and keep calling emergency CABs when there are justifiably critical updates, such as Heartbleed, or for the inevitable critical updates from MS every second Tuesday that impact your systems. The best tactic is to embrace ITIL and make it work for you, not allow them to make you jump through hoops and spend your time crafting unique documents for every patch. It also serves as a useful procedure check to make sure you don't mess up and have a contingency plan for when you do, and ultimately, if you get it right, you still get to dictate the schedule and make them do things in ways that you are happy to work with.

Re:Patching....

[MrNemesis]

I depends very much on the makeup of CAB and the company culture surrounding it. I've already mentioned in my other post the "fun" I had with a CAB in previous employment who were always obstructive about everything until we'd had a long string of changes that went exactly as planned (including some changes that were approved against our wishes and broke things exactly like we said they would) - if there's an adversarial relationship then even with excellent diligence it takes a long, long time to build up a sufficient amount of trust.

CAB at my current employer is brilliant. You submit your request along with your technical risk analysis and the people on the CAB are techy enough to understand those risks and how they relate to business risks. Submit, say, a zero-day and not only will they welcome it with open arms but they'll literally ask how fast-tracked you want it. Yes there's technically red tape as T's need to be dotted and I's need to be crossed, but it's not a hindrance. A good CAB should know when they need to be slow and especially when they need to be fast; anything else runs against the idea of having a CAB in the first place.

What helps...

[ProfessionalCookie]

ethanol.

Re:What helps...

[LifesABeach]

"...Is there already a product out there that will make my life a little less stressful on the admin side?..."

I was thinking along the lines of Meth, it's not going to change anything, but so what.

Re:What helps...

[OzPeter]

ethanol.

Yeah .. but methanol is a better solution - especially if its not you drinking it

perhaps

[dimko]

New product your comapny requires is called: junior admin? Expensive stuff but does the job.

Re:perhaps

[halltk1983]

Several of them. Expect each one to be able to write up a change per hour, so you need a junior admins for every 40 changes per week. That means you'll need probably 2 for windows patches, plus quite a few for linux patches, another couple for your other software packages you maintain, and a couple more for any software you have in house. Welcome to management, sir.

Always quit dumb jobs

[koinu]

You know that stress reduces your life expectancy? You have most stress with dumb supervisors/bosses. Go and quit there. This has also the effect that you've ultimately showed your position about it.

Re:Always quit dumb jobs

[Kernel Kurtz]

Absolutely stress is bad.

I see all kinds of stupid at my job and it does not stress me out at all. I do what they pay me to do and that is all.

Not my job to keep management from running it into the ground, even if I could.

I do this

[beezly]

I have to do this and it's no problem at all, although our change management process doesn't sound quite as onerous as yours (I suspect yours will adapt over time -- the CAB will soon get bored if they have to approve every single OS patch).

I have to do a risk analysis for each change that gets made to a system (not just patches). Sometimes this risk analysis is fairly informal, for example if the change is to add more RAM to a VM, it's very unlikely to have a significant adverse impact and is easily reversible, so low risk. Other times the risk analysis (and processes that come out of that) may take a long time and require significant co-ordination with other parts of the organisation I work in.

A good example is if we make a change to a service that impacts the look and feel of that service. It will require co-ordinating with our communications, helpdesk, training and documentation teams as well as other parts of the technical group I work in and the CAB really acts as a check to make sure all of that has happened properly.

There are still a few people in our organisation who see the CAB as a barrier to getting work done, but for me it is really a check to make sure we're delivering changes in a proper way.

I can recommend you take a look at The Phoenix Project by Gene Kim, Kevin Behr and George Spafford. http://itrevolution.com/books/... - I had quite a few "this is where I work" moments whilst reading it :)

Re:I do this

[beezly]

It needn't difficult at all and it doesn't have to impact your ability to apply security patches. For example, patches from Microsoft released on the 8th April were applied to roughly 500 servers on the 11th. A couple of hundred of our servers applied the software remedy for heartbleed within hours of it being released, without any intervention from a human at all.

A change management process should take into account an organisations appetite for risk. For us, we're keen to apply security patches quickly, so they are pre-approved by our CAB.

Re:I do this

[OzPeter]

I have to do a risk analysis for each change that gets made to a system (not just patches)

Which sounds like its straight out of the OSHA playbook for considering the health and safety aspects of a physical job before performing it. While it is a PITA sometimes, when the shit does hit the fan you are glad that you have all the correct responses ready to roll.

Re:I do this

[beezly]

Indeed. When we introduced our change management process I realised that I was informally doing this risk analysis anyway. The change management process and CAB just formalise it.

Risk analysis can be as simple as thinking "is this low impact" for a second and then deciding it is and continuing. Most of these types of changes are pre-approved by CAB and we just have to record the change. If we started creating outages from these types of changes then that pre-approval would probably be reviewed.

There are other times when that pre-approval is temporarily revoked when the organisation cannot tolerate the risk of any downtime caused by changes, but that only happens twice a year, and I get to put my feet up a bit and work on interesting hobby projects for a couple of weeks :) A few of my colleagues get irritated that they "can't get anything done", but if my employer chooses to stop me making changes and let me have a rest for a bit, I'm not going to complain!

Re:I do this

[mysidia]

The auditor does not care if the patch may cause a work stoppage, if they are not fully patched, they are not in compliance, if they are not in compliance they get written up

Sounds like they need to fire and hire new auditors.

Setup a WSUS server

[will_die]

Setup a WSUS server, you probably already have the licenses. From there you can pull the patches to it and then push it to needed servers as approved.
There are commercial products that can also this in a nicer manner but they cost money.

Re:Setup a WSUS server

[Anonymous Coward]

Also, get the CAB to set policy to pre-approval any patches marked "security" or "critical". This can be configured within WSUS. That will cut the workload/paperwork down significantly.

Ask one question about patches

[deongene]

Ask a simple question, will this patch cost lives if it is applied. If the answer is no then apply the patch. Justification of applying the patch , no people will die if the patch is applied.

Re:Ask one question about patches

[DoofusOfDeath]

Erm... I meant to write "lives", not "lies." Freudian slip.

Re:Ask one question about patches

[csnydermvpsoft]

I'm sure that the "well, at least no lives were lost!" response will fly really well when a patch causes the company to lose $100,000 in worker productivity.

Re:Ask one question about patches

[deongene]

Lets say you have servers that manage traffic lights , fail an update and traffic chaos that "might" cause an accident that someone dies in. now yes if traffic lights fail then normal road rules apply , approach intersection with caution etc ... but still... MS patch = Death and destruction ..... but if all it costs is money .... patch patch your might machines.

ask yourself *why* and do the right thing

[flinkflonk]

This is known as the change process in ITIL, and it does have a remedy. The remedy is pre-approved changes (standard changes), which should include patching the OS with patches approved by the vendor. It's meant for exactly this situation, and if your change process doesn't have them it's just a paper wall.
The ITIL change process is all about reducing risk. If there is a risk with patching your OS (there is, especially since you mention Windows, it's not that unheard of that a Windows patch makes your whole network inoperative) you have to weigh it against the risk of not patching it (meaning you leave known security holes in).
So, my advice is to get OS patches for your OSes pre-approved by the CAB, that is, when a vendor releases a set of patches you are allowed to patch your systems in the way and the order of that pre-approved change. Of course it's paper-pushing, but use it to your advantage and push some paper yourself. If a server gets compromised and you have the papers (changelog) to prove that you followed procedure, blame will be placed somewhere else. And things will be done differently from there on, since it has been proven that the procedure didn't work, and everybody wins.
Or you could go find another job (like some other posters recommended) where you are the sole *cowboy*-admin and nothing gets done properly. Your choice really.

Re:ask yourself *why* and do the right thing

[Maxo-Texas]

This lowered productivity at my last place from 2 days to 47 days for a similar change level for changes from 1 to 400 lines and from 3 months to 6 to 9 months to never for larger changes. Once the cost was recognized, it also resulted in a lot of small changes not being done because their benefit no longer justified the cost.

However, it lowered our critical errors which effected production from about 6 unscheduled downtimes per year to about 6 unscheduled downtimes per year so it was worth it.

All kidding aside- a few time a year, it prevented different departments from really stepping on each other's toes hard. As in, "But we have a major upgrade that's going to take 20 hours to install this weekend and you are going to have the system down for O/S patches the entire weekend!?!?!" and "But we have a major upgrade that requires Jane from your team. She's in Europe for the next two weeks!?!?!?!"

But other than those... seriously.. reduced unscheduled downtime from 6 a year to 6 a year. I.e. no benefit. All the successful testing in the world IS NOT PRODUCTION. Most companies can't afford to maintain a test system identical to production. It's always a subset in some way.

Re:ask yourself *why* and do the right thing

[Maxo-Texas]

Oh and the worst case scenario was that the cab meeting was a fixed length and the number of changes took to long so all changes not approved were pushed back to the next cab meeting unless you got a senior director to hold a special meeting for the projects. In one really bad time- a lot of critical projects slid over 90 days due to this.

But they were very serious about it. The CEO or president's ass was on the line if a change went in which wasn't approved or recorded. So it was a firing offense. You *did* follow procedures.

And those procedures changed... constantly. From cab meeting to cab meeting the procedures changed. The only notice was a series of emails. They did not maintain a central change procedure process document.

So the point here is that the people on your CAB may be very powerful and not follow the rules themselves and may change the rules with little notice to suit their own needs.

Re:ask yourself *why* and do the right thing

[Pikewake]

I've been involved in setting up ITIL processes for several organiztions and agree 100% with the above. The main benefit of a change process and a CAB is that you can get an overall picture of all incoming changes, compare it to the available resources and prioritize based on fact instead on who screams the loudest. Hope you have a compentent change manager who can keep that focus and avoid greasing the squeaky wheels.
If the CAB starts micromanaging they will self-destruct.
After you get your standard changes approved, just make sure that the CAB is aware of the time you spend on doing them. CABs tend to forget that any work they're not actively involved in approving needs time to get done.

Re:ask yourself *why* and do the right thing

[Krokant]

Totally agree with this! Best is to introduce a classification of patches, e.g. "minor", "significant", "major" and "emergency" patches. Agree that you get the "minor" patches pre-approved (those with minor risk & impact, "minor" to be defined in agreement with the CAB). Other patches like service packs (significant?) and OS upgrades (major) should really go through a CAB, even if it is just to inform the other IT staff members about what you are doing (and to give them a chance to point out that application X or Y can break). Finally, also agree up front who you can call at night to get a "carte blance" in case an emergency patch needs to be deployed to fix some 0-day expl0it.

Re:ask yourself *why* and do the right thing

[sjames]

What you needed was a CAB CAB to maintain the change procedure process document. And then, of course the CAB CAB CAB to maintain the change procedure document change procedure process document.

They might need to lay off their production people to afford another layer of CAB or two, but that's OK, with the constant change in the change procedure change procedure, none of them knew what they were supposed to be doing anymore anyway,.

micromanager jerk

[Anonymous Coward]

I bet your CEO or upper level boss is the typical dimwit/jerk, knows nothing about the business, microcontroller type of guy, stupid games of power, calls you on purpose once his secretary tells him you are out of the door. Small guy, stupid looking, may beard of a goatee, cheap-looking suit. Tell him to sod off and change jobs...

Run away!

[arcade]

Given your description, you're the sole sysadmin. This means you're the person who should take these decision - nobody else. If the company disagrees with this, then either you've done a poor job previously, or they don't trust you to do your job for some strange reason.

Now, if it's you that have fscked up on previous occasions, then it's understandable that they want the red tape.

If you haven't, then it's time to put down the foot and say "Nope, that's my job". If they disagree with that - linkedin should be a relatively short distance away, and after you find yourself a new job - simply hand in your resignation pointing out that you have no interest in having babysitters.

Re:Run away!

[SirDrinksAlot]

I think the real moral of the story is, not that it isn't his job but it's a job for a whole team. There should be an engineering team (testing updates, finding the issues and improvements etc) and a change/ops team who does the leg work when deploying these kinds of processes. One guy responsible for a big ass pile of servers and is also responsible for all of this other stuff is at least two full time jobs.

Re:Run away!

[MTEK]

While running away is an option, if your customer is the U.S. gov't, you can be the most competent admin out there, rational logic and stubbornness and won't make a damn bit of difference. The terms and conditions are dictated from afar and you do what you're told. Simple as that.

OTOH, if you're someone who takes pride in getting work done, making the customer happy, and find the experience of doing paperwork tedious and mind-numbing, there are less frustrating places to work.

They have a point

[distilate]

As a software developer I have multiple times had a development box screwed over by an IT department pushing unneeded drivers and patches that cause problems. I say prove they are good or needed before you waste other peoples time. If you just want to push any random patch that comes along then you should be forced to resolve all issues without the traditional reinstall the machine.

Re:They have a point

[Anonymous Coward]

As a software developer I have multiple times had a development box screwed over by an IT department pushing unneeded drivers and patches that cause problems.

I say prove they are good or needed before you waste other peoples time.
If you just want to push any random patch that comes along then you should be forced to resolve all issues without the traditional reinstall the machine.

Er, waste people's time?

As a software developer, you have no fucking idea how difficult it is to pick and choose patches and driver updates to get pushed out to machines while also trying to maintain any sort of consistency with patch levels across the enterprise, but apparently this is something you want me to waste my time on in order to ensure you've not lost a spare second on the rare and random occurrence that you experience a problem in 1 out of 200 patches (my patching track record over 15+ years shows the frequency quite less than that)

And if you're doing patching correctly, you're mainly concerned about patches deemed "critical", so again, you're not really afforded the luxury of picking and choosing here without risk.

As a seasoned sysadmin, I have a fix for you. It's called VMs. Play till your hearts content and press the rewind (snapshot) button when find the environment screwed up (shockingly, it's not always the IT department that screws computers up...yes, I know this is breaking news)

Re:They have a point

[Anonymous Coward]

And you sir, are why most people hate IT.

In short, yes, I do expect you to waste your time to pick and choose which patches so I can not lose that spare second. After all, it's YOUR JOB to keep the computers running well. If you can't be bothered to do it, then what's the point of you being employed? My job as a developer is to develop products, not to battle with my machine. By you not doing your job properly and by approving a patch that takes out my machine, I'm now unable to do my job. And likely, I'm not the only person who will have been effected by this issue, so thus, by you not doing your job, you may have now cost 10, 20, 1000 people a couple hours where they're not doing their assigned tasks.

If you can't be bothered to as a sys admin, then in my opinion, a company shouldn't be bothered to employ you.

Re:They have a point

[Bigbutt]

Seriously? You want to be involved in the patching process of all the servers? I appreciate that, but damn dude, talk about a waste of time. After a bit, getting e-mails on the patching of our 1,000 Unix systems might get a bit tedious. Especially when something breaks in production and it's traced to a patch that you signed off on. Do you really have that much free time?

See, I have no idea if a patch that appears to be necessary on my server will affect your application. So you'll get to see all the patches that I'm intending on applying. And since your environment needs to match mine... Well because your software will eventually be deployed into my environment. So I will insist your environment be as close as possible so you experience the problems before it gets into my hands. That way I'm not trying to figure out why your software isn't working or worse yet, the application guys aren't in an incident trying to figure out why the timing is off which will affect 17,000,000 users forcing a report to the FCC and an urgent change request is sent to you and your group to be worked 24x7 until the code works again.

Wouldn't you rather your server work like a production server? So when you're done with your code, you _know_ it works in production and you can move on to the next project without having to worry about being dragged back in to fix some problem?

[John]

Re:They have a point

[Bigbutt]

Humorously I am doing my job. My job is to ensure all the servers are patched appropriately and that the environment runs smoothly. This means regular patches, resolving hardware and software issues, working on projects to deploy new systems or apply new versions to existing systems, and of course the numerous meetings to discuss important issues.

I patch our lab environment before patching anything else to ensure production isn't adversely affected by the patches. Of course the lab can't exactly match production but we do our best. Next up is QA so incoming applications can be tested on a production like server. Next is Production once both areas are patched and standardized and no issues discovered. Finally, assuming the developers are willing to be patched, the dev servers are patched.

Now I don't always get to patch development servers in part because you're working on something or other and can't be bothered. And unfortunately there are times when you're done with a package and you present it to QA only to discover it doesn't work there. A few times because of a patch. Which means I have to break out of what I'm doing if possible (you're not my priority by the way) and patch your system so you can identify and fix the problem.

So I am doing my job.

[John]

Re:They have a point

[Bigbutt]

It's likely I was coding before you were a twinkle in your father's eye. I've done my share of programming in the past. As a sysadmin I find I can code to my heart's content without having some project manager or manager breathing over my neck to get some buggy code out. If you had actual systems admin skills, you'd understand that we need to work together to get your code out to production as seamlessly as possible. That means working within the constraints of the environment your project is destined for. What that doesn't mean is that you can pull some Linux distro of the week out of your ass and expect it to be deployed into production without issue or complaint. Depending on the urgency of the requirement, you might get it deployed, but I guarantee you, you will be working on the next version with a supported operating system.

If you can't understand that dev, qa, and ops are a team, then perhaps you should come out of your mom's basement and check out the real world.

[John]

Re:They have a point

[HornWumpus]

They can code. But everybody who's seen their code, wishes they didn't. Which is why they are digital janitors.

Isn't there a toner cartridge or backup tape that needs changing?

Fight absurd with absurd

[Anonymous Coward]

An absurd idea or ludicrous non-sense can be fought with even more surrealistic silliness.

Just apply this dogma.

Follow the instructions diligently, put the administrative burden on them with tons of notices and emails, and don't forget to ask for a quick answer every now and then, because security is at stake.

Re:Are you still partying like its 1999, or what?

[gbjbaanb]

oh god Remedy....I used that once.

But the concept is good- you need a 'bug tracker' where the requests for patches can be made to you, and you can then assign tot he CCB. Once they agree it, then assign it back to you for implementation.

Any dev bugtracker will provide you with this kind of audit trail - think 'requirements' for the CCB authorisation, 'development' for the implementation, 'test' for the testing. You might want to rename these though.

I'd make it web based so access is simple for everyone involved - last thing you need is a Excel based solution. I've used Mantis, or Redmine but Bugzilla would work too as would any number of web based bug/task tracker tools. Get one installed before someone on the CCB says "we'll use a spreadsheet", seriously.

Windows servers aren't always the best...

[chentiangemalc]

Scares the hell out of me how many SYS admins don't know what a Microsoft KB article is... And you are not paying attention not knowing about "patch Tuesday" and where Microsoft announces out-of-band patches... get WSUS and half the work done for you.

Ha-ha!

[hsa]

In the voice of Nelson from the Simpsons: Ha-ha!

They want to make your work more transparent. Apparently, they think you have too much spare time, too. Or you getting fired/outsourced, and this is a gentle reminder to document your work..

Since all the reports are similar, I would just create a script to handle the documentation needs. I would also do extra work: create report how much this affects the efficiency of patch / hotfix distribution and how time all these process changes take (and maybe inflate that number a bit, just a bit).

This would also be a great time to ask for an assistant to ease the workload.

As a Change Manager...

[Pete (big-pete)]

I work in Change Management for a major telco, I chair the IT CAB, and I oversee server and client patching (amongst many other changes!). When we patch clients, we are patching up to around 30,000 real and virtual desktops - when we patch servers, they also number in the thousands.

There is no way we would allow a sysadmin to patch anything at any time without some level of oversight, an individual admin has no oversight on other patches, hardware interventions, application releases, network upgrades, business campaigns, etc that may be happening on our environment at any given moment (this isn't their job to be keeping track of all of that info). For server and client patching is as light as possible, but we still maintain a close oversight.

On the Wednesday following the second Tuesday of each month (for example), I sit down with the Windows server guys and the Windows client guys, and we review their proposals to patch - usually we have a fairly rapid timescale that we can meet to ensure that the patches are deployed (including pilot testing, etc to catch any issues before everyone's desktop is broken!), sometimes there are other major interventions that overlap, and then we need to make prioritisation decisions and decide which has priority. We have made similar agreements with the Linux teams, where they have a special process to patch, and we have close oversight on Unix patches, as upgrading these servers with a reboot can be a very big deal.

The last thing you want is an application version release of a critical ordering application happening at the same time as a system software patch, and then to have an issue afterwards - is it the application version, is it the systems patch, was there some conflict with the activties being performed at the same time? Troubleshooting gets more difficult, teams point fingers at eachother, and the whole time the business is screaming blue murder.

Of course in an Incident situation there is more flexibility to get things fixed fast, and with security issues I am keen to break open the S-CAB process to expedite a rapid approval flow to ensure that security holes are fixed as fast as possible - of course most changes are encouraged to follow the rules though, the change calendar is published, and everyone knows when the "standard" slots for deployment are, and if most people manage to schedule their changes within those windows, then it minimises potential conflict for everyone.

Change management are not your enemy, they are your friend - once you register your change with them, they have your back, they will guard from other interventions clashing with you, will stop you from inadvertently upsetting the business, and will decrease change related Incidents. However, with great power comes great responsibility, and Change Management need to find the right process for the right type of change - we cannot have a full in depth investigation into every configuration change, every patch, every bug-fix, every new server to be provisioned. A good Change Management team will guide changes to the appropriate flow, and grease the wheels for certain types of interventions - it seems that the CAB mentioned in the summary are still finding their feet a little, and I am sure they will evolve over time as they start to understand which changes are high risk, and which can be allowed to pass with a lighter touch.

-- Pete.

Re:As a Change Manager...

[Anonymous Coward]

OP is managing 50 servers, you are managing tens of thousands of Systems - the situations are hardly comparable.

Like the OP I am the sole admin for our companies IT (60+ on prem servers, mix of WinTel and Linux, plus 10 Azure hosted servers) and I am in charge of patch management.

If a committee in the organisation came and told me they were taking responsibility for patching away from me I would either tell them to sod off OR I would hand over all the admin accounts and wish them luck.

Re:As a Change Manager...

[Anonymous Coward]

Exactly. While I wouldn't have a problem if it was just a matter of tracking when changes were applied for audit purposes, having to document each patch is completely unreasonable. In my case, we have various regulations that require us to patch our systems within a certain period of time, which we've translated into patching once a month. If I get asked for justification for installing a particular patch, it's usually "because if I don't the auditors will ding us for not installing it." If something needs to be done off-cycle or in an emergency (e.g., OpenSSL updates for Heartbleed), those get documented and rubber-stamped by our change control process, but beyond that it's assumed that regular patches are approved.

At one point it was suggested that for compliance I was going to have to not only document and justify all patches applied, but pre-document exactly which patches were being applied on each system AND show proof that those patches and ONLY those patches were applied on each system. Given the way that RHEL releases patches (not to mention Debian and Ubuntu, that would have turned a monthly patch cycle that takes at most a day or two into a full-time job. I pushed back, pointing out that it would do nothing other than create more paperwork and take time away from my ability to do anything else, and they eventually agreed. Yes, there can be a happy medium somewhere, and it's not the same for every organization, but the fewer technical people there are doing the actual work, the less this type of bureaucracy will produce functional results...

Re:As a Change Manager...

[catmistake]

There is no way we would allow a sysadmin to patch anything at any time without some level of oversight

Change management is not your enemy

First let me say I really like it when people like what they do, and it sounds like you like what you do, take it seriously, and probably do it well.

However, this being a techy site, the commenters, yourself included, seem to inflate the importance of the one thing that is irrellivant and incidental. And by that I mean, of course, the system, the OS, and whatever state it may be in. Let me repeat, the OS is irrellivant and incidental. Thus, any time, effort, meetings, plans, etc., focusing on them at the expense of the state of what is important are an incredible waste of time and resources. I don't mean to belittle your job, btw, but merely to point out what every commenter I have looked at on this story seems to miss. The damn systems don't matter! (I am a systems admininstrator, myself, btw).

What matters? The data.

Change management is, in its own way, important for the reasons it is important and not for what those who have positively described it here. It is important because it tells us: "What the Hell have we done? How did we get here? What the Hell are we doing? Now what?" It is history, and it is intent. But it should only be as important as the systems themselves. And, again, the systems themselves don't matter one whit, especially if you have emergency procedures for when something breaks (i.e. fix it, or if fixing it is going to take too much time or effort, just reinstall it). So if you have a change-management-heavy operation, you're wasting a lot of time and effort for no great benefit or reason at all.

Tell me, at your company, is there a group whose sole responsibility is to manage the Data, and provide oversight concerning its reliability or how it is used or managed by the systems or users? Well, then its crazy that any resources would be sacrificed for the sake of the systems.

Systems administrators and the IT department are like the teams of airline mechanics. They're very specialized and very good at what they do because, in a relative way of course, lives depend on it. But what is important is not the damn planes but the cargo... i.e. in getting someone or something from point A to point B successfully. Does it matter if the plane is 40 years old? Not really, if that 40 year old plane does an identical job to a brand new plane. And we can replace the planes with high-speed trains, or (hopefully someday) teleportation. The planes themselves, and the use of them, are incidental, and irrelevant; they don't really matter.

In conclusion, I wouldn't say to the OP "get a new job!" as others have. That's not very helpful. I'd say, if you have control over these systems, and now you have to champion each new patch and get every patch approved, then switch everything to a system that will require less patches. Running Windows servers? Migrate to linux or a *nix variant. You'll still get security patches and bug fixes coming down, but far far less than with a Microsoft based operation, and if you miss a few months of patches, nothing really bad will happen (like on a Windows system). The world will still turn and the work will continue.

Re:As a Change Manager...

[Pete (big-pete)]

What matters? The data.

Sure - and we take our data very seriously - really, I doubt we'd last long as a telco if we didn't. But the underlying fact is that the data needs systems, your data sits in databases, that in turn reside on servers - hopefully virtual servers that can be moved around to minimise downtime and impact, but still systems at the end of the day.

The new way of thinking is certainly going to be more data orientated in the future, and for sure, there are still a lot of improvements to be done, but there is certainly no lack of focus in that area of the business already - in fact any company that has a data warehousing team to take care of the long term broad reporting needs of the business is already well aware of the importance of data.

In the future there'll be more focus on data (security) protection in the world of IT - we're already mostly past the days where people are only just starting to think about the implications of data loss and data mining for information though. There is a lot of work that can be done in controlling data access, and ensuring that all data has the appropriate levels of protection, and that it is managed in the correct way, I agree with you 100% there.

-- Pete.

Re:As a Change Manager...

[catmistake]

I was speaking in hyperbole... and of course I figured you and team and co take the data seriously... but no one was mentioning it; I thought it worth pointing out a different perspective. But still, compared to the data, the systems, as long as they are caretaking the data properly, are incidental, and from this perspective, irrelevant, as far as what the systems actually are. Thus it (or it should) follows that spending more time focusing on incrimental patches to incidental systems at the expense of spending due time or rational contemplation focusing on the persistence of the vastly more important data seems counter, or skew, to how time should be spent. (Fwiw, I think change management is a very "good thing," but also I think that having groups of employees spending time in meetings, any meetings, kill a company. Nothing ever gets done in meetings... its all "ok, here's reports, and here's what we want to go and do now," when anyone that has ever taken meetings knows that it never (ug, I mean rarely ever) has anything to do with how some initiative works/plays out... its just a way to synchronize intellectual capital of human resources, which can be done far more quicky, effectively, with an email.)

Maintenance Windows

[kefalonia]

Probably all you are missing over there are scheduled maintenance windows.

You give them a list once per month about what is about to change, get a confirmation, proceed with them available on standby for fixes on the spot or, rollback.

Try to think the big picture: how would you maintain the systems, if they were life-supporting medical equipment? Why not give same quality of service?!

System Administrator Vs Change Advisory Board

[wonkey_monkey]

System Administrator Vs Change Advisory Board

50 quatloos on the newcomer!

1 of 3 possibilities...

[Anonymous Coward]

1 of 3 possibilities:
1. You are perfect. You NEVER screw up. In this case, the CAB is just being a PITA.
2. You can make certain types of updates quickly, with little or no risk, and you never screw up. The CAB should agree to make these standard changes with very low overhead. The other types of updates are likly to help YOU, not to mention everyone else in the company that depends on you.
3. It's hard to say in advance - most of the time, things work OK, but sometimes problems arise and there is unexpected downtime (it's NEVER your fault, however). Bit the bullet. You are not running a world class shop and you need help to improve. Anyway, downtime in production always takes more of your time than filing an RFC.

Ask for a good vulnerability scanner

[Anonymous Coward]

Turn this request proactive. Ask for a good vulnerability scanner, one that can perform authenticated scans. Qualys or Rapid7 would be good choices. The scanner will list out all of the vulnerabilities on each server including those that have patches available and those that don't. Let the scanner do the work and then present the report of both patchable and unpatchable vulnerabilities and let them work off that. This is how we do the CAB at our 300 server and 2,000 desktop bank.

After the patches are installed run the same scan again and now you have proof that the patches did in fact close the vulnerability. Both the "before" and "after" scan becomes part of the CAB documentation.

This in fact will seriously increase your workload for months because there are a whole lot more vulnerabilities that you know about and many of those will be configuration issues. But for fifty servers it should be less than six months and then you'll be in a good place. And the CAB will lighten up a lot as things show improvement. Too many sysadmins think that Windows Update and the RHN are the only tools they need for vulnerability management and that is not anywhere close to the truth.

Re:Are you still partying like its 1999, or what?

[sjames]

That's not the big leagues, that's the short bus.

yes, changes need to be documented. They should be deployed on a test server before going into production. The rest is just people who were presumably traumatized by falling out of a tree as a child seeking revenge.

Take the people in the CAB and replace them with extra admins who are bright enough to know what I said in the 2nd paragraph.

Change management can cover your ass

[Madman]

There is genuine value in a well-run change management program. Organizations need to know what is going on in their infrastructure, and plan things properly. In many industries there is a growing regulatory requirement to have change management, and auditors are looking for these things more often too. Many smaller shops are bringing in change control, so rather than handing in your badge my advice would be to deal with it and learn the lessons.
One lesson is rather than fight it, use it to your advantage. Yes, there's paperwork, however if you follow the system correctly they cannot blame you if things go wrong. What you thought of as freedom was also a risk to your own position as you had sole responsibility - change control means less freedom, but you are covered. Also, you can get budget for better management systems which will make your life easier. Put together a realistic list of what you need and get involved with setting up the change control process. If you stay silent or fight it you won't get a say.

Re:SCCM

[gl4ss]

sure, but how does that help with having to run the CAB through 102 patches?

I think go for easy solution. introduce the patches in batches for the board. ("monday updates for week 32").

the fucking board will not care after 2 weeks anyways so just do lip service for two weeks.

I'm not a fan of CAB

[Anonymous Coward]

I used to work for a Fortune 100 company. I'm not sure how CAB works at other companies but I get the impression that their implementation was flawed. 1) You could easily go around the process. 2) I'm certain nobody reviews the code - They just kind of discussed it. In my opinion this is a half-baked solution to prevent things from getting pushed to production which could cause problems (errors, leak sensitive info, etc). I am 100% confident that I could have gotten CAB approval for nearly anything. I understand the idea behind CAB but in my experience it isn't effective.

I actually quit that job partially due to things like CAB. Increasingly control was taken away from people in the IT department, and handed to things like CAB or to 3rd parties who managed our systems, databases, etc. The jobs of myself and others in IT staff were being reduced from "actually doing the work" to "submitting tickets and following up on tickets." Nothing like being on hold when calling the 3rd party for a critical issue you yourself know how to fix in 5 minutes. It's also a blast when I had to tell the support guy what commands to run because he wasn't familiar.

And no we didn't fuck up anything to deserve this treatment. It was dictated to us from upper management.

Re:I'm not a fan of CAB

[Anonymous Coward]

You shoudnt have said the guy anything. First rule of consulting, saying no more than it is required.

Sounds like you work for the federal government

[MikeRT]

Do exactly what they say to the letter. After the second "patch Tues" where they pound the ever lovin fuck out of Windows Server with updates and the CAB has a pile of paperwork big enough to roast a wild boar they'll suddenly regain a measure of common sense.

Get a vulnerability scanner

[dremspider]

Buy something like Tenable Nessus or Rapid7. Make reports very easy and works across Windows, Linux, Cisco, etc. If you get Security Center it will track changes over time and you can see trends over time with patching.

Re:Are you still partying like its 1999, or what?

[clickclickdrone]

Heck, we have a CR process for anything that touches a live server. I even had to go through the process to get details of a file as it would have resulted in an unexpected file write. By way of background, the server used to fill up during the day's processing and empty out overnight. It got very tight sometimes and when someone made a copy of a file without checking the size, it filled the filesystem and the server fell over. That particular outage cost several million given what the server did.

Must have missed the TLA

[Kahn_au]

Where I come from CAB stands for "Change Acceptance Board", they don't get to make dumb decisions...

Re:Are you still partying like its 1999, or what?

[Stumbles]

Ug. Remedy is such a bitter pill.

Pre-approvals

[AndyCanfield]

Seems to me that you need to establish a list of pre-approved changes. For example, if you're running Windows and IIS, make sure there's a clause that says anything that comes down the pipeline via Windows Update does not need formal approval. That way you can offload the responsibilty, and work, onto Microsoft. You can keep your core software up-to-date. Third party software, same thing for corporations. Student projects and your own shell scripts might need more examination; not a bad idea actually. But if there's a new version of Firefox, why in the world would a Change Advisory Board think it knows more than Mozilla?

Re:Pre-approvals

[happyjack27]

Why in the world would a change advisory board think it knows more than ANY company putting out an update?

fuck this site and popups

[Anonymous Coward]

fuck this site and popups

BYE

In my experience...

[amalek]

.. as the admin for a couple of hundred Windows servers, an efficient CAB is your friend. As another said, they have your back, and that of the business (and by extension, the poor guy who is up at 4am fixing any issues introduced). That said, I've also worked with companies and CABs that know how everything is written in the ITIL handbook, but with no clue of how to put it into (an efficient) practice. It sounds like your CAB just wants the paperwork done - did you bring on consultants recently? - and think/hope it will mitigate the risks involved with patching. Change request for patching on a development environment? Routine change. Keep up with the news for any issues from this month's patches. You patch dev, or your pre-prod environment or whatever you have, monitor for a few days and if all is good you apply the same patches to your production machines. This is enough risk mitigation for most, and it gets the job done at the end of the day. Make up a nice RACI chart (Responsible, Accountable, Consulted, Informed) for the whole process - you are probably R/A for successful patching, but, the CAB will provide the approval for you to go ahead. They won't allow you to do it if there's a big release, or some on-going issues. Then you only need to know how to push the patches and have a good engineer to fix anything that might occur on the night, and the accountability trail takes care of any finger-pointing and addresses any gaps in the process you might have noticed. Start slow, start small. Work your way up in volume as the becomes more like a routine change.

zOS Maintenance and CAB

[jacobsm]

I'm the zOS Systems Programmer at a Fortune 500 company. When we do system maintenance cycles our CRB just wants to know when the system environment is changing, not what's changing.

If anyone ever does want to know I do have detailed logs and a before and after image of the maintenance management database (SMP/E Consolidated Software Inventory) for them to peruse. They never do; since they don't understand zOS Systems Programming, and they shouldn't have to. It's their job to manage system availability and to ensure that proper testing and system validation activities were performed. It's my job to manage the environmental change.

For anyone who's foolish enough to ask for detailed documentation of every module, macro, load module, dataset, file in the Unix System Services file system that's being modified, well enjoy yourself.

What I won't stand for, is for someone to have veto power over what maintenance goes on. That's my decision, and since I'm the best person in the organization to decide, I do so.

Big company experience comes to small company

[erroneus]

Yes, I know how they are thinking and the pain you are feeling. To accomplish the implementation of this change management process you will need a lot of people working for you. Use this to your advantage. Quickly study up on the subject so your experience with the systems will not leave you with a dog pile of new bosses to tell you how to do your job. Instead insist that you need to hire more people to manage the overhead.

In the end that probably won't work and you'll be kept "at the bottom" where you are now.

These changes are going to be enormously expensive and despite all you have done, it will be perceived that you created this mess by not having a change management system in place to begin with. Of course, they will also see that you don't know about change management and will prefer to hire someone who already knows about it.

Now I'm not going to down change management processes. They can prevent problems and identify people who would otherwise deflect blame and hide in the shadows. But from what I have seen, you're just getting the beginning of the tsunami of changes.

Push for testing systems and additional hardware to support it. Of course it will also require more space and other resources. Try to get ahead of this beast.

Routine changes.

[Vermifax]

We got our CAB to agree to a certain class of routine changes that require minimum review. They don't need anymore detail than, Test servers updated on Tuesday, Production one week later per maintenance windows.

Re:Routine changes.

[essbase_nerd]

We do the exact same thing Vermifax, and after 10 years of installing Windows Critical Updates on servers, we've never once had a problem in production with this approach. In fact, I can only recall two times that a patch caused an issue with our applications in non-prod and we had to investigate prior to deploying (or not deploying) in prod.

I think you're at higher risk with all the red tape and heavy process potentially resulting in servers with unpatched vulnerabilities.

Change Management is good

[MrNemesis]

...and necessary* but that doesn't stop some change management boards being needlessly obstructive.

Years back, I was working at a company where all of our servers got patched at build and then never patched again "in case it broke something". Myself and the rest of the ops team begged and pleaded for the business to allow us maintenance windows, allowed to reboot the OS outside of business hours, install patches... all to no avail.

Until the company lost a bidding on a contract because they had no maintenance or patch management policy in place so the business comes running at us screaming why we don't patch our servers (they would listen to their potential clients about computer security and whatnot, but not to their own staff). Cue us showing them the dozen or so draft maintenance policies that we'd submitted over the years, all of which were rejected by the directors. Red faces all round in that meeting :)

So the latest draft gets pushed into force by a wheelbarrow full of cash and we go out and buy Shavlik, a really rather nice patch management solution... and then our change management board goes nuts when they see our report. Lots of w2k and w2k3 boxes had literally hundreds of service packs and patches oustanding before, and like the OP wanted an individual change raised for each patch going on each server. We then set up an email direct to the change board that gave them Shavlik's automated PDF thingy which gives a list of all the patches outstanding on a server along with a hyperlink to the MS KB or similar... but that wasn't good enough. They wanted a report on what each patch did, which files it altered, all the usual stuff. Now as another poster had pointed out, under ITIL this should all have been "standard change" without needing so much paperwork (seriously, they should be at least aware of ITIL even if they're not going to follow it to the letter) but we could sympathise with them that, even with our planned dependency-based staggered rollout over a 4 week period, this was both a radical shift in company culture and posed a significant opportunity for breakage... but still. Filing about 20,000 change requests it was to be.

So obviously, since we were dealing with obstructive officials, we did exactly that. Did a few dozen hacky shell scripts that took the PDFs that Shavlik made, CURLed down the contents of the link to the KB page and then posted it off into the change management system - one request per patch per machine. After about twenty minutes of this we'd submitted about 400 requests and the change management system (an in-house pile o' shite that wasn't so much written as congealed out of various bits of sharepoint and was universally hated) had slowed to a crawl enough that it took 10mins to open the page. It used funky whizz-bang ajax to load *all* of the pending change requests in the background ("who needs a LIMIT on this SQL parameter?! We're never going to have more than fifty open change requests!" The developer in question also seemed to think that using a LIMIT statement was akin to taking the go-fasta stripes off your car. Wonder if he's doing webscale development now). After some brief arguing where they actually suggested we should open a change request to submit changes - at which point we cackled at the prospect of submitting another 20,000 pre-change-request changes - and after finding their ITIL manual down the back of the sofa they finally agreed that yes, actually, they didn't need quite such a detailed report, and were prepared to accept our risk assessment report as a single change for the first weekend's rollout.

So about 20,000 patches/service packs were staged and installed over the next two months, and luckily we didn't have a single failure due to the patches (yes, I also thought this was miraculous considering the crufty applications). From then on, every patch cycle needed just four changes, one for each week. That's how it should be done.

* Yes, necessary! I've done more than my fair share of JFDI but that just do

QA

[NapalmV]

This makes no sense unless you also have a QA department were all these patches would be tested. Then the CAB would need to get a list of the patches description, justification, and impact to existing enterprise applications. Based on this list they could select what can be applied immediately, bundled in a weekly/montly release, scrapped or postponed until a remediation plan is completed. Without QA results the CAB is useless.

What went wrong before?

[bwcbwc]

In my experience a CAB usually gets introduced in a small organization if something really got screwed up under the old process. There are exceptions - you could get a CTO who is gung-ho for ITIL, or you may have a new, important customer who insists on "process". But a CAB is an attempt to manage change and prevent problems in the working environment. So unless you have a better solution that will prevent negative impacts from your change process, go do the paperwork, with special attention to any risks or issues associated with the change (extended maintenance window, complex install or backout process, partial or incomplete fixes that still leave issues open). You can probably half-ass the CAB and get your work done almost like the old days, but when the next failed change occurs and they find out you hid risks or didn't do proper research, your ass could be out the door.

OTOH, if you really hate bureaucracy that much, hauling your ass out the door could be your best option - as long as you have a different career in mind besides sysadmin.

Re:Are you still partying like its 1999, or what?

[mysidia]

That particular outage cost several million given what the server did.

The problem is not the admin actions it's "what the server did"

The application the business was dependant on to generate millions of dollars was designed in such a fragile way, that it could fail as a result of whatever happened to just one server....

You see... this is bad architecture. Servers are prone to failure, even when designed with redundant components.

It is improper for a business application that generates revenue to be sensitive to a single or double server failure. Critical applications should be architected with a level of robustness that reflects their level of importance.

a few things

[smash]

CM is there for both preventing fuck ups and dealing with them when they occur. First things first: do you have a test environment? If not, build one. Do you have documented processes? If not, document them.

Proper change management ensures that: 1. people in the group know what is going on. 2. you have a second/third set of eyes to ensure that you have both a plan, a backout plan (or plan B in case it can't be backed out) and a test methodology to ensure that a change hasn't broken things. 3. to make you think about the implications of what you are doing, and 4. that business stakeholders are informed and know how to plan around any impact both expected and unforeseen.

If you aren't doing all of those things already, sorry dude but you are just winging it. That's efficient, etc. until one day it all goes horribly wrong and you need to figure it out on the fly how to get back to normality, with unpredictable outage durations, etc. All of that should be worked out before going live with your changes.

Yes, it sounds like a lot of faffing about for no real benefit, but really, one day it will save your arse. And really, you will be surprised at just how many effects even a single change to a production system can have.

Solving the wrong problem

[davecb]

In a previous life, we passed around virtual machines rather than doing paperwork. Paperwork is to be sure you have a plan to solve the explosion-and-revert problem.Managing machines instead of paper allowed us to include a process for doing an immediate revert on explosion (;-))

The VMs we passed around were Solaris zones, so they were very lightweight. If I wanted to apply an emergency patch to production, I first applied it to an image, put an instance on pre-prod, a physical machine, and varied it into test. After the smoke-test, I varied it into the pool on the load-balancer, and watched it closely. If it fixed the problem and didn't explode, I put lots of instances on the production physical servers and put them into the load-balancer, quiescing the un-patched instances but not erasing them. If the patch blew up after all, I could revert to the previous buggy release as fast as the load-balancer could disconnect people. Not quite as fast as doing an atomic change on a single server, but fast.

This is a minor variant on some old unix norms: 1) you aren't prohibited from doing even silly things, as prohibitions will keep you from doing something brilliant. 2) You can do anything, but you can't hide what you did, 3) you can change things atomically while running, and 4) if you do something dumb, you can revert it immediately.

The process is a variant/predecessor of ITIL, with pre-set apply and revert steps for emergency changes, which are the high-value part of the whole ITIL change process. Non-emergency changes were a little more heavy-weight, as we tested the patch in an instance in QA, then did a simulated UAT overnight (it was automated, but exceedingly slow), reviewed the results and then the de-facto board decided if we could release the image to production, QA and dev. Your paper-oriented CAB does approve all patches to QA and dev, right? I'll bet they missed that part (:-))

--dave
I did once have a customer where I had to do paper-based CAB approvals, but that was because we weren't funded to have a proper dev, and had no QA at all. As you might guess, we still had at least one fiasco. I shortened the contract as much as I could without doing a no-bid in the middle.

my monthly nightmare

[deadweight]

We are a 100,000+ user operation. Our patch tracking and approval process is a giant paperwork nightmare that does nothing useful. I would get Microsoft Security Baseline Analyzer, run the report after Patch Tuesday, and send it to the management types. Say look at this nice list of required patches. If there are no objections, we will roll them out :D

One change

[Dynamoo]

A whole bunch of OS patches = One change
Replacing a server = One change
Reconfiguring some shared folders = One change
Replacing a whole bunch of printers = One change

There are a couple of advantages with a change process like this.. the first one is collective responsibility, so the poor sysadmin can pass at least some of the blame back to the CAB if it goes wrong. And then also there's the point that other people might have a legitimate input into the process, especially if there are things happening in the business on the same day as the proposed change that IT doesn't know about.

Automate it.

[1s44c]

So you told them it won't work and they didn't listen. Now show them it won't work. Script something to send them a request for each update for each server. When they get flooded with 100+ perfectly valid requests each day they will beg for mercy. Then file one request for 'ongoing ad-hoc security updates for systems' and watch how fast they approve that one.

Re:Thank Snowden for this

[1s44c]

The nightmare of change control has existed long before Snowden leaked anything. This is normal big company stuff.

Re:CABs are the HOAs of business

[1s44c]

HOA?

I find burying these people with more requests than they can handle causes them to back off.

Change for the sake of change

[Shadow99_1]

Things like this always annoy me. Someone has decided either that you don't know your job or that they need more layers of bureaucracy. In my experience it is usually because they think you don't know your job as a system admin. Do I really need a 'paper trail' or make work for things I'm already tasked to manage the risk for? And why would a group of business people (generally) think they are somehow better at mitigating IT risks than the IT person?

Part of what they are supposed to be paying me for is to know that if patch X breaks on the test server it is probably not a good idea to go live with it and I should also know already how to revert the changes in said patch if they have an adverse effect on a live server when they did not on the test server.

Things like this were they feel a need to micromanage things they don't really understand just annoy the heck out of me.

Home for the Otherwise Unemployable

[Anonymous Coward]

ITIL is for shit. What an awful program, and let me wipe myself with my THREE certifications.

The CAB is where the otherwise unemployable go to die. It never gets streamlined, the castaways from other organizations will find their home there, and those most lacking in intelligence will go here.

As others have told OP, he should honestly quit. Lone admin, they can stand up a CAB but can't hire more help. Bad sign.

Swim away.

God I hate IT.

Due Diligence

[ogrius]

You are maintaining 50 servers for multiple contracts and they want to know what is patched.

To me this would be a completely fair and normal situation.

I haven't worked with Windows Server in 8+ years. But WSUS was great for telling you what patches were needed and approving them to be installed.

I know that RedHat has similar technologies. Though you can also roll your own as well.

From my own company, I attend a Change Control meeting and one meeting a month has the Microsoft patch bundle as part of it. The patches get installed on a subset of the company on day X and they day X+2 they get pushed to everyone. This allows testing.

For production servers that customers are using to me it is a no brainer that the customer wants to know and approve what changes are happening to their server.

Depending how busy you are, you might have a resource issue which is fair to complain about.

But that they want to track and approve patches / changes? Suck it up buttercup.

Servers are for applications...

[GoChickenFat]

I've been an admin for a very long time. What I see is a lot of admins think the OS is the most important and fail to understand why the server even exists in the first place. If you patch simply because it was made available, you don't test or know what the application the server is hosting does at all, then are you really doing what is best? Yes, patches break things and often the patch "fixes" something that was low or no risk inside the corporate network to begin with. Too many admins fail to balance the risks with application uptime. ...and that's why you end up with a CAB - to keep everyone informed, to balance risk and to account for audit controls. These usually pop up after too many system outages or lack of information sharing. Admins have a bad habit of being too smart and too busy to keep others informed. I have worked with a lot of CAB's in many companies and the best way to work with them is to be proactive in keeping them informed and to build a trust relationship in advance.

Re:Servers are for applications...

[nbritton]

If we're talking about enterprise linux, if the system administrator does a point upgrade from say RHEL 6.4 to 6.5, and a 3rd party application breaks, it's because the application developers or application administrator screwed up. The API and ABI doesn't change between releases, this is the reason why the latest release of RHEL still uses a 5 year old kernel.

As an enterprise linux administrator, with 15 years of experience, I believe this level of change management is a boondoggle. Applying RHEL update packages in an la a carte fashion is precisely what creates system instability, and setting aside kernel updates, all RHEL patches must be applied. When I setup new RHEL servers I enable automatic updates; development and testing are set to update daily, preproduction is set to weekly, and production is set to update monthly. Kernel updates are the only thing done under the supervision of an administrator, this is to ensure systems come back up after reboot and 3rd party kernel modules get updated.

Updates to 3rd party software are what should go through the change management process.

Re:Servers are for applications...

[david_thornley]

Thing is, while you get hold of that third party whose application broke because they were dumb, top management still wants the invoices that application produces to go out on time. A sysadmin should keep things running, not screw the systems up and blame somebody else.

Re:Servers are for applications...

[Kalriath]

You would literally not last 30 minutes where I work - you'd be fired by the end of the day.

The application is the important part of the puzzle, not the OS. The OS is an easily replaceable cog that can be spun up on another VM in 30 seconds flat if there's an issue, the application is a behemoth that requires deployment, configuration, and all manner of babysitting.

Your "update the OS, fuck the application" approach is patently dangerous to business continuity, and is the sort of thing that results in pretty rapid disciplinary action in a real enterprise environment (where it's clear you've never worked).

Tools for patch management

[shuz]

MS SCCM and RH Satellite are the two OS vendor specific patch management solutions. However your licensing will end up being more expensive per server and could be cost prohibitive for a small company. You cheapest option would be to script patch groups. You could do this in Powershell and Bash. The CAB may not require you to list in great detail exactly what each patch modify's. They may only ask you to list out the patch numbers being applied. The point of a CAB is to make you slow down rapid poorly thought out changes, bring stability, and external oversight to IT changes. CAB may also have a purpose in letting your greater organization know what is going on. You will find the new requirements painful and often times annoying or illogical, however they will also make you and your organization stronger.

Give 'em what they want

[rajats]

It seems that the process is not that bad (even though your description does look a lot worse). Subscribe to the Microsoft Security Bulletins and they have a full description of each patch that they put out on Patch Tuesday (e.g., https://technet.microsoft.com/...). The same goes with RHSA. Subscribe to the updates that you are interested in; these will most likely be your OS, web servers, app servers, other software installed. Similarly, most vendors run security patch announcements. There will likely be a lot of noise but in a couple of months you will know how to extract the information the change advisory board needs. Here's the positive aspect of CAB: if you screw something up, you have someone else to blame! ;-)

WSUS or SCCM (Configuration Manager)

[TMYates]

I would recommend the use of either Windows Software Update Services (WSUS) or in combination with System Center Configuration Manager (SCCM). WSUS allows you to approve/unapprove all the updates you want to allow in your network. You can group specific computers to a specific set of approved updates if you would like. You can also use SCCM to manage the change control, what was approved, and what was installed. SCCM can also be used to deploy updates in certain circumstances.

Of both of the options, WSUS is free and can be installed on Windows 2k3 or newer. SCCM is now licensed through the System Center package which may or may not be worth looking into if you want to look at the other built in components to it.

There comes a time in every job to quit

[Roxoff]

Tell them to stick their change advisory board up their shiny rear end. For fifty servers, with updates applied separately for each, they'll never get anybody to come in and do that task voluntarily. They'd need a small team.

Microsoft do actually spend quite a bit of time ensuring that all the changes they apply are proper stable fixes or improve security. How could some advisory board know more about these proposed fixes than Microsoft's developers who are writing the damn things in the first place?

Re:SCCM

[Enigma2175]

Yeah, after a couple of weeks of having to run through a few hundred patches at a time (make sure you write at least a page for each patch!) they'll get the hint that this is fucking retarded and back off.

I think and your parent underestimate the ability of committees to do work that is fucking retarded. I can't count the number of fucking retarded processes at my company that people have been happily doing for years.

"Jodi reaps the whirlwind."

[stoploss]

The way to succeed as a techie is not about being technically brilliant any more, it is about how you can talk people round to your way of thinking and use evidence to back up your points of view.

Meh, sometimes it truly is nice to see people suffer the consequences of disregarding one's expert advice. One time I was ordered to have our production system send a copy of all log warning output to a low-level exec's email (that was hooked to her blackberry).

I stressed this was a very bad idea due to the sheer volume of email this would produce (thousands of warnings per nightly run, from midnight to 3 AM). The order was reiterated. I believe my commit log message for the config change was, "Jodi reaps the whirlwind."

I heard she put her blackberry under a pillow in a different room so she could sleep thanks to the ~2,000 emailed log warnings she was getting each night. Apparently, the notification on the blackberry was getting queued, so it would constantly notify for hours. I have no clue why she couldn't just put it on silent... phone call rings, maybe?

Regardless, she had to put up with it for several days before the config change to disable her email cc could be deployed.

It's not that bad an idea

[UrsaMajor987]

I spent a lot of years working for a company with a very structured tech environment. In all fairness to the company, they work in an industry that is heavily regulated. That said, it was a highly competent development team of SAs that decided what should be on the servers. A bunch of managers on a CAB will not be able to replicate that. With a single SA and only 50 servers, you have a pretty small shop. Sounds like maybe they have plans to grow the business? It sounds like there is no process in place right now except what is inside your head. Hope you never get hit by a bus! Servers are too important to the functioning of a modern business to leave things to that kind of chance. I think the company is doing the right thing but they are attempting too much too soon. Try to help them but start small; maybe define a standard build of each type of server and then use one of the automation tools to keep each server in conformance with the defined standard build. You might even then use one of the tools like to Tripwire to notify you when someone or something makes an unauthorized change in your servers. Basically, work with your management to improve the situation. The upside of all this for you is that the management in your company will realize that your job is a lot more complicated then they ever imagined..

Divide your changes into groups

[MAXOMENOS]

90% of your changes won't have any effect on production systems. Just lump those together under "Routine changes to UNIX/Linux production environments" and explain that you've tested those on your sandbox network.

10% of your changes will impact your production systems, even if it's just because it's upgrading Apache or some Perl module that your systems use. This can be as trivial as "updated Perl module; ran complete unit, load and regression tests, everything works fine." to "This is a kernel patch that requires us to power cycle each box. Here is our plan to do this in a way that generates no application downtime." Those are the changes CAB is meant to catch. Document each one in a different request. Document them clearly and thoroughly. Run them by people whom you trust to write good English. Make sure that your deployment, testing, and rollback plans are solid, and document them thoroughly in each request.

After a while, you'll get really good at this, and people will trust your requests.

Totally not sucking up, but...

[Plugh]

you should start browsing dice.com now It's escape-hatch time

Re:Are you still partying like its 1999, or what?

[Minwee]

It got very tight sometimes and when someone made a copy of a file without checking the size, it filled the filesystem and the server fell over. That particular outage cost several million given what the server did.

At this point, or better yet several months before you get to this point, it's a good idea to volunteer the information that additional disk storage would cost only several thousand and prevent these kinds of problems.

Depending on what the server really does, a complete spare system ready to take over in the event of even the smallest failure also looks like a good investment. Don't let the business wait until they lose several million before spending a hundred thousand to prevent it.

Re:SCCM

[afidel]

Why write anything? Include the full expanded content from the MS KB article for the update, they generally run 1-5 pages each if printed on 8.5x11/A4

Re:SCCM

[DickBreath]

Just include the link. Don't bother with the expanded content. Make them feel like they are doing real work by having to click the link.

Bureaucrats need jobs too! They are a help to the organization in the same way that leeches are a help to their host organism.

Re:find another job...

[DickBreath]

When something gets royally F-ed up, and eventually it will, who is going to get the blame?

Re:SCCM

[dowens81625]

I would suggest writing a php look up page where all you need to do is copy and paste the requisite KB Patch number, and it have it scrub the http://support.microsoft.com/k... article for related information and paste it into Re canned Letter.

Patch Request for KB

This patch is critical to maintain a stable and update to systems environment. Failure to approve and install this patch will leave your systems vulnerable to

Please note that after applying this patch

Please sign off as approved or rejected

Approved by

Printed name
Signature:
Date:

Rejected by

Printed name
Signature:
Date:

Sincerely your system admin,

Copy & Paste your KBs then proof read each letter make small adjustments where needed. Must most KB description articles are close enough with proper php scripting you should have no trouble pulling the relevant info from the page in the variables. and customizing a script with the info they want to see.

Print and repeat. Hand them hard copies, drink beer,

After they sign off on about 30 of them they will get tired and just say just do what you think is best and go back to doing your job.

Re:SCCM

[nblender]

What I've observed at the customer site that I've been at for 6+ years is that some mid-level administrator will realize his/her job is becoming obsolete so he/she writes a white-paper for management outlining a huge problem that is waiting to bite the company (that upper management might be personally liable for should it ever come to pass)... Management has a bit of a freakout and generates new policy. Voila, author of white-paper is seen to be the resident expert on this problem and is put in charge of handling it. A new department is born and people are hired/transferred. It used to be ISO-900x that drove this stuff, then SOCS came around and provided even better fodder...

Letter of Resignation

[Spazmania]

That's how I'd handle it. If they want patch reports, that's reasonable. If they want you to patch the test environment a week ahead so that the devs can check for problems and alert you not proceed, that's reasonable too.

If they want to micromanage your tiny components of your job they can get bent and good luck finding a replacement. No preapproval for routine systems administration activity.

patches really are an issue

[roc97007]

I'm not sure that CAB is necessarily the right solution, but patching really is a problem and can't be done blindly unless your business can take the occasional production hit.

Admin is outsourced at out company, (I'm a former sysadmin who now does application admin, still local) and the contract apparently specifies "current minus one", which means we patch frequently on all platforms. The problem is, the offshore admins have no context, no idea what server provides what resources, (and yes, we've tried to educate them -- the information gets "lost" within weeks or months) and no conception of the idea of patching first on dev, then test, then prod. They manage patches by version numbers not by environments, which means a collection of patches may be announced (to all and sundry because they refuse to use the contact list) is a hodgepodge of development, sandbox and production servers. Information is commonly that the servers "will be patched" but not to what version, which has caused contractual support problems (where a server is running a more recent version of the OS than is supported by the app). Other joys have involved bricking prod servers with firmware patches, because they didn't try them in test first, insisting on doing nonessential servers on the weekends instead of evenings (because, no context) and forgetting that when it's daytime over there, it's dark over here, and I'm probably not going to be at my desk at 0'dark thirty to give some last minute approval to take a server down.

It's a mess, and the CAB process, as obnoxious as it is (we sit through 150 -- 200 change descriptions every week) serves to catch many of the above issues. The outsourcing company is annoyed by this -- they just want to patch -- but we have the process as self defense against very real issues.

What I'd recommend to the OP is to hire someone to manage the CAB process. We did, and it worked out pretty good.

What's your role on the CAB?

[skydyr]

If you're in a small company but still managing 50 servers, what is your role on the CAB? You should advocate to be part of the CAB, at the very least, so that you can coordinate processes to streamline critical and security patches, and keep management informed of the process. If you are approaching this with a hostile or obstructive attitude, it doesn't reflect well on you and it injures your ability to get management to listen to you when it counts. A CAB can be a rubber stamp, for the most part, that ensures that there is at least documentation and a modicum of thought going into the maintenance of the company's infrastructure. The creation of a CAB is pretty reasonable, but the key is to be involved in its creation and ongoing existence as well so that you can eliminate the red tape while still documenting the process to CYA in case something goes wrong. And something, sometime, will go wrong.

It's possible that moving to a more validated and controlled environment will increase your workload, but again, documenting the volume of changes, etc. is to your benefit. That is the data you need in order to provide management the justification for getting a second admin hired. This is also the data that you can use to justify a big raise or promotion at your next review, instead of people wondering what exactly you do all day.

Also, as mentioned previously, you really should have WSUS or a similar solution to deploy patches across your organization, both to make administration easier and to help enforce a consistent environment across all the systems. If you patch everything piecemeal, it's quite difficult to tell whether a patch will break one system with a different set of patches than this other one.

Awaiting approval

[Culture20]

This post awaiting approval of Dice Holdings Change Control Management Department.

If you don't have a seat on the CAB, it's not goin

[Maniac_Dervish]

As the sysadmin you should have a seat on the CAB. That's it.

You're the person doing implementation, and you're the person most suited to evaluate the technical impact of the changes that you're making.

If you were not a lone sysadmin, it would be your director or their delegate who ought to be on the board; as a standalone sysadmin, though, including you is the sane thing to do.

What's the purpose?

[PPH]

I assume you have various individuals/groups who have an interest in the systems you administrate. Users, developers, etc. Also regulators. Don't forget the utility of a good documentation system when the auditors come around*. So you need a process to keep them informed of the upcomming system changes. So they can ensure that their product or process isn't going to be broken by a change.

If you have relatively few of thes interested parties, the communications could be mandles manually and by you. If that community is large, the procedures need to be formalized and possibly automated. Having a CAB to represent your user community can offload the communications task from you. At the expense of some paperwork.

On the other hand, I've worked in organizations where the CAB was a make-work task for a few layers of management. People whos only other job prospects are standing by an off-ramp with a cardboard sign*.

*At one of my previous jobs, this was the acid test of the utility of our CAB. I had to fill out stacks of paperwork and await their blessing to make a change. But strangely enough, whenever the FAA came around, they were nowhere to be found. I had to walk the auditors through our systems myself.

ITIL and CAB works when implemented properly

[The Veracruz]

The ITIL change control and CAB process are quite useful when used properly and facilitated by the appropriate staff. That IT changes cause the majority of datacenter outages is not a debate, proper change control shows us this. However, CABs use is to get all changes in a room so impact and stacked changes can be weighted to ensure these "change outages" do not occur or at least minimize the risk.

Example: I need to push changes across a WAN but the network team has a router upgrade also planed. Someone is going to fail here. CAB resolves these conflicts.

Where CAB is supposed to back off is when we have ITIL-defined standard changes which dont require approval but are still notified so others can be aware systems WILL be rebooting. Our admins responsible for patching, have a pre-approved and scheduled patching cycle of systems, and CAB is the method we notify other departments of these updates. We have a stage and production environment for most applications anyway so we use these for patching also. Well know if an OS patch or package update breaks because it broke in stage, not production. And CAB is only there to pause patching IF theres an issue or other scheduled maintenance window more critical.

We have anywhere from 10 to 30 people in each weekly CAB and it takes 15 minutes (video conf/dial-in bridge). We then have another 15 minute meeting the day of our major maintenance windows. Since doing this weve reduced unplanned outages by 75% simply due to proper scheduling between departments. It takes 30 minutes a week. I spend maybe 30 to 45 minutes a week writing my change requests, so in the grand scheme, the benefit outweighs the administrative overhead.

However, this improvement assumes ITIL was facilitated and maintained by people with enough technical knowledge (through director-level) to make appropriate decisions based off changes presented. If your CAB is run by paper admins, youre in a world of hurt because theyre going to make uninformed approvals regardless of the amount of data presented. In your case, the CAB should not require a write up of every KB and you should not have to "prove" each update. Instead, if an update of a "test environment" goes fine, the CAB should only be there to make exceptions to your planned maintenance window.

I dont know the size of your contracts but my 30 minutes of CAB and pre-maintenance meetings allow us to maintain several hundred applications across multiple VM and physical server farms with additional AWS infrastructure. If ITIL is new to you and your CAB partners, it does take time to smooth out the workflow. It should take almost a year to get a nice flow because it is quite a shift to everyones workflow.

Or, if you have a bunch of chuckleheads running the show now, like others said, dust off the resume.

Things to try

[whitroth]

First, as some other folks have said, give them a weekly list, not every day, or every time one's announced.

Actually, that might burn them out... or, they might decide to batch them on their own, and think they'll get to it eventually.

Here's one: give them a weekly list, AND INSIST on a weekly meeting to discuss it. EVERY WEEK, without fail, without cancellations. Tell them that you'll also want a spot meeting, when you get critical updates (like yesterday's Java from Oracle, with it's 4 that had a CVA rating of 10 on a scale of 10). Insist that if you get those, they need to meet that day, or the next, or give you the pre-approval to put those in without consulting them.

The weekly meetings will get to them in relatively short order, they being so busy and all....

Also, here's another pushback: do you have a testing group, that runs regression tests before regular updates, and especially on ermergency ones? If no, question the committee how they expect you to regression test everything. Also, do you have test, as opposed to development systems? If not, that's another budget item the board needs to approve..

Make them do that real job, professionally. See how much they hate doing it, and maybe it'll go away.

                    mark

Re:Things to try

[geekoid]

I strongly disagree. Give them a list every day, with a question so you force a response. Suck up their time until you get more man power.
If they don't respond say 'I didn't move with that patch becasue X hasn't responded yet.

I"m not against CABs,, in fact I am for them. I am against just tossing this decision out and not adding man power. Recipe for failure.

I agree with everything else. Man power up, or make it go away.

Just quit

[ahodgson]

Seriously.

Billing by the hour?

[otis wildflower]

If you're billing by the hour, this should be a GODSEND.

Otherwise, start updating your resume..

Re:Are you still partying like its 1999, or what?

[clickclickdrone]

It was a very old and very complex system that was midway through having its replacement built. The system was not something you could easily add resource too. Yes, it was a disaster waiting to happen (although it had DR) but as is often the case, trying to persuade the suits that they needed to spend millions on a system so that they'd get a new one that did the same thing, isn't very easy.

Re:SCCM

[afidel]

Bah, a pageful of links doesn't have the same weight as a ream of paper dropped on the table in front of each participant =)

Puppet

[MillerHighLife21]

One of the things you can do with Puppet is get a change record of what WILL happen, specifically so you can show it to a CAB, get it approved, and then apply it during a scheduled maintenance period.

Re:Are you still partying like its 1999, or what?

[Minwee]

I've heard that story before. I once started a job and was told "Sure, this system is a bit touchy, but don't worry about a thing, it's being replaced and won't be needed after next month." My new coworker pointed out that he had been told the same story a year and a half earlier when he joined.

A year later the power supply on the single main server running dontworryitwillbereplacednextmonth blew out and we needed to replace it. Only it ran a very specific version of Netware with a highly specialized database product that nobody knew how to work with any more and even though we had backups it was still next to impossible to even reinstall the base operating system on anything but the original machine. We wound up checking eBay for an exact duplicate of the dead server, raced out to the next city to buy it and then just swapped the hard drives, powered it up and hoped for the best.

It worked, and a quick meeting was held in which my team stressed just how lucky we were to be able to recover from this and that we needed a real solution that didn't involve waiting until next month. The CIO listened to this,nodded his head and suggested that we should try to buy two more replacement servers just in case it happened again.

Another year later, when the company was finally bought up and the office closed down, dontworryitwillbereplacednextmonth was still happily running in the corner of the server room, waiting for a new application that would do everything that it did.

Any month now.

First things first

[geekoid]

Welcome to hell. son.

You need to come up with a number of how much time it takes to patch, evaluate and test, turn around time, and a testing environment.
Because you are going to need at least one other people, more likely 2 or 3. Now they will need to justify their CAB decision against actual money.
When I did it, I was at least able to get a whole slue of 'standard' changes that needed CAB notification, but not approval.
Don't let this force you into work a single second more then you already do. When something can't ge done just say 'Sorry, I'm mandates to do all this extra work for the CAB, so I didn't have time to get to it". Also chase it up the chain like 'I"m trying to do X, but I can't becasue of Y, I need more people." Be the flag waver for more budget.
They need to be aware, and feel the impact of a decision this big.

Let them - they will be responsible

[petes_PoV]

Any IT professional worthy of the title can generate paperwork fatser than any team of 50 or fewer can approve it.

If they want to approve every change, then just flood 'em with paperwork. 1 day spent automating your process should keep them busy for at least 6 months. Meanwhile you won't have any changes that have been approved, so you can get on with the interesting stuff.

Oh and if anything fails, dies, gets a virus (presumably security updates and virus scanner downloads count as changes) or lets the world and his/her dog steal your company's secrets then it's not your fault: the board hadn't approved the change you submitted weeks ago.

The good thing is that the change board are taking on responsibility for the changes. By approving them, provided you execute them exactly as described, then they are to blame for any problems - as they gave approval. Make sure you keep a paper trail and have a record of everything you do.

They will quickly tire of the burdensome, boring and ultimately futile work. So enjoy the honeymoon period. It wn't last forever, but if you handle it properly, you can shed the blame for any problems for at least a year - even if the board disband. The confusion and lack of clear indications of who should have approved what can be spum out for a long time - in the right hands.

Meantime, you will have plenty of opportunity to look for another job.

And you sir are the reason why developers are NOT

[Poohsticks]

And you sir are the reason why developers are NOT sysadmins or typically given admin privileges on servers. Sysadmins DO evaluate the patches and updates. That's a requirement before putting them on the machines. Developers however rarely review the latest security updates and changes required by vendors as relevant to the core OS functions - because they don't have to. So they rely on 5 year old driver implementations (which SUCK) and outdated security models (because that's not their job - to deal with security - they write code and new products!). FUCKING BULLSHIT. I have had more developers take down their own machines than I can count. The original comment is right. If you're working with such brittle fucking code that you can't deal with patch deployments - then go work in VM environment where you can snapshot and rollback with a few clicks. Fucking developers always think then know everything about computers "because I make them dance!" Bullshit. I bet you never took one fucking class on OS development or kernel basics. Stupid fucking arrogance.

Manage by exception

[dave562]

We manage our patching process by exception. By that I mean, "bad" patches are held back and everything else goes through. I am responsible for about 1400 VMs running on 60 physical ESX hosts. We have a small subset of VMs that are representative sample of the environment. Those get patched two weeks ahead of time. If nothing goes wrong with those servers, the corresponding patches are pushed into production.

We have an exception for the web tier. Those get patched the weekend after patch Tuesday. They are higher risk due to being public facing.

We have some verbiage in our documentation that states something to the effect of, "We expect that the vendors will properly test and QA their patches before releasing them. We do not have the time to fully vet every patch before deploying it. Therefore we take the following steps to mitigate the potential damage to the environment caused by a bad patch...."

Snapshots are taken of all VMs before patching. That way in case something slips through the cracks, we can quickly roll back to a known good state.

If you need to go toe to toe with the CAB, make them provide you with a business case justification that details the perceived risk(s) and danger of not mitigating the risk. If they cannot do that, they are completely worthless.

Your counter argument then becomes, "Mitigating your perceived risk is going to take xx hours of time. If the risk were to actually occur, we would lose xx hours of time cleaning up."

At the end of the day, if the risk absolutely has to be mitigated and you do not have enough time with all of your other responsibilities, then they need to provide resources. They can do that by either assigning the task to someone else, or hiring a new employee. Ultimately that is your supervisor's call to make the business case for needing more help. All you can do is quantify the time required to comply, and then make your supervisor make a decision on what you will stop doing because you will now be dealing with the new mandate.

Try to understand where the CAB is coming from. They probably have a regulatory requirement, either because of the business that your company is in, or because of the business that your clients are in. They have to prove that they have a functional change management process. It seems like they are just going too far overboard with the process. A change management process just needs to show that people cannot make unauthorized changes to the environment whenever they feel like it. It also needs to show that changes that are made are documented. Potentially destructive changes that could impact application or service available should be discussed, or at the very least, procedures should be developed to mitigate any potential impact of a destructive change.

Meet them half way. Suggest constructive solutions to address their concerns.

WSUS

[AntiTuX]

WSUS allows for you to track patches and installed software much easier. It works as a pretty good gatekeeper for that sort of stuff. I'd recommend it.

As for dealing with CAB boards, just use logic and reason to destroy them and crush their spirits.

Change management gone wrong

[SirNAOF]

This sounds like change management gone wrong.

The idea of change management is to ensure that changes are tracked, but this sounds like bureaucratic crap. Setup WSUS so you can track what patches are applied where, and then talk to the CAB to approve monthly (or whatever schedule) patches en-masse. Otherwise you'll end up not patching, and that's an even worse result.

I don't mind change management when it's done with some amount of sanity.

Re:Change management gone wrong

[roc97007]

> The idea of change management is to ensure that changes are tracked, but this sounds like bureaucratic crap.

Yep, that often happens when the people who are putting together the change management system have little experience with the issues that a change management system is supposed to fix. You then get hilarity like systems that prevent reasonable reaction time to production outages, "blanket" notification for every single change, a process that can't be successfully negotiated in less than a week, or my personal favorite, a mandatory process with no owners.

The issues as I see them are (a) what does the change entail? (a Red Hat "patch" sometimes will change the RHEL version number, which may take your apps out of compliance) (b) what else does it affect? (Sun patches often installed a virgin sendmail.cf, rendering email inoperable) (c) have you tested this change in a non-production environment? (if you're going to brick a server, do dev first) (d) will we be able to look back later and figure out what change occurred when and by who?

But this often degenerates into the managerial equivalent of a hen party, driven by people with way too much time on their hands, and the value is lost.

Push for a Fast Track Process

[sinequonon]

After many years of working with a CAB, my suggestion is to work with them but try to push for a Fast Track process that will allow you to apply lightweight changes with low risk. It will cut your struggles with the bureaucracy considerably. Also, when appropriate, try to bundle changes together into larger block releases, rather than taking through many small revisions.

GOLD MINE! LOVE IT!

[mcrbids]

OP starts with: "I am the sole sysadmin for nearly 50 servers (win/linux) across several contracts. ..."

This implies that he's paid hourly. Contracts implies that he's a consultant. If there's anything that a consultant craves, it's billable hours...

Re:GOLD MINE! LOVE IT!

[mysidia]

If there's anything that a consultant craves, it's billable hours...

Not billable hours at the cost of doing a crappy job, and most certainly going to get blamed, when the customer blames his company; when the failures caused by consultant's CAB causes downtime.

Re:SCCM

[donaldm]

I think go for easy solution. introduce the patches in batches for the board. ("monday updates for week 32")

I won't comment on MS Windows, although I don't think what you have said will work very well, since I have never seen Production MS centric machines updated on a weekly basis.

Providing information for Linux (Redhat example) is very easy if you have the rpm's. All you need to do is run "rpm -qp --changelog " on every package associated with a particular kernel release/update and provide that information to the Change Advisory Board (CAB) which may result in 100's of package information. This is extremely easy to automate and should only take you a few minutes.

If you provide the above type of info to the CAB then I am quite sure they will do one of two things. 1) Throw a "hissy-fit" (grin) and never want to speak to you again, or 2) Thank you for the information and get back to you in a few months. Of course to keep on their good side you could just give them the changelog of the kernel you are going to install then explain that this is your reference and in the case of a Redhat distribution, which the company has to pay for, this should be enough although you may want to list all the packages that will be updated and let CAB decide if they need their changelogs as well.

I do have to state that in a production, QA, development and to a lesser extent a test and/or "crash and burn" environment you should have appropriate software contracts in place whether it be for a Linux, Microsoft or Unix solution or even some other OS. Having an appropriate software contract in place should save yourself allot of problems and you actually look good with management especially if you can give CAB the info they require (not necessarily want) that will get the job done quickly and efficiently.

In the case of Linux it is fairly easy to setup (approx a days works) an "in-house" repo "jump" server keeping in mind your network people need to get involved here since all target machines will need network access to this machine (or multiple machines if you have separate networked environments. On your "repo" server (appox 100GB+ needed) make sure the appropriate distribution are kept current (within a week) then create links to a staging area that the software updater programs (ie. yum or apt-get) on the target machines can see which contains the packages that will be updated against a kernel (changelog provided to CAB) that they will be reference against. It must be noted that emergency (ie. security) patches should always (you need to check) have the kernel that the patch came out with which means you should update all packages associated with that kernel. Google and your software provider is your best friend here.

Obviously in the case of a company you must follow "Change Management" procedures and if they don't have one (yes some companies don't have this) make sure there is one in place since this covers you if things don't actually go as planned, then you would need to fall back to the appropriate part of the companies "Disaster Recovery Plan" (your company does have one that is tested, I hope).

Sounds complicated, well it is and it isn't. Basically no company that is serious should be without "Change Management" procedures and an appropriate tested "Disaster Recovery Plan" should contain a section for backups and recovery processes and the policies covering them. I am aware that some people will disagree with me but put yourself in the the shoes of the System Admin who has to explain to Management why the Production machine crashed and/or data was corrupted or lost because procedures were not followed.

Patching

[donaldm]

If all the patches come from Redhat there is no approval its necessary to keep them up to date for security purposes. The same is true for patches pushed out from Microsoft.

Your kidding right! In any company environment you must follow "Change Management" procedures and that usually involves getting written approval from all project managers that are responsible for each project that is installed on the particular machine. On a Production and/or machines it is usually good policy to be at least one month (possibly six) behind Testing.

I am well aware Redhat are very professional however you should never just update without appropriate testing and management approval. As for Microsoft the same concepts apply. The "cowboy" approach may be ok for home use but put yourself in the shoes of someone who has to explain to a really pissed off management why something went wrong when you were not following "Change Management" procedures.

Missing info

[PensivePeter]

You don't say why a Change Advisory Board is wanting to manage every patch - is it over-zealous micro-management or is there a wider governance issue?

A Necessary Evil

[haxin]

Take everything I say with a grain of salt: I'm not in management and don't have 20 years of system engineer or system administrator experience. We recently implemented a change advisory board and while it's not perfect, it seems to meet our needs without requiring too much. While I haven't read every comment here, many are filled with cynical comments but no matter how cynical you become, it's never enough to keep up. But there are also loads of very helpful and useful comments too. It’s been a good couple of hours well spent so far. There was a time when we shot from the hip. A change would be made that would ultimately affect dozens/hundreds of users resulting in loads of calls to the help desk. At some point management would be alerted to the ‘trend’ in all the calls that would result in an investigation which often led to "Oh yeah, this 'tiny' change was made an hour ago." Now that the [potential] source was identified, the work was double checked by the responsible parties, often with a few managers standing nearby, until the problem was found & corrected or the change reverted. There was a lot of foot shooting going on. We’re not idiots, but we’re not perfect either which means that sometimes mistakes happen. And occasionally, even after having done all the research, risk & impact assessments, unexpected complications would arise. I'll admit, there was something nice about operating autonomously, without being micromanaged, scrutinized and often provided anything but constructive criticism; And it was great not having to deal with the bureaucratic red-tape one often has to go through to get a simple a change done. But as someone else pointed out, the catalyst that brought about this change was the perceived perception of an unstable system due to ‘lower than acceptable’ success rates when changes were made. When we adopted some form of change control, which later morphed into a change advisory board, trips to the ER for bullet wounds in the foot dropped dramatically. And when something did go wrong, we weren't fearful for having made an ‘unauthorized’ change. I don’t think I’m one to resist change. More often than not, I'm the one trying to drive a change and am rarely affected by someone else's change. And when I am, it usually doesn't require a massive cultural, routine, behavioral etc. change on my part. So when it came time to implementing some form of change control, I could understand how it was beneficial and why it was necessary. I’ll admit, it wasn't easy and required some getting used to, but I have an appreciation for it does for us. But IMHO, it sounds like, for many, the real crux of the issue is *how* a CAB is implemented. I realize every organization is different, but it goes a little something like this on this side of the fence: - Create your change request, which amounts to filling out an online form including things like who is doing the work, why are we doing it, how this affects our users, what’s the procedure to make the changes, what’s the testing process, what’s the back out plan etc.. You’re encouraged to include as much detail here as possible. Strongly. Encouraged. - Then you have to ‘socialize’ the changes with the [affected] departments/department heads. This is kind of a gray “wild card” area as it could be a number of individuals, and you could potentially find yourself repeating the same thing multiple times a day over several days. As such, I suggest holding a regular meeting a day or two before the cab, invite ‘the powers that be’ to go over your proposed changes. The ‘socialization’ step is arguably the most important one because if questions come up in the CAB, or if just one person isn't comfortable, it almost guarantees it’ll be denied until you work it out. Because of that, I personally think this is absurd and loathe the process, but I obey. - Finally on CAB day it should be a slam dunk beca