Ask Slashdot: System Administrator Vs Change Advisory Board

thundergeek (808819) writes "I am the sole sysadmin for nearly 50 servers (win/linux) across several contracts. Now a Change Advisory Board (CAB) is wanting to manage every patch that will be installed on the OS and approve/disapprove for testing on the development network. Once tested and verified, all changes will then need to be approved for production. Windows servers aren't always the best for informing admin exactly what is being 'patched' on the OS, and the frequency of updates will make my efficiency take a nose dive. Now I'll have to track each KB, RHSA, directives and any other 3rd party updates, submit a lengthy report outlining each patch being applied, and then sit back and wait for approval. What should I use/do to track what I will be installing? Is there already a product out there that will make my life a little less stressful on the admin side? Does anyone else have to go toe-to-toe with a CAB? How do you handle your patch approval process?"

Comments

Nonsense

[ruir]

They want bureaucracy, they make the paperwork. Tell them to track windows and distro security pages, the changes are there. I would be toasted with that kind of tape, I updated my servers in a pinch immediately after the first news of heartbleed at 3 in the morning. 0300AM right. How about dusting your resume and changing jobs? Let them play the shuffling reports game alone.

Re:Nonsense

[Anonymous Coward]

Any decent change control process should have an emergency change category and process.

Re:Nonsense

[Opportunist]

This. Ask them if they have taken care of things like this. The answer to this alone will tell you whether there is some kind of deep consideration behind it or whether some PHB had a consultant toss the cool buzzword "change advisory board" in his direction.

If it's the latter, run. Run like the wind.

Re:Nonsense

[timepilot]

Dr. Seuss: “Oh, the jobs people work at! Out west near Hawtch-Hawtch there's a Hawtch-Hawtcher bee watcher, his job is to watch. Is to keep both his eyes on the lazy town bee, a bee that is watched will work harder you see. So he watched and he watched, but in spite of his watch that bee didn't work any harder not mawtch. So then somebody said "Our old bee-watching man just isn't bee watching as hard as he can, he ought to be watched by another Hawtch-Hawtcher! The thing that we need is a bee-watcher-watcher!". Well, the bee-watcher-watcher watched the bee-watcher. He didn't watch well so another Hawtch-Hawtcher had to come in as a watch-watcher-watcher! And now all the Hawtchers who live in Hawtch-Hawtch are watching on watch watcher watchering watch, watch watching the watcher who's watching that bee. You're not a Hawtch-Watcher you're lucky you see!”

Re:Nonsense

[OzPeter]

So... the business made a stupid decision, and when they realised the error of their ways, rather than trying to reach agreement on the best way forward, you delighted in rubbing their noses in it, using processes designed to protect you to hurt your employing organization instead.

If he had said .. "OK .. sure I'll stop sending you those 400 pages of paper per day", then the policy would still have been left in place, and sometime win the future his employer could have used his inability to follow policy as an excuse to ream him over. Yes its CYA, but some employers are not above using any tool at their disposal to justify their actions.

Only by being a genuine PITA does the stupid police get removed, rather than ignored until convenient.

Re:Nonsense

[mysidia]

like this will just make you look stupid and change averse to your employer.

No... it's obviously just aversity to excessive, unnecessary and crippling micromanagement. It's obviously some idiots in suits who are change averse and feel they need to justify their existence by "approving" or "disapproving" of each and every required security update or patch or system admin action.

Which involves real costs. With this kind of bullshit, they need to hire additional system admins for systems to approach proper management just to deal with the reduced time efficiency and increased waste caused by bureaucracy.

Re:Nonsense

[Registered Coward v2]

So... the business made a stupid decision, and when they realised the error of their ways, rather than trying to reach agreement on the best way forward, you delighted in rubbing their noses in it, using processes designed to protect you to hurt your employing organization instead.

If he had said .. "OK .. sure I'll stop sending you those 400 pages of paper per day", then the policy would still have been left in place, and sometime win the future his employer could have used his inability to follow policy as an excuse to ream him over. Yes its CYA, but some employers are not above using any tool at their disposal to justify their actions.

Only by being a genuine PITA does the stupid police get removed, rather than ignored until convenient.

While what you say is true, it is not necessarily the best way to deal with stupid policies. In the end, the PITA gets remembered long after the stupid policy is forgotten; all people know is someone was a PITA and eventually will be made to pay for it. You could, for example, send all links on one page and have that be signed. After a few were signed its's time to broach the "is there a better way to do this" argument. Malicious compliance with rules generally results in pissing people off and payback at some future point; it can even be fun watching the PITA get a taste of their own when the opportunity arises.

Re:Nonsense

[MachineShedFred]

It should also have a method of implementing a "standard change" - meaning a change that repeatedly happens, and has a good track record of being successful with minimal fallout. Implement a WSUS test server for having a patch test environment, and point some less critical systems at it. Should anything go wrong there, you'll then know what patches to NOT apply automatically to the critical infrastructure.

Change management and review is there to help you, not be a pain in the ass. Remember - if they sign off on your documented change, they share the responsibility should something go sideways.

Re:Nonsense

[Ash Vince]

They want bureaucracy, they make the paperwork. Tell them to track windows and distro security pages, the changes are there.

Yep. They're the "experts". Just tell them the Microsoft KB number, that's all the information they need.

Yup, follow this advice and come across like an unhelpful douchbag.

Or, bend over backwards to help them. Provide them with a break down on every single patch (a few line summary with a link to the KB article for the full details), then give each patch a priority based on its impact and come up with different deployment routes for each one, then explain to your manager who allocates your time why patch management for the CAB board just became a full time job.

Also, if they ever reject any and you end up a dependency hell where you cannot install a critical patch because of a low impact one you rejected (you do test each patch deployment run on a dummy server don't you?) then explain why the process failed, politely, without saying thing like "I told you this was dumb years ago!".

Alternatively, if the system runs for a few months and every single patch sent to the CAB board has been approved then you can clearly demonstrate the do not really add anything and start making rational arguments to abandon the process from a sound basis while demonstrating you are an excellent team player who easily adapts.

But if you would rather come across like a non-team player who hates any interference in your system admin fiefdom then, just go with the douche bag option and watch your job get outsourced in 6 months.

In my experience the world of work is full of crap like this, times when processes that are overly bureaucratic are forced on us techies even though we clearly see them as a waste of our time. Unfortunately this is generally just stuff we have to lap up as part of our job, if you can, you generally end up earning more and with the greater long term job security that working as part of a larger company provides.

An excellent book on this sort of business related stuff is called "Who moved my cheese" and the gist of it is that you want to come across as and "enabler" rather than a "blocker". That often means trying your best to make what is clearly a stupid idea a success.

Re:Nonsense

[MrNemesis]

So refusing to comply with an order that's in direct violation of your contract is acting like an arsehole now? And you're happy for the rule you're being routinely forced to violate in the course of your professional duties to be left on the books to trip up the next person who doesn't have the guts to stand up and say "no, I won't shoot myself in the foot"? Will HR even remember you have a signed waiver before marching you out of the building for knowingly violating company policy?

Sorry, but no. If you're stuck in a Kafkaesque situation like the GP was, the only professional thing you can do is give them exactly what they want. Especially when you've explained to them why giving them exactly what they want will be bad and they've given a written response that amounts to "we don't care, do it anyway".

If you act like something that badly needs fixing doesn't need fixing and you're happy to see people and companies ruined over it, but all means keep thrusting ones cranium into the pulverised silicon dioxide. Some people might say you're acting like an arsehole however.

Re:Nonsense

[Ash Vince]

The crown was an insane rule that every new hyperlink had to be aproved not just by a department head but by the vice chancellor himself.

At that point you should have just emailed everyone on the committee, and copied in the vice-chancellor with some stats on exactly how many approvals this would generate on a daily basis. Include the actual statistics for the previous 7 days so if this was generating 50 pages per day you had some clear number to back this up while still in the planning stage. That was clearly why you were put on the committee, to stop a bunch of know nothings from coming up with a stupid policy, you failed.

The way to succeed as a techie is not about being technically brilliant any more, it is about how you can talk people round to your way of thinking and use evidence to back up your points of view.

Re:Nonsense

[plover]

"Yes men", or "enablers" as you call them, are the root cause of the out-of-control bureaucracy problem.

Management wants desperately to have processes that can be followed by minimum wage desperate people. They want to believe that they're incredibly smart and insightful, and that their knowledge of their business is so absolute and perfect that a process is easily applied to every situation. The enablers then create the real problems by saying "yes, we'll make this work," when they are actually lying. The problems then get worse, because the enablers also feel that every status report must be green, otherwise their process is not as perfect as they said it was.

The transparent lies breed more processes to control the mistakes that never seem to get fixed, despite the change review board processes. Nobody ever questions the process, because they'd be seen as a blocker. Change becomes impossible, because the middle-management process owners who have been lying "yes" will lose their jobs if their process is removed.

The process explosion reaches critical mass as each failed process begets two more to control it. The business goes into a death spiral of bloat and inefficiency.

Everything is painful. The smart middle managers flee early, leaving only the enablers behind, and they refuse to see or acknowledge any problems. The loyal people have their jobs turned into paperwork and outsourcing. With luck, the board will recognize the out of control costs, and bring in a lean outsider. An organization this bad off may need to fire 50%-90% of the people. Or the board may go the wrong direction, and outsource the whole damn mess, further eroding their ability to change.

This philosophy has caused more damage to business and productivity than any other idea, ever.

Avoiding this is simple, in theory. Tell the truth, and don't be afraid to change your mind if shown evidence that you're wrong.

Re:Nonsense

[Slashdot Parent]

Yup, follow this advice and come across like an unhelpful douchbag.

Or, bend over backwards to help them. Provide them with a break down on every single patch (a few line summary with a link to the KB article for the full details), then give each patch a priority based on its impact and come up with different deployment routes for each one, then explain to your manager who allocates your time why patch management for the CAB board just became a full time job.

So you advocate becoming a passive-aggressive douchebag instead? If you don't have enough time for a new job responsibility, then you need to say something right away, not burn up a bunch of time of menial tasks and wait until your boss notices the productivity drop.

Alternatively, if the system runs for a few months and every single patch sent to the CAB board has been approved then you can clearly demonstrate the do not really add anything and start making rational arguments to abandon the process from a sound basis while demonstrating you are an excellent team player who easily adapts.

You know, it's interesting. My current client has a CAB process, and they do a good job of balancing process and agility.

Anyway, they don't use it to micromanage every stupid little patch. They recognize that the Linux admins are trained professionals and know which patches to apply. Same with the DBAs and same with the Network admins and same with the storage admins. Nobody questions them down to the individual patch level unless there is a known conflict with a specific patch, which basically never happens.

What they use it for is for auditing and coordination. Multiple groups never patch during the same maintenance window without a good reason (e.g. Heartbleed. Network and Linux both patched at the same time to get the patch out ASAP. But this was a rare event.). In other words, let's say the DBAs and the Linux admins both have patches that need to be applied. Well, they never patch during the same window, so if something breaks, we know that the problem is isolated to the database patch vs. the Linux patch. Also, if we get a report that something stopped working right on, say, 3/8/2014, then we know that the issue was with a patch to their main web proxy server.

CAB type groups can be a valuable, but it's important to find the sweet spot between micromanagement and chaos. It's good to have all of the various groups know what each other are doing and can advise each other regarding potential issues. It's not like when the DBAs do patching that everybody discusses the Oracle changelogs in great detail for half a day or anything.

Re:Nonsense

[fair_n_hite_451]

Go directly to your Change Manager (not the CAB, but the person who is invested in the day-to-day management and running of the process).
Utter the following sentence "I wish to get regular patching of Windows Servers defined as a standard change, what do I need to do?"
Follow those instructions.

It might seem like more beauracracy for the first run through, but it'll be smooth and seamless after that. As a Change Manager for years, I can tell you that the people in that position are far more worried about project managers trying to push through craptastic updates at the last second than they are with competant domain support people. They'd love to get you off their radar just as much as you want to be off the CAB's radar.

Re:Nonsense

[MrNemesis]

This. Absolutely, 100% this.

As I've alluded to in my other posts, as soon as I graduated from cowboy sysadmin to a "proper" sysadmin that files change requests and writes project documentation, I've come to love change managers for precisely the reasons above. Change managers are under continual bombardment from non-technical project managers and developers that might well have deep, deep insight into a certain area but can't see past the end of their nose. A good change manager will often trot up to us sysadmins and say "So-and-so has submitted this change but doesn't think it needs approvals from you guys, can you take a gander?" to be met with either a "yeah that's fine" or a "Holy crappingon what-the-fuck in a god-buggered handbasket NO!". Good sysadmins in a constructive environment see a bigger picture than the project managers and the developers and, as far as CAB is concerned, submit better change requests as a result - because risk analysis is such an innate part of our job that most of us don't even realise we're doing it. But change managers see a bigger picture still because they're exposed to the sysadmins, network admins, security admins, user admins, mail admins, storage admins, admin admins, admin users, sysadmin networks, bread, eggs and breaded eggs.

Change managers exist to protect the business. Sysadmins exist to run the business' IT. Change managers realise that sysadmins are often asked to do dangerous or even outright impossible things by powerful people with only an inkling of what consequences such an action might have; it's a change manager's job to communicate with and understand the sysadmin (and everyone else) in such repsects, just as it's the sysadmins' responsibility to communicate to the business why change X is crucial or dangerous. In a properly functioning IT dept, sysadmins and change managers protect both each other and the business from stupidity, mis-co-ordination and lack of oversight. As a sysadmin, change managers are almost always on your side - either pushing for that change that's so essential, or holding you back where there's a risk. They're a highly valuable ally. When something goes to shit, they're the first people to step in and say "no, the sysadmins had nothing to do with this incident".

I'm MrNemesis and in the last three years I've learned to love my change managers.

perhaps

[dimko]

New product your comapny requires is called: junior admin? Expensive stuff but does the job.

ask yourself *why* and do the right thing

[flinkflonk]

This is known as the change process in ITIL, and it does have a remedy. The remedy is pre-approved changes (standard changes), which should include patching the OS with patches approved by the vendor. It's meant for exactly this situation, and if your change process doesn't have them it's just a paper wall.
The ITIL change process is all about reducing risk. If there is a risk with patching your OS (there is, especially since you mention Windows, it's not that unheard of that a Windows patch makes your whole network inoperative) you have to weigh it against the risk of not patching it (meaning you leave known security holes in).
So, my advice is to get OS patches for your OSes pre-approved by the CAB, that is, when a vendor releases a set of patches you are allowed to patch your systems in the way and the order of that pre-approved change. Of course it's paper-pushing, but use it to your advantage and push some paper yourself. If a server gets compromised and you have the papers (changelog) to prove that you followed procedure, blame will be placed somewhere else. And things will be done differently from there on, since it has been proven that the procedure didn't work, and everybody wins.
Or you could go find another job (like some other posters recommended) where you are the sole *cowboy*-admin and nothing gets done properly. Your choice really.

Run away!

[arcade]

Given your description, you're the sole sysadmin. This means you're the person who should take these decision - nobody else. If the company disagrees with this, then either you've done a poor job previously, or they don't trust you to do your job for some strange reason.

Now, if it's you that have fscked up on previous occasions, then it's understandable that they want the red tape.

If you haven't, then it's time to put down the foot and say "Nope, that's my job". If they disagree with that - linkedin should be a relatively short distance away, and after you find yourself a new job - simply hand in your resignation pointing out that you have no interest in having babysitters.

Re:Patching....

[rioki]

I totally agree with the above. These change review rigmarole is often done for reasons of security and operational stability. This is a laudable goal, but often the added red tape make the entire system more vulnerable when they want to decide which security fixes get applies. You need to hammer it home that each second between the time the security fix is published and the time the fix is applied the systems are vulnerable. This is because, once the security fix is published, every hacker knows about the issue too. If you have something worthwhile to protect, which is probably the reason why a change review board was established, you do not want add more time to that time window. If they need red tape, you should get a blanket agreement that you apply security fixes from vendors for critical software (OS, databases, etc.) ASAP and they get a notification of when and what patch was installed.

They have a point

[distilate]

As a software developer I have multiple times had a development box screwed over by an IT department pushing unneeded drivers and patches that cause problems. I say prove they are good or needed before you waste other peoples time. If you just want to push any random patch that comes along then you should be forced to resolve all issues without the traditional reinstall the machine.

Re:They have a point

[Anonymous Coward]

And you sir, are why most people hate IT.

In short, yes, I do expect you to waste your time to pick and choose which patches so I can not lose that spare second. After all, it's YOUR JOB to keep the computers running well. If you can't be bothered to do it, then what's the point of you being employed? My job as a developer is to develop products, not to battle with my machine. By you not doing your job properly and by approving a patch that takes out my machine, I'm now unable to do my job. And likely, I'm not the only person who will have been effected by this issue, so thus, by you not doing your job, you may have now cost 10, 20, 1000 people a couple hours where they're not doing their assigned tasks.

If you can't be bothered to as a sys admin, then in my opinion, a company shouldn't be bothered to employ you.

Re:Patching....

[N1AK]

If you have something worthwhile to protect, which is probably the reason why a change review board was established, you do not want add more time to that time window.

No, CABs often get implemented because someone is worried about the damage a borked patch/update could do and doesn't have confidence that it could be reliably fixed quickly. Most of the 'admin' in a change request is things like a process plan (which surely you already know if you're deploying an update to a critical live system) and a rollback process (which again, surely you should be considering before risking fubaring the system).

What I will say is that you should ensure that the CAB members are aware of the need to be able to handle emergency requests (meet, agree and deploy in hours) and should have some process to handle retrospective requests if a business critical update comes out and you can't wait for CAB approval. Normally the requirements for retrospective requests is that it's genuinely critical and that you send a completed request before the update. It might sound odd, but the idea is that they can use that to see if you had properly thought through the process and not just gone Rambo on it.

As a Change Manager...

[Pete (big-pete)]

I work in Change Management for a major telco, I chair the IT CAB, and I oversee server and client patching (amongst many other changes!). When we patch clients, we are patching up to around 30,000 real and virtual desktops - when we patch servers, they also number in the thousands.

There is no way we would allow a sysadmin to patch anything at any time without some level of oversight, an individual admin has no oversight on other patches, hardware interventions, application releases, network upgrades, business campaigns, etc that may be happening on our environment at any given moment (this isn't their job to be keeping track of all of that info). For server and client patching is as light as possible, but we still maintain a close oversight.

On the Wednesday following the second Tuesday of each month (for example), I sit down with the Windows server guys and the Windows client guys, and we review their proposals to patch - usually we have a fairly rapid timescale that we can meet to ensure that the patches are deployed (including pilot testing, etc to catch any issues before everyone's desktop is broken!), sometimes there are other major interventions that overlap, and then we need to make prioritisation decisions and decide which has priority. We have made similar agreements with the Linux teams, where they have a special process to patch, and we have close oversight on Unix patches, as upgrading these servers with a reboot can be a very big deal.

The last thing you want is an application version release of a critical ordering application happening at the same time as a system software patch, and then to have an issue afterwards - is it the application version, is it the systems patch, was there some conflict with the activties being performed at the same time? Troubleshooting gets more difficult, teams point fingers at eachother, and the whole time the business is screaming blue murder.

Of course in an Incident situation there is more flexibility to get things fixed fast, and with security issues I am keen to break open the S-CAB process to expedite a rapid approval flow to ensure that security holes are fixed as fast as possible - of course most changes are encouraged to follow the rules though, the change calendar is published, and everyone knows when the "standard" slots for deployment are, and if most people manage to schedule their changes within those windows, then it minimises potential conflict for everyone.

Change management are not your enemy, they are your friend - once you register your change with them, they have your back, they will guard from other interventions clashing with you, will stop you from inadvertently upsetting the business, and will decrease change related Incidents. However, with great power comes great responsibility, and Change Management need to find the right process for the right type of change - we cannot have a full in depth investigation into every configuration change, every patch, every bug-fix, every new server to be provisioned. A good Change Management team will guide changes to the appropriate flow, and grease the wheels for certain types of interventions - it seems that the CAB mentioned in the summary are still finding their feet a little, and I am sure they will evolve over time as they start to understand which changes are high risk, and which can be allowed to pass with a lighter touch.

-- Pete.

Re:Patching....

[Zocalo]

Blanket approvals and template documents that you can cut and paste notifications into are the way to go, especially when it's on a schedule like MS, Adobe & Oracle. If they push back, suggest a documented process (this is ITIL, right? You can avoid the need for a CAB if it's an approved and documented procedure) where you push the patches to a few test systems on Tuesday (in the case of MS) then deploy to the rest later in the week - whatever they are happy with - if there are no issues. Depending on your timezone Tuesday PM or Wednesday AM are good slots for weekly CABs to pick up this; push to the test servers on the day, then the rest at the end of the week. For *nix, i've done updates this way for anything that didn't require a reboot so only stuff like Kernel updates and major low-level libraries needed to get approval via a CAB.

For everything else, it's your call. Either the patch waits for the next regular CAB or you play the game and keep calling emergency CABs when there are justifiably critical updates, such as Heartbleed, or for the inevitable critical updates from MS every second Tuesday that impact your systems. The best tactic is to embrace ITIL and make it work for you, not allow them to make you jump through hoops and spend your time crafting unique documents for every patch. It also serves as a useful procedure check to make sure you don't mess up and have a contingency plan for when you do, and ultimately, if you get it right, you still get to dictate the schedule and make them do things in ways that you are happy to work with.

Re:Are you still partying like its 1999, or what?

[sjames]

That's not the big leagues, that's the short bus.

yes, changes need to be documented. They should be deployed on a test server before going into production. The rest is just people who were presumably traumatized by falling out of a tree as a child seeking revenge.

Take the people in the CAB and replace them with extra admins who are bright enough to know what I said in the 2nd paragraph.

Change management can cover your ass

[Madman]

There is genuine value in a well-run change management program. Organizations need to know what is going on in their infrastructure, and plan things properly. In many industries there is a growing regulatory requirement to have change management, and auditors are looking for these things more often too. Many smaller shops are bringing in change control, so rather than handing in your badge my advice would be to deal with it and learn the lessons.
One lesson is rather than fight it, use it to your advantage. Yes, there's paperwork, however if you follow the system correctly they cannot blame you if things go wrong. What you thought of as freedom was also a risk to your own position as you had sole responsibility - change control means less freedom, but you are covered. Also, you can get budget for better management systems which will make your life easier. Put together a realistic list of what you need and get involved with setting up the change control process. If you stay silent or fight it you won't get a say.

Re:SCCM

[gl4ss]

sure, but how does that help with having to run the CAB through 102 patches?

I think go for easy solution. introduce the patches in batches for the board. ("monday updates for week 32").

the fucking board will not care after 2 weeks anyways so just do lip service for two weeks.

Servers are for applications...

[GoChickenFat]

I've been an admin for a very long time. What I see is a lot of admins think the OS is the most important and fail to understand why the server even exists in the first place. If you patch simply because it was made available, you don't test or know what the application the server is hosting does at all, then are you really doing what is best? Yes, patches break things and often the patch "fixes" something that was low or no risk inside the corporate network to begin with. Too many admins fail to balance the risks with application uptime. ...and that's why you end up with a CAB - to keep everyone informed, to balance risk and to account for audit controls. These usually pop up after too many system outages or lack of information sharing. Admins have a bad habit of being too smart and too busy to keep others informed. I have worked with a lot of CAB's in many companies and the best way to work with them is to be proactive in keeping them informed and to build a trust relationship in advance.

Re:SCCM

[Enigma2175]

Yeah, after a couple of weeks of having to run through a few hundred patches at a time (make sure you write at least a page for each patch!) they'll get the hint that this is fucking retarded and back off.

I think and your parent underestimate the ability of committees to do work that is fucking retarded. I can't count the number of fucking retarded processes at my company that people have been happily doing for years.