Slashdot Mirror
Ask Slashdot: System Administrator Vs Change Advisory Board
thundergeek (808819) writes "I am the sole sysadmin for nearly 50 servers (win/linux) across several contracts. Now a Change Advisory Board (CAB) is wanting to manage every patch that will be installed on the OS and approve/disapprove for testing on the development network. Once tested and verified, all changes will then need to be approved for production. Windows servers aren't always the best for informing admin exactly what is being 'patched' on the OS, and the frequency of updates will make my efficiency take a nose dive. Now I'll have to track each KB, RHSA, directives and any other 3rd party updates, submit a lengthy report outlining each patch being applied, and then sit back and wait for approval. What should I use/do to track what I will be installing? Is there already a product out there that will make my life a little less stressful on the admin side? Does anyone else have to go toe-to-toe with a CAB? How do you handle your patch approval process?"
What we normally do is get a blanket approval if its coming from the OS provider with an understanding that patching will be done on a specific schedule.
IE. If all the patches come from Redhat there is no approval its necessary to keep them up to date for security purposes. The same is true for patches pushed out from Microsoft.
Then your only dealing with 3rd party applications. Even those the more common ones we get added to the blanket approval, ie. Adobe. This way you are only telling them you are bringing them into line with the latest set of patches provided by the OS vendor without having to list all the packages that are being updated. Then they only have to ask you if a program has or does not have a certain bug.
I have to do this and it's no problem at all, although our change management process doesn't sound quite as onerous as yours (I suspect yours will adapt over time -- the CAB will soon get bored if they have to approve every single OS patch).
I have to do a risk analysis for each change that gets made to a system (not just patches). Sometimes this risk analysis is fairly informal, for example if the change is to add more RAM to a VM, it's very unlikely to have a significant adverse impact and is easily reversible, so low risk. Other times the risk analysis (and processes that come out of that) may take a long time and require significant co-ordination with other parts of the organisation I work in.
A good example is if we make a change to a service that impacts the look and feel of that service. It will require co-ordinating with our communications, helpdesk, training and documentation teams as well as other parts of the technical group I work in and the CAB really acts as a check to make sure all of that has happened properly.
There are still a few people in our organisation who see the CAB as a barrier to getting work done, but for me it is really a check to make sure we're delivering changes in a proper way.
I can recommend you take a look at The Phoenix Project by Gene Kim, Kevin Behr and George Spafford. http://itrevolution.com/books/... - I had quite a few "this is where I work" moments whilst reading it :)
Setup a WSUS server, you probably already have the licenses. From there you can pull the patches to it and then push it to needed servers as approved.
There are commercial products that can also this in a nicer manner but they cost money.
Any decent change control process should have an emergency change category and process.
I work in Change Management for a major telco, I chair the IT CAB, and I oversee server and client patching (amongst many other changes!). When we patch clients, we are patching up to around 30,000 real and virtual desktops - when we patch servers, they also number in the thousands.
There is no way we would allow a sysadmin to patch anything at any time without some level of oversight, an individual admin has no oversight on other patches, hardware interventions, application releases, network upgrades, business campaigns, etc that may be happening on our environment at any given moment (this isn't their job to be keeping track of all of that info). For server and client patching is as light as possible, but we still maintain a close oversight.
On the Wednesday following the second Tuesday of each month (for example), I sit down with the Windows server guys and the Windows client guys, and we review their proposals to patch - usually we have a fairly rapid timescale that we can meet to ensure that the patches are deployed (including pilot testing, etc to catch any issues before everyone's desktop is broken!), sometimes there are other major interventions that overlap, and then we need to make prioritisation decisions and decide which has priority. We have made similar agreements with the Linux teams, where they have a special process to patch, and we have close oversight on Unix patches, as upgrading these servers with a reboot can be a very big deal.
The last thing you want is an application version release of a critical ordering application happening at the same time as a system software patch, and then to have an issue afterwards - is it the application version, is it the systems patch, was there some conflict with the activties being performed at the same time? Troubleshooting gets more difficult, teams point fingers at eachother, and the whole time the business is screaming blue murder.
Of course in an Incident situation there is more flexibility to get things fixed fast, and with security issues I am keen to break open the S-CAB process to expedite a rapid approval flow to ensure that security holes are fixed as fast as possible - of course most changes are encouraged to follow the rules though, the change calendar is published, and everyone knows when the "standard" slots for deployment are, and if most people manage to schedule their changes within those windows, then it minimises potential conflict for everyone.
Change management are not your enemy, they are your friend - once you register your change with them, they have your back, they will guard from other interventions clashing with you, will stop you from inadvertently upsetting the business, and will decrease change related Incidents. However, with great power comes great responsibility, and Change Management need to find the right process for the right type of change - we cannot have a full in depth investigation into every configuration change, every patch, every bug-fix, every new server to be provisioned. A good Change Management team will guide changes to the appropriate flow, and grease the wheels for certain types of interventions - it seems that the CAB mentioned in the summary are still finding their feet a little, and I am sure they will evolve over time as they start to understand which changes are high risk, and which can be allowed to pass with a lighter touch.
-- Pete.
Monochrome - Probably the UK's largest internet BBS
System Administrator Vs Change Advisory Board
50 quatloos on the newcomer!
systemd is Roko's Basilisk.
Back when I worked as a web administrator at my local university back in the early 2000s, the admin make-work types decided to bash out a web policy , mostly to keep standards up and guard against legal liability (Admittedly we had students setting up websites on chemistry lab pcs turned webserver with novel meth recipes and all sorts of shenanigans before that). All good and fine, I asked to be on the committee as an advisor, and so I was.
Then the whole thing went off the rails, every page needed to be approved by a department head, 10,000+ pages of previously existing data had to be retrofitted with full dublin core metadata descriptions, and so on and so on for about 400 pages of rules and policy that despite my best efforts I could not stop. These people had no fucking idea.
The crown was an insane rule that every new hyperlink had to be aproved not just by a department head but by the vice chancellor himself.
And so thats what I did, and I made sure it was done good and proper. I wrote a perl script that took all new pages on the webserver network (about 50-100 new pages a day) and then whenever a hyperlink appeared it spat out a 1 page document for approval *per link* requiring the vice chancellor and a lawyer to co-sign off on. All with witnesses. All in all about 400 pages a day of paperwork for the vice chancellor and a lawyer.
The policy lasted 3 days before I was dragged into the admin building to be ordered to stop producing the reports. I went in with my union rep. I said "Sorry , no , thats the official policy as passed by the university senate and the website will need to be shut down if this isn't done.". Since the next senate meeting was two weeks away, I made sure every god damn day that stack of paperwork was done by the vice chancellor for a glorious fortnight before the senate could revoke the whole damn policy.
It was a magical and golden time to be a union protected government (Universities are mostly run by the state in australia) employee.
For some reason later that year I was passed over for a promotion though. I wonder why, lol.
Excuse the Unicode crap in my posts. That's an apostrophe, and slashdot is busted.
There is genuine value in a well-run change management program. Organizations need to know what is going on in their infrastructure, and plan things properly. In many industries there is a growing regulatory requirement to have change management, and auditors are looking for these things more often too. Many smaller shops are bringing in change control, so rather than handing in your badge my advice would be to deal with it and learn the lessons.
One lesson is rather than fight it, use it to your advantage. Yes, there's paperwork, however if you follow the system correctly they cannot blame you if things go wrong. What you thought of as freedom was also a risk to your own position as you had sole responsibility - change control means less freedom, but you are covered. Also, you can get budget for better management systems which will make your life easier. Put together a realistic list of what you need and get involved with setting up the change control process. If you stay silent or fight it you won't get a say.
sure, but how does that help with having to run the CAB through 102 patches?
I think go for easy solution. introduce the patches in batches for the board. ("monday updates for week 32").
the fucking board will not care after 2 weeks anyways so just do lip service for two weeks.
world was created 5 seconds before this post as it is.
This. Ask them if they have taken care of things like this. The answer to this alone will tell you whether there is some kind of deep consideration behind it or whether some PHB had a consultant toss the cool buzzword "change advisory board" in his direction.
If it's the latter, run. Run like the wind.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Dr. Seuss: “Oh, the jobs people work at! Out west near Hawtch-Hawtch there's a Hawtch-Hawtcher bee watcher, his job is to watch. Is to keep both his eyes on the lazy town bee, a bee that is watched will work harder you see. So he watched and he watched, but in spite of his watch that bee didn't work any harder not mawtch. So then somebody said "Our old bee-watching man just isn't bee watching as hard as he can, he ought to be watched by another Hawtch-Hawtcher! The thing that we need is a bee-watcher-watcher!". Well, the bee-watcher-watcher watched the bee-watcher. He didn't watch well so another Hawtch-Hawtcher had to come in as a watch-watcher-watcher! And now all the Hawtchers who live in Hawtch-Hawtch are watching on watch watcher watchering watch, watch watching the watcher who's watching that bee. You're not a Hawtch-Watcher you're lucky you see!”
It should also have a method of implementing a "standard change" - meaning a change that repeatedly happens, and has a good track record of being successful with minimal fallout. Implement a WSUS test server for having a patch test environment, and point some less critical systems at it. Should anything go wrong there, you'll then know what patches to NOT apply automatically to the critical infrastructure.
Change management and review is there to help you, not be a pain in the ass. Remember - if they sign off on your documented change, they share the responsibility should something go sideways.
Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
They want bureaucracy, they make the paperwork. Tell them to track windows and distro security pages, the changes are there.
Yep. They're the "experts". Just tell them the Microsoft KB number, that's all the information they need.
Yup, follow this advice and come across like an unhelpful douchbag.
Or, bend over backwards to help them. Provide them with a break down on every single patch (a few line summary with a link to the KB article for the full details), then give each patch a priority based on its impact and come up with different deployment routes for each one, then explain to your manager who allocates your time why patch management for the CAB board just became a full time job.
Also, if they ever reject any and you end up a dependency hell where you cannot install a critical patch because of a low impact one you rejected (you do test each patch deployment run on a dummy server don't you?) then explain why the process failed, politely, without saying thing like "I told you this was dumb years ago!".
Alternatively, if the system runs for a few months and every single patch sent to the CAB board has been approved then you can clearly demonstrate the do not really add anything and start making rational arguments to abandon the process from a sound basis while demonstrating you are an excellent team player who easily adapts.
But if you would rather come across like a non-team player who hates any interference in your system admin fiefdom then, just go with the douche bag option and watch your job get outsourced in 6 months.
In my experience the world of work is full of crap like this, times when processes that are overly bureaucratic are forced on us techies even though we clearly see them as a waste of our time. Unfortunately this is generally just stuff we have to lap up as part of our job, if you can, you generally end up earning more and with the greater long term job security that working as part of a larger company provides.
An excellent book on this sort of business related stuff is called "Who moved my cheese" and the gist of it is that you want to come across as and "enabler" rather than a "blocker". That often means trying your best to make what is clearly a stupid idea a success.
I dont read
"Yes men", or "enablers" as you call them, are the root cause of the out-of-control bureaucracy problem.
Management wants desperately to have processes that can be followed by minimum wage desperate people. They want to believe that they're incredibly smart and insightful, and that their knowledge of their business is so absolute and perfect that a process is easily applied to every situation. The enablers then create the real problems by saying "yes, we'll make this work," when they are actually lying. The problems then get worse, because the enablers also feel that every status report must be green, otherwise their process is not as perfect as they said it was.
The transparent lies breed more processes to control the mistakes that never seem to get fixed, despite the change review board processes. Nobody ever questions the process, because they'd be seen as a blocker. Change becomes impossible, because the middle-management process owners who have been lying "yes" will lose their jobs if their process is removed.
The process explosion reaches critical mass as each failed process begets two more to control it. The business goes into a death spiral of bloat and inefficiency.
Everything is painful. The smart middle managers flee early, leaving only the enablers behind, and they refuse to see or acknowledge any problems. The loyal people have their jobs turned into paperwork and outsourcing. With luck, the board will recognize the out of control costs, and bring in a lean outsider. An organization this bad off may need to fire 50%-90% of the people. Or the board may go the wrong direction, and outsource the whole damn mess, further eroding their ability to change.
This philosophy has caused more damage to business and productivity than any other idea, ever.
Avoiding this is simple, in theory. Tell the truth, and don't be afraid to change your mind if shown evidence that you're wrong.
John
Yeah, after a couple of weeks of having to run through a few hundred patches at a time (make sure you write at least a page for each patch!) they'll get the hint that this is fucking retarded and back off.
I think and your parent underestimate the ability of committees to do work that is fucking retarded. I can't count the number of fucking retarded processes at my company that people have been happily doing for years.
Enigma
Go directly to your Change Manager (not the CAB, but the person who is invested in the day-to-day management and running of the process).
Utter the following sentence "I wish to get regular patching of Windows Servers defined as a standard change, what do I need to do?"
Follow those instructions.
It might seem like more beauracracy for the first run through, but it'll be smooth and seamless after that. As a Change Manager for years, I can tell you that the people in that position are far more worried about project managers trying to push through craptastic updates at the last second than they are with competant domain support people. They'd love to get you off their radar just as much as you want to be off the CAB's radar.
Reason why there is hope for the future generation #364:
"I wish my grass was emo so it could cut itself."