I agree, innovation is the way to go, but not at the expense of the business. The team has a lot of flexibility in the way they work. But there are always some places where they're not going.
I'm sometimes guilty of the latter, but not of the former. A manual change effected by a sysadmin typing into a shell is different from one embodied in a piece of code, and they're actually subject to different processes. Change/release management does (should!) indeed apply to sysadmin activities, but there are these things called 'standard changes' which cover the typically mundane things like user Add/Move/Changes, that you don't want to go to a Change Control Board each time to get the go ahead. However they also cover stuff your sysadmin does in his/her/(but probably his) day to day work. Like pretty much everything they do, in fact. The weakness is generally nobody knows how a sysadmin spends their day, which makes these things difficult to control. But anyway, one or two changes made by a sysadmin isn't what we're talking about here. We're talking about automation, which means complexity, which means unforeseen results. That must be held to a higher standard and policed accordingly.
Most sysadmins are playing with prod systems to some degree - even if they're just gathering stats, it can have an impact, because the system they are querying still needs to spend cycles responding to the query. I have seen situations where prod performance has been degraded significantly because of best intentions to gather meaningful data. They are rare occasions, but they are still very damaging.
Remember this line?
"Those are the approaches I've taken, although it will ultimately depend on your organisation's skill mix and personnel availability."
If you want to continue changing the parameters of the problem, you can find a flaw in any proposed solution. I'll leave you to move the goalposts on your own.
That depends on what the scripts are for. Most of our infrastructure is supported and monitored by relatively robust enterprise management systems now (which is a reaction to feeling the pain of a decade of un-managed scripting [before my time], and doing something about it) so there's not a huge amount of string-and-glue solutions needed at the moment. There's some low-risk stuff that can be done by *certain* sysadmins as the prime, but supported by a part-time BA and tester, with scripting expert resources (usually a coder) on tap as required. They all work within a defined process - that's the agreement we have with them, and they seem happy with it. For the more complex stuff, which is higher risk by nature, I'm keener to prime it with a coder who knows a scripting language inside out, and who has some level of infrastructure experience. They will have a part time BA & tester working with them, and a sysadmin on tap as required. Those are the approaches I've taken, although it will ultimately depend on your organisation's skill mix and personnel availability. This approach seems to work quite nicely, in that we're achieving predictability in our infrastructure operations.
Ability to cut code isn't a primary reason for hiring a sysadmin. It's absolutely a primary reason for hiring a coder.
And well done. The same rules should apply to everyone. Historically sysadmins haven't been held to the same standards as everyone else. So they have been able to get away with murder.
Absolutely. Knowledge of the lifecycle is one thing, putting it to use in an officially sanctioned capacity is quite another. Your third sentence isn't the case, so it's irrelevant in this context.
Fair call from the complexity standpoint. However the assumption that each program is unrelated to the rest is stretching it, I think. Most situations in a relatively simple environment would call for some level of daisy-chaining (I think the author of the article even suggested some level of interaction between his scripts outputs.) So the complexity moves from the length of an individual script to the number of interaction paths between multiple scripts. And of course, as the infrastructure changes, these scripts have to be maintained, etc etc. So yeah, not the WOAH moment the 15000 lines suggest, but it can still be death by a thousand cuts.
1 line of code, run 500 times, does not present the same level of risk as 500 lines of code run once.
I'm not saying don't run scripts. I'm saying scripts need to be developed carefully, within a process, and there's better people to do them than sysadmins alone.
You're quite correct - scripting *done properly* is the key.
The hobby scripting mentality is something that we have historically had some challenges with, and it has ended in tears. And blood. Lots of blood. Dev and test environments have been made available in much the way you describe, but it has been a case of you can lead a horse to water but you can't make it drink. In my experience some people, especially those who choose to work in the operations space rather than project side, seem to have problems working within a process that involves other people. To mitigate risk they need to be part of a bigger process that involves independent checking and some level of traceability, but they don't necessarily want to be. How else can they cultivate their image as a worker of miracles? A proper mindset is just as important at proper procedures and environments if you want decent scripts.
The guy that wrote the article was talking about network monitoring and licence monitoring, so I'd assume it's something important to keep humming, and would therefore be classed as a production environment from some angle. As above, the article's really focusing on how to automate mundane tasks, not how to work with developers.
I reckon every sysadmin I've ever known has been able to write scripts. Have all of them been of a level of quality that I'd want near my systems without some additional oversight? Nope.
You don't understand change and release management, which is fine. Having seen the mess people get into without it I'm fine with having such gates in place. They're not hard to get over as long as people understand that the impact of running a shittily-written script in production is wider than them having to debug it.
I think if you read the article again (or perhaps for the first time), you will find that actually the article's talking about how having a scripting language under your belt will help you automate a lot of the day to day tasks a sysadmin would otherwise need to be doing manually. This is about business process automation, not raising bug reports.
The sysadmin is one member of a team. The article's talking about 15000 lines of code, which isn't a small amount. That's dangerous territory for sysadmins to get into. Do they understand design for performance, security, error checking, maintainability, corner cases? Have they done the right analysis? Are they solving the right problem? Do they understand how to test? These are not rudimentary skills, even though people think they are. The likelihood is that they haven't done enough to get past the company's (not my) Change Management/Release Management process (which is a fact of life in most organisations that have any level of sophistication) and the coders will be able to do it better, faster and get it through first time. So the total cost of ownership is what I'm looking at, it's not just about solving an immediate problem - it's also about how much effort it will take in future to keep that problem solved.
You have failed to grasp that *anything* that could influence correct operation of a production environment, even a little 'harmless' script that manipulates production systems and/or their data, needs to be managed correctly, assessed for impact (by someone other than the guy that wrote it), tested properly and deployed carefully. Risks to production need to be managed, and the biggest risk I know is a sysadmin with a god complex and a hobby in scripting.
As an IT manager, I don't want my sysadmins to code. I want my coders to code. I want my testers to test. And I want my sysadmins to...ermmm...admin the sys.
Mixing roles & responsibilities can be really dangerous, especially when there's a strict development and deployment lifecycle in place (and if there isn't you're living in uncontrolled chaos), including change management and release management, that must be gone through properly. Uncontrolled changes, with sysadmins using production systems like they're test environments, is a bad idea and will end in tears.
Slashdot Mirror
I loved this guy in Kelly's Heroes.
I'll take Nazis over poor use of apostrophes any day.
They are always assholes.
An 84 inch television is a massive waste of wall space, and of life.
I agree, innovation is the way to go, but not at the expense of the business. The team has a lot of flexibility in the way they work. But there are always some places where they're not going.
I'm sometimes guilty of the latter, but not of the former. A manual change effected by a sysadmin typing into a shell is different from one embodied in a piece of code, and they're actually subject to different processes. Change/release management does (should!) indeed apply to sysadmin activities, but there are these things called 'standard changes' which cover the typically mundane things like user Add/Move/Changes, that you don't want to go to a Change Control Board each time to get the go ahead. However they also cover stuff your sysadmin does in his/her/(but probably his) day to day work. Like pretty much everything they do, in fact. The weakness is generally nobody knows how a sysadmin spends their day, which makes these things difficult to control. But anyway, one or two changes made by a sysadmin isn't what we're talking about here. We're talking about automation, which means complexity, which means unforeseen results. That must be held to a higher standard and policed accordingly.
Most sysadmins are playing with prod systems to some degree - even if they're just gathering stats, it can have an impact, because the system they are querying still needs to spend cycles responding to the query. I have seen situations where prod performance has been degraded significantly because of best intentions to gather meaningful data. They are rare occasions, but they are still very damaging.
Nope. You fail to understand what risk is, and how to characterise it. Stick to your knitting.
Remember this line? "Those are the approaches I've taken, although it will ultimately depend on your organisation's skill mix and personnel availability." If you want to continue changing the parameters of the problem, you can find a flaw in any proposed solution. I'll leave you to move the goalposts on your own.
That depends on what the scripts are for. Most of our infrastructure is supported and monitored by relatively robust enterprise management systems now (which is a reaction to feeling the pain of a decade of un-managed scripting [before my time], and doing something about it) so there's not a huge amount of string-and-glue solutions needed at the moment. There's some low-risk stuff that can be done by *certain* sysadmins as the prime, but supported by a part-time BA and tester, with scripting expert resources (usually a coder) on tap as required. They all work within a defined process - that's the agreement we have with them, and they seem happy with it. For the more complex stuff, which is higher risk by nature, I'm keener to prime it with a coder who knows a scripting language inside out, and who has some level of infrastructure experience. They will have a part time BA & tester working with them, and a sysadmin on tap as required. Those are the approaches I've taken, although it will ultimately depend on your organisation's skill mix and personnel availability. This approach seems to work quite nicely, in that we're achieving predictability in our infrastructure operations.
Ability to cut code isn't a primary reason for hiring a sysadmin. It's absolutely a primary reason for hiring a coder. And well done. The same rules should apply to everyone. Historically sysadmins haven't been held to the same standards as everyone else. So they have been able to get away with murder.
Absolutely. Knowledge of the lifecycle is one thing, putting it to use in an officially sanctioned capacity is quite another. Your third sentence isn't the case, so it's irrelevant in this context.
Fair call from the complexity standpoint. However the assumption that each program is unrelated to the rest is stretching it, I think. Most situations in a relatively simple environment would call for some level of daisy-chaining (I think the author of the article even suggested some level of interaction between his scripts outputs.) So the complexity moves from the length of an individual script to the number of interaction paths between multiple scripts. And of course, as the infrastructure changes, these scripts have to be maintained, etc etc. So yeah, not the WOAH moment the 15000 lines suggest, but it can still be death by a thousand cuts.
1 line of code, run 500 times, does not present the same level of risk as 500 lines of code run once. I'm not saying don't run scripts. I'm saying scripts need to be developed carefully, within a process, and there's better people to do them than sysadmins alone.
You're quite correct - scripting *done properly* is the key. The hobby scripting mentality is something that we have historically had some challenges with, and it has ended in tears. And blood. Lots of blood. Dev and test environments have been made available in much the way you describe, but it has been a case of you can lead a horse to water but you can't make it drink. In my experience some people, especially those who choose to work in the operations space rather than project side, seem to have problems working within a process that involves other people. To mitigate risk they need to be part of a bigger process that involves independent checking and some level of traceability, but they don't necessarily want to be. How else can they cultivate their image as a worker of miracles? A proper mindset is just as important at proper procedures and environments if you want decent scripts.
Yes, in the same way I trust a coder to write a Hello World program flawlessly. In anything longer, mistakes creep in.
Stupid professionals work on salary. Smart ones freelance on an hourly rate.
The guy that wrote the article was talking about network monitoring and licence monitoring, so I'd assume it's something important to keep humming, and would therefore be classed as a production environment from some angle. As above, the article's really focusing on how to automate mundane tasks, not how to work with developers. I reckon every sysadmin I've ever known has been able to write scripts. Have all of them been of a level of quality that I'd want near my systems without some additional oversight? Nope.
You don't understand change and release management, which is fine. Having seen the mess people get into without it I'm fine with having such gates in place. They're not hard to get over as long as people understand that the impact of running a shittily-written script in production is wider than them having to debug it. I think if you read the article again (or perhaps for the first time), you will find that actually the article's talking about how having a scripting language under your belt will help you automate a lot of the day to day tasks a sysadmin would otherwise need to be doing manually. This is about business process automation, not raising bug reports.
The sysadmin is one member of a team. The article's talking about 15000 lines of code, which isn't a small amount. That's dangerous territory for sysadmins to get into. Do they understand design for performance, security, error checking, maintainability, corner cases? Have they done the right analysis? Are they solving the right problem? Do they understand how to test? These are not rudimentary skills, even though people think they are. The likelihood is that they haven't done enough to get past the company's (not my) Change Management/Release Management process (which is a fact of life in most organisations that have any level of sophistication) and the coders will be able to do it better, faster and get it through first time. So the total cost of ownership is what I'm looking at, it's not just about solving an immediate problem - it's also about how much effort it will take in future to keep that problem solved.
You have failed to grasp that *anything* that could influence correct operation of a production environment, even a little 'harmless' script that manipulates production systems and/or their data, needs to be managed correctly, assessed for impact (by someone other than the guy that wrote it), tested properly and deployed carefully. Risks to production need to be managed, and the biggest risk I know is a sysadmin with a god complex and a hobby in scripting.
As an IT manager, I don't want my sysadmins to code. I want my coders to code. I want my testers to test. And I want my sysadmins to...ermmm...admin the sys. Mixing roles & responsibilities can be really dangerous, especially when there's a strict development and deployment lifecycle in place (and if there isn't you're living in uncontrolled chaos), including change management and release management, that must be gone through properly. Uncontrolled changes, with sysadmins using production systems like they're test environments, is a bad idea and will end in tears.
Come on then. Pics or it didn't happen.
Mostly liquid ethane, methane and propane, rather than water. But yeah, one would think a splashdown would be more forgiving.
I think we're unlikely to wake you. But not for the reason you think.