Slashdot Mirror
How 'DevOps' Is Killing the Developer
An anonymous reader writes "Python guru Jeff Knupp writes about his frustration with the so-called 'DevOps' movement, an effort to blend development jobs with operations positions. It's an artifact of startup culture, and while it might make sense when you only have a few employees and a focus on simply getting it running rather than getting it running right, Knupp feels it has no place in bigger, more established companies. He says, 'Somewhere along the way, however, we tricked ourselves into thinking that because, at any one time, a start-up developer had to take on different roles he or she should actually be all those things at once. If such people even existed, "full-stack" developers still wouldn't be used as they should. Rather than temporarily taking on a single role for a short period of time, then transitioning into the next role, they are meant to be performing all the roles, all the time. And here's what really sucks: most good developers can almost pull this off.' Knupp adds, 'The effect of all of this is to destroy the role of "developer" and replace it with a sort of "technology utility-player". Every developer I know got into programming because they actually enjoyed doing it (at one point). You do a disservice to everyone involved when you force your brightest people to take on additional roles.'"
Python guru Jeff Knupp should go find a job where he can program, and not worry about ops. Simple solution.
"First they came for the slanderers and i said nothing."
This sysadmin/scripter/system architect/DBA role exists in virtually every company that has a core business other than IT or software development. Even in a very large multinational, there's more utility in having one "Mr Wizard" in each business unit than there is in having a room full of software developers somewhere far away from the rest of the business.
It really is a support role and it's more an outgrowth of system administration than it is saddling your brightest software guy with managing the mail server. Of course, it's possible to get stuck in that role because there's nowhere to go from there, but it's a niche that suits some people. If it doesn't suit you, then move.
It's also a distinct role from the "do everything guy" at a startup, because at a startup everyone is multitasking and as the startup expands, new people are hired to take on some of these roles. DevOps is a role in itself.
Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
Seen this many many times before. Cheap companies that have lots of developers and are too cheap to hire experienced admins... or an IT shop that thinks they can have the IT guys program instead of hiring proper developers. "hey, you work with computers, you guys can all do the same stuff, right?" Wrong.
While I have known developers that can sysadmin, and admins that can program... they are the exception not the rule. Quality suffers when you force people into jobs they are not qualified for. Companies know this, and they simply don't care as long as the managers think they are saving money.
I'm a good cook. I'm a fantastic eater. - Steven Brust
Full-stack developer generally refers to a developer who can code the full software stack - UI, middleware, and backend. Also lots of QA people can code - automated testing is mostly coding - and lots of developers can't test at all. Author needs a bit more real-world experience.
You don't force your brightest people to take on additional roles - that is the whole point of a devops team in the first place. Making developers argue about deployments and sending builds to QA and managing your GIT server and development and QA databases and managing your bug tracker is exactly what your developers should *not* be doing, especially if those scripts necessarily have to be in a different language than your application. Sure, your lead developers and architects work with the devops team to support them so they can in turn support you, but that's as far as the relationship goes.
The way we used to do it, where every senior architect is also responsible for all of those other functions (and has to take the time from his team members below him to help build all that out), is exactly how you stop architecting your software: your leads spend so much time trying to automate the drudgery they aren't improving the app.
They aren't improving the app because all of their brainpower is no longer focused on the *customers'* problems, but rather their own and their teams. That isn't a good use of their time. Hiring smart people who need to understand the application and its environment, but are good at scripting these other languages of automation, frees up your team leads to doing what they did before and do best: focus on the application and getting the team to produce the code that serves the customers' needs.
"But remember, most lynch mobs aren't this nice." (H.Simpson)
-- Joe
There's definitely truth to what he's saying but it cuts the other direction as well. Having your lead guru developer swapping disk drives on a machine isn't the best use of his time. However, I've also seen environments where the developers can't/won't/aren't allow to do the system admin tasks and wind up waiting around or being frustrated when their development systems have a problem. Likewise, with QA - I've seen developers that will just toss any old crap over the wall and expect QA to catch all of their bugs. And, developing tests is often software engineering, often complex software engineering that needs an experienced developer to establish at least the outline of how everything works.
Personally, I expect any developers I'm working with to have at least basic sys admin abilities and know how to setup/fix any other part of the stack they might touch. Those skills should be used when working with the dev systems and in establishing the base line for production. I would then expect that someone who is more specialized in those other roles to actually setup and run production and also be available when the developers get in over their heads on system admin, hardware troubleshooting, etc. In the same way I would expect a systems admin to at least be able to write a script to automate something and not go running to the developers for everything.
For test development, I always like to set groups against each other and develop the test suite for each other's code. Most people are a lot more comfortable and eager to break someone else's code than they are their own.
I had a job where I did everything once, wrote a full-blown ERP system for hundreds of users, all by myself. Everything. Though I was salaried, sometimes I worked whole weekends, or to 2 in the morning - not because I had to, but because I wanted to. No politics, no being just a cog in a machine, no project management, no BS. Just me and code, giving people what they needed and making their jobs easier. It was my dream job, my first and best job, and I've never had anything like it since.
This DevOps movement the author speaks of... I've never seen it, not in all the years I've looked to find it again. He may complain that it's bad, bad for the industry, but I would take it in a heartbeat.
Is that what I was, a DevOp? I miss it so much I can taste it.
I am primarily a developer but I also like to understand the big picture, including software design and UX but also system administration, infrastructure, hardware architectures, and everything else that *directly affects the software I develop*.
Deep understanding of the big picture is key to developing quality software, IMO. You need to understand what comes ahead of you (requirements, business needs, etc) and where your work is headed afterwards... The best way to understand it is to wear these hats from time to time or have previous work experience in those fields. When recommending candidate for developer positions, someone who has system administration experience is a bonus.
Yes, many days I need to take on multiple hats and switch gears as shit comes up in prod and I need to fix a config on production servers or assist whoever has the hands but lack the knowledge. That's the start-up culture I guess, even though I work for an established 100+ year old company.
The point of devops is not to take jobs away from developers. The point of devops is to provide an interface between system administration and development. Development and system administration have always been at odds with each other - system administrators not really understanding or caring how the application works, and developers treating the systems as an infinite resource pool with no real rules or resources past "does my code run?"
The sole purpose of devops is to ensure efficient operation of the infrastructure in a way that allows for repeatable deployments and controlled versioning, and that also includes system software such as operating systems (sysadmins benefit too because they no longer have to do one off deployments of OSes).
This criticism strikes me as woefully underinformed as to what devops actually does, and I'm wondering if the author of this is a developer who is upset because devops is forcing them to actually use the software lifecycle properly rather than just doing cowboy deployments and hoping they work.
For linux tips: http://www.linuxtipsblog.com
A problem I have run consistently around is that developers rarely understand system administration, and operations in general. It makes their software suck a lot more. This is even more true with the Java-crowd, many of which cannot even use a commandline. The more this gets fragmented, the more people get specialized, the more problems arise in architecture, design and implementation, up to and including software that cannot even be deployed because of misconceptions on the developer's side.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
It's legitimate for a developer to throw a fit. They should have admin access on their local box. There's an insane number of plugins, libraries, and frameworks they may need to install or test to determine if it's the right fit for their deliverable.
However, I fully expect DevOps to question every single add on and make the developer justify it before deploying to production. Just because I as a Dev want Ruby for compiling SCSS to CSS on my local machine to save me time, does not mean Ruby needs to be installed on the production environment. That's what build servers and deployment scripts are for.
SysOps on the other hand is far too prone to blanket say "No" or "We only approve this one obscure outdated fork of the package you want."
DevOps is the perfect position to reconcile developer wants with checks and balances against actual needs and responsible deployment.
I'm out of my mind right now, but feel free to leave a message.....
I essentially have this kind of role within my organization. I design, develop, deploy, and support small to mid-tier systems (e.g., the planning system for a $XXXmio/yr global department, with 300+ direct users) while being one of my own customers, as I am actually a business planner (by role) as opposed to developer. I develop systems as a way to do my "day job" much more effectively. Typical tech stack would be Excel UIs, PostgreSQL data store, and whatever else I need in the middle (e.g., nodejs, tomcat, redis, whatever).
What I've found is that, in general, doing the right thing the "right way" is not worth the cost compared to doing the right thing the "wrong way". By definition, in either scenario, the right things is getting done. What most pure developers utterly fail to understand is that in trying to do the former, there is an overwhelming tendency to do the wrong thing the right way instead.
This is because, as Fred Brooks pointed out long ago--and as the "lean startup" movement is re-discovering today--for any non-trivial novel problem you cannot know in advance what the "right thing" is until you've actually tried to implement a solution. Brooks stated this understanding as the need to throw away the first try, and the lean startup movement is essentially defined by a corollary--you have to figure out how to try cheaply enough that you can afford to throw it away and try again (and again, and again if necessary), while progressively elaborating a robust definition of what the "right thing" looks like by using those iterations as experiments to test hypotheses about what the "right thing" is. Doing things the "right way" usually costs so much in time if not capital that you simply can't afford to throw away the first try and start over, or you cannot complete enough iterations to learn enough about the problem.
Now, I'm not saying that you should be totally ignorant of software engineering best practices, design patterns, etc. What I am saying is that there is a limit to how effective you can be in reality if you live purely within the development silo. Having a "DevOps" role (granted, self-imposed in my case) has been one of the best things that's ever happened to me as far as making me a better developer, right up there with the standard oldies like writing your own recursive descent parser and compiler.
In short, it is commonly-accepted wisdom among programmers (for good reason!) that you are more effective if you actually understand the technology stack down to the bare metal or as close to it as you can manage (even if only in abstract-but-helpfully-illustrative examples like Knuth's MMIX VM), and that this understanding can only be gained via practice. It should be obvious that the same is true in the other conceptual direction through deployment and end use.
.
Operations people work in a timeframe of minutes to days
Developers work in a timeframe of weeks to months
Researchers work in a timeframe of years to decades
When you take a developer, who thinks in terms of months, and task that person to think in terms of minutes and hours, you are wasting a resource.
When you make someone respond to an overly wide range of timeframe-based events, the short term events always crowd out the longer term events.
Have you ever noticed that companies locate their research divisions away from the day-to-day operations divisions? It is to keep the timeframes separate.
you used to carry your stack of punch cards to a Demi-priest, who passed them to the Priest who was the only one that was allowed to touch the computer. Funny, how even today that role still exists.. it's called a DBA.. these jerks won't let mortal developers near a database, even though the dev writes the script to be run against production, and DBA monkey just hits Run for you.
I'm not sure that's accurate. Programming skills are pretty important today and most people, including myself, learn to code so that their current skill set (and market value) can be enhanced. It used to be that if you need an application, you'd have to buy one or hire a contractor to build one. So, this route has some problems associated with it besides cost. For example, if what you buy breaks, it's harder to fix without calling the company that made it. If you work in telecom and need a call center wallboard, you can either buy one or make one. I chose to write my own with Debian, PHP, and Apache. And it works just the way we like it, we have full documentation on how to rebuild it if needed, it's an inhouse solution that was built by "DevOps". There was a steep learning curve to develop the skill set to do this - it took a few months of hard work and learning. But it was totally worth it.
If anything, the existance of DevOps is totally natural and was born from necessity. If I were a full time programmer, would I be bothered by this trend? Maybe.
Several problems with the original article. Jeff makes the assumption that Developers are at the top of a totem pole and all others are below this developer hierarchy. This is the problem with a lot of younger developers, most who are jumping into development with no structural background in the operating system or real production operations. This premise leads developers to think they are superior to everyone, rather than focusing on building applications that must run in an operational context of being well-behaved in the environment they run, and allow for maintenance work of the operations environment. This is why you see anecdotal articles coming out that talk about the frustration and friction between developers and operations people. The truth is, most developers will be better developers if they learn a little more about how their app will work in a production environment. They will build better applications because they understand the environment better. Your architecture should be loosely coupled, that is, it should not stack up on a single server but rather be broken up into smaller components that can be built for scale across multiple machines and be able to fail over to redundant systems. Ops people will be more than happy to help get you there, but you need to think about how your app will fail as much as how it will work properly. The developer also needs to stay away from dictating specific languages -must- be used to run operations. Rather, they need to understand that operations is best suited to a polyglot of different kinds of tools and languages, using the right tools and wares for the job, not being dictated in some kind of nanny state that says you must use this language or that for operations and all others are to be thrown out. All too frequently startups get hung up on dictating operations people must use Python or Ruby or the ops dude is a total loser. This is the kind of arrogance that shows how little most developers really know about operations. Any of the more popular tools should be available to the DevOps person so they can get things done, and developers should stay away from being nannies. In classic UNIX operating environments the system administrator is/was god, followed by an allowance to use the system granted to developers, database admins, etc. Applications have always been guests inside most Linux/UNUX-based systems, even Windows treats applications for what they are, programs that run inside the operating system. This has always been the premise of modern UNIX-based and even Windows-based environments. Ops people follow this naturally, but typically run into developers that don't want to follow the rules. Ops people expect you to write a well-behaved application. Developers expect an operations environment that runs their application without getting in the way. These two goals can be easily met if both sides work together and allow each side to exploit the system, but everybody has to allow the freedom to make things work best for the goal of building better systems.
Tim
Integrator/Architect here... I travel all over the country (USA) for $CFGMGMT software company doing implementations and training in said package. Further, I do professional services engagements to implement and deploy devops workflows and integration paths for both operational and development workflows. I can say with great confidence that after reading the 150-ish comments here that I've only seen one or two commenters that come close to "getting" modern devops.
As for the RTFA implications, many people saw right through his silliness. Developer in an ivory tower thinking his world is grand and majestic and lauded above all other disciplines. I guess that explains some of the crap in Python (but I digress).
I've been in this business since before the bubble. I have done systems work dating all the way back to the 80's. At that time there were many silos with few bridges between. The first iteration of "devops" I remember seeing can be quantified as developers seeking root access. Even to this day, the answer is continually a resounding "no". Why? Are devs automatically rejected due to what specific tower they are residing in? Are they rejected because of the "separation of powers" or because of "politics"? Are ops folks just dicks with a chip on their shoulder?
I think a number of things have happened and if you'll indulge me, perhaps we can break this conversation out more to see what's going on today.
Back in the day, we had an operations team, systems team, qa team, and dev team. Loosely defined, these were the folks in the NOC monitoring everything and pushing ticket events to the right teams via triage, doing prints and getting them ready for the requestor, managing backups and access to the computer room, etc... general "operational" tasks. Day to day "operations" of the center of computing for your facility. At some point, this title got applied to anything that smacked of someone whose fingers touched anything made of metal. When this seemed to change? Late 90's-ish. I don't have specifics, but that's my recollection. Remember also that different sections of the country progress at different rates, so YMMV.
Then there was the "systems team". Different sites had different monikers for this team from "engineering" (which PEs got pissed about) to "architecture" to "Systems Admins" to... well, whatever title HR thought was appropriate and met their organizational/talent distribution goals, but the work was generally the same. Storage, systems (mid range, intel, mainframe, etc.), capacity planning, backup systems and strategies, disaster recovery, and a myriad of other specializations within systems work. This is the specific "field" of systems work that breeds organizations like SAGE/LOPSA, etc. Those in these organizations dig into the very nuts and bolts of admin as a career and a field of expertise, following current research and methodologies and bettering themselves each and every day. I would offer that while dev folks can be quite competent in various systems tasks, they're just not a sysadmin (much less an engineer or architect). They really have no place in this area REGARDLESS of their quality at it.
Many times I've heard dev get bent out of shape because "ops is being a blocker" or "we can't get anything done because ops is saying 'no'". It's unfortunate to be that myopic and that selfish. What many devs don't understand is the ops person they're dealing with is supporting sometimes as many as 10 different DEV teams operating on the exact same hosts or clusters of hosts and one or two of those environments is the pet project of VP #13 that agreed to a bunch of junk on the golf course that isn't in the statement of work, but is being offered "on the sly". They don't understand that dev #22, who has relationship with VP #11 and can essentially get what they want has forced you to introduce "whiz bang package or library 1.2.3.4" while the safe, researched, CERT recommended package is 1.2.2.2 (and, incidentally is the one all the other dev teams are u
#!/Jerald
Horrible assumption that you think developers are top-dogs, Jeff. I've seen many cases were DevOps role FAILED for competent developers because, in the cases I've seen, this is true:
* Your smartest developer may be just that when it comes to language, software architecture and platform development, but their operating system, networking, hardware infrastructure knowledge + background is not even hobby-shop at best.
* They've always had an ops or engineering crew to throw their code at, figure out how to integrate it, and NEVER had to support it.
* Ego problems thinking they are 'above' remedial automation --- which most of the time doesn't involve a real development language, just scripting.
Out of those two things alone, I've always heard the: "Well we need a sys-admin/engineer now because we are spending more time trying to manage systems, not really sure how or what to automate, and it's really taking time away from me getting back to the kind of development I, as the developer want to do." Which is the polar opposite of the two points I mentioned above.
A working DevOps group should be an amalgamation of the Dev team and Ops teams.
Trying to forge one team where everyone knows everything simply sets a level of mediocrity; you can research so far down an avenue in a given time, and get only so good at it.
I've learned from the ground up (i.e. electronics, basics of VLSI, board design, basic OS design, all layers of the stack programming), and went from there on to system admin. Then did a stint as a developer using the knowledge that I had from my earlier history, and found that the sysop area of my knowledge atrophied in the detail (and the devil is _always_ in the detail) the more I concentrated on being a better dev.
Went back to more of the operator/business side of things, and lo and behold, the more I go into systems and how to put together a proper reliable, recoverable infrastructure, the more my dev is atrophying. I'm half management these days, which means the ops side _and_ the dev side are both atrophying. The guys that do it in a dedicated fashion are more familiar with the latest tech than I am..
You can be a jack of all trades.. But I seriously hope a company doesn't rely on you to get them out of trouble when the fecal matter hits the fan.. If you've been spending most of your time developing, with the nod to tuning the servers so your app runs better, you're not likely to have been able to put the time in to develop the wider infrastructure to support things going fubar, or had the time and concentration to really work out what is likely to get you.
Having a few people marked as DevOps would be useful when you need to populate a middle ground.. They can work with both dedicated ops, and dedicated dev to ensure that scalability is baked in, and resilience is baked in to the apps. When it comes to ensuring the boxes are kept in tidy order for everyone, and get to be able to recover from the smoking ruins.. That's where the dedicated ops shine. When you really want that app to do something really slick, that's when a dedicated dev shines.
Small scale, a DevOps person would work. The larger you scale, the less appropriate it becomes (as the only solution; a big company with the techs being solely a DevOps team would scare me).