Slashdot Mirror
DevOps: Threat or Menace? (Video)
The title above is a joke. Mostly. We've heard so much about DevOps -- good, bad, and indifferent -- from so many people who contradict each other, that we turned to Alan Zeichick, one of the world's most experienced IT analysts, to tell us what DevOps is and isn't, how it can help get work done (and done right), how it can hinder progress, and how to make sure DevOps is a help, not a hindrance, if you or your employers decide to implement DevOps yourselves at some point.
That's what "DevOps" is to me. A meaningless buzzword. I did once see a "DevOps guy" lauded as some kind of hero for changing a haproxy configuration and reloading it.
Devops is one of those meaningless bullshit terms that seeks to create something complicated out of something simple. Basically, if you want shit to work, which is the goal here I'm guessing, you put your developers and sys admins working the same room and get them to run something that will work in production. Simple.
Devops has also been used to give them impression that system administration can be abstracted away as some kind of quasi-cloud-development thingy. That is not the case nor will it ever be so.
Hi! Happy Tuesday
My understanding is that DevOps was coined by a manager at Etsy who recruited developers for managing IOPs and other costs in the Amazon cloud via software designed to do just that. DevOps meant someone who was saavy enough to write system level code.
Somewhere along the way this notion got morphed into being the system administrator and the developer.
DevOps:
1. Developers optimizing Amazon and other cloud environment costs by using application code specialized to manage system administration aspects of the cloud; including managing switches, spinning up VMs, etc.
2. Developers with system administration responsibilities.
The reality is that Etsy moved off of Amazon to an in-house data center and left us with a messy legacy of a term, DevOps. :-)
No, I will not take part in your survey of whether DevOps is a thing.
>> IT people don't work in teams
So..that's the heart of DevOps according to this guy? More meetings?
DevOps - What the operations team are called when they actually know something, not just the usual button pushers.
In my experience, the term DevOps is thrown around by two types of companies:
- Startups who use it to tout their agility and fast deployment
- Large company CIOs who want to get rid of their sysadmins or abstract all the complexity away into the cloud
What I do like about it is the fact that, when done right, it forces developers to know a little about the environments their creations will need to run reliably in. As a systems guy, my experience has been that developers don't really understand how stuff they works scales beyond their own computer or dev environment. Conversely, my development experience is limited to automation and scripting, and I know I could do much more with some formal programming training.
The problem is that DevOps is used the same way Agile is -- a magic bullet that will fix the problem of expensive sysadmins, expensive QA, production outages and delays in deployment all at once. DevOps works for startups because you're usually dealing with a single product, a small team who all know most of its features, and a group of people who are 100% focused on making it work correctly in production. For bigger companies it can work, but you need that level of focus that you might not have in a large organization juggling hundreds of different deployments.
What I don't like is DevOps being used as an excuse to remove any systems discipline. The cloud is great and you can spin up a million servers if you want, but just handing developers the keys and saying "go nuts" isn't the answer. Nor is locking down the environment so much that processes kill any hope of moving quickly. Like everything, the balance is in the middle, and sysadmins who know a little about the products they're taking care of, plus developers who know a little about the metal their code is running on make for much smoother deployments.
DevOps is a model where developers can spin up a VM via some cloud mechanism, throw in their code and launch without having to get an SA involved. This is great for lower environments, and lets you use CI tools like Jenkins to do a build/test/refine loop, but once you get near production, presumably you don't want the code churning so much. Then you want the SAs in the loop to manage environments for stability and to tune for performance. What the suits want is an "easy-button" solution that does all the provisioning/tuning/deployments so they can drop all those knowledgeable, expensive SAs and just let the developers handle it all. This is probably suboptimal as you don't necessarily get the benefits of performance tuning and looking at the overall structure.
DevOps is a specialization which used to be part of standard system administration. Developing custom tools to do custom tasks, in this case related to "Ops" (another specialty that used to be standard sysadmin territory). The term is a great dummy term, but really does not distinguish someone's ability to manage servers and infrastructure.
You seem to be the 'new guy' who preaches that everything should be run in a generic docker container, complaining about the 'old guy' who wants none. Meanwhile, most of the people worth their salt understand that sometimes generic works and sometimes it doesn't. If there was some magic perfect layout everyone would be using it. Instead, we have a huge array of both hardware and software being used in the market. Knowing a dictionary of buzz words does not make you good, and usually indicates just the opposite.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
devops was spawned from startup culture. imho it was never intended to be something that lasted longer than a few hundred users or internal need. The needs of ops and developers, while similar in some ways, vary dramatically in others. devops is a compromise designed to foster quick, often unscaleable delivery of a service. Once larger companies caught on that you could make devs to op work, and vice versa, the trend became unstoppable. its still an insufferable title.
As an op, i dont write the kind of code that would get into a formal repo with a standup and peer review. im writing a script to automate something that either pisses me off or takes up too much of my time. I write breakfix code too, but its not designed to upgrade or mod easily and i dont share a ton of comments or documentation because im busy handling the infrastructure. piping my one-liners and quick functions through the holy trinity of git/gerrit/jenkins for review by real developers is a demoralizing pain in the ass. it also slows down fixes and deployments by subjecting talented coders to a confusing ratsnest of code that isnt meant to become a fundamental part of how a user interacts with the brand.
and making devs subservient to ops isnt something you should do either. scrum and kanban and swimlanes are boring 15-30 minute reminders that marketing drives the bus. We need to be part of the rollout process to make sure things, if they break, are handled appropriately. its nice to have validation the codes been tested as well, but all these things are an email or a quick chat and dont need to involve a manager.
Good people go to bed earlier.
This guy's focus on job titles is the issue
Wow, it would be tough to work with this guy. His assumptions about developers is a sign of his lack of adapting to the newer smaller teams or the type of customers he only deals with. He has a strong belief in job title and domain knowledge isolation.
His views on the job title's perspective on what is important is pretty bad. If you have departments and priorities on security, login, and scale are not matching up you have issues with the company's culture and isolation. What it sounds like is he is describing a horrible work environment and the use of devops was put into place to allow others push buttons... yep, that to me sounds like the wrong way to go about it.
I wonder if he would be able to keep up with smaller companies with less isolation between these arbitrary job titles. Has anyone informed him the concepts of scale, security, and single sign-on are results of software engineers who have knowledge in this domain?
He must deal with outsourced teams and a very disconnected software to infrastructure workflow. Eliminate the isolation and you may have a better operations flow.
He talks about product teams shipping products that are not aware of security associates it to job titles and assumptions of responsibilities and knowledge domains.
What it means is if you are trying to do DevOps to resolve the issues he is pointing out. You are still going to have the same issues.
DevOps means "give those developers who can't write code stuff to do that that the programmers don't want to do, like maintain development systems and VMs and build machines".
I guess DevOps is shorter.
Really? Lets add Special to the word Special DevOPS, there NOW I feel really important!
I FEEL the NEEED for a CERTIFICATE! YES! A 'Special DevOPS' Certification
Woo Hoo! Now I can really write code! Just look at my piece of paper, it says I can!!
Idiots.
And I'm an idiot for responding.
So he is full of it. DevOps is not about ops engaging with devs. It's about making all devs part of ops. It's a model which existed in commercial banks and other places which could afford to overpay (a lot) to have people do work a few notches below the pay grade they are paid. But it's high stress and snail-pace progress. It reduces specialization which, by definition, makes experience less valuable. It puts all of the testing burden on the developers and removes testing specialists. In the most extreme cases, it flattens the most experienced developers and newly minted college interns into doing the same job. The result is that the quality of the product is always determined by what the least skilled members of the team can handle. Oh, and because everyone is resentful and knows that they are overqualified, it creates a frat-house-like environment. Everyone end up overworked and under accomplished. But because it's used in the cloud management now (an industry growing quickly so it has as much money to burn as the banks), it creates the illusion of being successful. It's not.
The cloud started before there was dev ops. And it would continue and succeed without dev ops. It exists because all development is network development (to at least some degree) because CPUs can't be made much more powerful, they can only be networked on the chip (aka multi-core CPUs, GPU). And if there is no distinction between pooling multi-computer resources and pooling multi-core resources, then scaling is accomplished by pooling a lot of them in near-real-time batch processing (aka pipeline). Which means the state of technology makes the cloud computing model make more sense than stand-alone units computing model. The state of technology is what drives the business incentive here. And the business incentive is what creates the illusion that the fulfillment model of the business incentive is the right one. Even if it's not organization model. Poorly chosen organization model in the environment of over-funding will not fail until the market place becomes more efficient and consolidates. And at that time, all the DevOps shops will be unable to support their costs and won't understand why everyone is switching to well-engineered (through full SDLC) products. They'll be saying it's not in vogue anymore. While the reality is that DevOps is just an inefficient organizational model to develop cloud services.
Any guest worker system is indistinguishable from indentured servitude.
Whether all you ops (and it seems mostly ops posting) understand it or not, devops is a type of dev consultant. It can replace ops in the cloud, but for the most part it is about making devs lives easier. It is basically being a release engineer, but one that architects staging and production deployments. Ops have a knack for locking down things and being scared to change, devs are not meticulous and have a tendency to deploy production code that accesses their home VPN to check the time. Somewhere in between the company needs to make new revenue. Some ops do devops, lots of devs do devops. In fact some large companies have dropped SDE/SDET and just have SDEs that do dev/testing/devops. I believe that is the future. Devops is a glue for legacy teams. It is BS in most startups. Few startups have a real need to scale and everything else is early optimization. Devops helps with microservices concepts, but it can easily be just a fancy NoSQL DBA + cloud ops job if you aren't working on improving processes. You need devs being accountable, but also making devs trust their dev environment is correct and that repros happen and are visible in every environment.
You mostly run around being some sort of six sigma consultant, preaching CI/CD and tweaking CM systems that make consistent environments. You also do scaling tests, cloud/in house provisioning, and implement instrumentation/health checks with dev. Ultimately, you enable and teach devs to become more devops themselves if they want to be more senior.
DevOps Manager, here. We help a several hundred developers on three continents and manage several thousand systems, covering a few dozen products. Anyhooo ...
These guys are having a good time talking about something that, like art, is pretty wide-ranging and loosely defined. It was fun to watch and listen to, but it didn't do much more than dwell on stereotypes in software development and IT.
Personally, I think DevOps is less focused on the hosting technologies and more focused around rapid build and deployment, along the continuous delivery model. (Think continuous integration, with a deployment step tacked onto the end (first to test, and then maybe to production).)
There is a lot of systems stuff that comes with it, and there is both systems and software configuration management involved, but it covers the tooling that is sprinkled all the way through... from the various repositories (for artifacts & code), to the build mechanisms (custom-tooled, jenkins/hudson), to the environments for functional and maybe integration testing, on to release and maybe deployment to production.
No two companies really have the same workflows, and agile (scrum, SAFe, Kanban, etc.) may or may not be part of the mix, and some shops like to have a lot of participation from developers, and some want more clear-cut boundaries.
At the root, though, I think the common threads are taking code from the time it's checked in, and getting it built and tested and deployed or released as rapidly as possible, accounting for all the systems administration and vendor relationship overhead along the way.
I should point out that neither I or my team wants to be the traffic cop or the "adult in the room" -- we hate that stuff, actually. My team just wants to do our jobs so well as to be easily forgotten, to keep things moving along and stable, to be ready for emergencies or disasters so that no time or coding effort gets lost, and to be ready to lay a clean path ahead for developers and testers to do their work in whatever direction gets decided by the architects or product people.
We work in the belly of the battleship, between the coal and the ammunition. If we are doing our jobs correctly (and well!) we don't need to know what direction the boat is heading in, what threats are next to target, or who is at the wheel.
I'm sure there are a couple dozen people around here who are dying to tell me how useless my team and I are, and how I'm not a real engineer or technician, because that's probably what feels good.
There is so much misunderstanding because there is not a universal, static definition of DevOps that everyone can point to and say "that is DevOps" or "you are doing it wrong!" This is because DevOps is ultimately defined by the capacity of the people who practice it and I think we can see (already in these postings) that many people do not have the capacity to define it.
The history of DevOps begins with the people who coined it: Patrick Debois and Andrew Clay Shafer's first discussion about Agile Engineering at a conference in 2008, which led to a Google group and then to the first community meeting as DevOpsDays Belgium in 2009. W#e can trace to the beginnings and primary source folks, so please stop demonizing and making DevOps anything more mysterious than a knowledge gap.
For an overview with my definition of DevOps, please see my blog post with talk and slides that I presented at Silicon Valley Code Camp earlier this month:
http://mlavi.github.io/post/de...
My opinions are my own, but you may share them!
I have no strong opinions about the word "devops", however I think there is a role needed that doesn't get met by developers, QA, or IT. The problem is that some things don't fall squarely in anyone's lap, so everyone can play the "not my problem" game and nothing gets fixed. As a developer i've had to fix so many problems that aren't "my job" and are next to impossible to get time for. They add up to the point where it's darn near a full-time job. I'd love to hear anyone's recommendations on what job title that would be, or just how to handle this stuff better.
Some examples:
- Our software builds were completely manual. Everytime we wanted to ship a new version, someone would go and manually build the software, package it up by hand, and give it to QA. This led to all sorts of problems, and encouraged everyone to play the 'ping-pong game' between development and QA. I had to setup a Jenkins server and rework all of our builds. This was not that easy, because when you automate something, you lift the rug up and expose all the problems that have been swept under the rug for years.
- IT had created VM images for QA to use, with all the software dependencies pre-installed. That was screwed up because it didn't reflect how the customer installs on new machine. Plus they never updated their VM images because it's a PITA, and devs would never tell anyone when they added a new dependency. I had to turn all our stuff into RPM's that required the correct packages instead and teach everyone to update the spec file when they added new dependencies.
- Same story as above, but in regards to development machines. I had to create provision scripts and teach everyone to update those.
Once he dismissed younger developers as kids who needed babysitting it became mildly offensive. I'm a middle-aged developer myself and I will never dismiss anyone who has a passion. I also think his description of DevOps was incorrect. Where I work, its Operations people who can develop and automate. It merges the disciplines and is not merely some developers working with operations people. We no longer hire people who don't have both skillsets. Its basically raising the bar and saying "automation first", so operations people better learn to code.
For the smartest people I know on the Internet, I find the disparity between the comments here and reality discouraging.
:)
I travel the country installing automation software, training people, and doing this whole "devops" thing (gad, I hate that word). I post here for info, not cred and I post here to tell what I see from the trenches. I also post anonymously because the bit $company I do this for wouldn't like to have my (highly visible in devops cicles) name plastered around, as it affects them too. (This is the first time in my career I've had to watch my mouth in public...It's killing me)
Quite awhile ago... about 10 years, I was working the Sysadmin/Eng world and enjoying it. This site was a Top-10 Webmonsters site... top 10 traffic receivers on the Internet for you guys not in the playground. In short, a bit site that provides an insanely popular service for the privilege of throwing a gazillion ads at the thundering herd. Hey, it's a living.
I was happily minding my store of all sorts of perl, shell, and wonderful glue of all sorts (we had even written a little orchestration platform that worked pretty well) but life was good. Then they hired "that guy" over in DEV. I think it was day 2 he wanted root. Well, we didn't "do that here" and weren't going to budge just for him. Logs were aggregated and made accessible via internal browsable sites, deploys were push-button and quick (all before Hudson/Jenkins) and the DEVs could iterate to their heart's content. They had no logins anywhere, but they had tools to do everything they wanted "root" for. This guy refused to see it.
He started yammering on about DEVOPS that long ago, but in his mind, his own little world, it just meant "give me root". As such, he had a one-track mind and whined and complained loudly to anyone who would listen. As chance would have it, our CTO was the "bee's knees" as they say and saw the writing on the wall. We were going to have to build something to shut this a-hole up. So here was the deal: "We'll give you your own environment that mirrors production. You have to care for and feed it yourself. You will have revision control to store code, push-button access to build deployment and a merge-structure that will allow you to transition target nodes from one environment to the other without friction... Your own little DEVOPS wonderland to do with as you wish. But there's a catch: You're unsupported. If you open a ticket to "fix" something, the "fix" will be restoring the snapshot of the working environment from provision time over whatever mess he had made. Since any sensible developer will commit early and often, that shouldn't be an issue. Restore, check out latest tag, re-run the process, and voila! You're moving again in minutes! (we even put the destroy & rebuild buttons on a web page via perl CGI so he could easily do it himself without the embarrassment of asking us. (not without a counter and an RRD graph to track his rebuilds, mind you). Plus, being PCI governed, he had to use Powerbroker to connect, which dutifully carted off his entire command line history to a different server for storage.
That's the backdrop. He definition of DEVOPS given to him on a silver platter.
Fast forward two days. TWO. DAYS. 34 rebulds and reimages later. TWO FRICKING DAYS!!! See where I'm going here? The early translation for DEVOPS was "give me root and leave me alone". Well, apparently that level of power causes you to completely restore your system from snapshots 34 times in two days. (but I digress... And THIS guy was a highly-paid, superman brought in for top dollar to train everyone else!!!)
So, here I sat with 15 years experience at the time and I had a hunch. I did some research into what he thought he knew, and I thought this thing was going to be big. REALLY big. IT automation, data-center based company infra being farmed out to you at a fraction of your hardware costs, streamlining the pipeline between code and d
DevOps: where lack of experience in process design, change management, configuration management, and capability maturity model is glorified. Where hap-hazard hack-it-'till-it-works is the order of the day, the only how, without thinking about far reaching consequences, because the only thing one knows is how to hack it up on the quick.
Yes, DevOps, where guerilla style, Viet-Cong, ho-ruk, partisan, chaotic mode of working is the order of the day, the one and only order of the day. Where Chaos theory and entropy rule supreme. Where planning is not 60% of the work, where process design causes eyes to glaze. Where concepts like "configuration management with operating system packages" are faux pas or far worse, outright politically incorrect.
DevOps is a movement not a team.
If you create yet another silo, you just lost.