Slashdot Mirror
DevOps: Threat or Menace? (Video)
The title above is a joke. Mostly. We've heard so much about DevOps -- good, bad, and indifferent -- from so many people who contradict each other, that we turned to Alan Zeichick, one of the world's most experienced IT analysts, to tell us what DevOps is and isn't, how it can help get work done (and done right), how it can hinder progress, and how to make sure DevOps is a help, not a hindrance, if you or your employers decide to implement DevOps yourselves at some point.
That's what "DevOps" is to me. A meaningless buzzword. I did once see a "DevOps guy" lauded as some kind of hero for changing a haproxy configuration and reloading it.
Devops is one of those meaningless bullshit terms that seeks to create something complicated out of something simple. Basically, if you want shit to work, which is the goal here I'm guessing, you put your developers and sys admins working the same room and get them to run something that will work in production. Simple.
Devops has also been used to give them impression that system administration can be abstracted away as some kind of quasi-cloud-development thingy. That is not the case nor will it ever be so.
Hi! Happy Tuesday
My understanding is that DevOps was coined by a manager at Etsy who recruited developers for managing IOPs and other costs in the Amazon cloud via software designed to do just that. DevOps meant someone who was saavy enough to write system level code.
Somewhere along the way this notion got morphed into being the system administrator and the developer.
DevOps:
1. Developers optimizing Amazon and other cloud environment costs by using application code specialized to manage system administration aspects of the cloud; including managing switches, spinning up VMs, etc.
2. Developers with system administration responsibilities.
The reality is that Etsy moved off of Amazon to an in-house data center and left us with a messy legacy of a term, DevOps. :-)
No, I will not take part in your survey of whether DevOps is a thing.
>> IT people don't work in teams
So..that's the heart of DevOps according to this guy? More meetings?
In my experience, the term DevOps is thrown around by two types of companies:
- Startups who use it to tout their agility and fast deployment
- Large company CIOs who want to get rid of their sysadmins or abstract all the complexity away into the cloud
What I do like about it is the fact that, when done right, it forces developers to know a little about the environments their creations will need to run reliably in. As a systems guy, my experience has been that developers don't really understand how stuff they works scales beyond their own computer or dev environment. Conversely, my development experience is limited to automation and scripting, and I know I could do much more with some formal programming training.
The problem is that DevOps is used the same way Agile is -- a magic bullet that will fix the problem of expensive sysadmins, expensive QA, production outages and delays in deployment all at once. DevOps works for startups because you're usually dealing with a single product, a small team who all know most of its features, and a group of people who are 100% focused on making it work correctly in production. For bigger companies it can work, but you need that level of focus that you might not have in a large organization juggling hundreds of different deployments.
What I don't like is DevOps being used as an excuse to remove any systems discipline. The cloud is great and you can spin up a million servers if you want, but just handing developers the keys and saying "go nuts" isn't the answer. Nor is locking down the environment so much that processes kill any hope of moving quickly. Like everything, the balance is in the middle, and sysadmins who know a little about the products they're taking care of, plus developers who know a little about the metal their code is running on make for much smoother deployments.
DevOps is a model where developers can spin up a VM via some cloud mechanism, throw in their code and launch without having to get an SA involved. This is great for lower environments, and lets you use CI tools like Jenkins to do a build/test/refine loop, but once you get near production, presumably you don't want the code churning so much. Then you want the SAs in the loop to manage environments for stability and to tune for performance. What the suits want is an "easy-button" solution that does all the provisioning/tuning/deployments so they can drop all those knowledgeable, expensive SAs and just let the developers handle it all. This is probably suboptimal as you don't necessarily get the benefits of performance tuning and looking at the overall structure.
DevOps is a specialization which used to be part of standard system administration. Developing custom tools to do custom tasks, in this case related to "Ops" (another specialty that used to be standard sysadmin territory). The term is a great dummy term, but really does not distinguish someone's ability to manage servers and infrastructure.
You seem to be the 'new guy' who preaches that everything should be run in a generic docker container, complaining about the 'old guy' who wants none. Meanwhile, most of the people worth their salt understand that sometimes generic works and sometimes it doesn't. If there was some magic perfect layout everyone would be using it. Instead, we have a huge array of both hardware and software being used in the market. Knowing a dictionary of buzz words does not make you good, and usually indicates just the opposite.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
devops was spawned from startup culture. imho it was never intended to be something that lasted longer than a few hundred users or internal need. The needs of ops and developers, while similar in some ways, vary dramatically in others. devops is a compromise designed to foster quick, often unscaleable delivery of a service. Once larger companies caught on that you could make devs to op work, and vice versa, the trend became unstoppable. its still an insufferable title.
As an op, i dont write the kind of code that would get into a formal repo with a standup and peer review. im writing a script to automate something that either pisses me off or takes up too much of my time. I write breakfix code too, but its not designed to upgrade or mod easily and i dont share a ton of comments or documentation because im busy handling the infrastructure. piping my one-liners and quick functions through the holy trinity of git/gerrit/jenkins for review by real developers is a demoralizing pain in the ass. it also slows down fixes and deployments by subjecting talented coders to a confusing ratsnest of code that isnt meant to become a fundamental part of how a user interacts with the brand.
and making devs subservient to ops isnt something you should do either. scrum and kanban and swimlanes are boring 15-30 minute reminders that marketing drives the bus. We need to be part of the rollout process to make sure things, if they break, are handled appropriately. its nice to have validation the codes been tested as well, but all these things are an email or a quick chat and dont need to involve a manager.
Good people go to bed earlier.
Really? Lets add Special to the word Special DevOPS, there NOW I feel really important!
I FEEL the NEEED for a CERTIFICATE! YES! A 'Special DevOPS' Certification
Woo Hoo! Now I can really write code! Just look at my piece of paper, it says I can!!
Idiots.
And I'm an idiot for responding.
So he is full of it. DevOps is not about ops engaging with devs. It's about making all devs part of ops. It's a model which existed in commercial banks and other places which could afford to overpay (a lot) to have people do work a few notches below the pay grade they are paid. But it's high stress and snail-pace progress. It reduces specialization which, by definition, makes experience less valuable. It puts all of the testing burden on the developers and removes testing specialists. In the most extreme cases, it flattens the most experienced developers and newly minted college interns into doing the same job. The result is that the quality of the product is always determined by what the least skilled members of the team can handle. Oh, and because everyone is resentful and knows that they are overqualified, it creates a frat-house-like environment. Everyone end up overworked and under accomplished. But because it's used in the cloud management now (an industry growing quickly so it has as much money to burn as the banks), it creates the illusion of being successful. It's not.
The cloud started before there was dev ops. And it would continue and succeed without dev ops. It exists because all development is network development (to at least some degree) because CPUs can't be made much more powerful, they can only be networked on the chip (aka multi-core CPUs, GPU). And if there is no distinction between pooling multi-computer resources and pooling multi-core resources, then scaling is accomplished by pooling a lot of them in near-real-time batch processing (aka pipeline). Which means the state of technology makes the cloud computing model make more sense than stand-alone units computing model. The state of technology is what drives the business incentive here. And the business incentive is what creates the illusion that the fulfillment model of the business incentive is the right one. Even if it's not organization model. Poorly chosen organization model in the environment of over-funding will not fail until the market place becomes more efficient and consolidates. And at that time, all the DevOps shops will be unable to support their costs and won't understand why everyone is switching to well-engineered (through full SDLC) products. They'll be saying it's not in vogue anymore. While the reality is that DevOps is just an inefficient organizational model to develop cloud services.
Any guest worker system is indistinguishable from indentured servitude.
Not in my old company. In my company, QA was the ones let go and then they made a new DevOps team to replace them. I guess this is part of the great thing about DevOps. It means something different at every single company. I guess I'll go ahead and put DevOps on my resume, since I undoubtedly have done the jobs of at least 50 or so of the different definitions.
If you are not allowed to question your government then the government has answered your question.
There is so much misunderstanding because there is not a universal, static definition of DevOps that everyone can point to and say "that is DevOps" or "you are doing it wrong!" This is because DevOps is ultimately defined by the capacity of the people who practice it and I think we can see (already in these postings) that many people do not have the capacity to define it.
The history of DevOps begins with the people who coined it: Patrick Debois and Andrew Clay Shafer's first discussion about Agile Engineering at a conference in 2008, which led to a Google group and then to the first community meeting as DevOpsDays Belgium in 2009. W#e can trace to the beginnings and primary source folks, so please stop demonizing and making DevOps anything more mysterious than a knowledge gap.
For an overview with my definition of DevOps, please see my blog post with talk and slides that I presented at Silicon Valley Code Camp earlier this month:
http://mlavi.github.io/post/de...
My opinions are my own, but you may share them!
I have no strong opinions about the word "devops", however I think there is a role needed that doesn't get met by developers, QA, or IT. The problem is that some things don't fall squarely in anyone's lap, so everyone can play the "not my problem" game and nothing gets fixed. As a developer i've had to fix so many problems that aren't "my job" and are next to impossible to get time for. They add up to the point where it's darn near a full-time job. I'd love to hear anyone's recommendations on what job title that would be, or just how to handle this stuff better.
Some examples:
- Our software builds were completely manual. Everytime we wanted to ship a new version, someone would go and manually build the software, package it up by hand, and give it to QA. This led to all sorts of problems, and encouraged everyone to play the 'ping-pong game' between development and QA. I had to setup a Jenkins server and rework all of our builds. This was not that easy, because when you automate something, you lift the rug up and expose all the problems that have been swept under the rug for years.
- IT had created VM images for QA to use, with all the software dependencies pre-installed. That was screwed up because it didn't reflect how the customer installs on new machine. Plus they never updated their VM images because it's a PITA, and devs would never tell anyone when they added a new dependency. I had to turn all our stuff into RPM's that required the correct packages instead and teach everyone to update the spec file when they added new dependencies.
- Same story as above, but in regards to development machines. I had to create provision scripts and teach everyone to update those.
For the smartest people I know on the Internet, I find the disparity between the comments here and reality discouraging.
:)
I travel the country installing automation software, training people, and doing this whole "devops" thing (gad, I hate that word). I post here for info, not cred and I post here to tell what I see from the trenches. I also post anonymously because the bit $company I do this for wouldn't like to have my (highly visible in devops cicles) name plastered around, as it affects them too. (This is the first time in my career I've had to watch my mouth in public...It's killing me)
Quite awhile ago... about 10 years, I was working the Sysadmin/Eng world and enjoying it. This site was a Top-10 Webmonsters site... top 10 traffic receivers on the Internet for you guys not in the playground. In short, a bit site that provides an insanely popular service for the privilege of throwing a gazillion ads at the thundering herd. Hey, it's a living.
I was happily minding my store of all sorts of perl, shell, and wonderful glue of all sorts (we had even written a little orchestration platform that worked pretty well) but life was good. Then they hired "that guy" over in DEV. I think it was day 2 he wanted root. Well, we didn't "do that here" and weren't going to budge just for him. Logs were aggregated and made accessible via internal browsable sites, deploys were push-button and quick (all before Hudson/Jenkins) and the DEVs could iterate to their heart's content. They had no logins anywhere, but they had tools to do everything they wanted "root" for. This guy refused to see it.
He started yammering on about DEVOPS that long ago, but in his mind, his own little world, it just meant "give me root". As such, he had a one-track mind and whined and complained loudly to anyone who would listen. As chance would have it, our CTO was the "bee's knees" as they say and saw the writing on the wall. We were going to have to build something to shut this a-hole up. So here was the deal: "We'll give you your own environment that mirrors production. You have to care for and feed it yourself. You will have revision control to store code, push-button access to build deployment and a merge-structure that will allow you to transition target nodes from one environment to the other without friction... Your own little DEVOPS wonderland to do with as you wish. But there's a catch: You're unsupported. If you open a ticket to "fix" something, the "fix" will be restoring the snapshot of the working environment from provision time over whatever mess he had made. Since any sensible developer will commit early and often, that shouldn't be an issue. Restore, check out latest tag, re-run the process, and voila! You're moving again in minutes! (we even put the destroy & rebuild buttons on a web page via perl CGI so he could easily do it himself without the embarrassment of asking us. (not without a counter and an RRD graph to track his rebuilds, mind you). Plus, being PCI governed, he had to use Powerbroker to connect, which dutifully carted off his entire command line history to a different server for storage.
That's the backdrop. He definition of DEVOPS given to him on a silver platter.
Fast forward two days. TWO. DAYS. 34 rebulds and reimages later. TWO FRICKING DAYS!!! See where I'm going here? The early translation for DEVOPS was "give me root and leave me alone". Well, apparently that level of power causes you to completely restore your system from snapshots 34 times in two days. (but I digress... And THIS guy was a highly-paid, superman brought in for top dollar to train everyone else!!!)
So, here I sat with 15 years experience at the time and I had a hunch. I did some research into what he thought he knew, and I thought this thing was going to be big. REALLY big. IT automation, data-center based company infra being farmed out to you at a fraction of your hardware costs, streamlining the pipeline between code and d
He views it how it is. Most people not only don't know outside their domain, but are horrible at the one they specialize in. Heck, half of our customers that have their firewalls blocking our ability to transfer files via SFTP, ask us how to configure their firewalls to open ports because they don't know how to do it. I'm talking about network admins. Freaking networking admins asking us how to do their job. Sometimes they just give us admin access to their firewalls and tell us to figure it out.
Another company we worked with had this nasty habit of sending us their private key every time they change their keys. It seemed like they changed their key pairs every few months because they didn't know how to keep their old one when they upgraded their servers. So they just generated a new key and had it signed. Then they'd EMAIL us their public and private key. A freaking Verisign signed private key that they use on their website.
Another company was freaking out that we didn't encrypt our data at rest, then requested that we communicated sensitive data via FTP and legacy zip format to encrypt the payload. Not only did it have known password issues that allowed even the strongest passwords to be easily broken but they required that we used a 6 digit all lower case password and refused to listen to us about it being incredibly weak. Luckily I brought this to the attention of the VP, and we were in the process of phase out all FTP support of any kind, and he jumped in an required SFTP. Not that their SFTP site that they provided with an 8 year out of date server version that used OpenSSL was going to add much protection, and their 5 char FTPS(Not SFTP, and uses SSL) password that was a shared account and had read/write access to a slew of files.