Slashdot Mirror


Dev vs. Ops: The State of Accountability (overops.com)

Here's an analysis by OverOps on how shared accountability affects the delivery of reliable software in a DevOps environment, and what are some of the top challenges teams face when it comes to building and maintaining quality applications. Conclusion from the report [PDF], which relies on a survey of over 2,000 IT professionals around the globe : At the center of this DevOps adoption chaos is the evolving relationship between development and operations. Many organizations are already taking a shared approach to accountability for application health, however they still lack the tools and application visibility needed to know who is ultimately responsible for addressing and fixing each issue. As the lines between these two teams continue to blur, organizations will need to focus on adopting tools that deepen visibility into their applications. Clarifying ownership of applications and services, and avoiding the "multiple owners = no owner" syndrome is a crucial for even the most bleeding edge organizations.

The "Dev vs. Ops: State of Accountability" survey revealed that as more organizations begin the transition to DevOps workflows, defining roles and processes becomes more difficult and more important. Furthermore, businesses of all sizes are building and releasing new code and application features faster than ever before, which adds additional pressure across the entire software delivery supply chain. Organizations going through the DevOps transformation are more likely to face visibility challenges that make it difficult to maintain or improve application quality and reliability.

92 comments

  1. And thats why by Anonymous Coward · · Score: 1

    ... devops actually sucks just like every other âoeagileâ fad that came before and produced nothing of value other than the ability to avoid accountability

    1. Re:And thats why by Bengie · · Score: 4, Interesting

      Devops the concept is great, "devops" the buzzword is just adding more complexity with little or no benefit. The idea of devops is the devs design their projects such that they maintain the operations. This leads to the desirable outcome of a moral feedback that forces the devs to actually care. Without it, devs just throw the responsibly to operate over the wall and divorces them from the consequence of poor design.

      Without devops, devs tend to product projects that are brittle and need to be micromanaged. Why would devs care if ops gets called at 1am on a Saturday? If the devs are getting called in the middle of the night, they'll start to care.

      Ops should be limited to maintaining infrastructure services like VMs and creating tools to allows devs to deploy to prod on their own in a controlled way.

    2. Re:And thats why by XXeR · · Score: 2

      Ops should be limited to maintaining infrastructure services like VMs and creating tools to allows devs to deploy to prod on their own in a controlled way.

      That's a valid approach for sure. What I've found in that case, however, is that taking Ops out of the loop and relying completely on developers for operational response to an incident in production yields poor results. Put simply, it's not their core competency, and most don't take to it well (although some do!). It's a very different type of pressure to which they're unaccustomed.

      In my experience, a hybrid approach is best, where Ops personnel quarterback a production incident (or Support, assuming it's a mature org), and escalate as needed to development.

      YMMV

    3. Re:And thats why by Cylix · · Score: 4, Insightful

      Unless your project is very small or your management is stellar this just builds a road to disaster.

      Developers who do not feel the pain of their actions are not incentivized to correct issues. In the beginning, there may be a few bright eyed engineers who take to heart the messages, but eventually that gets lost in the sea of priority. A number of things start to happen which can increase page count. It could be a hard to find bug that really only produces a few escalations, but combined with a number of those the issues can severely add up. Increasing escalations that don't get attention because individually they are not severe enough and well the operators can handle those.

      Perhaps the largest issue only hinted at is the brittle nature that tends to evolve. It may take the form of circular dependencies or poorly considered dependency trees. Oh, such as taking a dependency on a tier 2 service for your tier 1 service or perhaps no one discussed the volume of traffic they would be placing on this other service. Those types of failures start to creep in when you disconnect yourself from your platform and multiple specialized teams start making changes to a cohesive whole. The reality is, despite everyone's hopes, is that living in the service keeps you connected to the details. Personally, I witnessed this type of degradation occur so severely I was likely one of maybe three people who understand the entire platform in an organization of hundreds.

      The last service I helped build and design was built with the concept of "I don't want to be paged in the middle of the night." We didn't get it to that point by passing the micro issues to an oncall. No, the other teams did that and many of our pages are the result of those teams poor implementation or lifting work to our successful team.

      I've seen whole operation teams nearly up and quit after development teams were completely disconnected from their projects and shouldered maintenance. It usually goes that way or you get some very battered and dedicated people who eventually trickle out until the lynch pin fails.

      Pain is a great motivator to fix problems or at the very worst fix their poor alarms.

      --
      "You should always go to other people's funerals; otherwise, they won't come to yours." -- Yogi Berra
    4. Re: And thats why by Anonymous Coward · · Score: 0

      Nice to see some one with the right mindset, very few have. Wish we had you in my company.

    5. Re:And thats why by Aighearach · · Score: 2

      When I first heard about devops it was in the Ruby web dev community ~2007, and it referred to a role that was a liaison between the programmers and the sysadmins. Their job was to understand the dictates of the BOFH, and to help the programmers find a way to implement what they wanted in a way that was consistent with all the organizational rules. Mostly security or technology restrictions.

      Then whoever decided to pay to hype it changed it to mean some sort of management service, a type of telemetry for the PHB to measure the team. It doesn't seem to even be a role on the team anymore, but instead a service where you pay consultants to spy on your team. I guess the idea is to externalize the resulting hate? It doesn't seem to be working.

    6. Re: And thats why by reanjr · · Score: 1

      It's a balancing act, but if the devs can't work with the ops tools, the ops tools should be reworked until they can. Devs are generally smart. If you can't build tools they can use, then you probably aren't managing your ops issues well enough. You're probably reacting to problems instead of fixing them.

    7. Re: And thats why by reanjr · · Score: 4, Insightful

      Sysadmins mostly work at Amazon and Google and Microsoft now. So, devops is largely now the process of automating cloud services.

    8. Re:And thats why by AuMatar · · Score: 2

      Bullshit. I've never known a dev not to care about maintainability and stability of their service. What devops causes is burnout for developers, as people who never signed up for 24/7 on call get forced into it, and people who know absolutely nothing about being a sysadmin get forced to do it, and do it half assed as a result.

      Its one thing to have devops at a small startup, where you literally can't afford more people and need to wear multiple hats. At anything bigger, its a complete joke- you either have 1 person on the team doing 100% ops anyway, or you have a bunch of unqualified people hacking at it and doing it badly. The main cause of 1 am pages on Saturday is having devops instead of ops.

      --
      I still have more fans than freaks. WTF is wrong with you people?
    9. Re:And thats why by leonbev · · Score: 2

      DevOps is pretty cool when done correctly, where infrastructure is fully automated to the point where can you deploy new servers with the latest code and security fixes with just a mouse press.

      Of course, most organizations don't have the resources or technical skill to pull that off or maintain it correctly. Worse yet, some of those same organizations also tend to be staffed with clueless managers who think that "DevOps" means that they can hire a junior developer out of college to replace their senior systems administrator. These same people then wonder in amazement when they are the victims of a major security breach six months later.

    10. Re:And thats why by hey+hey+hey · · Score: 1

      Without it, devs just throw the responsibly to operate over the wall and divorces them from the consequence of poor design.

      I have no clue where you have been working (and hope never to work there), but in all the places I have worked, dev is *NEVER* divorced from the consequences. If there is a problem that isn't obviously hardware (failing disk, machine rebooting from kernel panics, router failing, ...), dev, almost always in conjunction with ops works on the problem. Sometimes the problem will still be hardware (a failing drive in one system manifested as a DB slowdown in a different system), sometimes it will be in the app.

    11. Re:And thats why by Bengie · · Score: 2

      I've always been interested in all aspects of the lifetime of software services. I am part of a small team in my company that deal with one-off custom projects, typically driven by SLA'd large contracts. We get district, state, and national level requests. We need to deal with many aspects of these demands. Our company does not specialize in contracted work, the work is all value add, it has to be treated as being throw away work, but at the same time we cannot train up Ops or any other team. Due to all of the requirements, it is impractical to get too many people involved.

      We analyse, design, implement, deploy, and operate our custom services as a 5 person team, in a multi-billion dollar company with hundreds of software engineers. We have tens of millions of users who tend to hit the myriad of services hard at roughly the same few times every year, and a low constant rate throughout the year. The systems need to be stable, performant, easy to configure, easy to diagnose, and difficult to not understand. It must lead you to doing the correct thing. Make it easy to do the right thing and difficult to do the wrong thing.

      Our team has several years of backlog, primarily in tooling enhancements to better support ourselves. Our biggest pain point is we're dependent on nearly every other services in the company. This means our services break when other services are not working to spec, assuming we're graced with a spec. We constantly deal with undocumented unreliable systems that are liable to change without warning. When we design a system, we go over nearly every possible case and either handle them or make it blatantly obvious what the issue is.

      I get peeved when I hear someone from ops putting in 60 hour weeks because the general services are constantly needing to be hand-tuned with a seemingly chaotic load pattern because of the.... "intricate"... inter-service dependencies and unreliable performance characteristics. I just want to tear out someone's throat when I ask "how many concurrent requests should I be making to your service and how long should we wait before considering it a timeout?" and getting "Measure it and see what's fastest. If you're getting timeouts, try increasing your timeout setting and see if that helps. If you're still having performance issues, open a defect." Yeah.. please find another line of work you code monkey. The best part is when someone responds like this, we almost invariably overload their services and ops yells at us. And boy do I hate making ops' life any harder than it needs to be.

      We quickly found that we needed to add multiple forms of rate limiting to our services. Engineering "load tests" their services and they claim everything is fine, but for some reason we blow their crap out of the water. It's been an interesting exercise in system design to make sure our systems are very gentle with the snow flake services because they seemingly spontaneously go from steady-ish response times to timing out every request for a minute or so without warning. Generally feels like garbage collection issues, but we never know.

      I found out ops has to manually scale many of the AWS services up and down because certain negative scaling characteristics. No AWS auto-scaling here. Adding more nodes to many of the clusters barely gives any increase to throughput, and response time jitter increases as the number of nodes goes up. The main benefit of adding more nodes is there are more instances to buffer events. Other services expect an event to be "serviced" in a reasonable time, so pulling an event out of the event queue makes it look like they are. Of course, if one of these nodes goes down while holding onto all of these events, the events are effectively lost without lots of manual intervention. And each additional node increases the risk of race conditions, to which they have created hundreds of scripts that walk the data during off hours to make educated guesses on how to fix data in an unknown state. But whatever, it's ops' issue now, right?

    12. Re:And thats why by Anonymous Coward · · Score: 0

      The length and lack of focus in your comment suggests that you are part of the problem in your own environment. I've been on both sides. The decent ones learn from the other side and the best teach the other side. The marginally skilled guys blame the other side because they don't even know how to do their own job properly. You sound marginal.

      If "Ops" asks "Dev" for the expected load, Dev should respond with "here's a script to run for about an hour which should roughly simulate about #X transactions, we want however many containers/servers/nodes gets it done in an hour. Please let us know what that will cost department Y if it isn't trivial (for our expected unit size)." Shitty developers run "chmod -R 777" on the /srv directory and then blame Ops when things don't work on a production system where global "777" isn't allowed.

      Good Ops teams get the build script from the Dev guys and can flag out the "777" bullshit to get the scope narrowed or point out that the Dev's design is inherently insecure. Shitty Devs whine that Ops/IT are "roadblocks" and that Devs are "customers" and "always right".

    13. Re: And thats why by Anonymous Coward · · Score: 0

      First off, you are supposed to be a team, asshole. You want a cookie for a successful DDOS?

      Second, tell your ops team to enable caching. Cloudfront, elasticache...also decoupling with SQS...there are tons of tricks. If you can do better than them, go do ops...because as someone that both codes and architects, anyone can write code, it is expendable, but not anyone can architect. It pays really well and can not be outsourced.

    14. Re: And thats why by Anonymous Coward · · Score: 0

      Then prepare to get outsourced if all you want to do is write code. Writing code is not special anymore...yeah, you might be better, but you sure arent cheaper, and eventually the bean counters will take that gamble.

      Suck it up and learn ops. Go architect your own shit and then have the lower ops guys take the call outs. Climb the ladder or keep eating shit.

    15. Re: And thats why by Anonymous Coward · · Score: 0

      Someone who gets it. There are still the large companies hogtied to Oracle, but eventually it willl all be public cloud.

    16. Re: And thats why by Anonymous Coward · · Score: 0

      Easy answer. Devops is not a development strategy. It's a hiring strategy. How competent engineers who know how operations works and how software development works.

    17. Re: And thats why by Anonymous Coward · · Score: 0

      Hahaha that's rich. I, and several others on my team, just quit a job where development was never subject to any requirements or accountability. Just loved seeing the Indians cause a problem, telling them how to solve/avoid it, seeing them ignore it, and having to deal with it over and over and over.

      I mean, I did DevOps before it was cool. The problem is that it's not a silver bullet. You can't hire hindu chimps and just say "go do DevOps." That doesn't work and it has nothing to do with DevOps.

    18. Re: And thats why by AuMatar · · Score: 1

      I'm perfectly prepared to be outsourced- I have all the manager's emails, and I'll let them know that my hourly rate is 4x what I charge as an employee to fix things after they fuck it up. As the inevitably will-- the percentage of oursourcings which work is way down in the single digit percents. The number that work but don't end up costing more in the end are even smaller.

      Not to mention your argument is non-sensical- if they can outsource dev, they can outsource ops. So it would make someone no safer.

      In the meantime, I'll continue writing code in the hottest market for devs I've seen since the dot com craze. I'm not going to call ops work easy (it takes its own skill set), but the pool of people who can do it at a high (or even mediocre) level is 5x the number of people who can write good code.

      --
      I still have more fans than freaks. WTF is wrong with you people?
    19. Re: And thats why by Anonymous Coward · · Score: 0

      They dont outsource the work that is the current architecture of the company, that is done by in-house devs, or at worst, a 3rd party with high paid devs in this country (so you coukd work there)

      Programming is turning into a code monkey position, CRUD and ETL. Specialized 3rd parties taking care of anything more advanced than that.

    20. Re: And thats why by Anonymous Coward · · Score: 0

      Indians arent trusted to do devops, thats why its a high paid position. Indians are, for financial reasons, trusted to do programming, which is why you do not want to solely be a programmer. Everyday you will have to justify why you are worth 5-10x the salary of Ackbar.

    21. Re:And thats why by Anonymous Coward · · Score: 0

      Spoken like a true infrastructure wonk that does the minimum to get the devs to leave you alone.

      Then complain when the piss-poor server and network cause application issues and the dev has to come back to you to get you to fix your crap.

      The entire point of Devops is to make one team own the entire process from start to end and avoid finger pointing of whose problem it is.

    22. Re: And thats why by Anonymous Coward · · Score: 0

      +1

    23. Re: And thats why by Anonymous Coward · · Score: 0

      > They dont outsource the work that is the current architecture of the company,

      Yes "they" do.

    24. Re:And thats why by Anonymous Coward · · Score: 0

      I'm my experience it is better the other way around. Proper development only happens when the developer is well rested. It is the same boat with trying to put professional services people on-call rather than just having them as an escalation point. The helpdesk fields the majority of the work, consequently they have the majority of staff. If they can't fix it they escalate to tier 4 which should know enough about development to produce proper bug reports. They then work with developers to resolve the issue. If the developer doesn't care about the organization and its problems then odds are you have the wrong developers on-staff.

      When Sr Engineers work with Sr Devs who then delegate the work that is needed you end up with a clear chain of command and accountability easy because accepting an issue means you are now accountable for it. So many organizations I've walked into Dev and Ops had adversarial relationships. I can usually fix this pretty quick by interfacing with project managers and talking to people on both sides. After I've gathered enough info I can talk to larger and larger groups of people to build consensus. That is usually the problem with most orgs. They don't bother to build consensus which leads to a Dev not taking and issue seriously because there really are bigger fish to fry. Issues can't just be ignored so you develop timelines and set reasonable expectations. Knowing the Dev will take care of the problem by next months gives Ops a light at the end of the tunnel and unit cohesion and start to take place.

    25. Re: And thats why by Anonymous Coward · · Score: 0

      Spoken like a (wannabe?) PHB.

    26. Re: And thats why by Aighearach · · Score: 1

      That doesn't differentiate between anything, though.

      We already had a word for the person that automates VM services; sysadmin.

    27. Re: And thats why by AuMatar · · Score: 1

      Sure they do. And ops != architecture at any rate.

      --
      I still have more fans than freaks. WTF is wrong with you people?
    28. Re: And thats why by Anonymous Coward · · Score: 0

      The classical error that everyone here seems to make is that Ops in this case does not reflect the operations of the entire infrastructure, but just the operations of the application itself.
      DevOps is not about installing patches and preventing MeltDown exploits.
      DevOps is about developing and operating an application, from concept, through production, all the way to decommissioning.

      The platform is a given (hence usually some form of cloud, or 'Platform as a Service'), operations is about maintaining the application and keeping it running in optimal form.
      DevOps decides to make things scalable, stateless and use a load balancer, it doesn't configure load balancers or assigns IP addresses.

    29. Re:And thats why by Anonymous Coward · · Score: 0

      I agree developers should operate their service and Ops should manage the infra, VMs, networking, certificates and other aspects that make the service operational. If Devs simply pitch code over a fence and make an ops team manage the service, it does lead to some weirdness that really impacts the end user the most.

    30. Re:And thats why by JoeDuncan · · Score: 1

      Devops the concept is great, "devops" the buzzword is just adding more complexity with little or no benefit.

      No, the whole thing is a dumb concept invented by people who don't know how to run a software shop properly.

      The idea of devops is the devs design their projects such that they maintain the operations. ... Without devops, devs tend to product projects that are brittle and need to be micromanaged. Why would devs care if ops gets called at 1am on a Saturday? If the devs are getting called in the middle of the night, they'll start to care.

      Again, what the hell? I deny your above premise scenario as complete BS (or incompetence). This is NOT the way ANY software dev shop should be run!!! I've been doing professional software dev for 20+ years and have worked in half-a-dozen places over that time - and in every single shop I have worked in, the devs have ALWAYS been "third line support".

      First line of support: customer service reps handling non-technical issues for users.

      Second line of support: ops personnel handling things like connectivity issues (routers, switches), server up time, backups, password resets, OS issues, permission, power issues etc...

      Third line of support: any issue with the software/application itself

      So in a competently run software shop, the devs ARE the ones getting called at 1am on a Saturday when the software breaks; it's happened to me a couple of times - but I strive to make software robust enough that I don't get those midnight calls.

      If your shop is calling the ops people to handle a development software issue in the middle of the night, well: "you're doing it fucking wrong". These same people would probably call a chauffeur instead of a mechanic when their car breaks down FFS.

      Ops should be limited to maintaining infrastructure services like VMs and creating tools to allows devs to deploy to prod on their own in a controlled way.

      Where the hell have you been working?!? This is *exactly* how it has been done in every software dev shop I have ever worked in, it was simply considered "standard operating procedure" (no one ever called it "Devops" though, they just thought it would be stupid to NOT do it that way)

      Lastly, most of the reading I've done on the "DevOps" concept seems to be more overly concerned with teaching sys admins how to code rather than getting coders to handle application maintenance (which is, again, *standard practice* in competent software shops). This is precisely what my boss did at one of my last jobs when he got excited about the "Devops" hype - he sent all the sysadmins and QA engineers and testers on "learn how to code" courses.

  2. When Buzzwords Clash! by Anonymous Coward · · Score: 0

    "Embrace the Agile paradigm ..."

    1. Re:When Buzzwords Clash! by Anonymous Coward · · Score: 0

      "Host a plethora of microservices in your open SOAPy back-end."

  3. Jumping to conclusions by Anonymous Coward · · Score: 0

    1st Bob: What you do at Initech is you take the specifications from the customer and bring them down to the software engineers?
    Tom: Yes, yes that's right.
    2nd Bob: Well then I just have to ask why can't the customers take them directly to the software people?
    Tom: Well, I'll tell you why... because... engineers are not good at dealing with customers...
    1st Bob: So you physically take the specs from the customer?
    Tom: Well... No. My secretary does that... or they're faxed.
    2nd Bob: So then you must physically bring them to the software people?
    Tom: Well... No. ah sometimes.
    1st Bob: What would you say you do here?
    Tom: Look I already told you, I deal with the @#$% customers so the engineers don't have to. I have people skills! I am good at dealing with people, can't you understand that? WHAT THE HELL IS WRONG WITH YOU PEOPLE?!

  4. TP reports by Anonymous Coward · · Score: 0

    Peter Gibbons: It's a problem of motivation, all right? Now if I work my ass off and Initech ships a few extra units, I don't see another dime; so where's the motivation? And here's something else, Bob: I have eight different bosses right now.
    Bob Slydell: I beg your pardon?
    Peter Gibbons: Eight bosses.
    Bob Slydell: Eight?
    Peter Gibbons: Eight, Bob. So that means that when I make a mistake, I have eight different people coming by to tell me about it. That's my only real motivation is not to be hassled; that, and the fear of losing my job. But you know, Bob, that will only make someone work just hard enough not to get fired. ”

  5. Reliable software? by Anonymous Coward · · Score: 0

    Apple and Microsoft could learn a lot about reliable software from Linux.

    1. Re: Reliable software? by Anonymous Coward · · Score: 0

      Apple has Linux as a kernel component. Hello! Users dont have time for all the Linux intricacies and often just want it taken care of for them

    2. Re: Reliable software? by reanjr · · Score: 1

      Apple is based on Unix, not Linux. And as someone who spends a lot of time writing shell scripts, I assure you they are quite different.

  6. More Crap!! by SirAstral · · Score: 4, Insightful

    IT and the "movement disease".

    I am sort of tired of this constant "revolution" garbage that surrounds the IT industry in general. I work in this shit industry, I am well paid for what I do, but one thing is always certain... It will always suck because everyone in charge of IT came from college where stupid is the only thing being taught when it comes to computer science.

    There is never anything innovative being done, by the time I am done listening to a sales pitch I realized I have heard all of this shit before, it's the same shit product emulating another shit product surrounded by proprietary technology that works like shit just in a different shitty way.

    There is also the problem that every industry has... 20% of the folks do 80% of the work. Do you know what else tends to happen? 20% of the people are the only ones that knows what to do or what is going on. Do you know what else? IT is not a meritocracy either... it is still the same brown-nosing ass licking who you know path to success, like every other department. Those in the know are constantly assaulted by their lesser skilled and capable "co-workers". Those in the know are constantly waiting for some other knob in a different department to do their own damn job. And all of this while management keeps not getting a fucking clue and piling on more and more work to the point that more than 50% of projects fail by either never having the proper amount of time & expense dedicated to it.

    These bullshit "cultural DevOps, ITIL, Agile, Waterfall, blah blah blah" are all stupid ideas people keep coming up with to address the problem of an industry that is riddled with incompetent management trying to rule over an incompetent group of pseudo intellectual nerds that know far less than they put on. And that is another problem as well... people hate IT personnel that do not sound "over confident" it is a practical requirement for IT pro's to act like they know every fucking thing there is to know and yet those of us at the top know different. We are all running around trying to figure out every little fucking thing on the fly because experience has taught us to just roll up our fucking sleeves and work it out... regardless of whichever newfangled fucking "operation ideology" that someone pushes.

    1. Re:More Crap!! by Anonymous Coward · · Score: 0

      Who asked you to whine?

    2. Re:More Crap!! by Anonymous Coward · · Score: 0

      There is never anything innovative being done

      But the illusion of innovation is ever present.

    3. Re: More Crap!! by Anonymous Coward · · Score: 0

      You need drugs.

    4. Re:More Crap!! by XXeR · · Score: 2, Informative

      You sound like you're referring to corporate IT throughout your comment, or at least that's my impression. DevOps concepts don't really apply there...or at best it's a square peg in a round hole situation.

      Building custom software, and more specifically SaaS, truly do have a lot to gain by adopting DevOps, especially when combined with Agile development.

    5. Re:More Crap!! by Anonymous Coward · · Score: 0

      Agreed. "People who think they know everything are a great annoyance to those of us who do." - Isaac Asimov

    6. Re: More Crap!! by Anonymous Coward · · Score: 0

      Eat shit and die you fucking clueless buzzword dropping fuckstick

    7. Re:More Crap!! by Anonymous Coward · · Score: 0

      The world is unfair. Tech is complicated. Our jobs are hard.

      Pretty much sum things up?

    8. Re:More Crap!! by Bengie · · Score: 3, Insightful

      Most of the buzzwords are real things that someone implemented very successfully, but the general population likes to treat it like magic and if they do some rain dance, all of their problems will go away. Of course they don't understand anything about the dance and just follow the path of least resistance, which is nearly guaranteed to be wrong. Devops and Agile are very real processes to dealing with certain types of problems. There is no one process for these. They're a class of processes. You need to tweak the process for your current issue. If you don't understand the process, you'll end up using the claw end of the hammer to hit nails.

      Technology, aka tools, can never fix human problems, like incompetence. Incompetence in the work force is reinforced by incompetent people hiring others more incompetent than themselves.

    9. Re:More Crap!! by AndrewFlagg · · Score: 1

      spot on and i love the language used to describe most large organizations. the problem you describe can't really get away with in it smaller teams less than 10 or 5. one of my friends, I call him Tony E. talked a lot like you write. we worked for a few years together, and another Andrew F., both different, both great... we had a few others and they were idiots and we fired them... kicked them to a state job or something more their sped.... one thing about your style, i hope this is active-pre-aggressive not passive aggressive communication. more importantly, keep up the great work. need more attitudes and thinkers and believers like you. happy new year 2019.

    10. Re:More Crap!! by SirAstral · · Score: 4, Insightful

      I have lived in both, once again... just because you change how things get managed it does not change the underlying idea I am putting forward.

      No matter what you put forward the basic problem with incompetent people running the show exists. DevOps will fail just like every other doctrine before it.

      I don't hate DevOps, Agile, ITIL, Waterfall or any of that stuff. They are all perfectly valid ways of doing things... and I do mean PERFECTLY VALID, the problem is that it is another smoke and mirrors attempt to move the goal post because no matter what you say or do... the blame has to go somewhere other than at the people in charge.

    11. Re:More Crap!! by AndrewFlagg · · Score: 1

      i appreciate the props. isaac reborn.

    12. Re: More Crap!! by lolo220 · · Score: 1

      then try to cast blame while someone else fixes it. Don't work for a tech company where the CEO isn't an engineer. https://8ballpool.onl/ https://discord.software/ https://omegle.onl/

    13. Re:More Crap!! by Aighearach · · Score: 1, Offtopic

      Who asked you to whine?

      I did. I was listening to Astral Plane by Jonathan Richman, and I thought, "gosh, are these devops guys just astral travelers, or what?"

      Now, thanks to SirAstral, we know the answer. Or to paraphrase, "revolution in, revolution out."

    14. Re: More Crap!! by reanjr · · Score: 1

      Why do you think software should be innovative? Software is largely the translation of human-focused business processes into machine-readable code. If you are spending time "innovating" in this space, you aren't servicing your employer.

      Some people get to work on innovative shit, but you shouldn't expect that to be the norm.

    15. Re:More Crap!! by Anonymous Coward · · Score: 0

      Devops only works if you go into it where your developers work with your support team to crank out the broken issues for the users. But if you think it will replace your support team you will fail. If you are renaming a different position 'devops' because the name sounds cool, you will fail. If you just put someone in that position 100% of the time. They will burn out, you will fail. DevOps takes work to make it work. When it does it can really make your customers think you are awesome. When it fails the users do not see anything different and you just inflicted yourself with a lot of busy work.

      The team I am currently on calls itself devops. It is failing at that because it was never defined up front what the hell we actually do. But it sounded 'cool' to have a devops role. I work with neither support or the users or the developers. It is just a separate team that just runs some software and makes a few scripts. You know what we used to call an 'it admin'. Support treats me like a developer (and ignores me), developers treat me like support and give me shit things they do not want to do, and the users I have never met.

    16. Re:More Crap!! by davecb · · Score: 1

      I've noticed that the technologies change quite slowly, but the buzzwords are redefined annually (;-))

      For example, Communicating Sequential Processes were first described in a 1978 paper by Tony Hoare, but only widely adopted by Golang folks in the 200Xs.

      --
      davecb@spamcop.net
    17. Re:More Crap!! by Anonymous Coward · · Score: 0

      And then there's the Slashdot Scatalogical Writing Disease...

  7. What bollocks by Anonymous Coward · · Score: 0

    I don't know how people like you can look at the past five years, with what's happened with Docker and containers in general, and cloud in particular, and how they've changed almost everything about deploying and running software, and say things like "it's the same shit product emulating another shit product". I mean, really?

    If everyone and everything around you is the problem, the problem might in fact be YOU.

    1. Re:What bollocks by Anonymous Coward · · Score: 1

      old concept learn to operate a mainframe and nothing fundenmentally new will be under your sun

    2. Re:What bollocks by plopez · · Score: 4, Informative

      Docker is getting more and more heavy weight, i.e. becoming full blown VMs (Though in the strict sense of the word they always were). Now part of the problem I see is that projects end up with containers scattered around like jack straws, like the "DLL hell" many experienced in the past, or plug-in hell. All docker does is allow the complexity to take a different form. Also stateless containers in my experience are pretty useless. Working on back end "heavy lifting" applications somewhere you need to maintain state and Docker and stateless containers, by definition, cannot do that.

      Cloud is just putting the application somewhere else and paying by the cpu cycle. No different than before, it just makes it opaque as to what is going on and who is responsible. There's really not much new under the sun and good ideas are basically reinvented over time. Mainframe == cloud hosted apps on your mobi or browser. Nothing to see here, move along...

      --
      putting the 'B' in LGBTQ+
    3. Re:What bollocks by SirAstral · · Score: 1

      Application Virtualization is nothing new.

      Just because you change it does not make it innovative.
      Innovative means NEW... application virtualization is not.

      I have been doing this for a while... sure I might be a problem, but likely only to the people recycling old hat like it is a new hat.

    4. Re:What bollocks by SirAstral · · Score: 1

      this is so spot on!!!

    5. Re:What bollocks by Anonymous Coward · · Score: 0

      Docker and containers in general are the epitome of fucking garbage. They're like getting veneers on your teeth: they might look really nice, but in the end you have even more fucked up teeth.

    6. Re:What bollocks by Aighearach · · Score: 1

      I thought you were supposed to maintain the state in whatever framework you're using to generate the docker containers?

      I guess that is the problem; the more common practice is to receive containers rather than to build them. So whatever state you need is tacked onto the side.

      IMO if you're using cloud computing to run your services, it should only be providing horizontal scaling. Nothing else should change. In that way of thinking, the statelessness is something you build into parts of your architecture by compartmentalizing the state in such a way that the parts that scale horizontally are stateless at runtime. This has to be done from inside the problem domain, it isn't going to be turn-key and also fit well. Docker is useful as a container, but a container isn't a framework. It is like having a nice lunch box; it tells you nothing about the lunch inside. And simply trading lunches with people who have nice lunch boxes doesn't imply that your personal dietary needs will be met.

      Cloud computing solves the problem, "What happens when you need two mainframes? Or maybe three? And what if it changes from hour to hour?" Unfortunately, a lot of devs treat it as "Geocities for elite programmers" and forget to also plan their architecture before they implement it.

    7. Re:What bollocks by Anonymous Coward · · Score: 0

      Cloud is just putting the application somewhere else and paying by the cpu cycle. No different than before, it just makes it opaque as to what is going on and who is responsible.

      Really? Just the fact that you can auto scale with just the click of a button or automatically on the cloud is different than before. You don't even need to maintain machines anymore.

    8. Re:What bollocks by 110010001000 · · Score: 1

      You can only auto scale if the APPLICATION is built to handle it. It isn't magic. That isn't new either. Machines need to be maintained: just not by you. That isn't new either.

    9. Re: What bollocks by Anonymous Coward · · Score: 0

      Except you can spin it up instantly. Want a server in Japan before cloud? Youd have to spec it out, make tons of calls, pay a small fortune, and hope you built it correctly.

      The benefit of cloud is time, everything goes up fast and can be changed on the fly. Seriously, anyone that hates cloud just fears making the transition, and rues the day their company forces them to do so.

    10. Re:What bollocks by Anonymous Coward · · Score: 0

      Of course it's magic. Want more requests per second? Scale up. Want more bandwidth? Scale up. No need to modify your code. Those that code better will just initially have more requests per second and less bugs to fix.

  8. don't fix what ain't broke too... by Anonymous Coward · · Score: 1

    I'm in a situation now where a new VP of IT was hired late last year at the smallish (approx 500 employees total) company where I work but he came from a large multinational. Since coming in he started introducing various changes to our development and deployment processes, wringing his hands frequently about how bush league we supposedly are and frequently invoking "well in other companies they _typically_ do X".

    The "funny" thing being that every time he introduces some new process that supposedly will improve delivery, it makes it worse and orders of magnitude more time consuming. We've gone from the past where deployments were total non-events and never posed any problem to a place where they are a walk through hell involving dozens of people for no good reason and they all bike-shed and conflate the unrelated so much that each and every deploy is now a fiasco.

    The other "funny" thing is he constantly spouts agile terminology while forcing us to undertake a process that is virtually the exact opposite of agile.

    1. Re:don't fix what ain't broke too... by Anonymous Coward · · Score: 0

      This is typical big corporate mentality. Get enough red tape in the way, and enough people that have to sign off on stuff that no single person (except the guys at the bottom of the totem pole) can take the fall for a major fuck up. Cause when one does happen 20 mangers, and VPs can all stand around pointing fingers at each other with no one being able to take blame. While the guy who pushed the button gets escorted out the door. After a couple weeks of "root cause analysis" it just gets swept under the rug and forgotten about with no one in middle/upper managements head rolling.

    2. Re:don't fix what ain't broke too... by pauljlucas · · Score: 1

      Any time a company brings in someone new at either the VP or C levels, that someone always has to change something to justify himself. Nobody gets hired to do more of the same as the previous guy even if everything works well already.

      --
      If you reply, do so only to what I explicitly wrote. If I didn't write it, don't assume or infer it.
    3. Re: don't fix what ain't broke too... by Anonymous Coward · · Score: 0

      PA state IT is going through this right now. L&I, who just got taken for $170M by IBM, keeps foisting more bureaucracy on everybody. Everything takes longer, fucks up harder, and we're drowning in AARs that change nothing. I saw the writing on the wall early and started filling out job apps. Complete shit show that I decided I wasn't going to put up with any more of than I had to.

  9. Re: Apple kernels by Anonymous Coward · · Score: 0

    Apple? Linux or UNIX kernel? XNU for Darwin was BSD based. Maybe you're thinking of Google/Android.

  10. .pdf.pdf by Anonymous Coward · · Score: 0

    Not starting out well when they cannot name the file correctly. Also was this created in macOS? Not good.

  11. Re: Apple kernels by Anonymous Coward · · Score: 0

    Did I say kernel? Last week... https://it.slashdot.org/story/... https://apple.slashdot.org/sto... Not to mention that clicking "check for updates" puts you on Alpha testing for Win10.

  12. Operations? by Anonymous Coward · · Score: 0

    Operations has been decimated in the past several companies that I worked at. Developers are running operations... an analogy would be the insane running the insane asylum.

    1. Re:Operations? by Anonymous Coward · · Score: 1

      Dev Ops is responsible for the Great Lab Hack where I work.

      Short version: because some genius thought Dev Ops means that no one needs people with actual operations experience, we wound up with hundreds of old VMs that were spun up during some ancient sprint, then left online and never updated, never properly secured, and using weak passwords. (Some genius dev-op declared that all dev VMs for his team should use the same weak root password so every developer could use them.) Eventually someone found the lab through the firewall and was able to get remote SSH access to one of the servers that was using a weak password and, since nothing was ever updated, basically got complete control of every VM in there.

      If you thought this meant that anything changed, then you clearly don't work in modern IT. We're still doing dev ops, there's still no inventory of VMs and what they're running (beyond what the VM software itself provides - which doesn't include what they're being used for even though there's a description field that's never filled in), and there are still no requirements to properly secure servers and retire old VMs that are no longer in use. Instead the hack was blamed on IT and the firewall.

      Welcome to Dev Ops.

  13. Dev Ops by JustAnotherOldGuy · · Score: 4, Interesting

    Dev Ops is an example of the willing, led by the unknowing, doing the impossible for the ungrateful.

    Our Dev Ops team has adopted the slogan, "Delivering Yesterday's Technology Tomorrow!"

    --
    Just cruising through this digital world at 33 1/3 rpm...
    1. Re:Dev Ops by AndrewFlagg · · Score: 1

      nice one. i love it. Sunday Funnies at its best.

  14. Monitoring by Herkum01 · · Score: 4, Insightful

    I was our company's monitoring department and was checking systems and applications and it fits this question quite well, and you know what, IT SUCKS!

    Between management that does not give two f**ks, developers who don't understand infrastructure and systems administrators who cannot manage applications, no one wants to be on the hook for anything. Just TRYING to get them fix issues without pointing a finger is a nightmare. It is like being the IRS, you never get a call from them saying you did a good job.

    Everyone is afraid of looking bad because management, which does not understand IT or process, falls to politics to address issues and everyone else is afraid to make a move that make get them into trouble.

    DevOps and Agile crap, will not fix broken management.

  15. Jacks by Dog-Cow · · Score: 1

    DevOps typifies the phrase "jack-of-all-trades, master of none". This is a trend I've started to see where I work. A push to get everyone at least a little knowledgeable about everything, so anyone can fill in for anyone else. The idea that specialized skills allow a good team to be more efficient and productive is mostly lost on my coworkers.

    1. Re:Jacks by Hognoxious · · Score: 1

      There's nothing wrong with knowing a bit of someone else's job in addition to your own. If nothing else it aids understanding.

      The problem is when you have a dev who sort of knows a bit about system administration because he kept the lights on that time the actual sysadmin was away on a course. Then some PHB decides you don't need the sysadmin any more, because the dev can do it. And then something crops up that isn't in the "For Dummies" book, and the safety rope - interrupting the guy on the course - isn't attached to anything.

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
  16. Docker, Docker, Docker! by bill_mcgonigle · · Score: 1

    Here's the thread where we talk about developers bundling ancient versions of libraries with vulnerabilities in their Docker images, calling Ops people obstructionists, and then blaming them for security failures. Also, people who refused to use CI, introduce breakage, and then try to cast blame while someone else fixes it.

    Don't work for a tech company where the CEO isn't an engineer.

    --
    My God, it's Full of Source!
    OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
    1. Re: Docker, Docker, Docker! by reanjr · · Score: 1

      Yeah, I'd say Docker might one day be a useful tool for production, but it's still a ways off. And some other shiny thing will probably appear before that happens.

    2. Re: Docker, Docker, Docker! by Anonymous Coward · · Score: 0

      Docker has its uses, but is still all too often used as a cop-out, a way for lazy devs to avoid making their apps robust to real-world situations in a native OS and to avoid the forethought required to make an application work in anything other than the dev's pet Linux distro. Let me compile from source, and it better be init-system agnostic or GTFO (looking at you, systemd). I am sick of projects that give, as install instructions, "download this docker image which is completely undocumented and just pray it works ..."

      Where I work we use docker *only* as a way to get a repeatable environment for limited unit and integration testing. It isn't a bloody package system and it's about time we stopped abusing it as such.

  17. Non-hierarchical delegated responsibility by reanjr · · Score: 1

    The complicated interconnected nature of this sort of thing means issues land initially in one expert's lap, only to be delegated to another expert. The key is to be flexible and responsive to issues and not worry so much about who is responsible for what, but who is responsible now as an issue progresses. The hand-off of responsibility is more important than defining the responsibility.

  18. Turns out Fred Brooks was wrong by Anonymous Coward · · Score: 0

    I guess there *is* a silver bullet, it's called DevOps! What wondrous times we live in.

  19. Smart people tight teams make popular software tha by Invisible+Now · · Score: 1

    Non-coding managers will always be be prey to anyone or anything insulating them from the unacceptable truth that making software system sing is an art form more similar to making music than a production line. Find the right artists and keep the band together and sweet music will flow to your customers.

    --

    "Knowing everything doesn't help..."

  20. Re: Trump 2019! by Anonymous Coward · · Score: 0

    Trump is a PEACE president. He has already tried ended 3 wars - Korea (!!!), Syria, and Afghanistan.

    How did the Democrats become the pro-war party??

  21. You are the cause that it becomes shit. by DutchDopey · · Score: 1

    I so dislike this attitude of 'I am not going to change' instead of looking at the methods and technologies and see how they can be of use to you. Some might be great for your organisation, some not. But dismissing everything on forehand is just so stupid and the reason some organisations are stuck in old, unmaintainable and inefficient IT environments while the world moves on.

  22. DevOps is a joke (mostly) by Anonymous Coward · · Score: 0

    DevOps is a joke - or at least, the implementation of DevOps is a joke. The concept is a great idea. Unfortunately when it comes to how companies actually implement it, they don’t see it as a way to improve collaboration between Ops and Developers. They either:

    1) See it as a way to reduce headcount by combining two jobs into one (HR/Management viewpoint)
    2) Developers see it as a way to remove “obstructions” put into place by IT/Operations, and give themselves direct access to production systems.

    The majority of today’s DevOps are re-inventing the wheel and re-discovering bad ideas that more experienced system administrators discarded decades ago.

    Case in point: I attended a talk by the DevOps team from Wave Accounting about their deployment strategies. In the past, whenever Wave pushed a new update of their software to production, it would take literally hours to deploy, and the deploy process was very fragile. It turns out that they were coordinating the deploy from a single server, which would use a config management tool to log into each server, check out the source, download all the dependencies, and run the build - umpteen times for umpteen servers.

    The point of their talk was to showcase their brand new build system, which did away with all that and built the software locally, *once*, and then packaged the build and deployed it to each server. They thought this was the best thing since sliced bread of course, meanwhile all the experienced sysadmins in the room were rolling their eyes since *this* is how you deploy software (and control the dependency chain if you’re smart about it).

    But hey, why listen to experience? Let’s just do away with the sysadmins and let the devs run amok. What’s the worst that could happen? (I mean, besides npm?)