Slashdot Mirror

← Back to Stories (view on slashdot.org)

No More QA: Yahoo's Tech Leaders Say Engineers Are Better Off Coding With No Net (ieee.org)

Posted by timothy on from the who-ya-no-longer-gonna-call? dept.

Tekla Perry writes: A year ago Yahoo eliminated its test and quality assurance team, as part of project Warp Drive, its move to continuous delivery of code. The shift wasn't easy, Yahoo tech execs say, and required some "tough parenting." But the result has been fewer errors because "when you have humans everywhere, checking this, checking that, they add so much human error into the chain that, when you take them out, even if you fail sometimes, overall you are doing better." And the pain wasn't as great as expected. Yahoo's chief architect and SVP of science and technology discuss the transition.

34 of 216 comments (clear)

  1. Sounds like an MBA plan! by Dunbal · · Score: 5, Insightful

    We were tired of being constantly bogged down by all these mistakes and bugs, so we got rid of the people who kept telling us about all the mistakes and bugs. Now our code is 100% mistake and bug free! Next step, get rid of our expensive experienced coders and replace them with cheaper outsourced coders with "equivalent" experience. We'll save so much money what could possibly go wrong? And the third and final phase of our plan is that in order to motivate our coders we will be paying a bonus that scales with the amount of code written. The more code you write, the better the bonus!

    Don't you just love management?

    --
    Seven puppies were harmed during the making of this post.
    1. Re: Sounds like an MBA plan! by bill_mcgonigle · · Score: 2, Interesting

      Did you get through TFS? The claim is that overall errors have been reduced but not eliminated.

      He could be lying, but if that's the counterclaim, then show some evidence for it.

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
    2. Re: Sounds like an MBA plan! by Junta · · Score: 5, Interesting

      While 100% was hyperbole, I think the point stands: if you remove QA then the number of *known* issues is going to go down. Whether your number of *actual* issues go down it's another matter.

      I view this as the nightmare scenario, management hears words like automated testing as part of CI and then assumes automation has taken care of everything and they don't need the QA people. When the QA people's input into the issue tracker goes away, the numbers of issues in the graphs go down. Management gives themselves a pat on the back as a problematic metric has improved, without too much thought as to why.

      --
      XML is like violence. If it doesn't solve the problem, use more.
    3. Re:Sounds like an MBA plan! by Earthquake+Retrofit · · Score: 2

      A handful of ordinary dumb users and a smart sixteen-year-old with authority issues should be able to find enough bugs to keep the QA department busy for quite a while.

      --
      Fifty years of Yippie! 1968-2018
    4. Re:Sounds like an MBA plan! by Lennie · · Score: 2

      The summary sucks, read the article:

      They got rid of the QA team and made the programmers create more unit-tests (this is key).

      Because the computers made less mistakes than the QA team.

      --
      New things are always on the horizon
    5. Re:Sounds like an MBA plan! by Junta · · Score: 4, Insightful

      The argument is actually pretty bad. By all means add more automated testing and hopefully make the QA process smoother, but skipping it entirely (they fired the QA team) is a recipe for disaster, and the only thing they have to measure the effectiveness of this is the number of issues in their issue tracker, which was *fed* by the QA team. It's like a company firing all their warranty people and bragging about how their warranty claims have gone to zero. The filed issues may be lower, but that does not speak to the final product.

      Testers don't always follow proper testing procedure

      This is a double edged sword in my experience. The testers are using the software much like a human would, and they do unexpected human things like the end user would be doing. This particularly helps identify usability issues, which isn't really an automation sort of thing. It's also a lot of fuzzy stuff that is usually not specifically laid out in the spec.

      robustness towards matching the spec

      Sometimes single minded march toward matching the spec is a bad idea, because the spec is bad and needs to be revisited. Sometimes this plays out in development, and sometimes this plays out in QA when humans clearly have a harder time with the interfaces than expected. Also generally speaking for me a spec is pretty high level, open to interpretation, and refined throughout the process as the details of the implementation get fleshed out.

      --
      XML is like violence. If it doesn't solve the problem, use more.
    6. Re: Sounds like an MBA plan! by Anonymous Coward · · Score: 2, Interesting

      He could be lying, but if that's the counterclaim, then show some evidence for it.

      I hold in my hand a bag containing a small, living, pink unicorn.
      What do you say? I'm lying? Well, if that's the counterclaim, then show some evidence for it.
      What do you say? You can't look in the bag? Well, my point exactly.

    7. Re:Sounds like an MBA plan! by naasking · · Score: 2

      By all means add more automated testing and hopefully make the QA process smoother, but skipping it entirely (they fired the QA team) is a recipe for disaster, and the only thing they have to measure the effectiveness of this is the number of issues in their issue tracker, which was *fed* by the QA tea.

      Which can be replaced by an easy issue submission process embedded in the program itself. Combined with a program trace of the user's actions, which Yahoo already uses as part of its distributed architecture, the developer would be able to reproduce precisely any behaviour seen.

      The testers are using the software much like a human would, and they do unexpected human things like the end user would be doing. This particularly helps identify usability issues, which isn't really an automation sort of thing.

      But you can automate entry of unexpected data too, without having to generate the data/tests yourself. See QuickCheck and other testing tools inspired by it, which invokes functions with a large set of permutations over the full domain of its parameters, starting with upper and lower bounds, ie. if your function accepts an int, it will invoke it thousands of times int.MaxValue, int.MinValue, and a number of randomly selected values between.

      Sometimes single minded march toward matching the spec is a bad idea, because the spec is bad and needs to be revisited.

      Sure, but you can't fix spec problems in the testing phase with a QA team that aren't domain experts. Only end users can validate something like this, which is where shorter feedback cycles with end users are key delivering a good result.

      Re: humans having a hard time using an interface, this is generally obvious to the developers too while they're writing and testing their own code. They generally just don't have a process available via which they can question the viability of the spec and be taken seriously.

      --
      Higher Logics: where programming meets science.
    8. Re:Sounds like an MBA plan! by evilviper · · Score: 2

      They got rid of the QA team and made the programmers create more unit-tests (this is key).

      Of course that's not really true, either...

      From the QA team: "Some started working on automation [for testing], and they thought that was great—that they didn't have to do the same thing over and over."

      So really all they did was move their good QA folks to different departments, where they are now writing QA tests, but just don't get called QA engineers anymore, so Yahoo can get PR with nonsense stories like this one claiming there's "no more QA".

      Yes, they automated QA testing as part of this, though it sounds like they did so in the most backwards and painful way possible, which means they necessarily reduced the size of the QA team in the process. But none of that would be news if they didn't lie about it, would it? I'm sure many other companies (like Amazon) had this model of fully-automated QA tests working for many YEARS before Yahoo thought-up the idea.

      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
    9. Re:Sounds like an MBA plan! by lucm · · Score: 2

      You forget the team lead: an older woman who doesn't understand the difference between click and double-click and has to do all her testing with one of those laptops that has a virtual scroll bar on the trackpad.

      --
      lucm, indeed.
    10. Re: Sounds like an MBA plan! by rgmoore · · Score: 4, Insightful

      It was up to me to make sure my own code was quality enough before releasing it, and that aspect terrified me enough that I did learn to write quality code (which basically means you are testing your own code thoroughly, doing your own QA).

      The problem with testing your own code is that you're likely to miss entire classes of bugs. You can be very effective at catching the kinds of bugs you can think of, but those are always the easiest bugs to catch in the first place. The tests you write for yourself will never do a good job of catching errors in your assumptions about what the code should do, what kinds of inputs it needs to handle, etc. Catching those kinds of conceptual bugs really requires adversarial testing from somebody who isn't starting from the same set of assumptions.

      --

      There's no point in questioning authority if you aren't going to listen to the answers.

    11. Re: Sounds like an MBA plan! by Anonymous Coward · · Score: 2, Informative

      The poster was close, when referring to the actual word "Mistakes" as opposed to its meaning, then the correct word was indeed "is".

      So to make it perfectly clear, quotes should have been used as such:

      "Mistakes" is a countable noun not a mass noun.

      ---
      Not APK

    12. Re: Sounds like an MBA plan! by sunderland56 · · Score: 3, Interesting

      The claim is that overall errors have been reduced but not eliminated.

      But with zero testing, how would they know?

    13. Re: Sounds like an MBA plan! by phantomfive · · Score: 4, Insightful

      Yeap. That's where pushing to live production is going to be your brutal tutor.....if you have cognitive biases that are hiding bugs from your psyche, then the system will show you to them (most likely in the most inconvenient time), and you will learn. It's harsh.

      You can get the same experience (but with less harshness) by giving yourself the goal to never let a bug get past you to QA. Then QA will be a kinder tutor that helps you learn.

      --
      "First they came for the slanderers and i said nothing."
    14. Re: Sounds like an MBA plan! by NormalVisual · · Score: 3, Insightful

      I can give a counter-idea: in my first job, I didn't have QA. It was up to me to make sure my own code was quality enough before releasing it, and that aspect terrified me enough that I did learn to write quality code (which basically means you are testing your own code thoroughly, doing your own QA).

      I think it's always necessary to test your own code via unit tests and whatnot prior to letting someone else have at it, but I think it's also important to have that second set of eyes on it from a behavioral standpoint. Often we're too close to the code to really beat it up properly and subject it to really weird edge cases, and good QA people are quite adept at doing that. What I've found works pretty well is having the code go through QA, but still being held to some kind of personal accountability for it as well. Even if it's nothing more than your co-workers laughing at your QA bug reports and saying your code sucks, it's some more motivation to get it right the first time, and if it passes QA you have at least some degree of confidence that it works properly.

      Another thing I've learned is that the stereotypical animosity between dev and QA is counterproductive. It's much better to think of them as two parts of a team working toward the same goal, and it's beneficial to foster an environment where the devs actively encourage QA to break stuff instead of trying to blow them off all the time.

      --
      Please stand clear of the doors, por favor mantenganse alejado de las puertas
    15. Re: Sounds like an MBA plan! by phantomfive · · Score: 4, Insightful

      Another thing I've learned is that the stereotypical animosity between dev and QA is counterproductive.

      Yes. QA is absolutely on your side!! Every time they find a bug for me, I thank them profusely, because better them than a customer.

      --
      "First they came for the slanderers and i said nothing."
    16. Re: Sounds like an MBA plan! by Anonymous Coward · · Score: 2, Insightful

      Call volume is unusually high. You are number [37] in line. Please be patient and listen to the pleasant music for approximately [109] minutes. [buzz buzz brrrrff buzz ggggzhxx buzz buzz ...]

    17. Re: Sounds like an MBA plan! by phantomfive · · Score: 5, Insightful

      Do you think that users' data security is the proper object of harsh lessons for coders?

      If you are depending on QA to make sure user data is secure, you're going to fail. That's not what QA does, and hacking is rarely in their skillset.

      When companies start caring about security, they usually start a separate team "Red Team" or something whose entire job is to try to hack and find security holes in the project. (I would argue this is not sufficient, because companies who have done this still fail. If you want secure software, it has to start from the beginning. You need to teach your programmers how to write good code, and security has to be a priority on every line of code that is written. Security can't be tacked on as an afterthought).

      --
      "First they came for the slanderers and i said nothing."
    18. Re: Sounds like an MBA plan! by gweihir · · Score: 2

      "Number of errors" is a meaningless metric. You need to take severity, cost of fixing, cost of finding, and damage done into account. But MBA bean-counters cannot do that, as they do not understand anything they are "managing".

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    19. Re: Sounds like an MBA plan! by chaboud · · Score: 2

      Keep in mind that the number of reported bugs in a given piece of software is proportional to the probability of a bug occurring, the likelihood of a user being sufficiently competent to recognize it as a bug, and the amount of usage of the given piece of software.

      The last two figures in that trio have been dropping off precipitously for Yahoo for quite some time.

      The problem with dropping QA (as if there is only one) is that you lose the adversarial position of quality. The tension of a QA org, refusing to sign off on shit software, helps that software be better, just as the tension of TPM's, riding the org to hit a schedule, helps the software value scheduling constraints. Giving people causes to champion is an extremely useful way to keep an organization from developing blind spots. A bunch of devs, together, can easily become dogmatic and over index on one aspect of their work.

      Domain expert QA also allows developers to solve problems that would have been incomprehensibly out of reach by providing conversational user perspective.

      I'm a developer, so a visit from QA might mean something landing on my desk with a thud. Still, skipping QA altogether is going to lead to more developer hours spent on work that would have been covered by (sadly, much less costly) QAE's and more blinds spots, in code and organizational operation.

      Love your QA. They are your safety net, fresh eyes, user advocate, exhaustive pounder, field-bug triage, and fire marshal.

      Besides, anyone taking advice from Yahoo's tech leadership at this point is woefully out of touch.

    20. Re:Sounds like an MBA plan! by naasking · · Score: 2

      Yes, just make your users your testers, great idea.

      User acceptance testing is the only meaningful testing of this sort. There's no way around it really. Black box testers don't have the domain knowledge that end users have, nearly any usability feedback is practically meaningless. How could a blackbox tester possibly really know whether a particular screen layout, or information organization, would make sense to an end user?

      Developers are frequently not very much in tune with their target end user perspective. Hence why I see my fellow developers get frustrated with testers and even clients, wondering why the users can't do things the way the developer thinks they should do.

      If users are confused, it's an impedance mismatch between what a developer thinks the user knows, and what the user actually knows. Which is a spec failure. The developer can't divine what a hypothetical user would know at any given moment by magic. How could a QA team know this information? And if they somehow know, why not just communicate this to the dev in the spec in the first place?

      --
      Higher Logics: where programming meets science.
    21. Re: Sounds like an MBA plan! by phantomfive · · Score: 2

      SDETs can do unit and white box testing, security testing, performance testing, etc. but they cost as much as a developer, since that's what they really are.

      An SDET who is good at hacking will cost more than a developer.

      --
      "First they came for the slanderers and i said nothing."
    22. Re: Sounds like an MBA plan! by stephanruby · · Score: 2

      He could be lying, but if that's the counterclaim, then show some evidence for it.

      You're missing the OP's original point. It's difficult to prove anything if you've stopped measuring things.

      May be it's just me because I'm not a great developer, but I work that much harder to find bugs within my programs because I know that QA will always try harder than me to find any bugs that I let through.

      Furthermore, even if the number of bugs stays constant even without QA (which I very much doubt), without QA those bugs are going to be found many days if not weeks later than usual. And as a developer, I am so much quicker at fixing a bug I introduced 30 minutes ago or a couple of hours ago, than many days ago. After all, I am only human, and it takes time for me to re-immerse myself into a particular piece of code if it's not already fresh in my mind.

      Also, Yahoo makes it sound as if they took the harder approach and now it's paying off for them. Not at all, laying off QA is the easier approach. It's just like stopping regression-testing. It's infinitely easier to do that, at least initially. QA people are not easy to deal with. Failing tests are not easy to deal with either. When confronted with either QA people or failing tests, there is also a sense of urgency about having to drop whatever you're doing and dealing with the problem at hand. Removing those QA people and removing those failing tests may give you a sense of reprieve, but it is certainly not going to help fix any of the underlying problems. If anything, it's just like an ostrich burying its head in the sand when it sees a predator coming.

  2. Do they know what QA is? by Razed+By+TV · · Score: 5, Insightful

    If your QA people are adding to the problems, you are probably doing it wrong.

    1. Re:Do they know what QA is? by Teancum · · Score: 2

      I couldn't agree more. It sounds like they were more interested in the QA process than actually having people as a part of the overall software development team that try to break code and find flaws.

      There are multiple ways to accomplish the goal of finding bugs, where the worst place to have them reported is by customers. Is that what Yahoo is hoping for now?

  3. Oh, they re-invented Test Driven Development. by mikelieman · · Score: 3, Insightful

    >It has also, he said, forced engineers to develop tools to automate the kinds of checks previously handled by teams of humans. An engineer might go through an arduous process of checking code once—but then would start building tools to automate that process.

    I suspect they're too dumb to realize this and just tell everyone, "We're saving money and delivering better code using TDD"

    --
    Technology -- No Place For Wimps! Grateful Dead and Jerry Garcia Chatroom -- http://www.wemissjerry.org
    1. Re: Oh, they re-invented Test Driven Development. by JoeMerchant · · Score: 5, Interesting

      Where are my mod points? TDD is not the elimination of test and quality assurance, it's the procedualization / automation of it - along with driving it further forward in the product development cycle.

      To do TDD, you need less (or no) QA people at the end, and more QA work in early development. If you choose to do this by firing the QA department, you probably are getting your product to market slower. If, instead, you transition the frustrated programmers who have been stuck in QA since graduating into design, implementation, and maintenance of automated tests for the project, you're probably winning big.

    2. Re: Oh, they re-invented Test Driven Development. by TWX · · Score: 4, Interesting

      I very much doubt that they were smart enough to begin the transition to TDD before they eliminated their QA department.

      I worked quality assurance at a very-much test level without really delving into code. Programmers can be effin' stupid at times. I was testing daemons for communications protocols. I took four approaches. Verify that it does what it's supposed to do per RFC. Investigate the behavior when confronted with stale RFCs. Investigate the behavior when it's confronted with non-RFC input that is commonly available in end-user applications (ie, non-malicious incorrect use), and investigate the behavior when malicious intent is used. I can tell you that programmers working on these protocols wrote them to current-RFC only and that they were quite upset when testing obsolete commands from previous versions of RFC broke things, or when the third-party applications sent stuff to the daemons and broke them. I would have been understanding if the software had failed gracefully, with a warning or a 500 code or the like, but when sending obsolete POP3 commands to a POP3 daemon causes it to crash hard, or when encoding an attachment with Mac BinHex instead of MIME makes it crash, both of which were user-selectable options in then-popular Eudora (the client that the company itself used) then there's a big problem. I didn't even have to get to the level of malicious testing things were so bad.

      The point of a good quality assurance tester is to concoct out-of-the-box but plausible tests to try to break it. It has a lot in common with Security; it requires a degree of white-hat maliciousness and a need to use other kinds of experiences from other disciplines to devise the scenarios to use. It also requires the tester to have good communications skills as they may have to work to convince the development team that what they've found actually is a problem and being too antisocial or too introverted might not get problems corrected even if they are important and are discovered.

      Yahoo is doing a hell of a job in convincing me that they're really not worth paying attention to anymore. They are taking a position that is risky and is arguably excessively risky relative to the size and position of the company.

      --
      Do not look into laser with remaining eye.
    3. Re: Oh, they re-invented Test Driven Development. by lucm · · Score: 2

      To do TDD, you need less (or no) QA people at the end, and more QA work in early development.

      Wrong. You need less QA people at the end and more *BA* work in early development. Those are two totally different things.

      --
      lucm, indeed.
  4. Money Quotes by coop247 · · Score: 5, Funny

    Hilariously full of idiot speak:

    "We forced excellence into the process"
    "caused a paradigm shift in how engineers thought about problems"
    "even if you fail sometimes, overall you are doing better"

    --
    //TODO: Insert catchy phrase
  5. Re:Awesome Idea! by TWX · · Score: 3, Insightful

    They still do a lot of end-stage testing though. Pressure testing, structural testing, failure-mode testing, load testing, load-shift testing, strike testing, and probably all other manner of tests. They do perform a lot of tests as they develop but that's no replacement for the final-stage testing that confirms predicted behavior or verifies that systems integrate as expected.

    To pretend that there's no end-stage testing is simply ridiculous.

    --
    Do not look into laser with remaining eye.
  6. Re:You can't make this stuff up by sunderland56 · · Score: 3, Funny

    A year ago Yahoo eliminated its test and quality assurance team

    Several years ago Yahoo eliminated it's customer base. That had a larger impact on their net worth than QA.

  7. Re: customer-reported problems by FlaSheridn · · Score: 2

    Management takes customer-reported problems very seriously.

    And how seriously do they take ex-customers not reporting problems because they gave up in disgust instead?

  8. We did this years ago! by Tony+Isaac · · Score: 2

    Back in the 80s, we wrote code and threw it over the fence to the users. If nobody complained, we assumed it was working fine! Ah, the good old days. What's old is new again!