No More QA: Yahoo's Tech Leaders Say Engineers Are Better Off Coding With No Net (ieee.org)

Tekla Perry writes: A year ago Yahoo eliminated its test and quality assurance team, as part of project Warp Drive, its move to continuous delivery of code. The shift wasn't easy, Yahoo tech execs say, and required some "tough parenting." But the result has been fewer errors because "when you have humans everywhere, checking this, checking that, they add so much human error into the chain that, when you take them out, even if you fail sometimes, overall you are doing better." And the pain wasn't as great as expected. Yahoo's chief architect and SVP of science and technology discuss the transition.

Comments

Sounds like an MBA plan!

[Dunbal]

We were tired of being constantly bogged down by all these mistakes and bugs, so we got rid of the people who kept telling us about all the mistakes and bugs. Now our code is 100% mistake and bug free! Next step, get rid of our expensive experienced coders and replace them with cheaper outsourced coders with "equivalent" experience. We'll save so much money what could possibly go wrong? And the third and final phase of our plan is that in order to motivate our coders we will be paying a bonus that scales with the amount of code written. The more code you write, the better the bonus!

Don't you just love management?

Re: Sounds like an MBA plan!

[Junta]

While 100% was hyperbole, I think the point stands: if you remove QA then the number of *known* issues is going to go down. Whether your number of *actual* issues go down it's another matter.

I view this as the nightmare scenario, management hears words like automated testing as part of CI and then assumes automation has taken care of everything and they don't need the QA people. When the QA people's input into the issue tracker goes away, the numbers of issues in the graphs go down. Management gives themselves a pat on the back as a problematic metric has improved, without too much thought as to why.

Re:Sounds like an MBA plan!

[Junta]

The argument is actually pretty bad. By all means add more automated testing and hopefully make the QA process smoother, but skipping it entirely (they fired the QA team) is a recipe for disaster, and the only thing they have to measure the effectiveness of this is the number of issues in their issue tracker, which was *fed* by the QA team. It's like a company firing all their warranty people and bragging about how their warranty claims have gone to zero. The filed issues may be lower, but that does not speak to the final product.

Testers don't always follow proper testing procedure

This is a double edged sword in my experience. The testers are using the software much like a human would, and they do unexpected human things like the end user would be doing. This particularly helps identify usability issues, which isn't really an automation sort of thing. It's also a lot of fuzzy stuff that is usually not specifically laid out in the spec.

robustness towards matching the spec

Sometimes single minded march toward matching the spec is a bad idea, because the spec is bad and needs to be revisited. Sometimes this plays out in development, and sometimes this plays out in QA when humans clearly have a harder time with the interfaces than expected. Also generally speaking for me a spec is pretty high level, open to interpretation, and refined throughout the process as the details of the implementation get fleshed out.

Re: Sounds like an MBA plan!

[rgmoore]

It was up to me to make sure my own code was quality enough before releasing it, and that aspect terrified me enough that I did learn to write quality code (which basically means you are testing your own code thoroughly, doing your own QA).

The problem with testing your own code is that you're likely to miss entire classes of bugs. You can be very effective at catching the kinds of bugs you can think of, but those are always the easiest bugs to catch in the first place. The tests you write for yourself will never do a good job of catching errors in your assumptions about what the code should do, what kinds of inputs it needs to handle, etc. Catching those kinds of conceptual bugs really requires adversarial testing from somebody who isn't starting from the same set of assumptions.

Re: Sounds like an MBA plan!

[phantomfive]

Yeap. That's where pushing to live production is going to be your brutal tutor.....if you have cognitive biases that are hiding bugs from your psyche, then the system will show you to them (most likely in the most inconvenient time), and you will learn. It's harsh.

You can get the same experience (but with less harshness) by giving yourself the goal to never let a bug get past you to QA. Then QA will be a kinder tutor that helps you learn.

Re: Sounds like an MBA plan!

[phantomfive]

Another thing I've learned is that the stereotypical animosity between dev and QA is counterproductive.

Yes. QA is absolutely on your side!! Every time they find a bug for me, I thank them profusely, because better them than a customer.

Re: Sounds like an MBA plan!

[phantomfive]

Do you think that users' data security is the proper object of harsh lessons for coders?

If you are depending on QA to make sure user data is secure, you're going to fail. That's not what QA does, and hacking is rarely in their skillset.

When companies start caring about security, they usually start a separate team "Red Team" or something whose entire job is to try to hack and find security holes in the project. (I would argue this is not sufficient, because companies who have done this still fail. If you want secure software, it has to start from the beginning. You need to teach your programmers how to write good code, and security has to be a priority on every line of code that is written. Security can't be tacked on as an afterthought).

Do they know what QA is?

[Razed By TV]

If your QA people are adding to the problems, you are probably doing it wrong.

Money Quotes

[coop247]

Hilariously full of idiot speak:

"We forced excellence into the process"
"caused a paradigm shift in how engineers thought about problems"
"even if you fail sometimes, overall you are doing better"

Re: Oh, they re-invented Test Driven Development.

[JoeMerchant]

Where are my mod points? TDD is not the elimination of test and quality assurance, it's the procedualization / automation of it - along with driving it further forward in the product development cycle.

To do TDD, you need less (or no) QA people at the end, and more QA work in early development. If you choose to do this by firing the QA department, you probably are getting your product to market slower. If, instead, you transition the frustrated programmers who have been stuck in QA since graduating into design, implementation, and maintenance of automated tests for the project, you're probably winning big.

Re: Oh, they re-invented Test Driven Development.

[TWX]

I very much doubt that they were smart enough to begin the transition to TDD before they eliminated their QA department.

I worked quality assurance at a very-much test level without really delving into code. Programmers can be effin' stupid at times. I was testing daemons for communications protocols. I took four approaches. Verify that it does what it's supposed to do per RFC. Investigate the behavior when confronted with stale RFCs. Investigate the behavior when it's confronted with non-RFC input that is commonly available in end-user applications (ie, non-malicious incorrect use), and investigate the behavior when malicious intent is used. I can tell you that programmers working on these protocols wrote them to current-RFC only and that they were quite upset when testing obsolete commands from previous versions of RFC broke things, or when the third-party applications sent stuff to the daemons and broke them. I would have been understanding if the software had failed gracefully, with a warning or a 500 code or the like, but when sending obsolete POP3 commands to a POP3 daemon causes it to crash hard, or when encoding an attachment with Mac BinHex instead of MIME makes it crash, both of which were user-selectable options in then-popular Eudora (the client that the company itself used) then there's a big problem. I didn't even have to get to the level of malicious testing things were so bad.

The point of a good quality assurance tester is to concoct out-of-the-box but plausible tests to try to break it. It has a lot in common with Security; it requires a degree of white-hat maliciousness and a need to use other kinds of experiences from other disciplines to devise the scenarios to use. It also requires the tester to have good communications skills as they may have to work to convince the development team that what they've found actually is a problem and being too antisocial or too introverted might not get problems corrected even if they are important and are discovered.

Yahoo is doing a hell of a job in convincing me that they're really not worth paying attention to anymore. They are taking a position that is risky and is arguably excessively risky relative to the size and position of the company.