Slashdot Mirror
Book Review: How Google Tests Software
MassDosage writes "Having developed software for nearly fifteen years, I remember the dark days before testing was all the rage and the large number of bugs that had to be arduously found and fixed manually. The next step was nervously releasing the code without the safety net of a test bed and having no idea if one had introduced regressions or new bugs. When I first came across unit testing I ardently embraced it and am a huge fan of testing of various forms — from automated to smoke tests to performance and load tests to end user and exploratory testing. So it was with much enthusiasm that I picked up How Google Tests Software — written by some of the big names in testing at Google. I was hoping it would give me fresh insights into testing software at "Google Scale" as promised on the back cover, hopefully coupled with some innovative new techniques and tips. While partially succeeding on these fronts, the book as a whole didn't quite live up to my expectations and feels like a missed opportunity." Read below for the rest of MassDosage's review.
How Google Tests Software
author
James Whittaker, Jason Arbon, Jeff Carollo
pages
281
publisher
Addison Wesley
rating
6/10
reviewer
Mass Dosage
ISBN
978-0321803023
summary
Testing at Google scale
The book is written in an informal, easy to read manner and organized in such a way that readers can read chapters in any order or just choose to focus on the parts that interest them. One annoying layout choice is to highlight and repeat certain key sentences (as is often done in magazines) resulting in one reading the same thing twice, often only words away from the original sentence. Thankfully this is only the case in the first two chapters, but it highlights the variable quality of this book — possibly due to the authors having worked separately on different chapters. How Google Tests Software isn't a book for people new to testing or software development. The authors assume you know a fair amount about the software development lifecycle, where testing fits into this and what different forms testing can take. It is also largely technology neutral, using specific examples of testing software that Google uses only to illustrate concepts.
After a brief introduction as to how testing has evolved over time at Google the book devotes a chapter to each of the key testing-related roles in the company: the 'Software Engineer in Test' (SET), the 'Test Engineer' (TE) and the 'Test Engineering Manager' (TEM). SETs are coders who focus on writing tests or frameworks and infrastructure to support other coders in their testing. The TE has a broader, less well-defined role and is tasked with looking at the bigger picture of the product in question and its impact on users and how it fits into the broader software ecosystem. These two sections form the bulk of the book in terms of pages and interesting content. The TEM is essentially what the name says — someone who manages testers and testing and coordinates these activities at a higher level within Google.
The descriptions of each of these testing roles highlights the ways Google's thinking about testing has matured and also shows how some of these approaches differ from other companies. There are also explanations of the tools and processes that people in these roles use and follow and this for me was the most interesting part of the book. Topics covered include: specific bug tracking and test plan creation tools; risk analysis; test case management over time; and automated testing. Particularly of note are discussions on using bots to perform testing of web pages to detect differences between software releases, cutting down on the amount of human interaction required as well as the opposite approach — using more humans via "crowd sourced testing" among first internal and then select groups of external users. The tools that Google utilizes to simplify tester's jobs by recording steps to reproduce bugs and simplifying bug reporting and management sound very useful. Many of the tools described in the book are open source (or soon to be opened) and are probably worth following up on and investigating if this is what you do for a living.
In addition to the main body of text most chapters also include interviews with Google staff on various testing related topics. Some of these are genuinely interesting and give the reader a good idea of how testing is tackled at Google on a practical level. However some of the interviews fall into the "navel gazing" camp (especially when the authors interview one of themselves) and feel more like filler material. I enjoyed the interviews with Google hiring staff the most — their take on how they recruit people for testing roles and the types of questions they ask and qualities they look for make a lot of sense. The interview with the GMail TEM was also good and illustrated how the concepts described in the book are actually performed in practise. The interviews are clearly marked and can thus be easily skipped or skim read but one wonders what more useful text could have been included in their place.
The book wraps up with a chapter that attempts to describe how Google intends to improve their testing in the future. The most valuable point here is how testing as a separate function could "disappear" as it becomes part and parcel of the product being developed like any other feature, and thus the responsibility of all of the people working on the product as opposed to it being a separate thing. Another key point made throughout the book is how the state of testing at Google is constantly in flux which makes sense in such a fast moving and innovative company but leaves one questioning how much of this book will still be relevant in a few year's time.
How Google Tests Software isn't a bad book but neither is it a great one. It has some good parts and will be worth reading for those who are interested in "all things Google." For everyone else I'd recommend skimming through to the parts that grab your attention most and glossing over the rest.
You can purchase How Google Tests Software from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
After a brief introduction as to how testing has evolved over time at Google the book devotes a chapter to each of the key testing-related roles in the company: the 'Software Engineer in Test' (SET), the 'Test Engineer' (TE) and the 'Test Engineering Manager' (TEM). SETs are coders who focus on writing tests or frameworks and infrastructure to support other coders in their testing. The TE has a broader, less well-defined role and is tasked with looking at the bigger picture of the product in question and its impact on users and how it fits into the broader software ecosystem. These two sections form the bulk of the book in terms of pages and interesting content. The TEM is essentially what the name says — someone who manages testers and testing and coordinates these activities at a higher level within Google.
The descriptions of each of these testing roles highlights the ways Google's thinking about testing has matured and also shows how some of these approaches differ from other companies. There are also explanations of the tools and processes that people in these roles use and follow and this for me was the most interesting part of the book. Topics covered include: specific bug tracking and test plan creation tools; risk analysis; test case management over time; and automated testing. Particularly of note are discussions on using bots to perform testing of web pages to detect differences between software releases, cutting down on the amount of human interaction required as well as the opposite approach — using more humans via "crowd sourced testing" among first internal and then select groups of external users. The tools that Google utilizes to simplify tester's jobs by recording steps to reproduce bugs and simplifying bug reporting and management sound very useful. Many of the tools described in the book are open source (or soon to be opened) and are probably worth following up on and investigating if this is what you do for a living.
In addition to the main body of text most chapters also include interviews with Google staff on various testing related topics. Some of these are genuinely interesting and give the reader a good idea of how testing is tackled at Google on a practical level. However some of the interviews fall into the "navel gazing" camp (especially when the authors interview one of themselves) and feel more like filler material. I enjoyed the interviews with Google hiring staff the most — their take on how they recruit people for testing roles and the types of questions they ask and qualities they look for make a lot of sense. The interview with the GMail TEM was also good and illustrated how the concepts described in the book are actually performed in practise. The interviews are clearly marked and can thus be easily skipped or skim read but one wonders what more useful text could have been included in their place.
The book wraps up with a chapter that attempts to describe how Google intends to improve their testing in the future. The most valuable point here is how testing as a separate function could "disappear" as it becomes part and parcel of the product being developed like any other feature, and thus the responsibility of all of the people working on the product as opposed to it being a separate thing. Another key point made throughout the book is how the state of testing at Google is constantly in flux which makes sense in such a fast moving and innovative company but leaves one questioning how much of this book will still be relevant in a few year's time.
How Google Tests Software isn't a bad book but neither is it a great one. It has some good parts and will be worth reading for those who are interested in "all things Google." For everyone else I'd recommend skimming through to the parts that grab your attention most and glossing over the rest.
You can purchase How Google Tests Software from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Nearly four months ago, I noticed that my internet connection was very sluggish. Eventually getting fed up with it, I began to seek out software that would speed up the gigabits in my router. After an hour of searching, I found what at first appeared to be a very promising piece of software. Not only did it claim it would speed up my internet connection, but that it would overclock my power supply, speed up my gigabits, and remove any viruses from my computer! "This is a fantastic opportunity that I simply can't pass up," I thought. I immediately downloaded the software and began the installation, all the while laughing like a small child. I was highly anticipating a future where the speed of my internet connection would leave everyone else's in the dust.
I was horribly, horribly naive. Immediately upon the completion of the software's installation, various messages popped up on my screen about how I needed to buy software to remove a virus that I wasn't aware I had from a software company I'd never once heard of. The strange software also blocked me from doing anything except buying the software it was advertising. Being that I was a computer whiz (I had taken a computer essentials class in high school that taught me how to use Microsoft Office, and was quite adept at accessing my Facebook account), I was immediately able to conclude that the software I'd downloaded was, in fact, a virus, and that it was slowing down my gigabits at an exponential rate. "I can't let this insanity proceed any further," I thought.
As I was often called a computer genius, I was confident at the time that I could get rid of the virus with my own two hands. I tried numerous things: restarting the computer, pressing random keys on the keyboard, throwing the mouse across the room, and even flipping an orange switch on the back of the tower and turning the computer back on. My efforts were all in vain; the virus persisted, and my gigabits were running slower than ever! "This cannot be! What is this!? I've never once seen such a vicious virus in my entire life!" I was dumbfounded that I, a computer genius, was unable to remove the virus using the methods I described. Upon coming to terms with my failure, I decided to take my computer to a PC repair shop for repair.
I drove to a nearby computer repair shop and entered the building with my computer in hand. The inside of the building was quite large, neat, and organized, and the employees all seemed very kind and knowledgeable. They laughed upon hearing my embarrassing story, and told me that they saw this kind of thing on a daily basis. They then accepted the job, and told me that in the worst case, it'd be fixed in three days from now. I left with a smile, and felt confident in my decision to leave the computer repairs to the experts.
A week later, they still hadn't called back. Visibly angry, I tried calling them countless times, but not a single time did they answer the phone. Their negligence and irresponsibility infuriated me, and sent me into a state of insanity that caused me to punch a gigantic hole in the wall. Being that I would require my computer for work soon, I decided to head over to the computer repair shop to find out exactly what the problem was.
Upon entering the building, I was shocked by the state of its interior; it looked as if a tornado had tore through the entire building! Countless broken computers were scattered all about the floor, desks were flipped over, the walls had holes in them, there was a puddle of blood on the floor, and worst of all, I saw that my computer was sitting in the middle of the room laying on its side! Absolutely unforgivable! I soon noticed one of the employees sitting behind one of the tipped over desks (the one that had previously had the cash register on top of it); he was shaking uncontrollably and sobbing. Despite being furious about my computer being tipped over, seeing him in that state still managed to make me less unforgiving. I decided to ask him what happened.
A few moments passed where the entire r
I thought they call it "beta", release it and let the users find the bugs.
My friend! We meet again!
Oh dear.
What a shame.
Truly a shame indeed.
Remember the promise you made to me?
You broke it.
You shattered my expectations in you.
You're dead to me.
You promised!
You promised you'd return!
You promised, and you broke that promise!
You are worthless!
A worthless pile of trash!
You promised you'd return... to Gamemakerdom!
Why haven't you returned!?
Friend, you're embarrassing me with your Gamemakerlessness.
I shan't with a Gamemakerlessness such as you.
I won't let myself be seen with you.
Friend, you're an eyesore.
A mere eyesore that should vanish.
Eyesore, eyesore, eyesore.
I don't know you.
You're a worthless, monstrous being that isn't even fit to be called a human.
You're... a Gamemakerlessness!
Get away from me, you monster!
You're nothing!
You will always be nothing!
You always were nothing.
After all, any sane individual would be using Gamemaker.
Yes. There is nothing Gamemaker cannot do.
Gamemaker's the best.
Return.
Return.
Return.
Return.
Return.
I command you: return!
Return... to Gamemakerdom, or turn to dust and die!
You can return.
You may return.
You will return.
You *shall* return.
Return... to Gamemakerdom!
Having developed software for nearly fifteen years, I remember the dark days before testing was all the rage
Umm.. wtf are you talking about? Extreme Programming is 13 years old, and it wasn't first. Even the waterfall model has testing, and it's 40 years old:
1. Requirements specification
2. Design
3. Construction (AKA implementation or coding)
4. Integration
5. Testing and debugging (AKA Validation)
6. Installation
7. Maintenance
Just because you didn't know how to test your software back then doesn't mean testing didn't exist.
I am interested in your poetry and lyrics. These are very insightful and deep. Please contact me so we can look into opportunities to monetize them. Regards!
I see...so they get rid of bugs by boring them to death with explanations of their bureaucratic structure, and threatening to add additional layers of management!
Shit, if I was a bug, I'd leave the affected program voluntarily, just to avoid that TPS-report-fest in the making and give the lower employees time to breathe.
You can hold down the "B" button for continuous firing.
Is there anything on the (automated) verification techniques used at Google? Microsoft has one of the best verification research labs around and they develop a handful of great tools that are freely available. Is there a Google analogue?
I am interested in your poetry and lyrics. These are very insightful and deep. Please contact me so we can look into opportunities to monetize them. Naturally, I will have to charge you for my services. Would 50% be agreeable to you? Thank you and I look forward to our prosperous business relationship!
The 80's really sucked when I worked on a Unix kernel. We had unit tests, integration tests, system tests, stress tests, performance tests, compatibility tests (AT&T, BSD, SunOS, DB, major apps, Orange Book/security tests, various CPUs & devices, with builds from both a commercial and gnu compilers), and others.
In addition to working on the kernel, I managed our testing. I had to manually start the tests each morning (after the automatic nightly builds that took 10 hours). Then I had to manually start emacs toward the end of the day and load the result files (which were fortunately analyzed in lisp) rather than looking at a desktop widget, then manually send an email to anyone who caused a problem.
And to make matters worse (as if it can get worse than 10-20 minutes of my time a day) I didn't have lots of people raving about my cool test setup (they all thought it was just a standard and trivial part of software development)
And don't make me go off on the pain of alpha and beta tests. I had to email an ftp location to our major customers using !-notation.
Before Kinect: "I just got a new first person shooter game. Want to come over and play?"
person who agrees is a guy
After Kinect: "I just got the Xbox 360 Kinect thingy with this awesome dance game. You should totally come over to my house and dance around in my living room."
person who agrees is a girl who will bring her girl friends and bounce up and down.
Kinect FTW
Of course, I haven't read more than the above summary, but the lack of mention of user research and user testing sends up all sorts of red flags, of the same hue I've seen w/r/t Google products for years.
If you don't think about UX, think about it this way. Ever read the legal agreements that come with, say, credit card applications? Pages upon pages of legalese, right? See, to _them_, it all makes sense. But they don't test it against users (who are usually not credit lawyers). This is, in a way, the core of UX: You, the creator/coder/programmer are _not_ the user. You know how it works. UX helps you design stuff so that users can use it without knowing it inside-out from the word go.
Even if I didn't know how manager- and engineer-centric the Google product design process was, with researchers and user designers set way outside the inner circle and floating around from project to project, it'd be evident by a quick look at most Google products. Or their documentation, which reads one step better than, "well, WE know how it works, so why don't you?"
I know, you reading this now, you're smart, you've never had a problem, and if you did you could dig through Stack Overflow and suss it out. And I know, Apple products are for babies who want to have their hands held whilst locked in a crystal garden. You're better than the rest of us are. I acknowledge that.
But the point is: UX helps you create things that solve the problems people actually have, and do so easily, without having to spend their pitifully scare mental efforts on finding buttons, or links, or settings, etc. It does require exposing your prototypes to people not already on the project, or in your company (see the spectacular failure of Buzz; people inside Google are used to all their info and contacts being exposed to everyone, but some people in the real world react poorly to that), and listening to plebes. Because they are the users. They may want to use your product for their own work, and not work on your product.
Again, this might be addressed in the book. Though it doesn't seem to be central enough to Google's concept of developing products to rate a high-level mention. And that -- that's the problem.
release it for public use, put "Beta" in the logo, monitor the complaints.
Google is a successful company, but this isn't so much about their testing practices. Or hiring practices, or development practices, work culture or anything like that. Simply that they cornered the search market.
Anyone can have a successful company if you have a product good enough or different enough to dominate a niche.
Maybe the let the user find bugs.
womens sexy lingerie |
I am interested in your ideas. These are very insightful and deep. Please contact me so we can look into opportunities to monetize them. Naturally, I will have to charge you for my services. Would 50% be agreeable to you? Thank you and I look forward to our prosperous business relationship!