Slashdot Mirror
Quality Assurance In The Games Industry
Thanks to NTSC-uk for their opinion piece discussing the perceived lack of 'quality assurance' in the videogame industry. Amid oft-repeated claims that "many games fall short of the mark" on overall quality, there are some more interesting arguments that QA testing "rarely promotes the criticism and fine-tuning of the most important aspect of design - gameplay." The author even goes on to suggest that hardware manufacturers should again get more involved in the quality of games on their machine: "Nintendo demonstrated during the 80s and early 90s how the power of the manufacturer can be used... to ensure that the design of new games, and particularly good gameplay, was top of the agenda - hence Nintendo's 'Seal of Quality'."
There are still a lot of console games released while suffering from major clipping and control issues, so some QA from console manufacturers would definitely be A Good Thing. Gameplay and content, on the other hand, are very subjective things, and things that console makers might be better off not dabbling in. Remember the santised RPG translations that appeared under the Seal of Quality in the NES days? I would hate to fire up FF:Crystal Chronicles and find Holy replaced with White, or bars replaced with cafes, as per the bad old days.
Vino, gyno, and techno -Bruce Sterling
I was always under the impression that the Nintendo Seal of Quality was pure BS. Back when I was a kid, I suggested to one of my friends (an avid reader of Nintendo Power) that he buy Afterburner for his NES. (Yes, there was a NES Afterburner cartridge, as far as I can remember :). At any rate, he says to me something to the point of, "But it doesn't have the Nintendo Seal of Quality."
Well of course it doesn't. Sega made it, and at the time, they were Nintendo's biggest competition. I had no doubts about the quality of Afterburner as a game, at least in comparison to every other NES title out there at the time. It seems to me that the Seal of Quality was mainly just a way to keep unlicensed cartridges from selling. Go figure.
Quality assurance is an oxymoron. The fact is, with the suits saying "We gotta ship game X by this date or else there shareholders will bitch", there's no way quality assurance can be taken seriously.
Games are consistently shipped with the "screw it, we can patch it later" mentality. Look back over the last few years. There have been games such as Pool of Radiance, Myth 2 etc... That shipped with fundamental bugs, not just in the game itself, but in the installer routine! The original versions of those games, if you installed to a non-default location and then used the automatic uninstall, it deleted the contents of your hard drive.
Kinda hard to believe in any quality assurance with glaring failures like that getting through.
QA in most non-game software developments involved creating automation to pound a particular piece of the software quickly and reliably.
This would be kind of hard to implement in games.
And if you've ever met a game tester, yeesh.
Vonal Declosion
And you know who we have to blame for all of this? UO.
They started the trend of releasing a game that realistically was still in beta. However, since it was so successful anyways, other companies decided, "HEY! if they did it, we can do it to!", and thus the trend began. And I know I only speak for the MMORPG genre, I'm sure similar things have happened with other genres as well.
Buy Steampunk Clothing Online!
Working for a some big corporations like consulting companies (like Accenture and IBM) and (when jobs were scarce) a gaming company (EA), I have found that while QA isn't perfect for video games; it is much worse for enterprise applications like banking software. Scary thought eh, that GTA is better tested than online bill pay...
Nintendo Power. Those two words used to go together. Today in the US however, they do not. If Nintendo were to request major changes to a game they would be laughed at. They just don't have the market share to command the respect they used to. I would wager that if Nintendo did something to slow a third party game down, the third party would just cancel the Gamecube release. "Well it was buggy anyway so we don't want it." Riiiiiiiiight, it makes me laugh to see someone argue that fewer games is better.
So that brings us to Sony and Microsoft. They can and do reject games because of quality. But they can't just go rejecting a game because they don't like it. If it functions according to their standards and as the publisher intends, what grounds do they have to not allow it to be released? Its a tight-rope, my friends.
So are we talking bugs or are we talking design?
If its a bug, really truly a bug that is going to affect many end-users experiences it just doesn't get out the door. Does clipping affect an end-users experience? A whiny end-user maybe. Are they going to return the product? Maybe, but they would find fault in anything at that point since that individual is just probably an unhappy person who sits around judging things like Comic Book Store Guy.
If its a design choice, who is the first party to stop it? Just because YOU don't like it, someone else might. And standards do exist to ensure a consistent user experience in all games that come out on a given console. Beyond that, its not really QA's job to make sure a game is fun. QA is QA, just like any other software company. Just "does it work?"
And who is NTSC-uk anyway? Does every rant by every random fanboy who bought Enter the Matrix and found a bug end up posted here now? Here's an idea NTSC-uk, if you can do better, open up a publishing house or a developer. I like how he sites (but doesn't link to) some random "programmer's" perception of QA. Just because the author alledges there is one arrogant bastard "programmer" out there (well I can tell you there are many) who hates QA, doesn't mean that he is #1 correct in what he says or #2 even knows what he is talking about.
On a final note, searching around the site, there is a little blurb that introduces the article:
"MSR, Outcast and Broken Sword. Just a few of the many games released to an unsuspecting public with bugs and flaws. Barry Ip takes a closer look at QA in the games industry"
MSR as near as I can tell (ANAICT) is a Dreamcast game, Outcast ANAICT is a PC game which they claim not to cover in their about us page, and there appears to be many games with at least the partial title of Broken Sword. Those 3 games are supposed to represent the whole console gaming industry? This is a very old article, being a gamer in the UK just sucks that bad, or the guy is a moron who does incomplete research.
News for nerds maybe, stuff that matters...no.
Thanks for still trying to bring us quality Nintendo. I for one appreciate it.
Superman Has the Nintendo quality seal, heck for that matter so does Aquaman, so does charlie's Angels for Game Cube and nfl blitz 2003 for gameboy advance.
So Here's how much a nintendo quality seal Matters
Console and PC development tracks are entirely different. Console manufacturers have stringent controls on what they allow to be published on their platforms. Obviously you can publish anything you want on the PC as you can publish it yourself.
Fatal bugs (ie: crashes) are a lot less common on consoles than PCs. That's because they often must undergo a "burn in" process. For example, Xbox games have to stay running for 7 days - being played - without crashing before Microsoft will send the game to duplication. Similar thing for Sony. This is why there's no Star Wars Galaxies or Planetside on the PS2. The PC versions crash. A lot. Even after patches.
As to the quality of the gameplay itself - there are more people working on today's games than ever before. Teams are larger, movie-scale assembly-line production techniques are being used that simply do not work well for high quality videogames. With many members of the team only focused on one small area of the game, it's far easier to produce a great seething mess of grotesque compromises. If it's just two guys working in a garage - as it often was in the old days of games - they would have a very clear vision.
I believe that great games are formed around great ideas - and keeping those great ideas in sight as the game is built. If a broken idea makes it into production, the game is probably doomed from the start. No amount of QA shine can change poop into gold.
For awhile there, console games were relatively bug free. On the Super Nintendo, Genesis, and to some degree the PlayStation, there were very few crippling bugs that every single person that played the game had to watch out for while playing. Lately, this has changed. Enter the Matrix was riddled tons of different bugs. Tomb Raider: The Angel of Darkness was filled with situations where Lara would get stuck in a jump and you would have to reset. Jak & Daxter had a bug that randomly occurred in the middle of the game that kept you from fully completing it. Knights of the Old Republic's bugs are just as infamous (though not as numerous) as Enter the Matrix's. Star Ocean: Til The End of Time, a best-selling Japanese console RPG, had crippling bugs. Yu-Gi-Oh! World Wide Edition is filled with bugs toward the end of the game, including a relentless crash bug that causes the game to crash during almost any battle with the final boss.
These are some of the best-selling games in their respective countries and consoles, but they're riddled with software bugs and glitches that, in some cases, ruin really great gameplay ideas. And these are just the ones that are popular! Play any of the less popular GBA titles, such as Megaman Battle Network 3 or the GBA port of Super Puzzle Fighter 2 Turbo, and you'll find many more of those GBA crash bugs.
We've actually gotten to the point where games made by veteran game developers like Capcom, Shiny, and Konami that have been certified by NINTENDO (of all companies!) are riddled with crash bugs, so I think gameplay is the least of our worries at this point. If you can't even play the damn game, then the gameplay doesn't really matter much.
(And as a brief side note, some of the practices that the article mentions have already been standard at Sony for years. Sony Computer Entertainment America has wielded its broad monopoly in the United States to keep what it sees as "below average" Japanese PlayStation and PS2 games from entering the US. Some notable victims are The King of Fighters 2000, a Metal Slug title or two, a Persona game, and Goemon.)
The Nintendo Seal of Quality wasn't worth the paper it was printed on. It was nothing more than a Nintendo Fee, and promise to promote the game in Nintendo Power. Ever read a Nintendo Power review? Ever hear them trash a game? Were all of the games good? Nope. Nintendo Power was just propaganda. There were plenty of great games that came sans Seals.
But feel free to step up now, i hear SCO is selling Seals of Quality for all the major Linux distros.
So that brings us to Sony and Microsoft. They can and do reject games because of quality.
Or they let it slip by, like Star Wars Galaxy--that wouldn't install. JUST HOW THE FUCK DO YOU LET SOMETHING LIKE THAT GET BY?!?
You make this big rant about what you think happens within Nintendo vs other companies. Then you use the noun glitch as a verb. A PROGRAM CANNOT GLITCH. NOTHING CAN GLITCH.
Please explain to me how to glitch. I want to go outside tonight and glitch. What does glitching look like?
So what does your use of the word glitch say to me? It says you have never worked in the games industry and have no idea what you are talking about. That you are just spouting random, hateful things with nothing to back it up.
Is the sorting bug with the boat's shadow and the water in The Wind Waker considered "glitching"? Wait, thats a Nintendo game.
Business software gets tested by the programmers, and the end-users (and occasionally there is a QA tester at the developer).
I would wager that if Nintendo did something to slow a third party game down, the third party would just cancel the Gamecube release. "Well it was buggy anyway so we don't want it." Riiiiiiiiight, it makes me laugh to see someone argue that fewer games is better.
So that brings us to Sony and Microsoft. They can and do reject games because of quality. But they can't just go rejecting a game because they don't like it. If it functions according to their standards and as the publisher intends, what grounds do they have to not allow it to be released? Its a tight-rope, my friends.
Nintendo, Sony, and Microsoft are all in the same boat on this when it comes to their consoles. If the game is buggy and just doesn't work the way it's supposed to, they might prevent it from being released, it really just depends on the level of demand for the title and what role they play in the process of that game. I know that Nintendo and Microsoft both will delay titles that they are publishing (as opposed to titles published by someone else for their consoles) for almost any reason, including gameplay, but then when they publish they're paying for a good chunk of the development. Sony may do so as well, but then I haven't heard a whole lot about Sony's publishing. If someone else is footing the bill to put the game out, chances are it'll go out the door unless it's so bad that they feel it will somehow bring the console's reputation down, and I really haven't seen that happen yet (despite such debacles as Enter the Matrix and the latest Tomb Raider).
So are we talking bugs or are we talking design?
If its a bug, really truly a bug that is going to affect many end-users experiences it just doesn't get out the door. Does clipping affect an end-users experience? A whiny end-user maybe. Are they going to return the product? Maybe, but they would find fault in anything at that point since that individual is just probably an unhappy person who sits around judging things like Comic Book Store Guy.
Clipping definitely effects the end user. However, the real question is, what choice does the end user have? If the end user is totally disappointed with the game, what are they going to do? Sony, Microsoft, and Nintendo already know the answer to this. It has no effect on the console makers unless it becomes a serious problem with a large number of games on their platform. Games can't be returned once they've been opened at most stores. At best they become used titles and the retailers end up with so many used copies that they don't buy more new copies to stock their shelves.
If its a design choice, who is the first party to stop it? Just because YOU don't like it, someone else might. And standards do exist to ensure a consistent user experience in all games that come out on a given console. Beyond that, its not really QA's job to make sure a game is fun. QA is QA, just like any other software company. Just "does it work?"
That last part is really the best point you make, that QA is there to make sure the game functions, and some portion of each publisher's QA team isn't going to like each title that they test (though most likely different portions based on the title). QA teams usually have to go through very specific steps, too, to test very specific portions of the game. Most QA work is not sitting down and playing through the game (though you'll have some people do that at some point), it's playing through a particular part of the game doing a particular thing. The real question is, do we need people to perform some sort of QA that answers questions like 'Is this fun?' and 'Does this control/play well?'. Personally, I'm not sure that it's really needed. Despite what people like to say, even at the height of Nintendo's power to control what was released for their console, there were a lot of crappy titles released with their 'Seal of Quality', it's just that people don't like to remember those most of the time.
-PainKilleR-[CE]
As someone who works in software QA, all I can say is thus:
If QA has the ability to block the release of something due to defects then this is an almost absolute way to ensure quality (other factors notwithstanding).
If QA doesn't have to give their seal of approval before something goes out the door, then things will be released with defects (some known, others not).
I was fortunate enough to work for a boss who stated to development and the project managers that he would not sign off on releases simply to meet deadlines. If the powers that be wanted something shoved out the door simply to meet customer expecations, they'd do so without QAs consent - and that we'd not take the blame.
Cruising the internet on my TI-99/4A @ a whopping 300 baud!
Actually, the soak test / aging test is usually 72 hours, not during active game play. Usually this means going in and out of "attract mode". Often that means playing a movie over and over again, and never running the game itself.
I suspect the real reson for the aging test is that MS avoid embarrassing public crashes on the in-store displays.
Thad
Anecdotal evidence doesn't count for much, but it can be entertaining, so here is a the "description" from a genuine bug report from a major US publisher, for a game I was working on a while back (not published):
This was submitted as a "class A", "In-Game GUI" bug.
It's a cut and paste. No typos introduced in the retelling.
Thad
Heaven's yes, let's go back to the seal of quality. There was certainly nothing wrong with that. As everyone knows the seal of quallity was definitely synonymous with quality.
I have a lot of opinions about Cyborgs and Architects
Surprise, surprise, Sony DOES deny stuff arbitrarily. They've had a rather strict policy of no 2D stuff on PS2 since launch. The policy goes back to PSX days-- Capcom had a LOT of trouble getting some of the Rockman titles okayed for US release and in fact could NOT secure rights for Rockman Complete Works around 1998-1999.
With pieces of buggy crap (see the Temple of Elemental Evil boards for a perfect example) basically ruining PC gaming, yeah...something has to be done. As it is now, you can't hold a company accountable because you can't RETURN buggy as hell software. It's quickly becoming impossible to do so because of the way the companies that MAKE the games tell stores they will do returns. It's a sad thing, because I can say that many good/quality games are tossed and never played because of the horrendous problems people have playing them.
This is standard practice in the console world, not a Microsoft conspiracy. Sony et al have similar soak test requirements that must be passed before release.
It's not an assumption. Nintendo used to be pretty rigorous. I'm talking 16 bit days (Super Nintendo). Even our own testers never thought to do anything so silly as to hit the reset button 30 times!
But since you have examples of games that crash, well, what we CAN assume is either Nintendo no longer is as stringent with their quality testing, or they just turn a blind eye to more than they used to, since they don't have the marketshare they probably have less leverage over the publishers.
ALTHOUGH, you mentioned GAMEBOY games, so it could be that having come out with so many variations of the Gameboy hardware, each of which is back-wards compatible, has perhaps caused problems in across-the-board compatibility. It's one thing to test a game for existing hardware, but then to guarantee it will work on a future hardware revision is quite difficult.
from the article..
<blockquote>And what about industry regulators? Organizations such as the IDSA and ELSPA fight piracy, and impose age ratings for new software, but nothing to raise awareness of the concerns caused by software glut and poor gameplay. To counteract this, trade bodies could establish their own independent departments employing experienced testers and gamers to evaluate the quality of new software. The outcome would enable developers to be given an independent assessment of their games, which again promotes software improvement within the industry. Furthermore, quality ratings could also be given, thus providing the consumer with reliable information of what the game is like without having to resort to reviews from questionable sources, and developers with extra incentive to make better games.</blockquote>
I really think this is what needs to happen. For the same reason my tech services team has to deal with security audits every year from a 3rd party that is LOOKING for things to break so they have something to write about in their report. If some company were to make a name for itself and off to put their 'seal of quality' on games that survive their QA tests, we could approach some kind of standards.
I'm sure there would be plenty of company that would spring up to fill this role, and it would be in their best interest to be honest.. because if you buy a game 'assured' by QA company X, and it is bugtastic, you probably wont trust their rating in the future.
1) Your analysis is based on bad assumptions so your result is way off. 2) You're a sick bastard for fucking a horse.
Why isn't there a Mystical Ninja sequel on the PS2? Because Sony is ghey.
The Morrowind crashes (screen freezes, loud buzzing from speakers, I seem to recall other games doing this once in a great while) after less than 7 days of play certainly seem to support your theory =) Say, that reminds me, I have to go sell my copy and buy the bug-fixed GOTY edition...
-insert a witty something-
Well, most QA in the software entertainment industry is SQA - for Software Quality Assurance - meaning we find bugs, perform all the necessary testing (according to test plans and test scripts), send reports, and track issues (and more...)
/I probably missed something somewhere... /draw your own conclusions...
Most testers are not lucky enough to have much of an input in the design of the game, implementation of ideas, or correction of bugs. And most are there to make sure hourly requirements are applied for each build. Alot of testers don't even work closely with their development teams (imagine that!).
Also, working with different development teams leads to different experiences - some are more open to QA feedback, some are not. These days, some games are open-beta and many games will even ship beta copies to select users for evaluation.
Many design ideas come from reading the web forums around games, and some ideas are just poorly thought up, or not implemented correctly, for whatever reason.
Dates are definitely an issue, and bigger games must ship, or else stock holders will be upset. Though it is quite common to see some PC, Xbox, and GC titles held back a week or more, but generally PS2 is almost always out at the door on-time.
Not to say QA is never at fault - I've seen some marvellously ridiculous things get missed. Alot of it comes down to (like any other job) hiring the right people for the right job. QA is also generally underpaid, overworked (7days), bored (many days are spent just waiting for a build that comes at 11:00pm and then is completely broken), and ignored, even though they may have some valuable insights.
The real shame about video game QA is that most people just want to get paid for playing video games, and don't develop that sense of ownership. I take great pride in my roles, do my job to the best of my ability everyday, and enjoy the benefits of my work.
The Nintendo Seal of Quality wasn't worth the paper it was printed on. It was nothing more than a Nintendo Fee, and promise to promote the game in Nintendo Power. [...] Were all of the games good?
The games with the nintendo seal of quality are tested on TECHNICAL quality, not subjective quality.
The nintendo seal of quality means that the game can be finished (!), that it runs correctly on nintendo hardware, that it has no major bugs (crashes), etc.
You can't take the sky from me...
Your assumption, as far as I've seen, is false.
I have tested games before they were submitted to Nintendo, and man, they test, and they test good.
I've had three GBA games with crash bugs
Please provide a precise description and reproduction steps that lead to 100% reproduction of the crash.
That is what a game tester would have to provide for the programmers to be able to correct it.
And bear in mind that maybe that game had hundreds of bugs found and fixed, and it had to ship sometime, and well, at some point, maybe they know the game can still crash, but they can't reproduce it at will, it just does sometimes, and maybe nintendo won't find it...
You know...
It's not an assumption. Nintendo used to be pretty rigorous. I'm talking 16 bit days (Super Nintendo). Even our own testers never thought to do anything so silly as to hit the reset button 30 times!
I was replying to the part where you said "(and I assume still do)". I'm well aware of Nintendo's nearly flawless quality control in the 16 bit days, as well as the admirable job that they did with the original Gameboy.
ALTHOUGH, you mentioned GAMEBOY games, so it could be that having come out with so many variations of the Gameboy hardware, each of which is back-wards compatible, has perhaps caused problems in across-the-board compatibility. It's one thing to test a game for existing hardware, but then to guarantee it will work on a future hardware revision is quite difficult.
I was talking about GBA games on a GBA SP that were produced after the GBA SP was released in the United States. I could excuse a Gameboy game not working with the GBA, or even a GBA game not working with the Gameboy Player, but a game designed for the GBA SP that has easily reproducible crash bugs on that system is not excusable. Expecting a game designed for the GBA SP to work on the GBA SP is not a ridiculous demand.
Pain, we are often on different sides of the arguement, but we are on the same page here. Are you trying to convince me of something? Very little of my post was opinion. 95% was up to the minute fact.
The only debate I could see is clipping, and I could see collision detection so poor that it could effect the end-user, but you would have to provide an example for me to evaluate. The best selling games (GTA and THPS) have some of the worst clipping out there, yet noone complains. In general, I stand by my statements.
Sony does and has rejected games cause they don't like it. I remember Sony wouldn't do a release of some Megaman game because they didn't want 2-D titles on the PS1 back in the day.
"There is no time, sir, at which ties do not matter," Jeeves, (Jeeves and the Impending Doom)