Slashdot Mirror

← Back to Stories (view on slashdot.org)

Exploring the Current State of Beta Testing

Posted by ScuttleMonkey on from the only-if-you-actually-learn-from-the-past dept.

Karen Hertzberg writes "Since the earliest days of MMO gaming, beta testing has played a pivotal role in the success or failure of our persistent worlds. We've come a long way since the initial tests of Ultima Online and The Realm, but what role do our current beta tests play in the potential outcomes of unreleased titles? To answer this question, Ten Ton Hammer turned to current and former beta decision makers at Cryptic Studios, NetDevil, Sony Online Entertainment, Funcom, and Mythic Entertainment. Some of their answers — and the information they reveal — may surprise you."

82 comments

  1. There Really Are Some Gems in This Article by eldavojohn · · Score: 5, Insightful
    That I wish the game companies and everyone would learn from:

    [Beta] seems to be more about marketing the game and hyping it up.

    So completely accurate.

    Honestly I feel that nowadays the developers get confused and can not find a happy medium between listening to every single testers opinion and listening to nobody. In some beta tests I have felt the community was highly ignored while in other tests I have felt that the developers tried to cater to every single tester. Both are and were recipes for failure.

    Beta tests now are glorified demos for the games. It used to be so much different. I really hate what's been done to them, and the only way to be a 'real' tester now is to get into the alpha's or in some rare cases early closed betas.

    Backing up the first quote and pointing out what Betas have become.

    I wish there were more people submitting bug reports, but that's the way it goes with beta, and weâ(TM)re still finding them regardless. Besides, I need all types [of players]. I need the exploits so we can find them ... I need the jerks.

    At first I was shocked they would want the scripters and botmakers on so early but I soon understood that you want to catch these serious things as early as possible to fix them because:

    By the time beta begins, you've made decision after decision that have compounded on each other. Your assumptions' assumptions' have assumptions about what your game is. The whole product, systems, content, operations, marketing, PR, community ramp, you name it -- is built upon them. Changing core assumptions about the product itself is unlikely to be possible without significant delays, costing progressively more money per month. (Remember, the months toward the end of the dev cycle are the most expensive ones by far.)

    Ultimately, you need to believe in your product before you conduct any sort of open beta or release a demo. You can message to players all you want that the game is a "work in progress" and that many things will change before final release, but that wonâ(TM)t stop them from making judgments about the title based on their beta experiences.

    I believe that's the fundamental reason behind Blizzard's horrid schedule slippages.

    It is disappointing to say, but testing has become a bit of a joke, and I feel that the current crop of recent games are a reflection of that. So many games are being released incomplete (as far as hyped features go) and containing issues that should have been picked up and resolved during the closed beta phase at the latest, but this isn't happening.

    It's the classic cash in while you still can mentality that has seen the release of so many unstable games only to have the servers shut off or merged down within a year.

    Here's to hoping the gaming industry finds and reads this article ... we're in a bad spot right now.

    --
    My work here is dung.
    1. Re:There Really Are Some Gems in This Article by Chabo · · Score: 4, Informative

      Beta tests now are glorified demos for the games. It used to be so much different. I really hate what's been done to them, and the only way to be a 'real' tester now is to get into the alpha's or in some rare cases early closed betas.

      Backing up the first quote and pointing out what Betas have become.

      IMO, the point of an open beta isn't to make client-side changes, because the developers can make all of those changes based on alpha or closed-beta testing, much earlier in the development process. By the time a product reaches beta stages, it's essentially done, and just needs a small amount of polish before release.

      The real point of an open beta, especially for MMOs, is stress-testing the game servers. An open beta by definition tries to get everyone possible to play the game, so in addition to being a demo for the game, you're also trying to debug any bugs in the server system that simply can't be found by testing with just a few clients.

      This was about all Blizzard was concentrating on with WoW during the open beta, and we can even see this outside the genre; when Valve had the pre-release demo of Left 4 Dead, they were testing what happens when thousands of gamers used their brand new matchmaking system. It had some issues that have been largely resolved by now, but they just couldn't test it that easily without that kind of hammering.

      --
      Convert FLACs to a portable format with FlacSquisher
    2. Re:There Really Are Some Gems in This Article by Darinbob · · Score: 3, Insightful

      [Beta] seems to be more about marketing the game and hyping it up.

      So completely accurate.

      It has annoyed me in the past when people have said they were "lucky" to be in a beta, or grumbling that only a select few were chosen. It seems that people treat being in a beta as a perk or reward, which means that they treat the beta like an opportunity to learn the tricks before everyone else, or use it like a badge of honor. That's completely the wrong attitude, as in normal software being a beta user means a lot of hard work and attention to detail. Being in a close game beta should not be about having fun.

    3. Re:There Really Are Some Gems in This Article by Pollardito · · Score: 4, Interesting

      You're not going to get "a lot of hard work and attention to detail" from the testers of your commercial product unless they're being paid. The fun of playing the game early is a form of payment, but if you're asking them to forgo that fun in order to only do the work part then you're insane. There's definitely an imbalance though, with some testers who don't bother to submit bug reports at all, and also a lot of companies that don't bother to listen to the feedback they do get.

    4. Re:There Really Are Some Gems in This Article by Anonymous Coward · · Score: 0, Offtopic

      Seriously though

      "
      wonÃ(TM)t
      "

      What in the fuck are you using to cut and paste that does this fucking SHIT!!@!!!

    5. Re:There Really Are Some Gems in This Article by Chabo · · Score: 1

      Slashdot's software takes the non-ASCII apostrophe and freaks out, and produces "Ãf(TM)".

      So he was merely trying to paste the contraction "won't".

      --
      Convert FLACs to a portable format with FlacSquisher
    6. Re:There Really Are Some Gems in This Article by DerekLyons · · Score: 2, Insightful

      You're not going to get "a lot of hard work and attention to detail" from the testers of your commercial product unless they're being paid. The fun of playing the game early is a form of payment, but if you're asking them to forgo that fun in order to only do the work part then you're insane.

      I'm guessing you've never actually part of gaming beta. I all of the ones I've been in (about nine so far), finding people willing to work hard and pay attention to detail has been pretty easy.

    7. Re:There Really Are Some Gems in This Article by BikeHelmet · · Score: 1

      when Valve had the pre-release demo of Left 4 Dead, they were testing what happens when thousands of gamers used their brand new matchmaking system

      I remember that!

      And the answer is: Nothing.

      Yep! Nothing happened when thousands of people tried to use the matchmaking system. We all just sat there waiting and waiting until we finally decided to play with friends on local servers.

    8. Re:There Really Are Some Gems in This Article by Anonymous Coward · · Score: 0

      I would agree with you on principle, as I think OP perhaps generalizes a bit too far wrt the lack of helpful beta participants.

      But I think what's worth pointing out is that there is probably some form of inverse relationship in the number of useful beta testers like you and I and the total number of people playing and/or how long the beta has been going on (and as a result of that, how hyped and marketed beyond all practical measures the game has been). Simply put, the closed beta phases have, in my experience, seen vastly more tester feedback relative to the total population of players than open beta phases. I'm not saying closed betas are completely full of useful testers, nor am I saying open betas lack them entirely (certainly the previous closed phase testers will probably stick around during the open phases), but that their usefulness is drowned out in so much worthless noise:
      "What's that you're trying to tell me? Possibly you may have even found an important bug in this 'wHY 1s my clAss teh sux!!1!' bug report, but it is utterly worthless to me because you couldn't follow the bug submission rules on EVERY GODDAMN PAGE..."
      And similar.

      I'm sure there are a number of factors contributing to this, and I know as a tester I too have tended to focus less on actually trying to be a good tester as more and more players would throw themselves at the games. I always appreciated the idea in days long gone that those forms one had to submit detailing specifically _why_ one might make a good tester were in theory being read by people and decisions made based upon them. To me that seems like a useful way of selecting for people who either genuinely want to to test or are smart enough to make it appear that they do, in which case you should at least end up with a relatively competent playerbase.

      Maybe this is all a result of the trend toward betas as nothing more than marketing tools to help push an obviously incompleted, bug-ridden product. This thought makes me sad. I'll go back in my cave and fondly recall the Asheron's Call 1 beta. Red server (Frostfell) for life!

      Now where did my walker go! Goddammit, ever since the nurse put tennis balls on it I can never hear when they take it away.....

    9. Re:There Really Are Some Gems in This Article by Austerity+Empowers · · Score: 4, Interesting

      I don't know, I've been in closed beta for several products lately, Warhammer Online, Age of Conan and Tabula Rasa.

      I submit bug reports out the wazoo (and believe me, all 3 of these have had a few bugs). I never see anything come of them, even the major ones. I try to view being in the beta as a privilege for which I owe good bug reports.

      On the other hand, you are supposed to have fun. Any time I am not having fun, is just as much of a bug as when things don't work.

      I've always got the feeling that the game was going to ship no matter what I found. I was just there to load the servers and stress the network so they could make the launch less hokey. I've found blatant exploits, item dupes, countless serious quest bugs, etc. I never saw any movement on any of those, nor was asked to clarify or repro. That kind of tells me a lot about how my input is being considered. It really is just a marketing thing that the developers have been able to put to small use. Beta's rarely start early enough, nor is there sufficient in-house support for beta testers for it to really be effective.

    10. Re:There Really Are Some Gems in This Article by fractoid · · Score: 5, Insightful

      I believe that's the fundamental reason behind Blizzard's horrid schedule slippages.

      (Warning: rabid Blizzard fanboy post ahead:) In the early days, yes, Blizzard had some pretty bad schedule slippages. Unlike most companies, they correctly identified the problem to be the schedule, not the slippages. The reason that their games are still regularly played over 10 years after release is the uncompromising attitude towards quality. That's why these days they keep everything completely dark until a game's already been in development for a couple of years, and they don't commit to a release date until a couple of months before that date.

      Games are art, and artwork isn't done until it's done. You can schedule an accounting app, because it has a strict function that's determined by an arbitrary set of rules. You can't schedule 'finishing' a game because the completion criteria are incredibly fuzzy and vauge. "It has to be fun." "The storyline needs to be engaging." "The player has to feel significant." "The world needs to be realistic." If you don't give this stuff enough time to be iterated over and refined, you end up with a bodgy game no matter how good your tech is.

      --
      Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
    11. Re:There Really Are Some Gems in This Article by fractoid · · Score: 1

      Bingo. But this was with a pre-release demo, which is a much better time to find problems like this than at release. If it were a post-release "yay I just paid $89 for this game and this is my first time playing it" situation you'd be a lot more upset than simply "meh".

      --
      Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
    12. Re:There Really Are Some Gems in This Article by metalcoat · · Score: 1

      I remember watching a video of the making of Halo 3. Microsoft used a new studio only for beta testing. They used input down to where players were being held up on the missions (can't find where to go to next). They had several instances of issues that I would have never thought about during development. I frequently get held up in games about where to go next but I figured I just wasn't paying attention.

    13. Re:There Really Are Some Gems in This Article by Merls+the+Sneaky · · Score: 1

      Tabula Rasa Had an issue with single core machines where if you were running it would constantly hiccup.(hiccup was caused due to CPU having to wait on terrain redraw) Issue was reported many on their official forums. Devs kept claiming it would be fixed, it was, about 6 months before shutdown.

    14. Re:There Really Are Some Gems in This Article by Opportunist · · Score: 1

      And that's basically what an open beta is about: A "live" test without anyone being able to complain.

      This may change, though, when the model some companies try to pull becomes more popular, i.e. charge premium for "beta access" or "headstart access" or whatever you want to call it. Those people will certainly want to get something out of their money besides "uh... sorry, login servers can't handle it... but it's gonna be ready when we start". They PAID to be let in before the crowd.

      I blame part of some product's failure on this behaviour.

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    15. Re:There Really Are Some Gems in This Article by Opportunist · · Score: 1

      Devs kept claiming it would be fixed, it was, about 6 months before shutdown.

      Damn pessimist, you could also have said 6 months after release!

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    16. Re:There Really Are Some Gems in This Article by Opportunist · · Score: 1

      Sadly, that's all that's left about being in beta: Hype.

      I haven't seen a single MMO (closed) beta lately (I've had my share, here's my badge...) that actually deserves the name.

      It seems the terms changed meaning. Alpha today is what CB was. CB is what OB was. OB is what the "10 days of stress test before release" used to be.

      So whether someone joins the CB to have "fun" doesn't really matter. First, what he learns is probably moot at release anyway. Second, his input goes to /dev/null 9 out of 10 times. And finally, if he really feels like waving his "I was in CB" badge, so be it. For some odd reason, it does not really impress me anymore.

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    17. Re:There Really Are Some Gems in This Article by Opportunist · · Score: 2, Insightful

      It has changed, though, if you ask me.

      I've been in a stretch of MMO CBs within the last decade or so. A decade ago, you had some pretty good chance to actually know a fellow tester or two. From another game, or because of the way the CB slots were distributed.

      Today, CB slots are more often than not some sort of raffle prize. You have people who join the game forum (or, IMO worse, some MMO forum) where then some raffle is held with CB slots as the prize for the lucky winners.

      What kind of "quality" do you expect to get out of that?

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    18. Re:There Really Are Some Gems in This Article by Anonymous Coward · · Score: 0

      I see. So you're saying Slashdot needed more beta testers to help fix these issues.... or did they have enough and simply didn't listen to the feedback? :)

    19. Re:There Really Are Some Gems in This Article by Faluzeer · · Score: 1

      "By the time a product reaches beta stages, it's essentially done, and just needs a small amount of polish before release."

      Hmmm

      In theory you are correct, in practice the majority of MMOG's need / needed more than "a small amount of polish" in beta. In most cases the games needed a large amount of work during the first few months after the game launched.

    20. Re:There Really Are Some Gems in This Article by jonaskoelker · · Score: 1

      You can schedule an accounting app, because it has a strict function that's determined by an arbitrary set of rules.

      I'd think that by the same token, you can schedule an OS. It didn't work too well for Vista, IIRC. In general, what I hear from people in the trenches is that schedules and budgets are almost never met.

      Games are works of art, and that makes them special. But regarding the particular management issues, they seem to do as well as everyone else (sadly).

    21. Re:There Really Are Some Gems in This Article by Anonymous Coward · · Score: 0

      So would that be when the game was half alive, or half dead?

    22. Re:There Really Are Some Gems in This Article by Chris+Mattern · · Score: 1

      Slashdot's software takes the non-ASCII apostrophe and freaks out, and produces "Ãf(TM)"

      Correction: Microsoft software produces directional apostrophes and quotes by using bizarre encodings that they call "Smart Quotes". Nobody else uses these encodings, so when "Smart Quotes" run up against anything that's not Microsoft, you get garbage.

    23. Re:There Really Are Some Gems in This Article by Chabo · · Score: 1

      Well, let me clarify. What I'm saying is that beta stages are not going to see any major gameplay or style changes; it's all about the bugfixes.

      Whether or not a game is horrendously buggy in beta (or even release!), it is essentially finished. If Blizzard is considering adding a new class, they'll wait for the expansion, they won't fiddle with that sort of thing in the beta.

      --
      Convert FLACs to a portable format with FlacSquisher
    24. Re:There Really Are Some Gems in This Article by Chabo · · Score: 1

      This instance may be a Microsoft issue, but I know Slashdot does freak out with Unicode as well, unless you use the correct reference.

      â(TM) is how Slashdot displays an XML/HTML-standard "right single quotation" mark, correctly displayed as ’ by using the character reference.

      --
      Convert FLACs to a portable format with FlacSquisher
    25. Re:There Really Are Some Gems in This Article by Anonymous Coward · · Score: 0

      So garbage in Garbage out basically

  2. Rant by Ethanol-fueled · · Score: 0, Offtopic

    but what role do our current beta tests play in the potential outcomes of unreleased titles?

    Companies just release their products anyway. Nowdays beta-testers are referred to as "early adopters". Sadly, that extends to the hardware world as well, as this and this (the forum is riddled with those kinds of posts) demonstrate. At least Google are honest about their products being in permanent beta, but their stuff works unlike the others ;)

    Disclaimer: I'm familiar with both items because my father purchased one of those laptops only to have it die within 1-2 months. I bought a Peavey Vypyr amp which was riddled with the bugs that you see in the forums, problems which firmware updates did not fix. I took it in to the repair shop three weeks ago and I just found out that the board my amp needs is backordered until mid-May! /rant, thanks for reading.

  3. Some Games Just Don't Catch On by qpawn · · Score: 1, Funny

    After extensive beta testing, I decided to scrap my game tentatively titled "Kill Orcs Instead of Talking to Girls"... which, coincidentally, is the title of my autobiography.

    1. Re:Some Games Just Don't Catch On by JerkBoB · · Score: 1

      I LOLed, anyhow.

      --
      A host is a host from coast to coast...
      Unless it's down, or slow, or fails to POST!
    2. Re:Some Games Just Don't Catch On by fractoid · · Score: 1

      See, you just needed to work on presentation. If you'd called it "Kill Orcs And Meet Hot Elf Chicks Who Really Are Hot Girls IRL" then you'd be onto a winner!

      --
      Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
  4. The Realm! by Kelz · · Score: 1

    Kudos for the mention of The Realm! I don't think there has been a purely stat-based graphical MMO since.

    *nostalgia's out*

    1. Re:The Realm! by awills · · Score: 1

      Ah, I miss hanging out in room 300....

    2. Re:The Realm! by daveime · · Score: 1

      I still owe that bastard who scammed my 4-banged Wrath !

    3. Re:The Realm! by Kelz · · Score: 1

      I do believe I quit with a FIVESD Wrath equipped :P Before they made it impossible. Glowie boots too!

    4. Re:The Realm! by Plaid+Phantom · · Score: 1

      scammed my 4-banged Wrath

      I have no idea what that means, and I don't think I want to.

      --
      All comments are properties and trademarks of the voices in my head. Not like I'm gonna claim them.
    5. Re:The Realm! by theskunkmonkey · · Score: 1

      I remember when this "world" was made up of 9 rooms. It was little more than a chatroom with legs a la Kings Quest/Leisure Suit Larry.

      Ah the memories. Gold Pants. Stephen dropping Baldrics. Tragluk and Shebok.

      The Jaxom Clone Army.

      Man, those were the Good Old Days!

    6. Re:The Realm! by Calmiche · · Score: 1

      Just as a bit of Nostalgia from me, I played the beta of the Realm when it went to 2.0 and again when it went to 3.0.

      Since I was one of only about 250 in the initial beta of Realm 3.0, I ran a website with information. (Mystro's Realm for anyone that remembers.) The programmers spent LONG shift for about two weeks with constant updates and fixes. It was one of the only beta tests that I've been involved with that there was almost instant response to fix bugs. We would find things that would crash the client or server and submit them and the programmers would fix them, pushing patches about once an hour from 8:00 a.m. to about 10:00 p.m.

      It was a fun experience.

    7. Re:The Realm! by Calmiche · · Score: 2, Informative

      Sadly, it's not as dirty as it sounds.

      The wrath was one of the rarest (early) swords in the game. You could enchant them with certain spells to increase strength and damage. However, the enchant spell had a chance of destroying the sword. I believe that a four enchant sword (4-banger) had a 80% chance of destroying the sword. For a time it was possible to get 5 enchants on a sword but the chance of getting 5 enchants was between 2% and 5%.

      So, if you played a lot and were high level, you could find 2-3 Wraths a month on average. Then, you could burn through about 20 to 50 of them to get a 5 enchant sword.

    8. Re:The Realm! by NemosomeN · · Score: 1

      wow, I'm surprised so many people know about The Realm, I loved the shit out of that game. I had 6x Glowies, 6x wraths, and tons of other stuff from way back when it was possible, and fleshies were still rare. I really wish I could get my account back.... not sure how much I'd be willing to pay. Actually, I still use my old Realm password/account name....

      --
      I hate grammar Nazi's.
    9. Re:The Realm! by Calmiche · · Score: 1

      You know, I e-mailed the new owners twice about old account reactivation but never got a reply. Not even a "No, that's not possible."

      I still feel nostalgic once in awhile and if I had ANY type of encouragement from the current owners, I would probably join up again. Ah well.

    10. Re:The Realm! by NemosomeN · · Score: 1

      Really? When was that? I emailed them about a month ago and got responses the same day. $100 for recovery of a very old account. I was tempted, but I restrained myself. They still have a free trial, but SirBryan really bollocksed it up. I miss the old days, classic interface, etc. I think it's cs @ realmserver .com ?

      --
      I hate grammar Nazi's.
  5. Once Beta, it's Done by MarioMax · · Score: 2, Informative

    At least in the case of SOE, once a game is in Beta, it's basically done. They just want marketing and bug reporting. The devs are extremely reluctant to change anything once their product is in Beta, no matter how fundamentally flawed the product is in the first place.

    1. Re:Once Beta, it's Done by awills · · Score: 1

      Unfortunately many companies are like this, and it has been this way for going on ten years now. I remember testing Dark Reign and the second build they mailed me didn't even include the single player component of the game. Guess which component needed patching as soon as the game was released.

    2. Re:Once Beta, it's Done by Shadow+of+Eternity · · Score: 1

      Frighteningly I'm disagreeing not because it isn't true but because I thought the Planetside betas were better than what came later. Then again SoE is out of their minds anyway.

      --
      A bullet may have your name on it but splash damage is addressed "To whom it may concern."
    3. Re:Once Beta, it's Done by muridae · · Score: 1

      I some how get into every Everquest Beta I apply for. I get the email the first day of closed beta. Then, about 3 days before it goes live, I get another one for the patcher so I can actually play. Some how, I've still managed to get in to the game before release, find a glitch, bug, wall-hack, or something that doesn't work, and then eventually see it patched a year or two later. It's the Everquest way.

  6. Just the beginning... by myowntrueself · · Score: 1

    We are really just at the beginning of the MMO genre.

    The game companies are still trying to figure out how to produce, test, market, maintain these games.

    Its a completely new model for them.

    Its bound to be a bumpy ride for the next few years, maybe even a decade or so.

    --
    In the free world the media isn't government run; the government is media run.
    1. Re:Just the beginning... by billcopc · · Score: 1

      The beginning ? MMOs have been around for decades.

      Oh, you mean graphical big-money MMOs... riiight!

      Joking aside, even "modern" MMOs have been around since the 90's. The formula has not changed much since Everquest. Perhaps the MMO producers are just too dense to actually think beyond their quarterly report.

      --
      -Billco, Fnarg.com
    2. Re:Just the beginning... by thePowerOfGrayskull · · Score: 1

      The beginning ? MMOs have been around for decades.

      Oh, you mean graphical big-money MMOs... riiight!

      Joking aside, even "modern" MMOs have been around since the 90's. The formula has not changed much since Everquest. Perhaps the MMO producers are just too dense to actually think beyond their quarterly report.

      age of human race: millions of years.

      age of computers: almost 100 years

      age of mmos: about 15 years.

      Yeah, I'd say that this qualifies as "the beginning".

    3. Re:Just the beginning... by Synn · · Score: 1

      Joking aside, even "modern" MMOs have been around since the 90's. The formula has not changed much since Everquest. Perhaps the MMO producers are just too dense to actually think beyond their quarterly report.

      Yeah, that's pretty much the problem. "Game X made it big, let's make ours just like it!"

      So we get the same sort of grind fest treadmills over and over.

    4. Re:Just the beginning... by Opportunist · · Score: 1

      When you put it that way you could excuse a few bad traits of the human race. After all, what's the measly 10k years of our civilisation compared to the 18b years of the universe?

      C'mon, try a more realistic timescale.

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    5. Re:Just the beginning... by Opportunist · · Score: 1

      MMOs are expensive. Very, very expensive. They're an investment unparalleled by any kind of game today. And with "normal" games studios are already wary of producing anything that breaks the mold. What do you expect to get?

      And any time someone tries something new he gets his nose bloody and has to withdraw in shame. From Neocron to Tabula Rasa, the only halfway successful non-vanilla MMO out there is EvE. Everything else, no matter what "universe" you put it in, works basically the same: tank-healer-dd. Sprinkle a little CC or buff/debuff here or there, but that's what 99% of the MMOs today come down to.

      Yes, that stinks. Yes, I hate it at least as much as you do (provided you're an opponent of the holy trinity treadmill as well). Yes, I like EvE and even liked TR and NC. Mostly because FINALLY someone did something new.

      But I guess we two ain't enough. It seems, the majority is quite happy with the stale, simple trinity play style.

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
  7. Halfway just for PR, Bad "beta testers" by TibbonZero · · Score: 4, Informative

    By the time most of these games hit Public Beta it's really just stress testing the servers a bit (but never enough) and working on some tuning stuff. Mainly I've seen this by this stage it's really just about getting a solid buzz going around the game. Most users will have broken their NDAs by this point (as we saw happen recently in WotLK), but the entire point is just to get hype going for the game.

    Until recently I was working for a game-industry related company, and we had a lot of close interaction with gamers and the game companies. I'm reading the article fully right now for some more of the developer/publisher-perspective details however.

    Half the problem is that most of these gamers suck at betatesting. They don't want to file bug reports, they want to play the game free/early so that their guild can get a head start on others. the number of users that I've seen rant about a game having downtime turning beta, doing server wipes, etc... They weren't complaining because they couldn't get enough bug reports in, but because they couldn't get into their Raid.

    Because of pressure on various fronts, most of these games are released with insufficient server architecture, horrid bugs, and critical balance issues. This is the stuff that should be stomped out during beta, but it isn't. Beta isn't about testing, its about PR and hype. Wish it was some other way. If i was developing an MMO I'd want to disable users accounts that didn't file bug reports properly, but I know that doesn't do well for the PR side.

    People feel entitled to their games, and even more entitled to a chance to play for free. As expected, its not uncommon for some big players in the game industry to give beta accounts to people who run big guilds, but don't necessarily put in bug reports.

    --
    Tibbon
    tibbon.com
    1. Re:Halfway just for PR, Bad "beta testers" by billcopc · · Score: 3, Interesting

      I think if game companies really wanted to get their value out of beta testers, they would implement simple, trivial changes like requiring a questionnaire after every N hours of game time. I'm not saying bug reports, but at least a 10-minute form where developers can ping the gamers for feedback, at least once or twice weekly.

      If the kids can't take 10 minutes of their time to help improve the game they sink 20 hours a week into, they can fuck right off and wait for retail. They will buy it anyway, and if they don't, well fuck 'em. That's business.

      --
      -Billco, Fnarg.com
    2. Re:Halfway just for PR, Bad "beta testers" by Bieeanda · · Score: 3, Informative
      In several of the betas I've been in over the last few years, I've been faced with brief questionnaires and comment spaces at the end of every mission/quest/hallucination and quite often upon logging in or going to quit. If anything, their frequency in the early game tends to be annoyingly high, thanks to all of the quests designed to baby-step you through the interface.

      But yes, most people who sign up for betas are absolutely fucking terrible at it. They have no QA training, and their posts on the official testing boards are usually 100% noise. It's even worse when they're dyed in the wool fanboys, because you'll never get useful criticism out of them.

    3. Re:Halfway just for PR, Bad "beta testers" by muridae · · Score: 1

      I wish SOE would do that. Instead, if you got into beta as a guild you are doing the raids when a dev clears them and warps you in. If you got in through application, you aren't going anywhere because no one knows who you are.
       
      Real, serious beta testing would do wonders for MMOs. Too bad they go about it in such convoluted ways.

  8. Opposite reaction in Guild Wars by TinBromide · · Score: 4, Interesting

    So, they talked about "good" betas and betas that opened miserably and killed a somewhat polished game, (auto assault, IIRC) that was opened too early. What about a game that the beta is BETTER than the released game?

    I was part of the open beta of guild wars(i.e. i pre-ordered and was part of the PR wave of beta). It was awesome, everything was fun, it was clear that it wasn't finished, but the missions were OK, the PvE was tolerable, but the PVP was phenomenal. When it was time to release, I fired up my copy and found like 2 skills at the first skill trainer. Approximately 750 in game hours later, it was possible to recreate the PVP experience I had during beta...

    750 hours in missions that are only OK and tolerable PvE that turned to miserable at the snails pace that they made you try it. Guildwars isn't a monthly thing based fee, so they gained nothing, absolutely zero, by forcing you to put 750 hours into the original campaign to get back to the fun of the open betas. By then, they had lost a very large portion of their user base and the beta users were not the the majority of the major adopters. If they had released the game we "beta tested", it probably would have been a runaway success instead of the third rate game it is today. Also, because the PVP players left for greener pastures (battlefield 2 so you can have an idea of what that crowd was), current PVP metagame is a pale imitation of what it once was and could be.

    For the record, about 2 years after release, they made enough changes so that a new player can jump into that old timey fun.

    --
    Is it sad that I am more likely to recognize you and your posts by your sig than your name or UID?
    1. Re:Opposite reaction in Guild Wars by Anonymous Coward · · Score: 0

      Warhammer did the same thing, experience gain was heavily nerfed at release making it a horrid grindfest which they slowly, partially undid.

      People abandoned chars at mid level due to the incessant grindy nature.

      Bait and switch seems to be part of the hype machine.

    2. Re:Opposite reaction in Guild Wars by Opportunist · · Score: 1

      I'm not a GW expert, but IIRC they got something out of forcing you into the treadmill, you could pay to get all the skills instantly.

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    3. Re:Opposite reaction in Guild Wars by TinBromide · · Score: 1

      It took them like 3 years to get the point where you could buy everything. They lost the very large portion of their base that would have bought the unlock everything right after the game was released. By the time they unveiled the option, people had enough time to either grind to unlock everything, unlock the skills that were necessary for what they were doing (farming, basic PVP builds) or didn't care about what skills they had.

      The market for the paid unlocks was precisely the people they lost at release. Those people weren't coming back. As a result of losing those people, pvp wasn't as diverse, there are 4 builds and you can play "competitively" in PVP with approximately 60-100 of the 750? (not sure nowadays) skills. That covers all 4 predominant builds. If they hadn't lost the PVP players, I'm willing to wager that there would be more builds, more strategies, and a full use for most of the unlocked skills (and more incentive to pay to unlock everything). Kind of a vicious cycle...

      --
      Is it sad that I am more likely to recognize you and your posts by your sig than your name or UID?
  9. One Day... by koterica · · Score: 3, Insightful

    One Day, Beta Testing will be out of Beta.

    1. Re:One Day... by RINGSMUTH · · Score: 1

      One day games in Soviet Russia will beta test you...

  10. long winded by bananaquackmoo · · Score: 1

    that article repeated itself and did little to explore the current state of beta. it could have been one page of just the developer quotes and it would have made more sense.

    1. Re:long winded by Chabo · · Score: 1

      Well, there was one good thing about the article; it may be multi-page, but at least each page has a whole standard page's worth of writing, compared with the usual one-paragraph-per-page article that's usually linked from /.

      --
      Convert FLACs to a portable format with FlacSquisher
  11. What's really funny? Cryptic's comments. by Anonymous Coward · · Score: 1, Interesting

    Their comments in the article are completely at odds with my experience with CO, and the experience of several other former closed beta testers I've spoken with.

    Their early closed beta was ugly. UGLY (Bolded, and blown up to 70 foot high glowing neon letters).

    Stuff was just nonfunctional, or intermittently functional. The game was unstable, even on hardware of exactly (or above) recommended spec. They had a whopping one zone in a semi-finished state.

    Bugs were ignored through repeated revisions. Stability issues were met with intimations that you shouldn't be on the beta and completely non-veiled threats to boot you off if you didn't just shut up, play, and give glowing feedback.

    I played through several months of release before I finally gave it the middle finger.

    It could be worlds better now. I don't care. I won't give them a dime.

  12. Duh, but good that it's being said by Anonymous Coward · · Score: 0

    This should have been (and was) obvious to most people who actually work in commercial software fields. By the time a company gets around to an open beta, they're maybe two weeks to a month from shipping the game. Chances are good that the master has already been sent off to manufacturing. They sure as hell aren't going to make functional gameplay changes at this point. The only thing that they can do now is test the network infrastructure over live conditions.

    For that to be effective, though, they actually have to take action on what they learn. Here is where Blizzard had their epic fail. When they ran the open beta for WoW, there were problems with connectivity due to the huge amount of traffic. But instead of adding more servers to deal with it, Blizzard instead reassured people that it wouldn't be like that for the "real" launch, because all the freeloaders would disappear. Then came the real launch, and a few days in, every server had queues. So, to some degree, stress testing is still only a partial motive, because the company doesn't necessarily care about the results.

    The only real motive is to drum up interest in the game.

    1. Re:Duh, but good that it's being said by fractoid · · Score: 1

      The rule of thumb is 1 to 100. Of your beta signups, only 1 in 100 of them will be online at any given time, and only 1 in 100 of THOSE will actually buy the game when it comes out. So if you have ten million beta signups, budget for roughly 1000 concurrent users on launch day.

      Blizzard managed to make WoW good enough, however, that a lot more than 1 in 100 of the beta testers went out and bought the game on launch day. And a server for 1000 concurrent users might be able to handle 2000 concurrent users if they bought a little abovespec. No matter how you slice it, though, it's not going to handle 50,000. And demand KEPT rising as they brought out new servers and people could finally get in, play, and show their friends.

      They seem pretty stable on the population front now, and have been for some time. I'd bet that most of the growth in accounts was due to their recruit-a-friend scam, though. That had everyone and their dog buying a second account to powerlevel their alts. Master move on Blizzard's part to harvest even more money, but it'd be interesting to see a drop in subscribed account numbers in the next few months, and all the "wow is dying, end of the world" waah-fest that accompanies it.

      --
      Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
    2. Re:Duh, but good that it's being said by Anonymous Coward · · Score: 0

      The rule of thumb is 1 to 100. Of your beta signups, only 1 in 100 of them will be online at any given time, and only 1 in 100 of THOSE will actually buy the game when it comes out. So if you have ten million beta signups, budget for roughly 1000 concurrent users on launch day.

      What school of math did you study where 1/100th of 10 million is 1000 ?

    3. Re:Duh, but good that it's being said by Canazza · · Score: 1

      The population is stable, but only if everyone's spread out all over the place. When the Ulduar Dungeon was released on wednesday, the server I was on, the instance servers basically died, everyone inside Ulduar (and any other instance) was kicked, and unable to log back in on those characters that were inside for about an hour. Meanwhile people who were still outside Ulduar couldn't get in, (and i'm talking hundreds of people) swarming around outside, reducing frame rates to nearly nothing.
      Their scaling & Redundancy systems work 99% of the time, but little seems to have changed since the end of open beta (when they started spawning raid bosses in the major cities and *everyone* gathered at the gates of stormwind/Orgrimarr) when exactly the same thing happened there too.

      --
      It pays to be obvious, especially if you have a reputation for being subtle.
    4. Re:Duh, but good that it's being said by fractoid · · Score: 1

      The school whereby it's 100 times as many beta signups as concurrent players, and 100 times as many concurrent beta players (since beta tests tend to have more focused, "please log on at time X" type events rather than everyone just logging on whenever) as there are sales on launch day.

      10,000,000 / (100 * 100) = 1,000. Capice?

      --
      Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
  13. Does anyone remember what "beta" meant? by swordgeek · · Score: 2, Interesting

    There was a time when alpha meant "feature-complete, but broken." Beta meant "OK, let's get the last few bugs out before release."

    Later, as code got more complex, beta usually went through a few phases. That was fine. Also, the beta testers were generally professionals, with some exceptions. (it's always a good plan to get real users who can break the unbreakable. Just make sure you don't count on them exclusively.)

    Nowadays, what companies put through 'beta testing' is rarely alpha-code. Feature complete? Maybe, maybe not. Realistically, the second release candidate onwards through the first post-release patch should properly be considered beta, because the number of products that are even usable until after the first patch are minimal.

    Ultimately, quality code doesn't pay in almost all commercial cases. Get enough untrained end-users to find the worst of the show-stopper bugs, and then release the code and start making money. Once in a long while, your product will be so bad that it falls on its face--but that's the exception, and will probably still cost less than professionally beta-testing all of your products to a high level of quality.

    --

    "People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
  14. Beta = Tribes2 (1st year) by binaryspiral · · Score: 1

    For those of you nodding your head in agreement - remember the bullshit that was Tribes2 development... endless patches and glitches, then finally after months of hell - a pretty darn fun game.

    So many games out there aren't developed enough or are rushed to market so fast to feed the hyped up masses... that the end result is the first few months of actual 1.0 is the *real* beta testing.

  15. Tester for the last 10 years by le_sean_moon · · Score: 2, Insightful

    I've beta tested plenty of games over the last ten years, but I didn't realize it until after I paid for them

  16. Funcom by k-macjapan · · Score: 1

    I beta tested Anarchy Online, it was a fairly shitty experience over all. Lot's of bug reports filed. I figured I would give the GA a try... I honestly do not think that they changed anything from the 'Beta' version. To this day I have not bought another Funcom game.

    EA Sports Madden franchise seems to be in beta for another 4 months after GA as well.

  17. Ship it ship it ship it by Anonymous Coward · · Score: 0

    Just shows how clueless the developers are. They've forgotten that the purpose of Beta is to ferret out unforseen usability issues.

    Known issues should not be released to the user base.

    You've got no business putting a product into Beta unless you would be willing to ship it as-is if you received no bug reports.

    Up to that point, all testing should be performed by paid testers under NDA.

  18. The future isn't pretty by Arimus · · Score: 2, Funny

    Just thinking to the future; when we get games with direct neural interfaces or even decent force feedback systems who will want to the beta tester then?

    One wrong function call and the word 'headache' could gain a whole new meaning...

    --
    --- Users are like bacteria -> Each one causing a thousand tiny crises until the host finally gives up and dies.
  19. Beta testing? That is SO web 1.0... by damn_registrars · · Score: 1

    At least that is what I have learned from the way code has been rolled out untested around here. Besides who needs beta testing when you already have a large base of readers who will put up with shitty code as long as the information they seek is out there somewhere?

    --
    Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
  20. flash-based MMO by Aklyon · · Score: 1

    i've only beta tested one game so far, AdventureQuest Worlds; probably the only MMO (of any kind) made completly in Adobe Flash. its fun. and funny. but you could be in the (now long over) Alpha too! both are over, though.
    http://aq.com/

    --
    I reserve the right to have a physical object so I can sell it later, and recover my money.