Slashdot Mirror
WoW Downtime Interview at Penny Arcade
The short answer is "The game is capable of supporting this many players," but it would probably be helpful to provide some background information. Based on our market analysis, we made some initial calculations about the size of the massively multiplayer online games market in the United States. We then accounted for new customers to the genre based on our previous games. Looking over this data, we did believe that there was the potential for an extremely sizable interest in a Blizzard MMOG. According to our research, other successful MMOGs in the U.S. had achieved roughly 300,000 subscribers after 12 months of operation. What ended up happening with World of Warcraft is that we achieved double these numbers in approximately the first six weeks of launch. We absolutely can support the number of copies we put on shelves, but we believed it would take us longer to get to this number in terms of players purchasing the game and logging on.
We had not anticipated this amount of growth in such a short time; however, we did have a backup plan that was deployed rapidly. In the first week of launch, we more than doubled our number of game servers and server infrastructure to accommodate the demand. The fact that we had planned to grow the service over the first 12 months of operation was evident, as we had server hardware waiting to be deployed. We just anticipated that this server rollout would be gradual. Copies of the game were being purchased at a much faster rate than anticipated, so we had to abandon our slower-paced plan and go into rapid deployment to accommodate these additional customers. This meant we also had to advance our timetable for additional server purchases.
With such a rapid growth of the network, we started to see several bottlenecks in the infrastructure that exposed themselves very quickly when the expanded hardware immediately took on massive load. These bottlenecks were solvable, but they required additional upgrades to the backend systems to accommodate the load--which, again, we hadn't planned to see, even with the extreme estimates, until later in the year. Regardless, server stability has remained our number-one priority, and so we acquired and deployed even more equipment as part of the process of addressing these issues. All of this new hardware also required additional software and operating system upgrades on the backend. The problems that some players on the 20 or so most populated servers (out of the current total of 88 servers) have been experiencing are related to some of the upgrades not functioning as desired. We are working diligently with our vendors and internal technical staff to get as quick of a resolution to the problems as possible, and we believe there should be noticeable improvements soon. When our community team commented that people are working 24/7, they weren't exaggerating.
2. If it's true that the server problems are related to the overwhelming number of players, why was no effort made to better distribute players evenly across realms, or allow players and guilds to transfer to less populated servers?
We actually did have a number of checks in place at launch to distribute players as evenly as possible across realms. When a new account logged in, the game would ask what realm rule set and time zone the player preferred, and then it would suggest the realm with the lowest population that matched the selected preferences. That said, we're definitely working on resolving the overpopulation problems that ended up occurring on some realms despite our preventative measures. A realm-transfer option that would allow players to move from their high-population realm to one with a low population is one of the things we're investigating. We're exploring this option fully and hope to be able to communicate more detailed information about it to our customers in the coming weeks.
3. Currently, large scale player raids involving large groups of players experience a huge amount of latency. How do you plan to compensate for this in your upcoming PvP Battlegrounds feature?
The player raids often have hundreds of people per side in one area; that area is on a server that is also running the rest of the continent, and that can result in the latency you describe--depending, as well, on the total population of that server. We're continuing to look into the issues surrounding this dip in performance. Battlegrounds, on the other hand, will run on the instance server, so there should be no such issues. Additionally, players will be unable to "zerg" in Battlegrounds; there will be a limit to the number of players per side.
4. What accounts for the frequent "emergency" maintenance downtime? What issues are you attempting to resolve?
The emergency maintenance periods are to restore stability while we continue to narrow down the cause of the problems. Some of them are also to deploy temporary fixes to various in-game systems while we continue to develop a longer term, more stable solution. World of Warcraft delivers many complex features that are unique to MMOGs. Features such as the in-game mail system, auction houses, player inventories, flight paths, quest states, etc. use a lot of server bandwidth, which makes pinpointing problems on the server infrastructure much more complicated.
Recently, the extended emergency downtime for a certain number of realms was needed in order to better accommodate our growing player base. Some of the upgrades that we planned for all of the realms were made to these realms first, as they are among the most populated and thus most in need of aid. We set the realms up on the latest top-of-the-line hardware and made the software upgrades accordingly, but some unforeseen issues cropped up with the database that resulted in the problems players currently see. This is no fun for our player base, of course, and we don't want to keep the realms running in a condition that frustrates our customers when we can attempt to fix things . So, these downtimes have been used to change hardware and apply fixes that will hopefully alleviate the issues. We have not yet resolved the problems, but we're working on this around the clock.
5. What issues are you experiencing with your login/authentication servers? It is often the case for myself and the people I play with that we cannot access realms our friends are already logged into.
These types of issues stem from the problems described above. Conflicts occur between some of the internal applications running in the background, and the end result can take the form of temporary login issues. We're working to resolve these conflicts so that they are no longer a factor.
6. When do you expect to have the worst of these problems resolved?
We'll be constantly working on these issues each day moving forward until they're resolved, but we don't currently have a set date for when that will be. We're doing all we can to make sure these problems no longer occur -- it's our top priority, and we hope to have the issues fixed as soon as possible. We'll continue to provide players with regular updates on our progress.
7. Will the European launch utilize the same realms, or will these players be hosted on all new equipment? If they are hosted on new servers, what have you done to ensure that the launch will be free of the problems mentioned above?
They will be on their own set of hardware, as with our Korean release. Our teams are learning from the experience of our North American launch and are applying that knowledge to the servers in Europe. We hope to provide them with a smooth launch.
8. What would you have done differently?
It would be easy to speculate about what we could have done differently, but that wouldn't turn back the clock. Right now we're extremely focused on the issues at hand, and this focus is helping us methodically chase down the problems that are causing frustration for some of our players. The foundation of our company is based on providing a top-notch game experience and an equally top-notch level of customer satisfaction; we won't be happy until we feel we're consistently meeting those standards.
No, it's commonly used by players, as well. It means you get rushed by a bunch of players. If this were the spelling bee, I'd say:
Nation of Origin: English, (more likely Korean)
Definition: To be rushed by several players.
Use it in a sentence: "So we were attacking Tarren Mill in Hillsbrad when the hordies called in all their guild mates and zerged us."
We don't have a state-run media we have a media-run state.
The people at Penny Arcade act like they are investigating a space shuttle disaster. It would be nice to see what level of respect the people who play games 24/7 and actually care about intermittent downtime would give a customer paying them $14 / month. I bet most of them would tell the customer to "F off you n00b".
It is a $14 a month service for unlimited entertainment, you can't expect that every single kink will be ironed out at launch.
Real world devices have real world problems, and a whole host of gamers, like Tycho fail to realize that.
Imagine if Tycho had to deal with 100,000's of people complaining that one stroke in one particular comic was 1 pt off.
The problem is that if Blizzard put the resources into making the game so that there were no problems at launch and that they had the server infrastructure to support the entire planet logging into same screen all at once the subscription fee would be so incredibly expensive that no one would play the game.
Manufacturing defects are a trade off, yes Blizzard could build a game with no bugs, but how many players would want to pay $5,000 for a copy, and $1,000 a month?
Translation: Bob's nifty 12 line perl hack doesn't seem to scale.
Sometimes my arms bend back.
1. You say that you sold six hundred thousand units. Is the game not capable of supporting this many users?
No it isn't. We had an untested backup plan that did not work either.
2. If it's true that the server problems are related to the overwhelming number of players, why was no effort made to better distribute players evenly across realms, or allow players and guilds to transfer to less populated servers?
We recommended a lower population server, but did not explain the implications of the choice, so people just made uninformed desicions. (Note: I would take a higher populated server if I didn't know it meant more lag. Do you want to play in a void??)
3. Currently, large scale player raids involving large groups of players experience a huge amount of latency. How do you plan to compensate for this in your upcoming PvP Battlegrounds feature?
Battlegrounds won't lag because we limit the number of people in a battle.
4. What accounts for the frequent "emergency" maintenance downtime? What issues are you attempting to resolve?
Once again we did not compensate for the high number of users. Do you think we had 600k people in our test environment?
5. What issues are you experiencing with your login/authentication servers? It is often the case for myself and the people I play with that we cannot access realms our friends are already logged into.
Once again we did not compensate for the high number of users. Do you think we had 600k people in our test environment?
6. When do you expect to have the worst of these problems resolved?
When enough people get pissed and leave and we finally upgrade. So in about 6 months.
7. Will the European launch utilize the same realms, or will these players be hosted on all new equipment? If they are hosted on new servers, what have you done to ensure that the launch will be free of the problems mentioned above?
We're not sure.
8. What would you have done differently?
Better market research, I guess. We'll get it fixed, don't worry.
In respectful defense of our colleagues at Blizzard, I think it's safe to say there's virtually no way to have predicted the kind of volume they saw. My hat's off to them for having 88 servers ready to go around launch time. Despite all of the servers, and the huge open beta they ran.. the problems they are seeing simply don't surface until you see scale on the order of what they're seeing. In my opinion they are doing right by their customers and are being respectful their players downtime by adding free time. In this business, you can spend a lot of time planning.. and even a lot of time in beta only to find some crucial piece of technology doesn't scale as well as advertised. I for one appreciate and respect the way they're dealing with this situation. I realize the frustration of their customers.. but I think it's safe to say that the Blizzard team is filled with stand-up people that are just focused on fixing the problems as quickly as possible. John Smedley President, Sony Online Entertainment
John Smedley President, Sony Online Entertainment
Having been an MMO play for a long time, they won't have to find something new to complain about. A few examples (Copy/pastes from the Ashen Empires forums, mind you):
Example 1 (Context: SoE is the Staff of Enervation, the most powerful mage staff in the game. It's magical wood, meaning it can be repaired by a Mind based repair spell. A database error during the shift from TKO Software back to private ownership made it Nature repair - which is for magic wood - a few weeks ago, and that was corrected shortly tehreafter. This post came a while after that)
WTF SOE IS MIND REPAIR NOW???
by (removed)
YESTERDA YIT WAS NATURE REPAIR WTF U CHANGE IT FOR?
by (removed)
I sure hope not that's gay I don't want to have another skill. Well not really I have 100 in all mage skills but its gay because I need to use my mind runes now.
by (removed)
WTF FIX THE FUCKING BUGS BEFORE YOU POST A NEW UPDATE
Another example:(Context: New land area was added recently, which is only suitable for parties of high level characters. A level 15 should not be there for any reason. The area is open, and is accessible from both lawful and criminal cities, although the walk from Desprail or Redwake makes it hard on criminals, but just about anybody can get there if they want)
Re: WHY do i see CRIMINALS running free in the new lands?
by (removed)
what city? there's no city, just a lot of grass and big ogre things that kill me. why the hell do the monsters so fucking high level? hows a level 15 supposed to bow in there geeze its not fare nemore
The following posts are by the same person, less than four hours apart: (Context: the update was scheduled for the day before, but was delayed due to a bug)
Where's the UPDATE????
by (removed)
Just put it on the servers so we can play it and we'll find the bugs stop slacking we pay your salary.
Wtf's with the fucking BUGS!?
by (removed)
FIX THE FUCKING BUGS before you post a patch. you should NEVER post any update unless it is ONE HUNDRED PERSENT BUG FREE goddamn incompetents. you can't run a fucking game. We pay your salary now do your jobs.
Frankly, in this game, I consider this sort of brattyness worse. The game is fairly small, and has been struggling to pay the bills its whole existence. Twice, they licensed larger firms (most recently TKO Software), and both times they got screwed. The first time, the developers broke the contract, gave up ALL the money the game had made for nearly a year, and went without a lot (Lead developer even sold his car) to keep the game alive. The second time, it was a bit worse. They all got canned by the new owner of the game, and the game was going to be shut down. They stepped in and paid - out of their own money - to license the game back and keep it alive. For more than half of the last two years, the game has been free for one reason or another, and for the rest, it was only $8 a month.
Granted Blizzard doesn't have these woes, but people will still complain about the exact same things. Even if they're shown that the problems are being fixed, it won't be enough, because it's not being fixed instantaneously.
Once, AE had a very strange bug. It was very difficult to reproduce, and the circumstances that caused it were never properly pinned down. Less than three hours after it was found (At 4 AM yet, the devs were probably asleep), people were already complaining about why there hadn't been an update to fix it. When there was an update, they complained that the server was down for almost "FIFTEEN MINUTES OMG I R WANT MI MONIE BAK."