Slashdot Mirror
WoW Downtime Interview at Penny Arcade
The short answer is "The game is capable of supporting this many players," but it would probably be helpful to provide some background information. Based on our market analysis, we made some initial calculations about the size of the massively multiplayer online games market in the United States. We then accounted for new customers to the genre based on our previous games. Looking over this data, we did believe that there was the potential for an extremely sizable interest in a Blizzard MMOG. According to our research, other successful MMOGs in the U.S. had achieved roughly 300,000 subscribers after 12 months of operation. What ended up happening with World of Warcraft is that we achieved double these numbers in approximately the first six weeks of launch. We absolutely can support the number of copies we put on shelves, but we believed it would take us longer to get to this number in terms of players purchasing the game and logging on.
We had not anticipated this amount of growth in such a short time; however, we did have a backup plan that was deployed rapidly. In the first week of launch, we more than doubled our number of game servers and server infrastructure to accommodate the demand. The fact that we had planned to grow the service over the first 12 months of operation was evident, as we had server hardware waiting to be deployed. We just anticipated that this server rollout would be gradual. Copies of the game were being purchased at a much faster rate than anticipated, so we had to abandon our slower-paced plan and go into rapid deployment to accommodate these additional customers. This meant we also had to advance our timetable for additional server purchases.
With such a rapid growth of the network, we started to see several bottlenecks in the infrastructure that exposed themselves very quickly when the expanded hardware immediately took on massive load. These bottlenecks were solvable, but they required additional upgrades to the backend systems to accommodate the load--which, again, we hadn't planned to see, even with the extreme estimates, until later in the year. Regardless, server stability has remained our number-one priority, and so we acquired and deployed even more equipment as part of the process of addressing these issues. All of this new hardware also required additional software and operating system upgrades on the backend. The problems that some players on the 20 or so most populated servers (out of the current total of 88 servers) have been experiencing are related to some of the upgrades not functioning as desired. We are working diligently with our vendors and internal technical staff to get as quick of a resolution to the problems as possible, and we believe there should be noticeable improvements soon. When our community team commented that people are working 24/7, they weren't exaggerating.
2. If it's true that the server problems are related to the overwhelming number of players, why was no effort made to better distribute players evenly across realms, or allow players and guilds to transfer to less populated servers?
We actually did have a number of checks in place at launch to distribute players as evenly as possible across realms. When a new account logged in, the game would ask what realm rule set and time zone the player preferred, and then it would suggest the realm with the lowest population that matched the selected preferences. That said, we're definitely working on resolving the overpopulation problems that ended up occurring on some realms despite our preventative measures. A realm-transfer option that would allow players to move from their high-population realm to one with a low population is one of the things we're investigating. We're exploring this option fully and hope to be able to communicate more detailed information about it to our customers in the coming weeks.
3. Currently, large scale player raids involving large groups of players experience a huge amount of latency. How do you plan to compensate for this in your upcoming PvP Battlegrounds feature?
The player raids often have hundreds of people per side in one area; that area is on a server that is also running the rest of the continent, and that can result in the latency you describe--depending, as well, on the total population of that server. We're continuing to look into the issues surrounding this dip in performance. Battlegrounds, on the other hand, will run on the instance server, so there should be no such issues. Additionally, players will be unable to "zerg" in Battlegrounds; there will be a limit to the number of players per side.
4. What accounts for the frequent "emergency" maintenance downtime? What issues are you attempting to resolve?
The emergency maintenance periods are to restore stability while we continue to narrow down the cause of the problems. Some of them are also to deploy temporary fixes to various in-game systems while we continue to develop a longer term, more stable solution. World of Warcraft delivers many complex features that are unique to MMOGs. Features such as the in-game mail system, auction houses, player inventories, flight paths, quest states, etc. use a lot of server bandwidth, which makes pinpointing problems on the server infrastructure much more complicated.
Recently, the extended emergency downtime for a certain number of realms was needed in order to better accommodate our growing player base. Some of the upgrades that we planned for all of the realms were made to these realms first, as they are among the most populated and thus most in need of aid. We set the realms up on the latest top-of-the-line hardware and made the software upgrades accordingly, but some unforeseen issues cropped up with the database that resulted in the problems players currently see. This is no fun for our player base, of course, and we don't want to keep the realms running in a condition that frustrates our customers when we can attempt to fix things . So, these downtimes have been used to change hardware and apply fixes that will hopefully alleviate the issues. We have not yet resolved the problems, but we're working on this around the clock.
5. What issues are you experiencing with your login/authentication servers? It is often the case for myself and the people I play with that we cannot access realms our friends are already logged into.
These types of issues stem from the problems described above. Conflicts occur between some of the internal applications running in the background, and the end result can take the form of temporary login issues. We're working to resolve these conflicts so that they are no longer a factor.
6. When do you expect to have the worst of these problems resolved?
We'll be constantly working on these issues each day moving forward until they're resolved, but we don't currently have a set date for when that will be. We're doing all we can to make sure these problems no longer occur -- it's our top priority, and we hope to have the issues fixed as soon as possible. We'll continue to provide players with regular updates on our progress.
7. Will the European launch utilize the same realms, or will these players be hosted on all new equipment? If they are hosted on new servers, what have you done to ensure that the launch will be free of the problems mentioned above?
They will be on their own set of hardware, as with our Korean release. Our teams are learning from the experience of our North American launch and are applying that knowledge to the servers in Europe. We hope to provide them with a smooth launch.
8. What would you have done differently?
It would be easy to speculate about what we could have done differently, but that wouldn't turn back the clock. Right now we're extremely focused on the issues at hand, and this focus is helping us methodically chase down the problems that are causing frustration for some of our players. The foundation of our company is based on providing a top-notch game experience and an equally top-notch level of customer satisfaction; we won't be happy until we feel we're consistently meeting those standards.
No, it's commonly used by players, as well. It means you get rushed by a bunch of players. If this were the spelling bee, I'd say:
Nation of Origin: English, (more likely Korean)
Definition: To be rushed by several players.
Use it in a sentence: "So we were attacking Tarren Mill in Hillsbrad when the hordies called in all their guild mates and zerged us."
We don't have a state-run media we have a media-run state.
Nope, "zerging" has become a gaming term for massive assaults that overwhelm the opponent by shear numbers.
WWII Online
Check em out
You're acting like this is just a single customer out of the many that is interrogating Blizzard, as if to ask "how would you like it if every customer did this to you for $14/month?". Correct me if I'm wrong on that. In response though, this is not just a customer, this is a well known site that a lot of people get a good dose of their gaming information from. Blizzard knows this and hence why they have responded at length and with an air of (sometimes painful) honesty.
...or having a phone line that you might be able to call someone on... once in a while. If this were just a case of bugs like, an NPC not giving you the quest you need or items not working in game, but you could still play the game and enjoy the rest of it... I don't think people would complain 1/10th as much as they are.
This has already been addressed in other replies, but I'll state it again here. Yes, real world devices have real world problems, and when real world services go down, people get pissed off, switch services, ask for refunds, reimbursements, pro-rated bills, etc.
100,00's of people complaining that one stroke in a particular comic was 1 pt off would be like 100,000's of people complaining that a building in the game was misaligned slightly, but still compeltely usable. What people are really complaining about in reference to Blizzard and WoW is not being able to access the service at all... like if people were PAYING to see Penny Arcade comics and then the site was down all the time and people couldn't see the comics or the site would be up, but viewing the comics required a login and it rejected all paying customers.
If you had bothered to read the article this post is referring to (which you obviously can't, with your head so far up your ass), you'd know that the problem had nothing to do with not spending enough money on the server infrastructure and more to do with Blizzard's extensive market research showing that even in very generous, extreme estimates, the number of people signed up to play this game would be less than HALF of what it really turned out to be.
Again, I just want to stress that we're not talking about annoying software bugs here, we're talking about not being able to use the service they are paying for. It's like buying an operating system that has a 50/50 chance of booting. >.>
Personally, I LOVE this game and like Tycho's previous posts about the game, the only thing that makes me really really mad is that I can't play! I have NO intention of returning the game. I just want the server instability (putting it lightly) to go away, so I can enjoy everything else about the game.
Access to a product that you pay a sign up fee and monthly fees for is not a priveledge, but part of the contract you enter into when you agree to pay them every month for the service they have told you you were getting for that monthly fee.
Yes. This has been known about for a long time, ever since the beta. If you want to play with folks in Europe you need to either buy a European copy which connects to the Europe server or have your friends in Europe buy a NA copy which will connect to the server in NA.
Q.
I doubt uptime has any effect on advertising -- it's usually charged by page views or clickthroughs. If the site is down for an hour, then they missed out on an hour's worth of pageviews to charge for.
Recursive: Adj. See Recursive.
Well, the way WoW's servers are set up is kind of odd, each "shard" is actually three seperate pieces of hardware. There's one for Kalimdor, one for the Eastern Kingdoms, and one for all the instance dungeons. Battlegrounds will be run on the instance server. This is also why you have a load screen going into places like the stockade.... it's not so much to load the textures and geometry as to cover up the fact that you're jumping servers.