Slashdot Mirror
WoW Downtime Interview at Penny Arcade
The short answer is "The game is capable of supporting this many players," but it would probably be helpful to provide some background information. Based on our market analysis, we made some initial calculations about the size of the massively multiplayer online games market in the United States. We then accounted for new customers to the genre based on our previous games. Looking over this data, we did believe that there was the potential for an extremely sizable interest in a Blizzard MMOG. According to our research, other successful MMOGs in the U.S. had achieved roughly 300,000 subscribers after 12 months of operation. What ended up happening with World of Warcraft is that we achieved double these numbers in approximately the first six weeks of launch. We absolutely can support the number of copies we put on shelves, but we believed it would take us longer to get to this number in terms of players purchasing the game and logging on.
We had not anticipated this amount of growth in such a short time; however, we did have a backup plan that was deployed rapidly. In the first week of launch, we more than doubled our number of game servers and server infrastructure to accommodate the demand. The fact that we had planned to grow the service over the first 12 months of operation was evident, as we had server hardware waiting to be deployed. We just anticipated that this server rollout would be gradual. Copies of the game were being purchased at a much faster rate than anticipated, so we had to abandon our slower-paced plan and go into rapid deployment to accommodate these additional customers. This meant we also had to advance our timetable for additional server purchases.
With such a rapid growth of the network, we started to see several bottlenecks in the infrastructure that exposed themselves very quickly when the expanded hardware immediately took on massive load. These bottlenecks were solvable, but they required additional upgrades to the backend systems to accommodate the load--which, again, we hadn't planned to see, even with the extreme estimates, until later in the year. Regardless, server stability has remained our number-one priority, and so we acquired and deployed even more equipment as part of the process of addressing these issues. All of this new hardware also required additional software and operating system upgrades on the backend. The problems that some players on the 20 or so most populated servers (out of the current total of 88 servers) have been experiencing are related to some of the upgrades not functioning as desired. We are working diligently with our vendors and internal technical staff to get as quick of a resolution to the problems as possible, and we believe there should be noticeable improvements soon. When our community team commented that people are working 24/7, they weren't exaggerating.
2. If it's true that the server problems are related to the overwhelming number of players, why was no effort made to better distribute players evenly across realms, or allow players and guilds to transfer to less populated servers?
We actually did have a number of checks in place at launch to distribute players as evenly as possible across realms. When a new account logged in, the game would ask what realm rule set and time zone the player preferred, and then it would suggest the realm with the lowest population that matched the selected preferences. That said, we're definitely working on resolving the overpopulation problems that ended up occurring on some realms despite our preventative measures. A realm-transfer option that would allow players to move from their high-population realm to one with a low population is one of the things we're investigating. We're exploring this option fully and hope to be able to communicate more detailed information about it to our customers in the coming weeks.
3. Currently, large scale player raids involving large groups of players experience a huge amount of latency. How do you plan to compensate for this in your upcoming PvP Battlegrounds feature?
The player raids often have hundreds of people per side in one area; that area is on a server that is also running the rest of the continent, and that can result in the latency you describe--depending, as well, on the total population of that server. We're continuing to look into the issues surrounding this dip in performance. Battlegrounds, on the other hand, will run on the instance server, so there should be no such issues. Additionally, players will be unable to "zerg" in Battlegrounds; there will be a limit to the number of players per side.
4. What accounts for the frequent "emergency" maintenance downtime? What issues are you attempting to resolve?
The emergency maintenance periods are to restore stability while we continue to narrow down the cause of the problems. Some of them are also to deploy temporary fixes to various in-game systems while we continue to develop a longer term, more stable solution. World of Warcraft delivers many complex features that are unique to MMOGs. Features such as the in-game mail system, auction houses, player inventories, flight paths, quest states, etc. use a lot of server bandwidth, which makes pinpointing problems on the server infrastructure much more complicated.
Recently, the extended emergency downtime for a certain number of realms was needed in order to better accommodate our growing player base. Some of the upgrades that we planned for all of the realms were made to these realms first, as they are among the most populated and thus most in need of aid. We set the realms up on the latest top-of-the-line hardware and made the software upgrades accordingly, but some unforeseen issues cropped up with the database that resulted in the problems players currently see. This is no fun for our player base, of course, and we don't want to keep the realms running in a condition that frustrates our customers when we can attempt to fix things . So, these downtimes have been used to change hardware and apply fixes that will hopefully alleviate the issues. We have not yet resolved the problems, but we're working on this around the clock.
5. What issues are you experiencing with your login/authentication servers? It is often the case for myself and the people I play with that we cannot access realms our friends are already logged into.
These types of issues stem from the problems described above. Conflicts occur between some of the internal applications running in the background, and the end result can take the form of temporary login issues. We're working to resolve these conflicts so that they are no longer a factor.
6. When do you expect to have the worst of these problems resolved?
We'll be constantly working on these issues each day moving forward until they're resolved, but we don't currently have a set date for when that will be. We're doing all we can to make sure these problems no longer occur -- it's our top priority, and we hope to have the issues fixed as soon as possible. We'll continue to provide players with regular updates on our progress.
7. Will the European launch utilize the same realms, or will these players be hosted on all new equipment? If they are hosted on new servers, what have you done to ensure that the launch will be free of the problems mentioned above?
They will be on their own set of hardware, as with our Korean release. Our teams are learning from the experience of our North American launch and are applying that knowledge to the servers in Europe. We hope to provide them with a smooth launch.
8. What would you have done differently?
It would be easy to speculate about what we could have done differently, but that wouldn't turn back the clock. Right now we're extremely focused on the issues at hand, and this focus is helping us methodically chase down the problems that are causing frustration for some of our players. The foundation of our company is based on providing a top-notch game experience and an equally top-notch level of customer satisfaction; we won't be happy until we feel we're consistently meeting those standards.
I got server for you!
Don't blame me, I voted for Durga.
Bout time they got round to this. Kudos for Blizz. for using the word Zerg in something about WoW. Freudian slip?
cmd-q.co.uk - some sort of stupid fucking internet bullshit
In addition to games, politics, yro, etc it seems there should be a WoW section, surely it would be more popular than other sections, e.g. BSD, for as we all know BSD is dying.
The people at Penny Arcade act like they are investigating a space shuttle disaster. It would be nice to see what level of respect the people who play games 24/7 and actually care about intermittent downtime would give a customer paying them $14 / month. I bet most of them would tell the customer to "F off you n00b".
It is a $14 a month service for unlimited entertainment, you can't expect that every single kink will be ironed out at launch.
Real world devices have real world problems, and a whole host of gamers, like Tycho fail to realize that.
Imagine if Tycho had to deal with 100,000's of people complaining that one stroke in one particular comic was 1 pt off.
The problem is that if Blizzard put the resources into making the game so that there were no problems at launch and that they had the server infrastructure to support the entire planet logging into same screen all at once the subscription fee would be so incredibly expensive that no one would play the game.
Manufacturing defects are a trade off, yes Blizzard could build a game with no bugs, but how many players would want to pay $5,000 for a copy, and $1,000 a month?
Blizzard actually made an effort to let the community know what was going on... Now the forum trolls will have to find something else to whine about. Something tells me it won't be pretty.
...I wonder why they never moved beyond the "realm" concept and put together a single, huge, continuous world (where different areas would be physically located on separate servers, even with several servers serving a single area or sharing area serving wherever needed).
As far as I know, the only MMOGame that ever attempted that approach was EVE-Online, and they have a record of a bit over 10,000 clients logged on and playing at the same time.
Personally, I find the entire concept of realm "sharding" to be archaic and absolete.
By reading this signature you agree to not disagree with the post you just read.
Translation: Bob's nifty 12 line perl hack doesn't seem to scale.
Sometimes my arms bend back.
They shipped an estimated 2 years worth of product at release? Even if you expected to be more popular than the other MMORPG, it seems silly that you would create so much, when it would be considerate to remaster the game at some point so that new users aren't forced to download a huge patch.
It sounds like the answers are spin for we weren't ready to handle a game of this scale. We maybe soon, but you all are going to have to put up with us through the learning process.
By the way they had to release early not because of their publisher, but they needed to release near EQ2 or lose lots of potential subscribers.
whether or not Starcraft: Ghost will be what they originally portrayed, a spiritual successor/equivalent to BattleZone 2, or if it's just going to be yet another crappy FPS.
If you can read this sig, congratulations, you have your glasses on!
that Blizzard hired the president's speech writers? "Well we don't dwell on the past and are only concerned with the future. Mistakes? I see no mistakes here. These aren't the droids you're looking for. Move along" - Long time geek, first time poster.
"It's difficult to meditate on amphetamines." - Joe Walsh
It's great that they responded and all; but I have no idea why I keep expecting something different from an MMO company response when we always get the same stuff.
It's what we already knew: They 'could' have supported many players, they 'tried' to evenly distribute players and they 'will' fix the problems at some unknown date.
I'm not sure what I was expecting, but a watered-down version of what we already know that doesn't cast any blame in the slightest wasn't it. Where is that 'Blizzard Difference' everybode keeps ranting about?
Does anybody know the specifics of the hardware they use, and what specific problems they are having? All the responses seem to be vaguely worded. I imagine that the majority of WoW players are technically minded, and want to know the specs and problems......
1. You say that you sold six hundred thousand units. Is the game not capable of supporting this many users?
No it isn't. We had an untested backup plan that did not work either.
2. If it's true that the server problems are related to the overwhelming number of players, why was no effort made to better distribute players evenly across realms, or allow players and guilds to transfer to less populated servers?
We recommended a lower population server, but did not explain the implications of the choice, so people just made uninformed desicions. (Note: I would take a higher populated server if I didn't know it meant more lag. Do you want to play in a void??)
3. Currently, large scale player raids involving large groups of players experience a huge amount of latency. How do you plan to compensate for this in your upcoming PvP Battlegrounds feature?
Battlegrounds won't lag because we limit the number of people in a battle.
4. What accounts for the frequent "emergency" maintenance downtime? What issues are you attempting to resolve?
Once again we did not compensate for the high number of users. Do you think we had 600k people in our test environment?
5. What issues are you experiencing with your login/authentication servers? It is often the case for myself and the people I play with that we cannot access realms our friends are already logged into.
Once again we did not compensate for the high number of users. Do you think we had 600k people in our test environment?
6. When do you expect to have the worst of these problems resolved?
When enough people get pissed and leave and we finally upgrade. So in about 6 months.
7. Will the European launch utilize the same realms, or will these players be hosted on all new equipment? If they are hosted on new servers, what have you done to ensure that the launch will be free of the problems mentioned above?
We're not sure.
8. What would you have done differently?
Better market research, I guess. We'll get it fixed, don't worry.
I thought WoW had like 500K beta accounts/testers?
The bean counters were either sleeping or stupid.
How could they NOT have seen this coming?
Are they are full of BS?
or more likely:
This is an example of a large corporation with very slow turning wheels when it comes to planning/changing plans.
I give them credit for having a backup plan in the first place... but personally, I expect MORE of Blizzard. So many beta accounts should have been an obvious indicator for what was in store, I would think..?
Now someone needs to do an interview with Tycho and Gabe about why the penny-arcade.com forums can't seem to stay available for longer than fifteen minutes at a time.
"Infants flesh will be in season throughout the year." -Swift
7. Will the European launch utilize the same realms ?
They will be on their own set of hardware, as with our Korean release. Our teams are learning from the experience of our North American launch and are applying that knowledge to the servers in Europe. We hope to provide them with a smooth launch.
Does this means people in north america won't be able to play with our europeans friends?Gee, a critical response in a discussion - (You know, the other side of the argument?) gets modded flamebait because the moderator doesn't agree with the opinion. Such a surprise.
Yes. This has been known about for a long time, ever since the beta. If you want to play with folks in Europe you need to either buy a European copy which connects to the Europe server or have your friends in Europe buy a NA copy which will connect to the server in NA.
Q.
This reminds me of the Star Wars: Galaxies roll out on June 26th, 2003 (if I remember the date correctly). I had my copy purchased, and installed, patched, and clicked on that 'Connect' button, and all of the servers were offline. Every single one. Apparently they severely underestimated the number of players who would be logging in that first day, and for some reason, nobody could.
Their answer? They gave folks 1 extra day of play time before their subscriptions had to be renewed.
People were complaining about service on the first day. However, the fact remains. Regardless of studies, analyses, etc., nobody can really know whats going to happen on the first day of a huge rollout like this. There's no possible way to test it adequately. Even the beta testers for SWG said that it was never that bad for them, but then again there was only a fraction of beta testers compared to the number of players on rollout day.
I ended up playing for a long time, finally got on the path to Jedi, then once I figured out it would have taken me like 7 months just to BECOME a jedi (let alone progress through the Jedi ranks) I decided I'd rather have the $15 a month.
Oh, those were the days. Now I just play Starcraft and America's Army (A free mmorpg!)
And they said zombies weren't real!
This is blizz's learning curve - they've never made a MMORPG before. Who in God's name could have predicted this kind of response? 600,000 people!!!
Bottom line - All MMORPG's take time - balance, cheating, whatever. Hasn't been one yet that didn't have a rocky launch, and I don't want to play one where the dev's think their game is perfect out the door. At least bliz is familiar with processes to fix games after their release.
As for the ppl who are crying that they'll drop their accounts... Drop away. It'll free up the servers for the rest of us, and I really dont like the sound of your whiny voice. If you are dissatisfied with a product, stop paying, voice your opinion to the company, and tell ONLY THOSE PEOPLE WHO ASK FOR IT your opinion. Thx in advance :)
Physics is nothing like religion. If it was, we'd have an easier time trying to raise money!
This is not flamebait. It is a critical analysis of responses laden with evasive language. The game might be capable of handling all the players but they did not plan correctly and blew it. The poster's comment on #2 is DEAD ON. #6 is probable. #7 is accurate. #8 is a great summary as well, somehow blizzard failed to realize that WoW was just about the most anticipated game of the year, maybe after Halo 2. "Here is the pulse, and here is your finger, far from the pulse, shoved straight up your ass."
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
I think Blizzard got a double whammy of success - 600,000 new players in two months and players loving the game so much that they play every night.
One thing that Blizzard touted early was the ability to play it casually soloing - you can play once a week and easily gain levels doing quests. By following the flow of quests, you can often do two in an hour. However, that rapid pace of advancement makes the game fun in general. You get rewarded with new skills, new environments and new challenges constantly. So rather than having a large number of casual gamers, folks are on every night hooked on the next quest.
I was not a MMORPG player previously. I bought the game shortly after it came out. I quickly found myself sinking 4-5 nights a week into - sometimes only an hour at a time - but still several nights. Around Christmas, some friends were intrigued and they bought it as well. One of them is now on every night - and I do mean EVERY night - for several hours at a time. Another friend of mine spent most of the first weekend online after he bought it.
Because you can sit down and play for just an hour, I think many people do just that. However, that side quest adds on another half hour, then you have to go to the auction house, then you chat with someone who needs help with another quest, etc. In an odd way, I think the quick pacing actually encourages people to play more rather than plan out a few sessions.
Actually, i THINK yo may simply be able to edit one text file which determines where the realm list is downloaded from. This may, ofc, not work as with warcraft 3 the server list was checked by the exe and if it was changed you couldn't play. Why they'd allow the server list to be changed in the registry/ini file then lock it in the exe is beyond me, however, Why not just hard-code the servers into the exe?
This is a joke. I am joking. Joke joke joke.
I can't think of a better way to handle the situation than Blizzard has so far. They're a business, not a Entity of Gaming Awesomeness that has a crystal ball to see market demand in the future. They did some research, saw the numbers, and bought the number of servers that they thought they'd need. When a problem came up, they let everyone know, allowed another free month to some, are working hard to fix the problem, while still keeping in touch with the public with what's going on.
What more do people want? They have tried their best based on what they know, and when things went wrong, they responded very quickly. Several MMOs have problems and are dickheads about it.
The fact that Penny Arcade yanked the Game of the Year award away from WoW is just immature, in my opinion, given Blizzard's response to the situation.
By the way, I've been noticing that Penny Arcade takes a shitton of time to load. I demand they fix the problem instantaneously! I don't care what it takes! I'm going to yank their links from my site unless they get on it right now! I'm sending "terse questions" next week.
1. You say that you sold six hundred thousand units. Is the game not capable of supporting this many users?
The short answer is "The game is capable of supporting this many players," but it would probably be helpful to provide some background information. Based on our market analysis, we made some initial calculations about the size of the massively multiplayer online games market in the United States. We then accounted for new customers to the genre based on our previous games. Looking over this data, we did believe that there was the potential for an extremely sizable interest in a Blizzard MMOG. According to our research, other successful MMOGs in the U.S. had achieved roughly 300,000 subscribers after 12 months of operation. What ended up happening with World of Warcraft is that we achieved double these numbers in approximately the first six weeks of launch. We absolutely can support the number of copies we put on shelves, but we believed it would take us longer to get to this number in terms of players purchasing the game and logging on.
We had not anticipated this amount of growth in such a short time; however, we did have a backup plan that was deployed rapidly. In the first week of launch, we more than doubled our number of game servers and server infrastructure to accommodate the demand. The fact that we had planned to grow the service over the first 12 months of operation was evident, as we had server hardware waiting to be deployed. We just anticipated that this server rollout would be gradual. Copies of the game were being purchased at a much faster rate than anticipated, so we had to abandon our slower-paced plan and go into rapid deployment to accommodate these additional customers. This meant we also had to advance our timetable for additional server purchases.
With such a rapid growth of the network, we started to see several bottlenecks in the infrastructure that exposed themselves very quickly when the expanded hardware immediately took on massive load. These bottlenecks were solvable, but they required additional upgrades to the backend systems to accommodate the load--which, again, we hadn't planned to see, even with the extreme estimates, until later in the year. Regardless, server stability has remained our number-one priority, and so we acquired and deployed even more equipment as part of the process of addressing these issues. All of this new hardware also required additional software and operating system upgrades on the backend. The problems that some players on the 20 or so most populated servers (out of the current total of 88 servers) have been experiencing are related to some of the upgrades not functioning as desired. We are working diligently with our vendors and internal technical staff to get as quick of a resolution to the problems as possible, and we believe there should be noticeable improvements soon. When our community team commented that people are working 24/7, they weren't exaggerating.
This is a total utter lie, i played the beta and the same problems still exist in the retail that they had in the beta. The only exception to this is that now the problems are worse.
2. If it's true that the server problems are related to the overwhelming number of players, why was no effort made to better distribute players evenly across realms, or allow players and guilds to transfer to less populated servers?
We actually did have a number of checks in place at launch to distribute players as evenly as possible across realms. When a new account logged in, the game would ask what realm rule set and time zone the player preferred, and then it would suggest the realm with the lowest population that matched the selected preferences. That said, we're definitely working on resolving the overpopulation problems that ended up occurring on some realms despite our preventative measures. A realm-transfer option that would allow pla
http://www.DaveNet.biz/
One thing that people don't often mention is that, while the lag was getting pretty terrible, the game was still mostly playable. It might take several minutes(!) for an Auction House query to return, or for an auction to be created, or for your email to show up, but the transactions DID eventually happen.
:-)
Instead of just blowing up completely, by and large, WOW fails fairly gracefully. The engineering in that is non-trivial. I don't think people realize just how good that code is. Getting a system that stays reasonably stable under completely untested loads is really, really hard. I am VERY impressed with the quality of their design and code work.
And, yes, some data did get lost... the servers did eventually seize up and crash completely, and often there'd be some data lost. But, at least in my case, it was never much more than a skill point or two, or a few hundred experience.... twenty minutes of lost playtime. I'm sure some people had a worse experience than I did, that's the nature of random data loss, but I wasn't badly affected.
When you consider the sheer scale of what they're doing.... they had TWO HUNDRED THOUDAND PEOPLE AT ONCE playing their game not long ago. The scale of that just boggles the imagination. If you assume 32kbits/second down, and 2kbits up (probably a bit skinny, but I'll try to err against Blizzard here), that's an aggregate total of 10,880,000,000 bits per second. Roughly 10 gigabits, or total saturation of an OC-192. Just the FIREWALLING on that kind of traffic is a HUGE project! Admittedly, they've broken that up into 3 or 4 datacenters, but doing firewalling and connection tracking on a mere 2.5gigabits is still pretty daunting. And that is completely ignoring the application servers, the load balancing, the inter-server communication, the databases..... just 1% of this project is a HUGE BIG DEAL.
The fact that we were able to pile in there at that kind of speed and the game didn't seize up and die completely is a resounding, amazing success. I'm sure the Blizzard guys aren't feeling too great about how things went by now, but.... guys, you kicked ASS. You did somethiung that has never been done before, a level of complexity nobody else has ever reached, got loaded down with about four times as much traffic as you were expecting.... and STILL mostly succeeded.
I have every faith that you'll work out the remaining kinks and bottlenecks.
By the way, the last couple days, on Uther, have been quite good... I think I got one disconnect in two days, and there really hasn't been any lag. They may nearly have the problems fixed. I haven't been thinking about bugs, just about slaughtering beasties.
You're right about the 'z', and the 'k' sound is usually somewhere between our 'g' and 'k', and it does get a little bit harsher at the end of a syllable. What sometimes happens in Korean, though, is that when there is a syllable with a heavy-sounding consonant, it gets moved to the beginning of a new syllable with the "uh" vowel sound attached (sometimes "ee" with the 'j' consonant or 'ch' combo).
So, instead of "jerk" (with the 'j' sounding more like the soft 'g' in 'rouge'), it comes out as "jer-guh". 'Big' is sometimes "bi-guh", although if they choose to keep it in the same syllable, the 'g' sound, while more closely resembling a 'k', is barely audible by English-pronunciation standards, so it sounds less like "bik" and more like "bi". Same with 'bag' -- either 'ba-guh' or 'ba' (with only the faintest hint of a 'k' sound).
("orange" becomes "oran-jee" and "watch" becomes "wa-tchee", btw. It's tough to teach them out of that practice).
Which is assuming that the term started with them. I'd be surprised if 'zerg'-ing came from Korean, though, since they don't tend to treat their nouns as verbs (as much as English people do, anyway).
(Please note, this is mostly related to the way things are pronounced in Daegu (central Korea), and off television from Seoul. Other parts of Korea are known to have widely differing dialects.)
There is one problem, and one problem only with World of Warcraft-- Blizzard isn't allowing character transfers across realms. All the problems we're talking about happen on something like 10% of the servers. When there are problems with my main character's server, I go play on a server that works. No one else seems willing to do this. To me, it's not a valid argument to demand your money back. There ARE servers up, just not the one with your primary character on it. If you're alliance, go do some horde quests (they're fun!).
Blizzard WAS a bit dumb when they were allowing folks to sign up to realms. If most people have the attitude "I want to be on the same server as my friends" and/or "I don't want to be on an empty server" it should come as no surprise that there was severe overpopulation of a few servers. On the flip side, folks would be complaining if there was 100% uptime and they couldn't get on the same servers as their friends, so what are you going to do?
Words like "u g0t pwn3d" and "Terrorists win!" come to mind here. Here's another (cached) site with the e-mail address, showing an apparent response to a *ahem* well-worded e-mail (not for children).
I wouldn't be shocked. Executives notice Slashdot, but they're busy making products, deals, and legal action--and filtering out e-mails like the one on the link above, I'm sure.
You can hold down the "B" button for continuous firing.