Slashdot Mirror
World of Warcraft AQ Gates Open!
Tayman writes "Wow...who didn't see this one coming? The players on the World of Warcraft Medivh server opened the gates to AQ. What happened next? The server crashed repeatedly. Why create content the servers can't handle?
The very first time I read about this patch, I knew the servers would crash. The more people who open the gates, the more angry customers Blizzard will have in my opinion. With 5million+ subscribers, you would think Blizzard would have the best servers/connection money can buy. Although, I'm sure it's more complicated than simply plugging in a few ram chips and faster processors though.
Most of the people involved in the raid are having a great time though. Could this be the most epic battle ever introduced to the mmorpg market? All signs point to yes. Let's see how long the mobs will respawn. Hopefully, the people of the Medivh server haven't seen anything yet.
Either way, I would hate to be a network admin for Blizzard atm. ^_^
Here are some pics of the event. Thanks go out to all of those who took these pics.
World of Warcraft AQ Pics Check out MMORPG Veteran to keep up with the events as they unfold." Update: 01/23 13:44 GMT by Z : Additionally, brandor wrote in with a link to some video of the event.
What AQ is supposed to be (for those that don't play WoW).
An expansion? Just a new dungeon? What's so special about it that it causes such server overload?
retrorocket.o not found, launch anyway?
Whoa. Generally, I take slashdot to be a lighthearted place to read interesting techish things. This post really does give me pause though. Not only does it sound like a mad World of Warcraft player's blog entry, it doesn't even explain the elements of what happened. If it wasn't for the fact that I play World of Warcraft I wouldn't have had a clue what he was on about.
I agree, Blizzard should have tested that part of the patch more specifically. Apparently, the gate was already opened on the test server (this is what I've heard from other players, I never did test the patch) which would leave me to suspect they never tested opening the gate very much.
I actually expected this crash.
Proceed with Format (Y/N)? Y
Yes development, testing, and operations is hard. Yes it is a huge game. But you know what - customers don't care. You don't get to sell a service and then say "ooh sorry it sucks, this is harder than we thought it would be" without offering some sort of refund or free play time or something.
Had you ever played Everquest in a mid-high end guild, during the Luclin+ era, you too would consider that 40 man raid for 6 hours IS casual.
"The way we can tell it's C# instead of Haskell is because it's nine lines instead of two." -- wadler
I work for blue chip companies setting up websites that are the busiest in their respective industries, including full connection through to back-end systems.
When the systems die at peak trading, it's 10s of millions in revenue lost. An hour.
My current company provides video downloads off our main sites. We service several hundred retail outlets. We offer very complex product search capabilities, and obviously we permit those products to be purchased. We're dealing with exceedingly large bandwidth, CPU and memory use. We have IBM mainframes, more Sun kit than you could fit in your house and more Wintel boxes than I'd like.
All of this is being provided for less than Blizzard's monthly subscription revenues. Far less. In fact, 3-4 months of WoW subscription revenue in Europe alone would cover the IT costs of our entire business.
So for Blizzard to be unable to handle the loads involved is frankly astonishing. Their systems architecture clearly isn't adequate. Their bandwidth isn't reliable. Heck, they can't even keep their website up and running at peak times - quite a simple website, at that.
This is despite being live for well over a year now. They know how much bandwidth each user needs. They know how many users they have. They know what the capacity on each server is. They already have logon queues at times of peak load, to control the numbers of logged in players.
I have no sympathy for Blizzard on this matter, because they've had plenty of time to get this sorted, and consistently fail to deal with it. This isn't rocket science, you don't need to steal Google's employees to find a solution, just get someone competent in and fund the necessary infrastructure.
These games are the biggest regret of my life. Seriously. I once spent over 1400 hours in one year playing one (back in the text days). That's seventy days, if you are counting.
I could have been getting good grades, chasing chicks, and figuring out what the "#$# to do with my life. I seriously messed up all three. Instead, I just had the coolest equipment in some worthless game. A couple people I know failed out of school entirely because of these games.
You can do better.
I'd like to thank Blizzard for making a server wide event that opens an instanced dungeon that 2% of the player base might get to. Sure, there is also a 20 man AQ instance, that maybe 8% of the player base will see. (It's on par with zul gurub). So instead of making another instance like Upper back rock spire that everyone can do, they continue to cater to the EQ fans who like 4-8 hour instnace runs on saturday nights.
WoW crashes when it gets only a few hundred people doing anything meaningful inside a zone. In total it handles a few thousand people connected to any given server at any given time, with some of those a sizeable portion of those people off-loaded onto other servers via instances most of the time.
In addition, they're well aware of their scaling problems, and have added things in to prevent the type of occurances that caused crashes(City Raids), as well as scaling back the max server populations(hence queues). So not having prepped properly for a World Event of this magnitude, especially given their revenues is inexcusable.
My guess is the situation at blizzard is the following:
Most of the core devs went onto other projects
Like most MMOs their network code is laughable.
Their code doesn't parallellize well, so they can't just toss more hardware at the problem
They can't fix the above without a drastic redesign, and by the time they did that it would probably cease to matter.
And yes, it's doable, I've seen MUDs/MUSHs written as hobbies that handled several hundred concurrent users on hardware from the mid-90s. An MMO doesn't push *that* much more traffic than a text-game that saturates a 56k connection(as many did), and it certainly doesn't do many more back-end calculations. Considering how much hardware has scaled and how much further we've gotten in various areas, there just isn't any excuse for several hundred people in a single world-segment causing the server(not the client) to go OMG and crap out.
No web-based business craps out under that kind of traffic. How to cope with it is well-known at this point. I mean shit, this is the type of crap DIKU's massive list of socket descriptors did under load, and that was written over 10 years ago!
Imagine a phone switch doing this! That's tech from the 70s that handles waaayyyy more traffic than one of blizzard's servers. Google easily copes with orders of magnitude more traffic every moment of every day, and holds up like a champ. Stop being an apologist for a drastic lack of planning and poor engineering.
The key to the enjoyment of pop music is to replace any instance of "love" with "C.H.U.D."
Or, you could have spent those 70 days doing something equally stupid.
The only problem I have with your logic -- or anyone that heavily criticizes people for spending too much time on any one activity -- is the assumption that if they did other activities, they would inherently have more value.
I know people that spend hours a day, pretty much all of their leisure time, watching sports on TV. Is that really any better or worse than playing WoW for an equivalent amount of time? I don't think so (especially given that ESPN costs more).
I'm willing to bet that most people who are on WoW, if Blizzard went under tomorrow, would find something equally useless to do in their spare time. This idea that people who play games are all going take up triathlon training or feed the homeless in their spare time, if games weren't available, is dumb. In all likelihood they'd just watch TV.
I'm not arguing that too much of anything can't really mess up your life -- when people stop going to class or work to play games (or watch TV, or whatever), it's a real problem. However I'm not sure that games are much worse in this regard than any of several "time wasters" that I can think of, it's just that you don't hear about the other ones.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
None of what you mention is nearly as dynamic and complex as a high-traffic MMORPG server back-end.
I'm quoting you because you're AC:
I just have to say that:
1. I was an EQ addict
2. I replaced my TV time with Game time...
Now I dont play EQ anymore, but I have returned to TV..
What was better?
This is exactly my point. I think the answer to your question of 'which is better' is "whatever works for you." As long as it doesn't keep you from going to work/class/school, and doesn't damage your health, that is. I think computer games get a bad rep, when they're really no better or worse than spending an equivalent amount of time watching TV, which many people do habitually anyway.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
"I know people that spend hours a day, pretty much all of their leisure time, watching sports on TV. Is that really any better or worse than playing WoW for an equivalent amount of time? I don't think so"
I agree. But those people have the same problem. You need to diversify your activities. Have other hobbies other then WoW. Get out and play catch, or go for a walk in a park, or paint a painting, write a story, read a book. Something other then screentime.
Too many people make a lifestyle out of their entertainment these days. Its shallow and meaningless. When being told this, many players get violently upset, not being able to handle that this insect-like specilization simply isnt healthy.
No, There is nothing WRONG with playing WoW, or videogames, or watching tv, but you have to do it in moderation. The problem with WoW is that almost noone I know that plays it does so in moderation. It becomes all-consuming. And anything that does that, weather it be screen entertainment, work, religion, food, drugs, sex, etc is unhealthy.
When the systems die at peak trading, it's 10s of millions in revenue lost. An hour.
While WoW is big, it's nowhere near that big. Blizzard does not make tens of millions an hour in revenue from this game. Half a million subscribers paying $12 per month gives around $8000 or so per hour. Revenue is only lost if people cancel their accounts, not if they don't get to play for a few hours.
"I have never let my schooling interfere with my education." - Mark Twain
You are comparing an orange to a 12 course chinese banquet For one average player blizzard need to track the following constantly changing data: world position, stats (aprox 10 different ones), status (in combat, out of combat who in combat with), repution, equipment...(and this is most likely only a few of the things) all in real time (something no web site has to do) None of this can be stored on the client as would be hackable Average player in wow probably generates per minute 100 times more database hits (and this is where most of the slowdown and crash's occure with wow, the databases) than average web site user generates in an hour. While blizz could deal better with some situations (new caracter creation on that server should have been turned off days ago for accounts that do not already have one there, they have the code for that already and have implimented it for other reasons elsewhere)the overall situation currently has no solution and this is why, without exception, every other MMO that experiences these kinds of loads have the exact same problems.
Also serving static web content is trivial compared to tracking the state of 5 million clients and letting them see each others in real time is so far beyond web hosting that it is laughable.
I worked at video game companies (Turbine) and I worked at some of IBM's large server farms (Poughkeepsie, Southbury) doing performance balancing. As far as software goes I have to say video games server technology makes web content delivery look like the stone age. The only thing that even compares in complexity is when IBM hosted the Olympic coverage. Trying to compare simple web content to a system where clients are all making updates to each others environments in real time is impossible.
I hate it when the Wow server's crash, but I have had my ego battered by what the guys at Blizzard have managed to do. They have done some great work and I am curious to see other game companies surpass the work Blizzard has done.
Nothing here is trivial. If it was it would have been done right the first time.
It's quite simple, really.
When you are playing these games for such a long time you start to recognize the pattern involved. The game is *designed* to take up as much of your time as possible in order to extract the most money out of my wallet. No game out there is as profitable as WoW.
But that's not the worst. The worst is that when you click to sign into your character, you have that sinking feeling in your stomach that you could be doing something better to enrich your existence on this Earth. You could be volunteering, you could be putting in extra time with your family, you could be going to the gym and get in the best shape of your life (what I did).
I'm still a gamer to the core. And without WoW I have SO MUCH MORE TIME for more games than ever before. So this isn't about gaming versus real life...I will never give up gaming, and I get to enjoy more of it and remain a happier and healthier human being than I ever was during those 8 months I was addicted to WoW.
Because we are immediately suspicious of any transaction where one side has a financial interest in how you spend your time. If MMORPGs did not have subscription fees, I'd be all over them.
We have that reaction because on some level, we realize that publishers of MMORPG's are the electronic equivalent of tobacco companies. They have a direct vested interest in making their games as addictive as possible, and they are going after kids. Sony hired shrinks to make sure EQ put asses in seats and kept them there. They didn't care how many lives they ruined (see EverQuest Widows), they don't care about player experiences (see all comments about grinding) - they only care about whether you keep playing.
Bottom line: we don't trust the publishers of these games to deliver the best quality experience to us. There are too many incentives for them to do otherwise.
But tell me, which is "worse" spedning 4 hours in from of the tube with a controller in a completely self absorbed activity...
Which has an ending.
or spending 4 hours in an MMO where you actually can speak and interact with actual people.
Good thing you didn't say "actual women." That would have been humorous.
Edith Keeler Must Die