Wikipedia Explains Today's Global Outage

gnujoshua writes "The Wikimedia Tech Blog has a post explaining why many users were unable to reach Wikimedia sites due to DNS resolution failure. The article states, 'Due to an overheating problem in our European data center many of our servers turned off to protect themselves. As this impacted all Wikipedia and other projects access from European users, we were forced to move all user traffic to our Florida cluster, for which we have a standard quick failover procedure in place, that changes our DNS entries. However, shortly after we did this failover switch, it turned out that this failover mechanism was now broken, causing the DNS resolution of Wikimedia sites to stop working globally. This problem was quickly resolved, but unfortunately it may take up to an hour before access is restored for everyone, due to caching effects."

Comments

DNA? Really?

[Anonymous Coward]

How 'bout proofreading titles

Re:DNA? Really?

[Anonymous Coward]

They're too busy downmodding nigger jokes.

Wow!

[Anonymous Coward]

DNA resolution failure

Re:Wow!

[suso]

Maybe the reverse DNA wasn't set right.

Re:Wow!

[Midnight Thunder]

DNA resolution failure

Clearly they thought captchas were too easy to defeat.

Re:Wow!

[monkeySauce]

No, the RNA is just fine. The real problem is a DNA CNAME pointing to an A molecule that isn't resolving.

Re:Wow!

[Anonymous Coward]

Nice try, but I think you stretched that one a little too far. And besides, pointing a CNAME to an A record is ok, its pointing a CNAME to a CNAME that is forbidden.

Re:Wow!

[DA-MAN]

Nah, its standard practice.

$ host download.microsoft.com
download.microsoft.com is an alias for download.microsoft.com.nsatc.net.
download.microsoft.com.nsatc.net is an alias for mscom-dlc.vo.llnwd.net.
mscom-dlc.vo.llnwd.net has address 208.111.161.113
mscom-dlc.vo.llnwd.net has address 208.111.161.89

Re:Wow!

[omnichad]

No, it's pointing an MX record to a CNAME that's "forbidden." It's done all the time, but it's technically not allowed.

This was terrible!

[Anonymous Coward]

Because of this outage, I actually had to work this morning.

Re:This was terrible!

[DeadDecoy]

Then why are you on slashdot?

DNA caused outage?

[xata_boy]

Human or otherwise?

DNA DNS?

[Anonymous Coward]

I could see why the failover didn't work... They should try resolving names instead of nucleic acids. :\

Oh teh noes!

[Anonymous Coward]

Its the T-virus, run!

errr

[yerpo]

The blog link explaining the whole thing of course doesn't work for us Europeans, either.

Test, and Test Again

[Jazz-Masta]

However, shortly after we did this failover switch, it turned out that this failover mechanism was now broken, causing the DNS resolution of Wikimedia sites to stop working globally.

Good thing Wikimedia pays their System Administrators well enough to test their backup systems.

Re:Test, and Test Again

[X0563511]

I know people who work in the Florida DC. They do, and they are smart people. Don't assume incompetence.

Re:Test, and Test Again

[cryfreedomlove]

You say test and test again. I say that this is true only when the cost of an outage outweighs the cost of testing. What does this one hour, once per year really cost wikipedia?

Re:Test, and Test Again

[commodore64_love]

Free media publicity.

Re:Test, and Test Again

[Dahamma]

True, and the cost was probably fairly minor, as they are not advertising based... so only the cost of any people so pissed off with the downtime that they refuse to donate :)

Re:Test, and Test Again

[Jazz-Masta]

I actually wasn't assuming incompetence, the hallmark of many SysAdmins is being understaffed, overworked and underpaid, and thus do not have the resources to properly test all backup and redundant systems.

As consultants and contractors in the area of System Administration, you get let go if anything like this was ever to happen. This is why they charge a little bit more.

Whatever happened, it failed. A good lesson for next time. Not knowing exactly the cause, but it is safe to safe there were too many eggs in one basket. Multiple geopgrahically diverse load-balanced DNS servers? Why was there an overheating problem in the first place? Only one air conditioner?

Wikipedia has had a few failures, not all their fault. In 2006 Cogent pulled a block of IP addresses that were leased to Wikipedia.

Re:Test, and Test Again

[geniice]

Going by past statsitics the cost of downtime to wikipedia tends to be negative since donations rise. Not that this is something wikimedia aims to do.

Re:Test, and Test Again

[Anonymous Coward]

I know people who work in the Florida DC. They do, and they are smart people. Don't assume incompetence.

I'm going to assume incompetence. The only question is whose incompetence: the admins, or the folks higher up the food chain who didn't give them the resources they needed. But I have no doubt somebody was incompetent somewhere, how else do you explain the failure? Can you answer that instead of telling people what to think?

Re:Test, and Test Again

[geniice]

Wikipedia has a fairly limited budget and has historically accepted the odd few hours of downtime now and again as the natural result of this. The number of such incidents have reduced over the years though.

Re:Test, and Test Again

[Anonymous Coward]

I'm going to assume incompetence. The only question is whose incompetence: the admins, or the folks higher up the food chain who didn't give them the resources they needed. But I have no doubt somebody was incompetent somewhere, how else do you explain the failure? Can you answer that instead of telling people what to think?

Wow. For someone who probably uses the service and doesn't pay for it, you're sure griping a lot. They don't serve ads (well, except to solict funds to keep their servers up and running), they rely on private donations (which sets them apart from even public television, which gets some of it's funding from the government).

When you pay for an SLA with Wikipedia (signed by someone with the authority to make such an agreement) then you have the right to throw rude accusations around.

Re:Test, and Test Again

[Critical Facilities]

That may be true. Although, this whole thing has me wondering if they're re-thinking their "Green Technology" push. According to this article, they've recently partnered with a European firm that specializes in Green Data Center Technology, most specifically, using "air side economization" cooling techniques (cool the data center with outside air as opposed to mechanical cooling). Now, while I think this is a viable and worthwhile technology and strategy (and before anyone flames me into oblivion for being a naysayer when it comes to becoming more energy efficient), I do have to raise the question as to whether they had fully commissioned this facility. Had they determined at what point(s) supplemental cooling was brought online? Had they tested the mechanical cooling systems to ensure they would come online successfully and in enough time? Did anyone ask any questions relating to these issues?

Just makes me wonder. I hope they share the information so that we can all learn from it.

Re:Test, and Test Again

[Chris Mattern]

I'm going to assume incompetence.

Damn right we're going to assume incompetence! Why aren't we getting the uptime we're paying for?

Oh, wait...

Re:Test, and Test Again

[petermgreen]

I doubt this was a case of wikimedia deliberately going green. I suspect it's far more likely that they happened to be in the right place and happened to make an offer that wikimedia liked.

Re:Test, and Test Again

[FuckingNickName]

Wow. For someone who probably uses the service and doesn't pay for it, you're sure griping a lot.

0. For someone who is going off on a rant based on a reasoned assumption, you sure aren't setting off on the right foot by starting the unjustified assumption that the poster uses Wikipedia;

1. You don't have to pay for or be a net consumer of something in order to criticise it - all you have to do is provide a reasonable explanation for the criticism. The alternative, that only the paying consumer should have a voice, is irrational and harmful;

2. All this said, maybe the poster has donated time and/or money to Wikipedia - you do realise it's produced by thousands of (sometimes even well-meaning) volunteers, right?

They don't serve ads (well, except to solict funds to keep their servers up and running),

So they don't, except when they do. At least regular adverts give you the opportunity to learn about some product. Huge banners telling you that the Child in Africa will die from not knowing all the Pokemon characters if you don't donate are quite pathetic.

When you pay for an SLA with Wikipedia (signed by someone with the authority to make such an agreement) then you have the right to throw rude accusations around.

I know America has such a macho culture that it's considered life-destroying to receive public criticism, but it's actually useful to be told that you're incompetent when you're incompetent. It's the first step to finding out where you've demonstrated incompetence, which is the precursor to fixing (i) your approach; (ii) the problem. Brushing the truth under the carpet by sounding the "I/Wikipedia admins have the right not to be offended!" klaxon solves nothing.

Re:Test, and Test Again

[Anonymous Coward]

Minus the cost of people donating so there's less downtime

Re:Test, and Test Again

[Anonymous Coward]

Don't forget that unqualified "CTO" they hired this year.

Re:Test, and Test Again

[Dencrypt]

Pays them what? $3.3M isn't really much in a year paying of 30 people.

Re:Test, and Test Again

[Anonymous Coward]

Just to nit pick, smart != competent though the two usually go together. It many cases of failure, it is a matter of the sysadmin not growing a pair and knuckling under to so know-nothing MBA. "Told ya so" won't cut it, even though the MBA forced the order down the throats of the IT dept, it will still be IT that gets the blame.

Re:Test, and Test Again

[David Gerard]

Wikimedia is a charity. $8M to run a top 5 website is approximately NOTHING. My suggested slogan for the last fundraiser wasn't used: "Give us money. Or the homework GETS IT."

Re:Test, and Test Again

[David Gerard]

There's nothing quite as uplifting as the complaints of someone getting something for free. Not even for ads.

Re:Test, and Test Again

[David Gerard]

It is actually true that downtime used to be by far our most profitable product ;-)

Re:Test, and Test Again

[similar_name]

For someone who is going off on a rant based on a reasoned assumption, you sure aren't setting off on the right foot by starting the unjustified assumption that the poster uses Wikipedia;

How is jumping to incompetence a reasoned assumption but assuming someone has used wikipedia unjustified. It's at least reasonable.

You don't have to pay for or be a net consumer of something in order to criticise it - all you have to do is provide a reasonable explanation for the criticism.

So what was the useful criticism in assuming incompetence over lack of funding?

All this said, maybe the poster has donated time and/or money to Wikipedia - you do realise it's produced by thousands of (sometimes even well-meaning) volunteers, right?

If so then according to his assumption of incompetence is he partly to blame?

So they don't, except when they do.

That's rather pedantic. I think (most) everybody knows the difference between a non-profit fundraising banner and an ad.

I know America has such a macho culture that it's considered life-destroying to receive public criticism, but it's actually useful to be told that you're incompetent when you're incompetent

Very true. I work with people you should be told so more often. However, it is equally true that thinking human error is the result of incompetence and that getting rid of people who make mistakes will lead to a more perfect system. I good system deals with human error, it doesn't assume it won't exist.

I think it is fair to say that someone may have made a mistake and that Wikipedia may not be funded well enough to deal with human error.

Rumor was..

[cybrthng]

Some government pencil pusher mixed up wikileaks with wikipedia... after all the "strange tweets" from @wikileaks it sounded feasible ;)

Re:Rumor was..

[Anonymous Coward]

Wikileaks is part of wikimedia, so it went down too (along with wikinews, wikispecies, etc.).

Re:Rumor was..

[Anonymous Coward]

Wikileaks is part of wikimedia, so it went down too (along with wikinews, wikispecies, etc.).

Wikileaks is certainly NOT part of Wikimedia. You can see such at http://wikimediafoundation.org/wiki/Our_projects

Re:Rumor was..

[David Gerard]

This is on the press coverage bingo card.

Do we accept this...

[Al's Hat]

...as proof of global warming?

Re:Do we accept this...

[alta]

+1 So True...

Re:Do we accept this...

[Pete Venkman]

It came from Wikipedia so it must be true--right?

Re:Do we accept this...

[smooth wombat]

Not necessarily, but this might be.

Re:Do we accept this...

[Al's Hat]

Possibly lame attempt at humor on my part. I'm more worried about ocean acidification as a more immediate threat.

Re:Do we accept this...

[afabbro]

Do not meddle in the affairs of sysadmins, for they are subtle, and quick to anger.

Wow - is that the worst sig on Slashdot or what?

• "Do not meddle in the affairs of sysadmins, for they are subtle..." The sysadmins or their affairs? And why would subtle affairs (or subtle sysadmins) be either significant or threatening? Oh my God, he's...he's...SUBTLE! RUN!
• "..and quick to anger." Angry affairs? Angry sysadmins?
• And to top it off, your URL points to a slimey affiliate site that promotes "Free Advertising System" and "Copy the Super Affiliates".

Re:Do we accept this...

[Anonymous Coward]

Troll, or woefully ignorant of Tolkien? I can't tell.

Re:Do we accept this...

[jeffasselin]

Either you should read more or you have some serious linguistic credentials.

It's a reference to a quote in The Lord of the Rings by JRR Tolkien, something said to Frodo Baggins by Gildor Inglorion in The Fellowship of the Ring (tome 1) in chapter 3 "Three is Company":

"Do not meddle in the affairs of wizards, for they are subtle and quick to anger."

The quote is of course in reference to the wizards of Middle-Earth. The user sig which you tried to disparage is an attempt to make an analogy between sysadmins to wizards.

My problem here is that you weren't attacking the analogy, but the syntax of the sentence. A sentence crafted by JRR Tolkien, one of the most well-known scholars of the English language, and one who was named "Author of the Century" for the last century. I think he knew English syntax better than you do. Unless of course you're a well-known, published writer who has studied the English language extensively and you have the diplomas to prove it on the wall in your office.

Re:Do we accept this...

[Anonymous Coward]

^Try not failing English 101. Sysadmins are the subject of the entire sentence. Thus, sysadmins are subtle, and quick to anger. HTH

Re:Do we accept this...

[fuzza]

The version I used to have was: "Do not meddle in the affairs of sysadmins, for they are quick to anger and have not need for subtlety."

Re:Do we accept this...

[Jupix]

While we're on the subject of temperature, can anyone enlighten me as to the point of having a data center in Florida? Wouldn't you want hardware like this in a climate that's naturally as cool as possible?

Re:Do we accept this...

[alta]

I like that version :)

Re:Do we accept this...

[alta]

All I can say is, thanks for learning what's at the site, you just earned me a quarter. Have a nice day Portland.

Hate

[olddotter]

Hate it when my DNA doesn't resolve.

Sorry I know its just a type-o, but its funny to me.

Re:Hate

[Locke2005]

I don't have a problem with DNA not resolving.
I have a problem with getting it out of the sheets.

welcome to the cloud

[Anonymous Coward]

With both stormy and sunny days.

Good choice on the article you're linking to...

[Anonymous Coward]

I don't know which is more awesome - that this article came up just as I was wondering what happened to Wikipedia, or that the post links to an article which I CAN'T READ BECAUSE WIKIPEDIA IS DOWN.

Didn't know DNA could cause an outage

[jmdevince]

When did we start using DNA to resolve domain names? I mean we can fit a butt-load of information in a DNA strand but I think the overhead would be too high for DNS resolutions. (Should be DNS)

Re:Didn't know DNA could cause an outage

[wizardforce]

Just to give an idea of just how vast DNA's information storage is, the average human cell contains about as much information as most DVDs can store. So hypothetically, if you could reliably transport DNA like we do electrons on the internet, the bandwidth would be enormous (1 gram DNA can store ~10^21 bits) although lag might be a problem unless you can route these DNA packets at relativistic velocities.

Re:Didn't know DNA could cause an outage

[Andy Dodd]

I'm assuming that material containing large amounts of DNA gummed up a cooling fan, causing the overheating. :)

Re:Didn't know DNA could cause an outage

[Anonymous Coward]

Hey, they really love their servers!

DNA resolution failure???

[Anonymous Coward]

>>due to DNA resolution failure. ...Also known as mutation...or X-men

wait what my dna is being resolved?

[Anonymous Coward]

cant we simply hack that by modifying gens.conf?

rndc flush

[ls671]

I noticed wikipedia wasn't resolving this morning.

Flushing my "DNA" cache fixed it ;-))

rndc flush

Re:rndc flush

[ls671]

I will add that this is a good thing this article was posted. It caused me to stop investigating the possibilities of somebody hacking into my "DNA". ;-))

Re:rndc flush

[Aladrin]

That is disgusting. :D

Re:rndc flush

[Sigma 7]

Flushing my "DNA" cache fixed it ;-))

Not for everyone, since some ISPs cache DNS lookup results.

Re:rndc flush

[ls671]

> Not for everyone, since some ISPs cache DNS lookup results.

It should have been obvious that you needed admin access to your own "DNA" in order for this fix to work... ;-))

Also your ISP must not intercept your "DNA" queries (redirecting deoxyribonucleic acid #53 to their own DNA)

Re:rndc flush

[icebraining]

Why, are you forced to use your ISP's DNS servers? here.

Re:rndc flush

[compro01]

I prefer level3's DNS servers (4.2.2.1-4.2.2.4). I've heard rumours of them planning to block public access to them, but never heard anything more about it. Works great for me.

Re:rndc flush

[ls671]

Here is the list of DNS to query when you run your own DNS, as I stated in my OP. You obviously need to run your own DNS in order to be able to flush the DNS cache as I mentioned in my OP ;-)

This list of root DNS is guaranteed to remain free for public access. These DNS only return pointers to other DNS and are the foundation of how name resolving works on the internet so you are guaranteed to get the correct data as far as it is possible to get it.

In short, no third party is required to run your own DNS. Some will say this is slower because you have to first populate your cache doing multiple queries but I have never noticed any slowness so I do not care about that. Once your cache is populated it is much faster anyway because you do not have to go to the network at all to resolve a name.

It is easy and free to setup your own DNS on most OSes and it could be safer because you get the information as accurate as it can get. My DNS process uses about 13 Meg of ram + the configured cache size which is very light.

List of root DNSes, most of those IPs use "anycast addressing and routing to provide resilience and load balancing across a wide geographic area", so you always query a root DNS close to you anyway:
http://www.internic.net/zones/named.root

Wikipedia documentation:
http://en.wikipedia.org/wiki/DNS_root_zone

I realize that you probably already knew this, I am just posting to clarify for others ;-))

DNA resolution failure

[TheCreeep]

Guess it resolved to a chimp?

DNA resolution?

[gad_zuki!]

Whoa, why is the DNS resolving dATP.dGTP.dCTP.dATP?!?

Hour Delay

[Reason58]

This problem was quickly resolved, but unfortunately it may take up to an hour before access is restored for everyone, due to caching effects.

If you don't want to wait an hour for it to update, you can open a command prompt and type "ipconfig /flushdna".

Please be warned that this may also revert you to some sort of single-celled organism.

Re:Hour Delay

[Dancindan84]

I /flushdna all the time. Hasn't had any noticeable effect except clogging my toilet.

Re:Hour Delay

[shoehornjob]

LMFAO wish I still had mod points +1 funny.

Re:Hour Delay

[mrdogi]

OK, I'm somewhat worried now. I was going to make a snarky comment on how I can't seem to find the ipconfig command on my Mac, but it *actually* has one! Mac is following Windows?!?

At least I'm still safe with not having on on my Solaris boxen...

Re:Hour Delay

[Shemmie]

Not often I laugh out loud at /. - thank you.

Re:Hour Delay

[BertieBaggio]

I /flushdna all the time. Hasn't had any noticeable effect except clogging my toilet.

...?!

I'd recommend you see a doctor about that.

Re:Hour Delay

[Hurricane78]

I AM a single-celled organism, you insensitive clod!

And: I’m also your single-celled overlord! So bow to me!
No! Not to wipe me away with your... sponge...! Please no! Aaaaahhhh!
*wipe*

Overheating

[Anonymous Coward]

Obviously this was caused by Global Warming.

DNA?

[MrTripps]

With DNA resolution problems, apple.com resolves to 64.38.232.180 (oranges.com).

Re:DNA?

[omnichad]

Nah...it points to orange.co.uk - which is a UK mobile phone company that offers the iphone.

FTFA

[Rik Sweeney]

We apologize for the inconvenience this has caused.

[Citation needed]

Re:FTFA

[macbuzz01]

We apologize [citation needed] for the inconvenience [citation needed] this has caused [citation needed].

Re:FTFA

[kenj0418]

I think they should have at least put up a page that had, in large, friendly letters "Don't Panic".

Actually an RNA problem

[Anonymous Coward]

RNA actually translates the name into an IP address.

You could read up on this at http://en.wikipedia.org/wiki/Translation_(genetics) if wikipedia's ribosomes weren't down right now.

Oops

[girlintraining]

You see guys, this is why you regularily test your backup plans and failovers. This is equivalent to building maintenance making sure the fire extinguishers aren't expired... it's basic to IT. Unfortunately, Wikipedia just reminded us that what's basic isn't always what's remembered. Someone just lost their job.

Re:Oops

[cryfreedomlove]

I doubt anyone lost their job over this. What is the real cost of a 1 hour global outage for wikipedia if it only occurs once per year?

Re:Oops

[Anonymous Coward]

Wikipedia does not profit from traffic so they actually save money for every hour the site is down. Looks like someone just got promoted!

Re:Oops

[ArundelCastle]

Wikipedia just reminded us that what's basic isn't always what's remembered.

TFA quote did say it was a standard procedure. Seems like an accurate description leading to the common SNAFU, or "Administrivia" if you prefer. It's the weird shit you're always checking on.
Building maintenance is an interesting comparison to use. Every year I see plenty of elevator licenses and fire extinguisher tags in many, many buildings that are expired.

Re:Oops

[Anonymous Coward]

For every hour? Really? With that logic they should just keep it down 24/7 then.

Re:Oops

[geniice]

Since wikimedia's server admins have long since been divided into two departments known as wing and prayer they can probably avoid any job loses by blaming each other.

Re:Oops

[Arthur Grumbine]

For every hour? Really? With that logic they should just keep it down 24/7 then.

Only when combined with the premise that profit is a goal for them. Which it's not.

Re:Oops

[VTEX]

Someone just lost their job.

I highly doubt someone lost their job over this - and they shouldn't. There are no perfect systems out there, period. Given Wikipedia is a not for profit corporation, they very likely have limited resources and the IT staff does the best with what they have. Even with a virtual unlimited amount of resources things can still go wrong in a "Perfect Storm".

If anything, the System Administrators should be commended for their quick actions to get the site back up and running as soon as they did.

Re:Oops

[Yvanhoe]

Someone does an awesome job at having a failover procedure for such an incredible non-profit project. And for resuming access within one hour. For heaven's sake, they don't even make money keeping the biggest encyclopedia of all History online, give them a break !

Come on wikipedia, fix this, but rest assured that we all love you !

Re:Oops

[Anonymous Coward]

This is equivalent to building maintenance making sure the fire extinguishers aren't expired...

It's as if they were making sure that the fire extinguishers paints weren't faded or peeling, rather than checking the mechanism and pressure.
Can't have rotten looking fire extinguishers now, can we? Nobody would want to pick them up if there was an actual fire, they might get all icky and covered in paint shards.

Re:Oops

[Anonymous Coward]

Expired buildings?

Re:Oops

[u38cg]

Since donations spike after an outage, they profit from downtime :p

Re:Oops

[tlhIngan]

I doubt anyone lost their job over this. What is the real cost of a 1 hour global outage for wikipedia if it only occurs once per year?

Having to deal with the students who couldn't crib their report off Wikipedia an hour before it was due?

(Yes, I'm joking. But I suppose we should continue this thread with other fun things we couldn't do with Wikipedia... like make bets about something on Wikipedia - only having edited the article in your favor minutes before).

Re:Oops

[Anonymous Coward]

Survival and dominance is though. Not for profit organisations still need money to run.

Re:Oops

[BitZtream]

Yea, the problem is people tend to 'regularly test' during the work day in my experience which results in the exact same event happening anyway.

It generally only happens once, either accident or during testing, and gets fixed. Unless you're going to do ALL your testing during off hours, which is really hard to define for a global operation, then any test that fails is just the same as a failure during non-test conditions.

Testing for no reason other than testing is not always the brightest of ideas, contrary to what you've been told.

Re:Oops

[FunPika]

Yes, and to boot the site being down for 1 hour will give people incentive to donate thinking it'll stop it from happening again. :D

Re:Oops

[David Gerard]

This is literally true! But we swear we don't take the site down deliberately ;-)

Only to Be Expected

[PingPongBoy]

Nothing to see here. Overheating was normal behavior after I updated the Pr0n article.

Re:Only to Be Expected

[ls671]

Are you Polish ? I thought the movement was abolished in 1989...

Pron:

http://en.wikipedia.org/wiki/Patriotyczny_Ruch_Odrodzenia_Narodowego

I just assumed DNS was reverted...

[Anonymous Coward]

...to the old setting by some Admin who edited it the last time, and who would be damned if he let anyone else get in the last word.

genetic level?

[Anonymous Coward]

DNA resolution failure? sounds serious.

Serves them right

[Anonymous Coward]

Wikipedia admins need to get out of their basement anyway.

makes sense

[nomadic]

Judging by traveling through Europe in the summer, they've never discovered "air conditioning."

Uptime is dumb

[Anonymous Coward]

Guess what Wikipedia, you are a free service. You could be up 10 hours a day or 24, existing is sufficient. Damn demanding internayz morons.

Edited?

[DarkKnightRadick]

Well, looks like all the DNA jokes are now -1 off topic

Well played /., well played.

I saw some issues with wiktionary...

[Qubit]

But when I got to the wiktionary.org main page I didn't see any kind of note or warning.

Couldn't they have at least put up some kind of warning box, hopefully with a list of IP addresses underneath so that one could directly access the services when in dire need?

.
.
.
.
.

(I'm not really sure what constitutes "dire need" of wikimedia services, but I'm sure someone can come up with a list of relevant circumstances)

Re:I saw some issues with wiktionary...

[PPH]

I'm not really sure what constitutes "dire need" of wikimedia services, but I'm sure someone can come up with a list of relevant circumstances

You could look up 'Dire Need' on Wiki..... oh, never mind.

Run both systems live at half capacity

[Colin Smith]

active/passive systems are a pain in the arse. The whole concept of testing failover in an active/passive situation is wrong. Anything which relies on human beings doing this and that and that and that is a bad solution.

Just run active/active and load balancer over both sites. If one fails it's tests, you just pull it.

 

Re:Run both systems live at half capacity

[rmm4pi8]

For systems that can be stateless, this is always the best approach. master-master replication with conflict resolution isn't always that easy, however, especially when you think about something like the way wikipedia edits can potentially interact. So developing a conflict resolution scheme can be extraordinarily expensive, and MySQL isn't the most stable in multi-master anyway. Thus while you're right in principle, the expense can be prohibitive.

Re:Run both systems live at half capacity

[GNUALMAFUERTE]

Yes, I agree. But the main issue with that paradigm is that many times the expense of one of your locations (and the quality of that location) is substantially lower than the other.

Example: I run servers on the US, Brasil and Argentina. The US server has better, cheaper bandwidth than the other two. Also, since this are VoIP servers, sometimes the services I send the calls to are in the US anyway, so even if the call goes originally to Argentina's POP, I'm still forwarding it to some IP in the US anyway.

So, in that case, I want the Arg/Brasil locations for other traffic (that's why there are there), and for local connectivity, but balancing our main traffic there makes no sense from any point of view. So, I only failover to those servers when I have an issue in our main location.

Sometimes, you have many resources you can use in emergencies, but you don't want to use them when the main location is clearly cheaper and better.

Re:Run both systems live at half capacity

[xaxa]

Ping [Amsterdam wikimedia cluster]: 30ms
Ping [Florida wikimedia cluster]: 130ms

That's from London. It's obviously better if I normally access the Amsterdam site.

Re:Run both systems live at half capacity

[Colin Smith]

powerdns geo backend.

Which they're already using.... Which means it looks like the problem may be more related to automation of the testing of the sites and the subsequent automatic (vs manual) pulling of a site from the dns when it fails.
 

Administrative problem

[PPH]

They couldn't get to the Wiki page about failover testing.

From hot to hotter

[LoRdTAW]

From the Summary:
"Due to an overheating problem in our European data center many of our servers turned off to protect themselves"
"we were forced to move all user traffic to our Florida cluster"

I think Wikipedia needs to build some data centers further north.

Re:From hot to hotter

[Anonymous Coward]

Maybe they should have spent some money on their data center instead of hiring Danese Cooper as CTO.

Deleted?

[Grishnakh]

I thought maybe they had simply deleted Wikipedia because some admin decided nothing on there was "notable".

I disagree.

[Colin Smith]

You build your systems to be fault tolerant. They automatically continue with half the components missing. Automatically disable those which fail the continually running tests.

Build your backup tests into daily procedures. i.e. don't copy/scp files to other locations/servers/sites, restore them to the other location. Autorestore DB backups to the staging/test/dev/reporting systems daily.

Computers are there to do stuff automatically. Getting human beings to do them is prone to failure.
 

Re:I disagree.

[RAMMS EIN]

You make some very good points in your post.

At the end of it all comes the realization that planning for crisis is complicated, and getting it right is hard. It's also something that every organization I have ever worked with has underestimated considerably. From what little information I have about this incident with Wikimedia (I noticed nothing, myself), they did considerably better than average.

But you are right: the right approach is not to prepare for contingency, but to make recovery part of the normal flow. If failure and recovery are part of the everyday routine, you will know what things are broken before disaster strikes, and when it does, you will be prepared. Nothing will make your organization infallible, but at least you will have procedures, people who know how to execute them, and experience with doing so.

Funny

[Anonymous Coward]

I didn't understand some terms in the summary, so I was about to wiki them... *sigh*

backup failure doesn't mean a failure to test

[rritterson]

I see lots of comments stating that this would not have happened had admins run regular tests on the failover mechanisms. That seems a poor assumption- if the system happens to fail and then an outage occurs before the next scheduled test, one may not be aware of it.

We had this problem recently where we were testing our backup generator. Normally, we cut power to the local on-campus substation, which kicks in the generator and activates a failover mechanism, rerouting power. Well, the generator came on no problem but the failover mechanism was broken, so every server in the datacenter spontaneously lost power. Had we known the failover was broken, we would have not done the regular test. However, the last test on the failover (done directly without cutting power), a mere month prior, had shown the failover mechanism was fine.

Point being, unless you are going to literally continuously test everything, there is still some probability of an unexpected double failure.

Re:backup failure doesn't mean a failure to test

[BitZtream]

As you pointed out, testing can (and in my experience with data center failures is usuaully) be the cause of a failure.

The only time I've ever had an 'outage' in a data center, it was during a test cycle. While thats great that it was during a test cycle, it STILL resulted in an outage. Had the tests not been performed, no service disruption would have happened.

Testing software in a test lab ... you test continuously.

Testing a production environment ... you do it only when you have a real reason to suspect a possible problem, and only then if you can perform the test in such a way that a failure during the test will be less harmful than at some random time.

For a global operation, there often isn't a 'less harmful time'.

Denmark is still without Wikipedia

[Anonymous Coward]

20:47 UTC+1, we are still without Wikipedia probably due to poor DNS propagation

great replies

[vxice]

if only their blog had mod points. all the comments are of the form "still down where ever I am"

Wikipedia goes off the air...

[Anonymous Coward]

...and nobody really gave a damn.

So why are they considered relevant again?

Read that wrong

[leamanc]

Darn, I thought Wikipedia was going to explain today's global outrage.

School assignments

[bjb_admin]

How many kids will go to school tomorrow and say they couldn't complete an assignment because Wikipedia is down?

Distributed Wikipedia

[RAMMS EIN]

Speaking of Wikipedia, an idea that has long been in my mind, but that I have never sat down and worked out is distributed hosting of Wikipedia. The idea is that volunteers each contribute some resources (network capacity, storage space, RAM, and CPU cycles) to host and serve part of the content.

This way, we should be able to reduce the load on the (donation supported) Wikimedia servers, as well as increase the redundancy in the system.

Is anybody already working on this or are there perhaps even already implementations of this idea?

Re:Distributed Wikipedia

[u38cg]

Attempts have been made at the general case, but it is a hard problem: how do you ensure fair resource sharing and reliability?

Re:Distributed Wikipedia

[BitZtream]

Its hard enough keeping a bunch of nodes that you control online and functioning properly (hence the failure) ... trying to run anything reliable when you give any control you had to other random people on the Internet is doomed to fail.

The only reason distributed computing projects like SETI@HOME and distributed.net work is because the server gives clients data to process but it doesn't need a quick response, nor does it have to trust that the data returned is actually valid ... its going to have another host check it at some point anyway to be sure. Those clients are used to weight the data so the master server only processes the most likely packets that may match and need authoritative checking.

Doing that for a web server would ... well, a complete and total waste of resources as its likely to be worse in every single way, including reliability.

Re:Distributed Wikipedia

[Anonymous Coward]

So, freenet without the anonymity? ;)

Re:Distributed Wikipedia

[David Gerard]

It's really hard keeping the databases distributed. Basically, all WMF wikis are served from three large database clusters in Florida. The parallelisation is having those large DB servers feed lots and lots of Apaches (which run PHP and render the pages into HTML) and worldwide Squids for reverse proxying.

Wikileaks was mooting plans for a distributed MediaWiki backend - they have serious need for such a thing - but they haven't managed it either.

There are perennial experimental projects to put something like git as the back end instead of MySQL, but none of these are in any usable condition as yet.

China?

[zorro-z]

And here, I thought that the Great Firewall of China had been blocking access to politically-charged Websites again.

My bad! Sorry...

[citab]

I was looking up curry recipes ... the really hot ones.

Man, I never thought my tapeworm would cause a global outage.

Global "Outrage"?

[sxedog]

I started reading the article and wondered why there was such global outrage about dns resolution on Wikipedia, then I went back and looked at the title again...

Ha

[mikazo]

I happened to be looking up "DNS Failover" on Wikipedia at the time of the DNS failure

More donations = More uptime

[saibot834]

Wikimedia is terribly understaffed. They have about 35 employees, for one of the 5th largest sites on the Internet (and that includes legal/finance/MediaWiki devs/etc. staff). Basically the site is run by a dozen guys. Compare that to any other Top 10 site, this is just crazy.

Given their limited resources (both human and financial), it is amazing that Wikipedia is down so rarely. If you want the site to be more reliable, there is something you can do: Donate to the Wikimedia Foundation

Re:More donations = More uptime

[troll8901]

Well said both GP and PP. Someone kindly mod them up.

I donated a meager US$20, and I have not a single word to criticize Wikimedia. I'm just grateful that it exists to serve me, and has been serving me for FREE for many years.

Overheating?!?

[Anonymous Coward]

Overheating?!? Must be Global Warming - ahem - [man made] Climate Change!

Jokes aside, how do you feel when you lost Wiki?

[porky_pig_jr]

I was rather pissed. And the only thing I was going to do is to look up a few math terms. Ended up using PlanetMath and few other sites, but when Wki came back, I check them as well as guess what: they had the most comprehensive and informative articles. That's the first outage I remember since I started using Wiki.

Re:Jokes aside, how do you feel when you lost Wiki

[owlnation]

It was though there was a great calming came over the Force. Like the dawning of a new age, one based on freedom and facts. One where people were free to write articles without fear of deletion and condemnation. Or edit articles without fear of biased reversion, or banishment.

Suddenly people saw the wood from the trees, and realized there was an whole Internet out there with truth and beauty in it, where jack-booted book-burners were not only not in control, but not welcome either.

And then they brought wikipedia back up again; the black flags flew, and the click of heels was once again heard around the net.

Jokes aside. Indeed, I wish I were joking. There is a lot of pure evil at the heart of Wikipedia.

content-addressed content

[jonaskoelker]

trying to run anything reliable when you give any control you had to other random people on the Internet is doomed to fail.

I've heard a talk from someone who suggested moving to content-addressing: instead of giving you a URL, I give you a sha1 hash of the page you want (and maybe an URL to tell you where to start looking). Then, you don't care from where you get your data, as long as it matches. You can grab the page from the originating host, or from a local cache, or from a bunch of different peers, or from... well, you name it. As long as you get the bits that match the hash, you're happy.

I think the idea is (1) good; (2) pie in the sky; and (3) applicable here.

Re:content-addressed content

[David Gerard]

That's why a git backend is such a tempting idea. Leading to many abortive attempts to write such a thing.

must be nice to have a job where you can f-up

[Anonymous Coward]

that bad and instead of having to apologize, you just have your buddies brag about how smart and competent you are and how well payed.

us little people down here in meatspace... when we f-up that bad, we get this thing called 'fired'.

Re:must be nice to have a job where you can f-up

[X0563511]

Generally there are lots of other little things you do wrong on the road to that firing.

Too bad for you. Do your job right and you tend to not get axed at the next convenient excuse.

Re:Jokes aside, how do you feel when you lost Wiki

[David Gerard]

You realise of course that pretty much all of Wikipedia is multiply mirrored ... answers.com, Google cache, Bing ...