Wikimedia Moving Main Data Center To Ashburn, Virginia

hydrofix writes "The Wikimedia Foundation is preparing for the transition of its main technical operations to a new data center in Ashburn, Virginia. This is intended to improve the technical performance and reliability of all Wikimedia sites, including Wikipedia. The current target windows for the migration are January 22nd, 23rd and 24th, 2013, from 17:00 to 01:00 UTC. Since 2004, Wikimedia sites have been hosted in the main data center in Tampa, Florida, a location chosen for its proximity to Jimmy Wales at the time. In 2009, the Wikimedia Foundation's Technical Operations team started to look for other locations with better network connectivity and more clement weather. Located in the Washington, D.C. metropolitan area, Ashburn offers faster and more reliable connectivity than Tampa, and usually fewer hurricanes."

Comments

"and usually fewer hurricanes."

[rueger]

Wow. Talk about tempting fate....... some things you just shouldn't say out loud.

Re:"and usually fewer hurricanes."

[drinkypoo]

Wow. Talk about tempting fate....... some things you just shouldn't say out loud.

So if they're wrong, D.C. gets wiped off the map? A situation with no drawbacks.

Re:"and usually fewer hurricanes."

[Anonymous Coward]

If they had any brains they'd move it outside of the United States, and not because of weather...

DC will get hit this year and there data center wi

[Joe_Dragon]

DC will get hit this year and there data center will go down they are just asking for it.

Virginia weather

[b5bartender]

Remember when electrical storms in Virginia knocked out Amazon's east coast data center multiple times?

Re:Virginia weather

[craigminah]

I'm just down the road from Ashburn and we were fine...no major power outages from last summer's Derecho or the earthquake other than traffic getting even worse. It was Maryland that got FUBARed from storms. They lose their power a lot across the Potomac but Virginia is solid.

Re:Virginia weather

[Skapare]

You mean this ... http://en.wikipedia.org/wiki/June_2012_North_American_derecho ... ?

Re:Virginia weather

[oodaloop]

Uh, I'm also right down the road from Ashburn and we were without power in most of my city for about 5 days. As I remember, about 7 million people in Virginia were without power.

Re:Virginia weather

[Anonymous Coward]

My employer's data center is in Ashburn and when the Derecho event happened, we didn't lose access to our resources. It's a modern data center; reliable backup power was part of the decision to house servers there. Not infallible, but proved itself this time. We have more issues with our ISP having failures and configuration errors.

Re:Virginia weather

[craigminah]

Not sure what to tell ya...our power almost never goes out here while my coworkers who live in Maryland have to shower at the gym at work. Power's funny like that... I bet if you looked at the power grid across the USA it's look like a gnarly mess of spaghetti...

Re:Virginia weather

[Anonymous Coward]

my company's cage is literally right next to the wikimedia cage. We haven't had any issues in there with any of the storms that have hit recently. Amazon East's problems are of their own making. They didn't design/test their power system properly and have let operator error take out their systems.

Re:Virginia weather

[petermgreen]

I've got the distinct impression that most power cuts are a result of problems with the distribution wiring (especially if the local distribution wiring is overhead) on a fairly local level (in britan the 6.6KV, 11KV stuff and the 240/400V low voltage stuff, not sure what typical voltages are for similar systems in america). Afaict the 33KV and up stuff has much greater clearances and is on much strudier supports so it's rarely affected by bad weather even if it's routed overhead (which it usualy is because putting it underground is very expensive).

Though frankly for a datacenter power reliability is probablly not top of the priority list since they can easilly put generators in place. Having reliable and independent communications links and a lot probability of weather that will close the roads (it's hard to fix stuff if you can't get trucks in and out) is probablly more important.

Re:Virginia weather

[petermgreen]

lot probability

That should have said low priority.

Re:Virginia weather

[petermgreen]

No it should have said low probability.

Note to self: posting in the early hours of the morning isn't the worlds best idea...

Re:Virginia weather

[Anonymous Coward]

I live down the road from Ashburn, too, and my power didn't even so much as flicker, but I'm along that short stretch of the Route 7 corridor that has utilities buried all the way to the substation. Data centers, being large consumers of power, don't have as much distribution infrastructure between them and the electrons, and it tends to be the big stuff which gets fixed first when it goes.

Recovery from the derecho took so long because it was a death-by-a-thousand-paper-cuts event. This was actually a good thing to have at some point because it got rid of a lot of the weak timber and is probably why Sandy wasn't so bad for this area. If you're going to be out of power for a week, I'd much rather the summer than fall or winter.

Re:DC will get hit this year and there data center

[Anonymous Coward]

Actually, it'll be Congress's provoking the WRATH of the ALMIGHTY.

The only question is which particular thing will be blamed.

I'll put 50 bucks on violent video games and tolerating homosexuals, with a rider on Social Security Reform.

Hurricanes?

[Bodero]

Tampa hasn't been directly hit by a hurricane since 1921. Not to say it couldn't happen, but I just don't get the 'weather' argument. I remember the reassurances from Amazon Web Services last year when the 'Frankenstorm' headed for Virginia.

Re:Hurricanes?

[oodaloop]

Who cares if it's direct? When a hurricane hits Florida, as many do each season, it covers the whole state.

Re:Hurricanes?

[Billly Gates]

No, but Pasco County in the Tampa area is considered one of the highest lightning capitals in the world. During monsoon season hail and dangerous lightning occurs practically every other day. THat can knock out data easily. I would not be surprised if lightning hits around the facility at least 30 or more times for the summer months.

Virginia gets them too but not so much as it is surrounded by warm 80 degree water on one side in the summer, but not 88 degree water on all 3 sides spawning these thunderstorms.

California obviously would be the best for inclement weather., but has an extremely expensive costs for land, rent, and labor with high taxes and earthquakes.

Re:Hurricanes?

[Anonymous Coward]

Florida hasn't been hit by a hurricane in about 7 years. The closest we've come is a couple tropical storms that brushed one side of the penninsula, while the other side had a sunny day.

Also, just in case you missed it: There is an implied "You idiot!" at the end of the above paragraph.

Re:Hurricanes?

[Anonymous Coward]

Florida is the lightning capitol

Re:Hurricanes?

[Anonymous Coward]

50 mph less wind speed vs a direct hit. You sell insurance don't you?

Re:Hurricanes?

[Y.A.A.P.]

California obviously would be the best for inclement weather., but has an extremely expensive costs for land, rent, and labor with high taxes and earthquakes.

It's not just earthquakes. Taking the state as a whole, California is subject to every natural disaster known to man save for volcanic eruptions. Anybody moving facilities there, when the intent is as close to 100% up-time as possible, is bloody freaking nuts.

Re:Hurricanes?

[k8to]

No, we have volcanoes. See: http://www.nationalatlas.gov/dynamic/dyn_vol-ca.html

Just because they haven't been active lately isn't any terribly good guarantee for volcano timescales.

However we are completely lacking in hurricanes, also blizzards (unless you go pretty far up into the mountains), and tornados.

The usual suspects like floods and fires we have, and more than the country's share of earthquakes. Also more than our share of infrastructure issues.

Re:Hurricanes?

[RabidReindeer]

No, but Pasco County in the Tampa area is considered one of the highest lightning capitals in the world. During monsoon season hail and dangerous lightning occurs practically every other day. THat can knock out data easily. I would not be surprised if lightning hits around the facility at least 30 or more times for the summer months.

Virginia gets them too but not so much as it is surrounded by warm 80 degree water on one side in the summer, but not 88 degree water on all 3 sides spawning these thunderstorms.

California obviously would be the best for inclement weather., but has an extremely expensive costs for land, rent, and labor with high taxes and earthquakes.

Most lightning maps draw a corridor right up I-4 from Tamps to Orlando. The second-highest probability zone is, well, just about the entire rest of the state. Lightning doesn't occur "every other day" in Summer, it occurs almost every day. People routinely get killed or injured.

Still, we know how to handle lightning and storms that take down power. Data centers with smaller budgets than Google are routinely built with motor-generators, backup batteries, power conditioners, etc., not to mention buildings capable of taking Category-4 winds, which is sufficient for almost everything but a head-on hit straight from the ocean of a major storm.

As natural disasters go, Florida actually isn't bad. Most of the really bad stuff can be predicted and planned for in advance. A lot of the scary stuff isn't as bad as it sounds - hail is rarely more than dime-sized, tornadoes are almost never Kansas-grade ferocity. Mildew, on the other hand...

Re:Hurricanes?

[dr2chase]

Lightning (especially Florida lightning -- higher frequency AND higher currents) makes Florida special for two reasons. #1, though we can mitigate power failures, it's not a risk-free operation; increase the rate of power failures, and sooner or later there will be a mistake. #2, if the lightning actually hits your facility all bets are off, and lightning strikes buildings at a decent rate (my parents' house got struck every couple of years when I was a kid -- it was on high ground) and hit my in-laws in Tampa at least once. Once it hits, you get random electrical violence -- in one memorable strike, it got in the fuse box, and blew screw-in fuses straight out of their sockets hard enough to dent the door and knock it off its hinges. Also took out the compressor on a freezer with fish in it, discovered two weeks later.

To make me certain of surviving a strike, I'd want a beefy Faraday cage with power supplied through a motor-generator with a non-conductive transfer shaft. Cooling water would need to traverse an air gap on entry and exit.

Huh. That doesn't raise any flags at all.

[Anonymous Coward]

Virginia, huh? Hmm... What else could be in Virginia, I wonder...?

Re:Huh. That doesn't raise any flags at all.

[Anonymous Coward]

Virginia, huh? Hmm... What else could be in Virginia, I wonder...?

Santa Claus?

Re:Huh. That doesn't raise any flags at all.

[evafan76]

Virginia, huh? Hmm... What else could be in Virginia, I wonder...?

No one, really. Just the CIA, the Pentagon, a lot of the Federal Bureaucracy (although much of it is moving back into D.C.), The FBI Academy, lots of Military Bases, most of the Government Contractors, etc.

Re:Huh. That doesn't raise any flags at all.

[smellotron]

Virginia, huh? Hmm... What else could be in Virginia, I wonder...?

NASDAQ's backup data center. Huh, maybe that means that Ashburn, VA is a reasonably good location for data centers. Spooks can install taps for local snooping in any data center, but getting connectivity to the rest of the world is expensive.

Ashburn

[Narrowband]

Also most of Ashburn is fairly new construction compared to other locations nearby and most of its power lines are buried... that probably contributes to it keeping power on when older communities in Fairfax and Maryland don't. I suspect also the choice of location may have something to do with proximity to others. UUNET (or whatever they are these days), AOL, Verizon, etc., etc.

Re:Ashburn

[TubeSteak]

Haven't most of the recent & large internet infrastructure failures been centered in Northern Virginia?
Amazon in particular has been repeatedly bitten by outages in VA.

Re:Ashburn

[niftymitch]

Not clear about new but old centers like Fort AOL are in N. VA.

Given the data transfer mandates to the TLA centers it makes complying with the law and regulators less vexing if your location is In No. VA.

I'd rather heare they were going distributed

[Bruce Perens]

No one location is safe from natural or human disasters. So, I'd rather hear that they were going to a more distributed architecture and that they'd be able to sustain a complete loss of one data center.

Re:I'd rather heare they were going distributed

[Trepidity]

They do maintain servers in other data centers, in Amsterdam and San Francisco, and use them for offsite backups and read-only Squid caches. They don't live-replicate DBs to them, though, I believe due to the decreased normal-case reliability and performance that you get when trying to replicate DB servers between data centers on different continents. The architecture of centralized DB with worldwide caches performs a lot better and more reliably. But if the VA servers were offline for an extended period of time, they could fail over to the Amsterdam cluster.

Re:I'd rather heare they were going distributed

[rtb61]

It is never a measure of safe, it is a measure of safer. So not risk versus no risk but risk versus lesser risk. Of course major snow storms versus hurricanes, can be a tricky balance of risk. Governors would be above ground power and comms versus below ground power and comms. In this case of Wikipedia access to volunteers would also count, more possible volunteers versus fewer, especially could be tied to the number of Universities in relatively close proximity. Of course land values could also be a major driver and how much support the local community could provide.

Non-US back-up

[aNonnyMouseCowered]

I haven't done much Googling, but it appears that Wikimedia's data centers are all located in the US. Is this still correct? Shouldn't Wikimedia be thinking of expanding its essential operations to other relatively "free" countries like Sweden, as a safeguard against possible natural disasters or human-induced server shutdowns?

Re:Non-US back-up

[Trepidity]

They have a major datacenter in Amsterdam, which backs up all the data, and runs Squid caches to reduce the read latency for European readers.

As a Northern Virginia resident...

[evafan76]

...I have absolutely no idea why anybody outside of the Federal Sector would want their data center in this area. We get Severe Weather (Tropical Storms and Snow Storms) on a semi-regular basis, and traffic tends to jam with a slight dusting of snow or a moderate rain to the point where it can take 3 hours to drive 2 miles, and the Utility Companies are not always the greatest at keeping the power running during these times. Neither of these things can be good for maximizing uptime and minimizing downtime.

Re:As a Northern Virginia resident...

[Anonymous Coward]

Well, you have to go back to the origin of the internet(or 'internetworking' as it was known to early users) to understand why a lot of the physical infrastructure is located where it is. Remember, what we call the internet now started out as a government funded project(ARPANET). Starting in 1985, the National Science Foundation inherited the responsibility for nurturing the internet from DARPA. This network was known as NSFNET. Eventually it was decided to decommission NSFNET and transfer routing responsibilities to various(yet to exist) Network Access Point(NAPs). The first Network Access Point to go into production was the Washington, D.C. NAP (MAE-East, the Metropolitan Area Ethernet).

Fast forward a few years and Jay Adelson and Al Avery(then working at DEC) come up with the idea of starting a Neutral Internet Exchange(NIX). DEC then invested a few million dollars in setting up just such a NIX which became known as the Palo Alto Internet Exchange(PAIX). Adelson and Avery then started a company named Equinix to manage and run PAIX but the deal fell through, and the next best option as they saw it was to compete with MAE-East. But by this time the Fairfax County(where MAE-East is located) planning officials were starting to become hostile to the constant digging that was going on to facilitate connections to MAE-East. However, neighboring Loudon County officials were eager to get in on the action and granted rights-of-way to dig in and lay fiber to Equinix. Thus the first Equinix IX ended up in Ashburn and it currently has the fifth highest amount of data traffic of any IX in the world according to Packet Clearing House.

See https://prefix.pch.net/applications/ixpdir/?show_active_only=0&sort=traffic&order=desc

If you want to move a lot of data and move it fast, you want to be in Ashburn, or very close by.

Policies based on U.S. law

[tepples]

A lot of Wikimedia wikis' policies are based on United States law. The applicable laws differ from country to country. Case in point: Copyright terms for some works are longer in Sweden than in the United States. For example, copyright in any work published before 1978 and more than 25 years before the author's death expires in the United States before Sweden or other EU countries. Putting a datacenter in Sweden would affect which images could be declared public domain on Commons based on its practice of using the later of copyright expiry in the datacenter's country and in the country of the work's first publication.

Re:DC will get hit this year and there data center

[dhall]

Since I moved down to NoVA over 2 years ago, I've seen a hurricane larger than Floyd and an earthquake and just missed a massive snow storm. The trees decided to have a massive orgy leading to one of the worse allergy seasons ever and now we've got a pretty bad flu season. Nothing surprises me anymore.

Re:DC will get hit this year and there data center

[chill]

You forgot the plague of stink bugs.

No hurricanes but ...

[Anonymous Coward]

Ashburn is within the blast zone of DC. Most government operations centers have moved out past Martinsburg WV to be out of the blast zone.

Re:No hurricanes but ...

[Anonymous Coward]

What is that, the blast zone from a 50MT ground burst?? Just wondering, since THAT is the *only* thing that will fix the issues that plague DC...

Re:No hurricanes but ...

[Anonymous Coward]

Ashburn is within the blast zone of DC. Most government operations centers have moved out past Martinsburg WV to be out of the blast zone.

Wha?

That's gotta be one of the dumbest statements I've seen on the internet.

Blast radius from what? Ashburn's damn near 30 miles/50 km from DC.

Re:No hurricanes but ...

[Anonymous Coward]

I am currently sitting in my office doing planned system upgrades. I am within sight of the white house grounds from my 12 floor K street office. If something happens around here, i hope it is on one of my days off.

Redundancy and geodiversity

[Anonymous Coward]

The rest of the world is trying to get redundancy and resilience, and working to that goal through geodiversity, content-distribution-networks, clouds of their own or others (AMZ, Google, etc.). And yet, Wikimedia is moving from data center A to data center B.

The rest of the world is moving away from being in one country's borders, including Google, Amazon, Mega. And yet Wikimedia is moving to the burbs of our nation's capitol.

Has the US Government starting running Wikimedia so they can get a headstart on future Wikileaks, or is it just that Wikimedia really really wants ONLY ONE DATACENTER!!! Because, what's all that stuff about redundancy, resiliency, geodiversity, and reliability REALLY worth.

M

Re:Redundancy and geodiversity

[Chuq]

I was trying to work out what on earth you were talking about.. then I realised you don't know the difference between Wikileaks and Wikimedia.

BTW, as mentioned earlier in these comments, WMF does have additional datacentres in San Francisco and Amsterdam.

Re:Redundancy and geodiversity

[Anonymous Coward]

the guy said "future wikileaks" either way good questions.

Re:Redundancy and geodiversity

[smellotron]

then I realised you don't know the difference between Wikileaks and Wikimedia.

They are one in the same. Ward Cunningham is a mastermind dispatched from Planet X.

Re:DC will get hit this year and there data center

[Anonymous Coward]

And the normal infestation of douchebags.

Quincy

[codecore]

I recommend Quincy Washington. There are a few data centers here already. The power is cheap. The thermal management is good. http://www.coloandcloud.com/editorial/quincy-wa-big-data-centers-leverage-abundant-inexpensive-renewable-energy/

If you want to avoid hurricanes

[Osgeld]

Move it more inland

just sayin, as it stands they are about the same distance from the cost that gets hit the most

Indiana on the other hand ... but I dont know about connectivity

Conveniently located near CIA headquarters

[Animats]

A surprisingly large number of key data centers and control points have been relocated to locations in Northern Virginia near CIA HQ. AOL is there. The Iridium satellite control center is there. (It used to be in Schaumburg, IL, near Motorola HQ) Ashburn alone has four Equnix colo facilities, two AT&T data centers, two Net2EZ facilities, and a few other major centers.

A few miles away in Vienna, VA, even closer to CIA HQ in McLean and less than a mile from "Liberty Crossing" (Homeland Security HQ) there are six more big data centers.

They do have a hot failover

[saibot834]

I think they do replicate DBs to Florida (the former main data center), because they wrote it's a hot failover. The other data centers are just caches, because that's most of the requests.

Virginia has had 90% of all requests anyway already, they are serving bits.wikimedia.org (JS, CSS, ...), upload.wikimedia.org (images and media) and I guess also Squid+Varnish. The "only" thing missing is the actual mediawiki software, databases and things like memcached. Here's the checklist

Tampa

[Nethead]

For those familiar with downtown Tampa, it's the building with the gecko on the side, 10th floor, second cage on the left as you come in. I spent 3 weeks working a few cages down and got to chat with the Wikipedia tech.

So yes, I have gone to Wikipedia.

possible offline copy? how big is it?

[Anonymous Coward]

personally, I'd like to get an offline copy, just in case. how big is the current image (head of all revisions)?