Seattle Data Center Outage Disrupts E-Commerce

1sockchuck writes "A major power outage at Seattle telecom hub Fisher Plaza has knocked payment processing provider Authorize.net offline for hours, leaving thousands of web sites unable to take credit cards for online sales. The Authorize site is still down, but its Twitter account attributes the outage to a fire, while AdHost calls it a 'significant power event.' Authorize.net is said to be trying to resume processing from a backup data center, but there's no clear ETA on when Fisher Plaza will have power again."

Comments

Failover Planning (and this broke FiOS too)

[Cysgod]

Apparently Verizon has a single point of failure for much of its FiOS for the metro areas of Western Washington state in this building as well so the FiOS customers are offline as well right now.

• Clownshoes: Have no failover plan and be singly homed.
• Meh: Have a failover plan.
• Good: Have a failover plan that requires humans and exercise it regularly.
• Better: Have a failover plan that is automated and exercise it regularly.
• Best: Eliminate single points of failure so failover is turning off the flake or fail and going back to drinking a beer.

Hot/Hot is always a more ideal solution than Hot/Warm or Hot/Cold for disaster recovery (and increasing equipment utilization/ROI), and this event demonstrates why.

Fisher Plaza is a disaster response center

[Anonymous Coward]

Fisher Plaza is supposed to be a regional telecomm / communications / medical care hub for the Seattle area. It was designed and built to *not* crash, even in a magnitude 9.5 quake. Sounds like they've got work to do ...

System failure

[ErkDemon]

There are four main factors that can take a part of a society's key infrastructure offline.

1: ACTS OF GOD
Meteor strike, lightnight strike, extreme weather ...

2: ACTS OF MALICE
War, terrorism, extortion, employee sabotage, criminal attacks ...

3: WEAK INFRASTRUCTRUCTURE
Underpowered networks, inadequate UPS backups, skeleton staffing, the shaving of safety margins as an efficiency exercise, inadequate rate of replacing old hardware ...

4: MANAGEMENT ARSINESS
This is when a problem starts, and the people in charge either don't know how to react, don't care, or prioritise face-saving over actual problem-solving. This happens when you get an outage, and instead of system management promptly calling all their critical clients to inform them, and warn them that there's maybe twenty minutes of UPS capacity in the routers if the system's not fixed by then, they instead cross their fingers and hope that things'll work out, and worry about what to tell the clients afterwards.

Fisher Plaza seems to have suffered from a case of #4 recently, so it's not surprising that they've gone down again. The first time should have been the wakeup call to show them that their human systems were in need of an overhaul. Without that overhaul, you're setting up a dynamic in which the second time it happens, things are even worse (because now people are locked into defensive mode).

No matter how advanced your technological systems, if the people running it have the wrong mindset, you're gonna go down. And when you go down, you're gonna go down far far harder than necessary.

Geocaching.com too

[dickens]

And on a holiday. Bummer. :(