Lessons Learned From Skype’s Outage

aabelro writes "On December 22th, 1600 GMT, the Skype services started to become unavailable, in the beginning for a small part of the users, then for more and more, until the network was down for about 24 hours. A week later, Lars Rabbe, CIO at Skype, explained what happened in a post-mortem analysis of the outage."

Comments

Deployed Soldiers.

[puterg33k]

For us it's nearly our only way to speak to our loved ones at home. I'm just glad it's back up...

lesson (hopefully) learned...

[smash]

... relying on dodgy peer to peer VOIP telephony for business purposes is retarded.

we've got people bitching at work about how it doesn't work from time to time, and why I've blocked its ability to do voice/video at the firewall. If you want VOIP, use something that uses standard SIP or some other documented, configurable traffic.

Re:lesson (hopefully) learned...

[smash]

Just let me clarify: corporate networks are different to your home network. your home network? fine, use skype. in the office, where you've got several hundred PCs that may/may not have malicious software and/or users at the helm - allowing all outgoing connections is just begging for trouble.

Egress filtering is a good thing.

Making your day at work "less boring" by enabling you to do non-work related shit with company resources is not what my job is about. It is about ensuring the continued operation of the company's network - and skype is a liability.

Re:Blogspam

[Jurily]

But how else will aabelro promote his own site on Slashdot?! It's just good business sense.

And people wonder why we don't RTFA.

Re:you are kidding me

[TubeSteak]

If you are a node-based company worth several billion, charge for services, and don't even run enough of your own supernodes and monitor them in such a way that they cannot handle an outage effectively, you need serious help.

No one expects 40% of a globally distributed network to crash at once. No one.
FTFA:

The initial crashes happened just before our usual daily peak-hour (1000 PST/1800 GMT), and very shortly after the initial crash, which resulted in traffic to the supernodes that was about 100 times what would normally be expected at that time of day.

Not even a multi-billion dollar company would have a disaster plan that provisions 100x capacity as a hot/cold spare.
Though I bet their new plan includes automatic spawning of nodes on EC2 or some other distributed CDN.

Public Post-Mortem

[Enderandrew]

You can bitch they didn't QA the release. You can bitch that you don't like a P2P topology. But it is nice to see a public post-mortem.

Re:Blogspam

[John Hasler]

My workplace is so backwards they still use old-fashioned telephone lines rather than internet phones.

And consequently you had reliable service while all the "modern, forward thinking" Skype users were down.