Slashdot Mirror
Skype Blames Microsoft Patch Tuesday for Outage
brajesh writes to tell us that Skype has blamed its outage over the last week on Microsoft's Patch Tuesday. Apparently the huge numbers of computers rebooting (and the resulting flood of login requests) revealed a problem with the network allocation algorithm resulting in a couple days of downtime. Skype further stressed that there was no malicious activity and user security was never in any danger.
Care to elaborate, Hercule Poirot?
"You don't need a weatherman to know which way the wind blows." - Bob Dylan
I am not a MS fanboy but it needs to be pointed out that Skype blamed a flaw in their self-healing algorithm that was highlighted by patch Tuesday. They took responsibility.
That's what Skype says. Doesn't sound like they're blaming anyone but themselves.
In Soviet Russia, I ruled you
This wasn't exactly the first ever Patch Tuesday. And didn't skype break on a Thursday anyway?
Yeah, but Patch Tuesday usually involves a dozen patches or less, any handful of which (2-3) might apply to any one system. This one included more than 50 patches, 12 of which were needed by most computers in my office.
ACs are modded -6. I don't read you, I don't mod you, I don't see you. Don't like it? Don't be a coward.
It was just a few days ago the Open Source elders asked people to stop bashing Microsoft. Skype did not blame Microsoft for the outage. They admitted the fault was in their software. We are not children here or part of a cult. This type of child play is no appreciated here.
Skype blames global warming on Colonel Mustard. In the conservatory (greenhouse). With the pipe. Since Colonel Mustard callously smashed all the windows in the greenhouse, it released all sorts of greenhouse gases into the environment thus dooming all the gay, baby polar bears unless the polar bears cooled themselves off by running the AC units of their Hummers at full blast. Why does Colonel Mustard hate the environment?
Well, there's spam egg sausage and spam, that's not got much spam in it.
Something was different last week wrt Microsoft. I had six servers reboot that had autoupdates turned off. My desktop system running 2003R2 and my laptop running XP also rebooted w/o my permission. We have quite a few pissed-off customers because of the updates. It was an unusual situation.
Given that this baby was steamrolled through the Congress two weeks ago, the outage seems coincidental.
Consider that Skype could not tell the users of the real reason even if they wanted to: the law mandates that the forced cooperation be kept in secret.
Obama likes poor people so much, he wants to make more of them.
Does the reboot occur at, say, 2AM local time? If so then reboots would be spread out by the (at least) 24 timezones.
I think this demonstrates the goofiness of a p2p telephone system. If I use Skype, I depend upon my data flowing through other users' computers because I am too dumb to allow incoming VOIP connections to my computer.
VOIP connections should be direct encrypted connections from my computer to the computer of the person whom I wish to contact. Period.
That's when the patches occurred.
I had to leave town and usually leave Thunderbird up and running to filter my mail on my IMAP account so my laptop syncs without having to redo all the filters I have in place. After no reboot on Tuesday I was relieved that I wouldn't have an issue with a down T-bird unless the power went out - which never happens unless I leave town (happened only once before).
Sure enough, none of my mail is filtered after Thursday. Come home this morning and see "Your computer has been recently updated" balloon.
if you steal from one source, that is plagiarism, if you steal from many, well, that's just research.
Skype Blames Microsoft Patch Tuesday for Outage
For the love of God editors, I understand that it is fine to write a sensationalist title on some articles but that is blatant FALSE. It is a complete LIE. People at Skype specifically stated that the fault was in *their* log-in mechanisms.
Really this kind of journalism is disgusting... I am tagging this story as LIE which I hope other people do as well, unless editors change the title.
I find hard to believe Slashdot has got so low... this and the speculative digg-like "articles" ending with a question mark "?", What the fuck.
Ubuntu is an African word meaning 'I can't configure Debian'
Reminds me of the late 90s where AOL's crashing mail servers ended up bringing down my universities server (and many other organizations) because of the surge of load when AOL came back online and started sending backlogged mail.
Perhaps it would be troubling if they were blaming Microsoft. In this case they explained that the large number of simultaneous reboots and subsequent logins simply stressed their servers. They further stated that their "self healing" did not function as designed. It is strange that earlier "patch Tuesdays" did not cause this to occur, but as I write code I find that many behaviors I see in my applications are strange until I truly understand their root cause. It may have been that the software was resilient to a point and then just fell over. Perhaps the point that it fell over was when the "self healing" kicked in and hit its fatal bug.
Load testing is hard. I know. I used to do it. It is hard to anticipate what your peak load might be. It can also be hard to generate the right kinds and volumes of loads that your service might experience. Proper load testing requires a realistic test bed with enough machines running client simulation scripts to sufficiently load the machine. This requires a deep understanding from management that spending large amounts of money on non-production systems is essential. Your setup might deal with some kinds of load well and fail on others. Perhaps Skype had considered what might happen during a natural disaster with a large number of calls originating at the same time, but neglected to see login as a significant risk, especially if they had weathered that storm before.
My least proud moment in quality assurance was seeing my company's service go down for a weekend due to excessive database load. We had a new version of our web service software that required significant database changes to each user account (including database structure redesign...go ahead and wade through that hard book on database principles before you start coding my friends...funny its what I'm doing right now as I go from QA dude to programmer). We made an upgrade script that ran when each user logged in, which brought the user's data up to date with the current version of our software. The thing is I knew about the risk, measured a high load at user login, notified engineering about the potential problem, but didn't demand that the upgrade be placed on hold until the issue could be better quantified. Ah, live and learn.
-Jon
Hey look, if I'm a skilled corporate comms officer -- and I have no doubt Skype has one of those --, and I have to lie about an outage, I'd do it so that it would be believable. All they had to say was:
We recently upgraded our login server authentification routines, and in spite of our testing, we missed something.
The underlying problem with Skype has always been the auth server: everything has to go through it. Worse, when a supernode goes down (e.g., reboots due to a planned install), everything connected to that supernode has to go through it. Now, Skype has been growing pretty fast, pretty much every week their auth servers handle more traffic than the previous week. Your average user might not reboot all computers at the same moment, but what about big enterprises?
And how does Skype pick its supernodes? We know one of the criteria is bandwidth. So let's say in some part of the world where a bunch of little skype clients are wired to a few big bandwidth providers, patch Tuesday hits, and a bunch of those supernodes reset at the same time. The Auth server is hit with the traffic, not from the rebooting supernode, but from all the clients connected to it. That's "peak load" for your auth server, and it increases every patch Tuesday.
It just goes to show that you DON'T have control over your machine when it's running Microsoft Windows and it's on the internet. We have seen problems that result from this level of consumer trust in Microsoft before. I just have to wonder how much more will consumers tolerate? Seems like plenty since most people thing that anything Microsoft does is normal.
How do you know your phone service has never been out in 60 years? Do you monitor it? How many calls a day do you make? Are you home 24/7 and do you use the phone all the time, as in more than 10,000 minutes per month?
Sure, you've never been affected by an outage of your phone service, but that doesn't mean it hasn't been out of service ever.
Plus, you pay for it too. At $30-40/month per line, you expect minimal outages. When you are paying $30/year or even nothing, a two day outage, while annoying, isn't surprising, especially when operated on a public network. Your phone line is on a private, dedicated network. You simply can't compare the two when it comes to uptime.
If all of Skype's customers paid $30-40/month, I'm much more confident that they wouldn't have had this outage.
TossableDigits.com: Temporary Phone Numb
I don't remember where/when this happened, so it might be an urban legend. But the story is that many years ago an earthquake rattled a California town. No major damage was done, but it killed all the phones in the town for several days.
:)
The earthquake had jostled thousands of telephones off hook. The central office switches survived the quake just fine, but crashed due to a bug that seems eerily like the one Skype just described. Basically the switch kept a list of phones that were off hook. The switch is responsible for playing "dial tone" to those phones, but the central office only had a certain number of units that could play dial tone and listen for dialing. So the first "n" phones off hook got dial tone; the rest were put into a FIFO list of phones waiting for dial-tone equipment.
There were so many phones off hook due to the earthquake that the FIFO list overflowed, crashing the switch.
When the switch rebooted, it had to figure out which phones needed dial-tone. So it had to examine each phone line in turn, putting the ones that were off hook into the queue for a dial tone...thus overflowing the list and crashing the switch again. And again. And again.
After a while the telco folks figured out what was wrong, but then couldn't tell anyone about it...since the phones were down. They eventually had police and fire trucks driving all over town, stopping to hang up all the pay phones that were jostled off hook, and blaring over megaphones for people to hang up their phones.
Eventually enough phones were hung up so the switch could reboot without crashing - end of crisis.
Good times.
Under this circumstance, I think it was funny, that they recommended leaving the client running in order to reconnect automagically again once the login service was fixed. Sounds like a bad idea while having login issues...