Apple Offers No Explanation for 7-Hour Outage (nbcnews.com)

Apple services went offline for up to 7 hours Thursday -- and the company has yet to offer an explanation. An anonymous reader writes: The outage affected the App Store, iTunes in the Cloud, Apple TV, Mail Drop, Find my iPhone, and Photos. During the outage, Apple responded to complaints on Twitter, "Thank you for the information. We're aware of this issue and are investigating," Tech Times reports that the iCloud Music Library had also experienced an outage on Wednesday, and that just weeks ago Apple released an operating system update which bricked several iPad Pros. And yesterday Amazon also experienced a service outage.

Comments

this kind of thing is usually a DDoS

[YesIAmAScript]

When companies have outages like this and don't want to talk about it it's usually a DDoS.

They don't want to brag they can be DDoSed nor do they want to help pump up the group who did it. They don't want to give any info which is feedback about how well it worked either so the attackers can tune their attacks or gauge their chance of success if it was a dry run.

Re:this kind of thing is usually a DDoS

[Anonymous Coward]

They don't want to give any info which is feedback about how well it worked either so the attackers can tune their attacks or gauge their chance of success if it was a dry run.

Nor do they want to risk exposing the possibility that they paid a ransom or protection money to halt the attack or "prevent" a future attack, if something like that was part of this particular occurrence, that is.

Re:this kind of thing is usually a DDoS

[yoshi_mon]

The main reason they don't want to talk about it is because they keep wanting every end user to believe that "the cloud" is some sort of mythical thing. That it will always be there to have their data and that they should pay their monthly subs to have that privilege.

Of course in reality we IT professionals know that "the cloud" is nothing new and not even remotely secure. Local data is always better than remote data and we have the tools to make that a very secure platform for end users. However that is just a one time cost and well...we can't have that now can we Win10?

Re:this kind of thing is usually a DDoS

[Anonymous Coward]

There is no cloud.

https://d21ii91i3y6o6h.cloudfront.net/gallery_images/from_proof/3442/large/1418280711/die-cut-stickers.png

Re: this kind of thing is usually a DDoS

[Rosyna]

While Apple isn't obligated to state while something was does, there are reasons other than a DDoS that seem more likely.

One is that various Apple services use both Amazon S3 and Microsoft Azure for file storage. Given that not every Apple service was down for everyone, it's possible the Amazon outage was related to the Apple outage.

Secondly, numerous other services on the internets were down, including numerous .gov websites belonging to states (California.gov, Connecticut.gov), it's possible a regional Internet backbone was having issues.

Although the most likely reason remains Apple that was working on some backend changes for new features and/or services they'll be announcing at WWDC and something went wrong.

Re: this kind of thing is usually a DDoS

[Rosyna]

Err, "While Apple isn't obligated to state why something was down".

Sigh, I wish Slashdot supported preview mode on mobile.

Re:this kind of thing is usually a DDoS

[quantaman]

When companies have outages like this and don't want to talk about it it's usually a DDoS.

They don't want to brag they can be DDoSed nor do they want to help pump up the group who did it. They don't want to give any info which is feedback about how well it worked either so the attackers can tune their attacks or gauge their chance of success if it was a dry run.

Or motivate more attacks when the media starts digging and finds out Apple payed a ransom.

Re:this kind of thing is usually a DDoS

[Hognoxious]

I reckon their server was installing Windows 10.

Re:this kind of thing is usually a DDoS

[swillden]

Local data is always better than remote data

In what way? Reliability of local data managed by a typical user is generally much worse than that of well-managed remote data. You talk about security, but fail to define what you're securing the data against. Depending on the threat model there are pros and cons, but I'd say that in the case of the typical person their data is more secure against theft and/or leakage when it's stored in a professionally-managed data center behind a good authentication system.

There are advantages to local data, but cloud storage also has significant advantages. There is no one perfect option for all situations, and for many situations cloud storage is better. It facilitates multi-device access, is resilient against device loss or damage and enables collaboration. It can have availability problems, but those can be mitigated with local caching.

All of my important data lives primarily in the cloud, automatically synced to multiple devices. Should the cloud servers become permanently unavailable, I still have local copies, but I strongly expect the real failure modes to be in the other direction -- my local copies may get destroyed, deleted, etc., but the cloud copies will be safe. And for the average user who doesn't think carefully about how to manage their data, good remote storage is an order of magnitude safer and more reliable.

Re:this kind of thing is usually a DDoS

[TheGratefulNet]

I used to think the cloud was great.

then I learned that clouds suck. now I use RAIN, Randomly Available Internet Nodes.

oh, wait. nevermind.

Re: this kind of thing is usually a DDoS

[Anonymous Coward]

When multiple govts and apple services both go down and they share the same provider (Amazon), my conclusion is Amazon got DDos attacked. The simple and obvious conclusion to me lol

Re:this kind of thing is usually a DDoS

[myowntrueself]

When companies have outages like this and don't want to talk about it it's usually a DDoS.

They don't want to brag they can be DDoSed nor do they want to help pump up the group who did it. They don't want to give any info which is feedback about how well it worked either so the attackers can tune their attacks or gauge their chance of success if it was a dry run.

Its like dealing with misbehaving toddlers. You don't tell them off, you don't make a fuss, you don't punish them. Thats all attention.

The only way to make it not worth their while to misbehave is to completely ignore them. Once they've grown up a bit you can use other strategies but for people who are, for all intents and purposes, gigantic toddlers, you have to just ignore.

Re: this kind of thing is usually a DDoS

[myowntrueself]

Spoken like a true modern parent.

So what are you going to do? Indulge them? Encourage the behaviour? Punishment isn't a deterrent; its encouragement.

Re: this kind of thing is usually a DDoS

[Hognoxious]

Servers dont run win10.

Don't go giving them ideas.

Re: this kind of thing is usually a DDoS

[trparky]

Yeah... ideas are bad.

Re:this kind of thing is usually a DDoS

[yoshi_mon]

Local data in the very real sense of keeping your personal data to yourself is a thing. Do I need to go on an NSA rant about how anything you transmit over the internet is compromised?

Further local data is just that local. If they go offline they have their backups right there. They don't have to worry about going online to put their data back in place. They can do it offline if needed.

I understand your arguments for server (cloud) data and have no problem with using server storage when it is appropriate. But we have the ability to have large amounts of data at our local fingertips. Why would I want to give it away when my 1st point is very real? If you could say that the NSA, and all other bodies, were not there then you might have a case. However you do not. The NSA and all the other things are snooping on your data right now. Thank you no.

Re:this kind of thing is usually a DDoS

[swillden]

Do I need to go on an NSA rant about how anything you transmit over the internet is compromised?

In practice, what does that mean to the average person? I fully agree that we should take political action to stop NSA spying, on principle and as a precaution against future abuses, but it has zero impact on my personal security if the NSA reads everything I write, and that is the case for the vast majority of people. So when it comes to figuring out how to manage my data, I'm going to focus on what actually matters. More than anything, that's the probability that the data will be lost due to user error, device failure or catastrophe (house fire, etc.). For the subset of my data that's actually a little sensitive, e.g. my financial information, I also want good access control in addition to reliability.

Seriously, if you have to choose between reliable data which the NSA may look at and unreliable data, which do you choose? And that's not a false dichotomy, that is the choice faced by most people, because they don't have the knowledge required to make sure their data doesn't get lost or destroyed.

Awful editing

[Anonymous Coward]

1) It's really late to post this story

2) Wow, there's a ton of tangential or unrelated crap in the summary

Holy shit the editing sucks now. Amazon's downtime is almost certainly unrelated to Apple's outage. The part about Apple's bad OS update is completely unrelated. Why add that crap to the summary?

Re: Awful editing

[topham]

Yep. They can't let the facts get in the way either. None of my Apple services were down. Lots of people were affected, but it certainly wasn't universal.

Re: Awful editing

[U2xhc2hkb3QgU3Vja3M]

How can the Apple TV be down anyway? It's a device, not a service. I didn't experience any problem watching Netflix with it.

Re:Awful editing

[Anubis IV]

Amazon's downtime is almost certainly unrelated to Apple's outage.

As I recall (and I may be wrong), a lot of Apple's services are still built on top of AWS and Azure. They've been transitioning away from Amazon and Microsoft to their home-grown solutions, but a lot of their core services actually use their competitor's back-ends. As such, it's entirely probable that an outage at the one may be related to an outage at the other happening at the same time.

Re:Awful editing

[dgatwood]

Yes, and one problem with that approach is that too many of Apple's services are centralized unnecessarily. I mean, a sizable percentage of iOS users already have Time Capsule base stations that could easily support iOS backups, but Apple won't let us use them, instead forcing us to either use these cloud-based solutions or back up manually to our laptops and then back up those backups to our Time Capsules.

Taken one step further, Find My iPhone could just as easily do some sort of wide-area Bonjour registration with that Time Capsule even when you aren't at home. This would require some centralized service for some users, but the infrastructure would be much simpler for that case (basically DynDNS), and for folks with public static IPs (whether IPv4 or IPv6), the infrastructure requirements would be nonexistent.

Same goes for photo storage and iTunes in the Cloud, potentially, depending on how fast your home Internet connection is.

And there's no reason you can't use DropBox or other services if Mail Drop goes down.

So basically, out of this entire list of services, had the system allowed decentralization, the only service that would have had a widespread impact would have been the App Store.

Re:Awful editing

[Impy the Impiuos Imp]

Reminds me of Michigan State's on-campus electricity generator. It could burn fossil fuels, but that's too expensive, so they boil water electrically, from the local electric utility. Cutting out the middle man? Forget it.

Re: Awful editing

[MichaelJ]

Actually during that timeframe I found that my Apple TV 3 would not connect to Netflix, while every other device in the house could. So I restarted the ATV3, and all I got were two icons onscreen: Settings, and Computers. At the time I had no idea what was going on. Now it makes unfortunate sense.

Wait a minute, does it have to explain?

[bogaboga]

Apple services went offline for up to 7 hours Thursday -- and the company has yet to offer an explanation...

Can someone tell us whether Apple's "contract" with its fans, or users of its services, contains any clause that requires it to explain incidents like the one described?

Re:Wait a minute, does it have to explain?

[Calydor]

Given frequent stories about iTunes deleting your offline library of music and choosing for you that it is better to download it every time you want to listen to it, having iTunes suddenly be unreachable to download your music is worthy of an explanation, yes.

Re:Wait a minute, does it have to explain?

[ATMAvatar]

It probably doesn't. However, there has been a trend where many tech businesses will do public post mortems of downtime. Some of it is to let people know what happened, but mostly, it's a way of assuring those who use the services that the business knows what happened and that measures have been put into place to prevent it from happening again.

It's a pretty cheap and easy PR move (since you have to do the post mortem analysis anyways) that can turn what would normally be perceived as a huge negative to much less negative (and perhaps even positive!)

While in aggregate, Apple is perceived very highly by the public, they have been getting beaten up lately by very publicly-visible issues with some of their services -- iTunes in particular. I can find no fault in someone wanting some reassurances after such a long outage, and I am a little curious why you do.

Re:Wait a minute, does it have to explain?

[geekmux]

Apple services went offline for up to 7 hours Thursday -- and the company has yet to offer an explanation...

Can someone tell us whether Apple's "contract" with its fans, or users of its services, contains any clause that requires it to explain incidents like the one described?

Does Customer Relationship Management mean anything anymore, or are consumers that much in denial about being nothing more than mindless lemmings, blindly accepting of anything from their service masters?

Re: Wait a minute, does it have to explain?

[BronsCon]

Not if you only have to deal with them one at a time.

maybe im paranoid

[LodCrappo]

apple, amazon, teamviewer and maybe more have had similarish seeming outages recently. one theory in the teamviewer incident was that attackers manipulated dns to gather credentials that the teamviewer client leaked when redirected. another is that the LinkedIn compromise combined with users using the same password on multiple sites lead to the teamviewer exploits and maybe some on Tumblr and myspace too.

spooky week for security things. if this is an organized thing and now apple accounts are targeted, about to get worse. but maybe its nothing, i honestly have no idea.

Re:maybe im paranoid

[antdude]

Same for others according to this news article, but who knows how accurate that is since no other sites seem to mention it too.

A dev is claiming responsibility

[drunkenoafoffofb3ta]

(He uploaded a series of >2Gb app updates at the same time). https://m.facebook.com/story.p...

Re: A dev is claiming responsibility

[Anonymous Coward]

Moleorlrofkeofekofrkfkekf.

His code must be shit. 2 apps both 2gb in size. And ones a map app? You have to be kidding me.

Re:A dev is claiming responsibility

[SeaFox]

Pfft. Yeah, he wishes he had the power to bring down a milti-billion dollar company's cloud infrastructure.
"I uploaded a couple app updates at once and look what I did! Hurr durr."

Cause you know, Apple has so few developers they rarely get that kind of load. So it only takes a few simultaneous transfers to crash their system.

Re:A dev is claiming responsibility

[dgatwood]

Post hoc ergo propter hoc.

Re: A dev is claiming responsibility

[Gary Perkins]

The apps' share a common codebase. The data is different.

Windows 10 upgrade in progress :)

[Anonymous Coward]

You didn't think that Apple runs only Apple computers, did you? Looks like there's been another unintended upgrade. Funny, same thing might be happening to Cricket Wireless today.

Re:Apple, the only certification-lacking cloud...

[ledow]

I was once asked to gather EU data protection guarantees from various cloud companies. Basically, a new law came in that meant that we couldn't just take their word that they only stored and processed our data on EU datacentres, but an actual written guarantee. ISO 27001 was apart of that.

At the time, Apple were the only large cloud company completely unable / unwilling to supply one. I don't know if that's changed because, well... since then we've only ever supplied fake personal information and/or disabled iCloud on products that we use, and the decision was made that Google Apps would be the cloud services of choice because they complied and were free (for schools).

Hilarious that people point fingers at Google for privacy and data processing, and yet Apple was the one to fail hardest on this, for a period of many months, forcing our hand.

What did Apple users do?

[Art Challenor]

Only slightly tongue in cheek... What did the users do? 7 hours without TV, Tunes and Photos. There has to have been some major angst.

Re: Apple, the only certification-lacking cloud...

[ledow]

iPads.
Education.

Though I agree with your sentiment, someone should tell Apple that they aren't targeting education. And the rest of the world that also believes that (because they are TOLD that).

Re:Because Fuck You

[jazzis]

Yeah. Amazon servers have outage. Affected Appl servicese.

So did Cricket Wireless and AT&T

[chill]

I'm not sure of the total duration, but Cricket and AT&T both had major, multi-hour outages on Thursday starting about 4:15 p.m. Eastern. From the chart below, it looks like the majority of it lasted...about 7 hours.

http://downdetector.com/status/cricket-wireless

Things that make you go "hmmmm...."

Reality

[DaMattster]

Technology, i.e. mechanics and electronics, is failure prone. Since major failures don't occur often, we become psychologically conditioned to believe that technology should never fail. I used to battle this all of the time when people would complain about emails not being delivered. It is (and always has been) a best effort service. The SMTP protocol makes allowances for delivery of an email to take 7 days but because most of the time delivery is instantaneous, people have expectations that this will always be. I loved some of the end-users complain that it worked yesterday. Well, yesterday is gone and today is today. I had a PHB once complain about email so I cited the very white paper written on the protocol. I was out of hot water but clearly he did not like the answer.

The obvious answer

[wierd_w]

Seriously-- Itunes and all associated services run on storage controllers.

To me, the obvious answer is a botched OS upgrade on the storage controller head units.

If memory serves, Apple uses Netapp controllers. By now, the support for OnTap 7 should be nearing the end of the legacy support stage. That means data migration to clustered mode in Ontap 8.

Most likely, they established a snapmirror relationship with the new filer equipment, and got the data transferred, but couldnt get the VIF configuration right-- or their 7 mode filer pair went into a takeover-giveback loop cycle of panics.

Not saying netapp is bad equipment, just saying things can and sometimes do happen in a deployment upgrade.

I dont have access to anything that would give me insider knowledge of exactly what happened, but these seem like plausible explanations for an extended, unplanned 7 hour downtime.

It could also be a fibrechannel switch deciding it needed to drop its configuration data and act all goofy, taking out a big chunk of the fabric.

Or worse still, a combination of the two.

I seriously doubt it was planned or malicious.

I smell government hackers

[samantha]

Forces in government are miffed with Apple. On a rogue basis or not they sent a message. Call me paranoid if you like.

Re: Because Fuck You

[jazzis]

Correction: It's a AWS iFeature