"Iron Man" Release Brings Down Paramount's Servers

secmartin writes "Shortly after the release of Iron Man on Blu-ray on October 1, people started complaining of defective discs; the problem turned out to be that all the Blu-ray players downloading additional content brought down Paramount's BD-Live servers, causing delays while loading the disc. Which really makes you wonder what will happen when they decide to shut down this service in a couple of years."

Comments

Why is it downloading at all?

[Loie]

With all the storage capacity available on these blu-ray discs why should there be any downloading of additional content? Does the movie really fill up the whole disc? Forgive my ignorace, I still haven't made the blu-ray jump.

Web isn't Really for National Media

[hhawk]

On interactive TV forums I've written extensively talking about how web infrastructure isn't really for national TV and large events with not 100 or 1000 but multiple millions of people try to access the same data within a few seconds of each other.

This is on a smaller scale but certainly proves the point; I do feel there are solutions for pre-caching to tiered servers through the network fabric; but some day when SuperBowl XXX runs and 200,000 TV sets try to access the same JavaTV Applets at the same time... that real fun begins.

Evil or incompetence?

[MojoRilla]

Here's a better story by the way.

So I'm trying to decide if this was evil or just total incompetence.

On the evil side, we have:

• Release a disk with mandatory downloads. They would have to know this will end up bricking the movie for fans. Perhaps they are thinking they can sell "upgraded" disks to the same fans again years later without the stupid download.
• Since no one could be so stupid as to not plan for heavy traffic and use a CDN for content (which they now are), perhaps they planned this failure to get some press about the release of their disk.
• Tell people that it shouldn't happen again, but you have provided a menu to skip the download as if that should make people happy. The fact that they could change the menu means it had to load the menu from their web site. So it still can have timeout issues.

And on the incompetence side.

• Stupidly release a movie which not only downloads mandatory content, but doesn't time out if the download fails. Internet 101 here. 10 minutes trying to connect to a server. Please.
• Don't scale your servers to anticipate traffic. Using a CDN to serve this content is absolutely a no brainer.

Hard to tell. Both are unbelieveable, yet this happened. Thankfully, there is a solution. Don't connect your Blue Ray player to the internet. That will work for now, until they start tying DRM into BD-Live. Idiots.

Re:Not FUD, More Like Therapy For Xbox/HD-DVD Fanb

[friedmud]

I, for one, am one of the people that invested in HD-DVD and hoped it would win out (I thought it was superior technology).... but the same week that Toshiba gave up the ghost... I went out and bought a PS3 and have never looked back.

People need to get over it. Bluray + PS3 = Really Good Platform. The PS3 just does so much more than just playing movies or games... I don't see how anyone with an HDTV and sound system gets by without one...

Anyway... I agree with the GP. I popped in Iron Man last night (rented) and it asked me if I wanted to download the BD-Live stuff. I didn't care so I just clicked "No" and we were able to watch the movie without issue.

BTW - What is the BFD about this movie? I waited to see it from Netflix like usual... but I was really anticipating a great movie from all the hype it got when it was released. Both my wife and I agreed that it was a mediocre movie at best. It had a lot of ridiculous plot elements and quite a few instances of bad acting. The camera work felt cliche and the dialog was uninspired. I just don't get it. I had a friend of mine say that he liked Iron Man more than the Dark Knight... but I don't think they're even in the same league...

Friedmud

Re:PS3

[fuzzyfuzzyfungus]

preview trailers are something you skip over, not something you waste bandwidth on.

Not if the studios have anything to say about it. Remember good old user operation prohibition? And remember how it was only ever used, pinkie swear, for those FBI warnings in the beginning, never for commercials?

I don't know whether they have done so yet; but the studios would love nothing more than to cram a new set of ads into your eyeballs before every showing.

Re:PS3

[SanityInAnarchy]

i thought the BD live content was extra content only downloaded once when you first play the disc--things like bonus scenes, soundtracks, ringtones, and other promo material--rather than just video streamed live each time you play it.

Neither, actually, unless they're being particularly stupid. More likely, downloaded once, whenever either you choose to download them, or the disc does -- and then saved, so you can watch them again. Live streaming would be reserved for places where it actually matters -- as in, content which is also being generated live.

preview trailers are something you skip over, not something you waste bandwidth on.

Preview trailers are also the most trivial, and the most useless, of the things that are possible with this.

I worked on some client-side programming for HD-DVD, before it died. Basically, you've got a little bit of local storage, an Internet connection, and a script engine. You can download small videos and play them, or you can run a program overlaid on top of the movie -- this is how menus were done, but we were doing a lot more than just menus.

Now, from what I remember of Paramount's discs, they pretty much re-downloaded several megs (at least) worth of data on boot -- including every single file needed for said scripts. The only exception was actual media, as in audio and video.

So, they're basically replacing a bunch of data that was already there on the disc. Unlike some other discs, you have no choice -- you will update, before you watch the movie.

That's not really "defectivebydesign", as it's got nothing to do with DRM. It is, however, a defective design. Subtle but very important difference.

It's possible none of this applies to the Blu-Ray, but I suspect it's very similar, and I very much doubt that any of it involves re-downloading the same trailer over and over.

trying to kill first-sale

[Anonymous Coward]

This definitely breaks First-sale Doctrine. coming straight from Wikipedia:

"In 1997 in Novell v. Network Trade Center 25 F. Supp. 2d 1218 (C.D. Utah 1997)[2] purchaser is an "owner" by way of sale and is entitled to the use and enjoyment of the software with the same rights as exist in the purchase of any other good. Said software transactions do not merely constitute the sale of a license to use the software. The shrinkwrap license included with the software is therefore invalid as against such a purchaser insofar as it purports to maintain title to the software in the copyright owner. Under the first sale doctrine, NTC was able to redistribute the software to end-users without copyright infringement. Transfer of a copyrighted work that is subject to the first sale doctrine extinguishes all distribution rights of the copyright holder upon transfer of title."

and

"In 2008, in Timothy S. Vernor v. Autodesk Inc.[2], a U.S. Federal District Judge in Washington rejected a software vendor's argument that it only licensed copies of its software, rather than selling them, and that therefore any resale of the software constituted copyright infringement. Judge Richard A. Jones cited first-sale doctrine when ruling that a reseller was entitled to sell used copies of the vendor's software regardless of any licensing agreement that might have bound the software's previous owners [3]."

Re:Sony could have learned from Microsoft

[mcrbids]

Any service built to assume to 100% uptime is really bad architecture.

If that's the case, then there are lots of really bad architectures in use.

For example, Airline reservation systems. Or Slashdot. Or anything web-based. Or your Visa card, Mastercard, ATM card. Strange how effective those electronic debit machines are, even though they assume 100% uptime of the Visa/MC/Debit back end systems? What about your phone? Doesn't it assume 100% uptime of the call routers and connection?

There are too many examples to mention.

Assuming 100% uptime is only bad architecture if you can't reliably assume near-100% uptime. The important factor is the relative cost of downtime, not the assumption of uptime.

For example, TCP provides a "guarantee of delivery". It overcomes many connection errors in the IP protocol, such as dropped packets, etc, intermittent connection errors, misrouted packets, out-of-order packet delivery, and so on. But no amount of algorithmic magic can change the fact that if somebody trips over a network cable, the destination server has been taken offline for a while.

So we see the real issue isn't whether or not you can count on 100% uptime, but whether or not having downtime in your "100% available" costs all that much.

Are you serving personal pictures on a home DSL line? If so, 99% uptime is probably for you. What's the real cost of a few days of unavailability per year?

Are you serving data commercially? If so, the cost of anything more than maybe 99.9% uptime may not be worth it. (That's about 8 hours of downtime per year) Think about the freebie web server on your local ISP. If it's down for a couple of afternoons per year, is anybody going to complain much?

Are you serving financial records for a state government? If so, the cost of anything more than maybe 99.99% uptime may not be worth it. (That's just under 1 hour of downtime per year)

Are you serving cash Visa for nations? If so, anything more than 99.999% uptime may not be worth it. (That's about 5 minutes of downtime per year)

Each of these "nines" costs exponentially more. A home computer running the latest consumer grade O/S can generally maintain 2 nines without too much difficulty. A basic server running a server O/S (EG: Linux) can generally sustain close to 3 nines without difficulty. When there's a problem, you can drive to the local colo to reboot the server. Keeping a spare server handy and reliable backups means you can recover in less than 8 hours or so. It gets pretty spendy at 4 nines: 99.99% gives you just under an hour. That means you are hosting a fully redundant cluster, with lots of realtime "auto-recover" options. And 99.999% uptime is insanely expensive. Not only are you fully redundant, but you are actually watching each individual process to ensure that it completes, even if the hardware/process dedicated to it fails.

5 nines, along with high performance, can be ridiculously expensive.

As a hosting provider, we're working hard on that "next nine" to do better than 99.9% uptime to achieve 99.99% uptime. When you have to deal with scale, and high performance, it's harder than you think.

Paramount's own fault

[DrXym]

Paramount decided that every BD-Live player should automatically connect to their servers as soon as it was played. With every PS3 offering BD-Live it meant they were deluged with requests. On top of that, they never bothered to make the disks indicate progress or gracefully handle timeouts so lots of people thought their disks had frozen.

There is nothing in the spec that requires this. If they had wanted they could have tested if the player supported networking and added a new menu which allowed users to manually connect to their servers for extra content.

Frankly this is all Paramount's own fault. Aside from the technical fuckup, I have to question the whole ethic of a disk that automatically "phones home" just by inserting it. For starters it means Paramount are tracking usage of this title. It also means the experience could change every time its loaded. Could we see adverts or new trailers being inserted onto disk? Or studios prominently promoting their own online stores or other content? What happens in 10 years if the website bitrots? Will the disk even play any more or will it hang like it did here?

I think it's very telling that the first prominent user of BD-Live immediately abuses it. BD-Live is IMO a waste of time and will continue to be while it used in such superficial and intrusive ways. Every 2.0 player should have the option to disable internet on a global and per-disk basis. Maybe some day a disk will produce a compelling use for it but nothing comes close yet.

Re:PS3

[Lars T.]

Sony has already demoed that, in a scene of "The Departed" that had an empty counter in the theatrical release, they showed a can of "Red Bull" standing on it.

You don't know what you're talking about

[TrekkieGod]

There is no requirement to actually utilize the BD-Live features to watch this title. I picked it up the other day, popped the disc into my PS3 and let it load. You know what happened? A screen came up ASKING wether or not I wanted to download the additional content. I chose not to, and it continued on its merry way to the main menu and I was able to watch the movie without any issues whatsoever.

Well, I picked it up on Tuesday, popped the disc into my PS3 and let it load. It didn't. A blue flashing circle showed up on the screen, and no message whatsoever about what was happening appeared. I took the disc out. I cleaned it and put it back on. I changed the input on the tv and WATCHED A 45 MINUTE TV EPISODE on my DVR, then went back to it to see if it had finished doing whatever it was that it was doing. It hadn't. Eventually I figured it out that it was due to BD-Live servers and changed the PS3 settings.

Here's what happened with you. Either you picked it up before their servers started messing up / after their servers were back up by which point they added the additional menu option that asks if you want to download the additional content or you had your PS3 set up to "ask" before connecting to BD-Live.

Under BD settings for the PS3 there's an option on whether or not to allow the Blu-Ray discs to connect to BD-Live servers. Here's the fucking catch. The two options are "allow" and "ask". You can't set it to I never want to fucking connect . So those of us who were tired of having the menu pop up every single time we put a blu-ray on the PS3 asking if we could allow it to connect to BD-Live had given up and set it to "allow." Then the servers were overloaded, the disc menu never said it was connecting to the net, so I didn't think to turn that feature off, and we had a horrible experience. So no, it's not fud.