ILM's Datacenter

kylegordon writes "CGW has inside scoop on Industrial Light and Magic's facilities after they moved from San Rafeal to San Franciscos Presidio. With 3000 disks, it can shift 170Tb to 5000 rendernodes over 10GbE and 1GbE network links. It's an impressive system, for impressive films."

Comments

Nice network

[liliafan]

They seem to have a really nice set up there, I would be curious about how their 'hybrid' NAS/SAN benchmarks, and see some comparisions against some of the big boy equipment like IBM sharks.

Re:Nice network

[flaming-opus]

That's a funny question because I used to work at ILM's (San rafael, much less shiny) lab, benchmarking raids, including the first version of the IBM shark. At that time we came to the conclussion that the IBM raid was reliable, and reasonably fast, but the price was so far out of line, that it wasn't a real contender.

The shark, and many of the high-end raids, are really designed around transaction oriented applications (databases). ILM's application are classic video codes, which work better on a classic raid5, than they do on the data-sprinkler style raids like the shark, eva, clariion, etc. Netapp makes pretty decent storage boxes, and they're highly configurable, so I'm sure they have them fine tuned to the apps' preffered i/o size.

Furthermore, the nas/san has more to do with the spinaker software than the raid of choice. Back when I worked there, ILM was testing cluster sollutions, but the renderfarm was a bunch of sgi origins. The storage was hung off of a couple of 8-way irix boxes, and pushed around with NFS. Since then they've upped their compute capacity by a factor of 30, there's no way they'd be able to do all that I/O with NFS to a couple of big servers. The san setup lets them distribute the NFS load to a large number of servers, all sharing access to the storage on a san. A lot of other cluster filesystems allow this too.

From the benchmarking I've done of these types of storage clusters, you don't get the same single stream performance as you do from a big-iron server setup, but the aggregate across a large number of nodes is pretty good. Managing the mess, and reliability can be problematic. I've never used spinaker, but I've used almost all the other products in this space, and they're all in the "pretty good" category. My current favorite is apple's xsan, because it is really inexpensive, and so is the hardware.

By my calculation ...

[LaughingCoder]

According to Mapquest a trip from San Rafeal to San Francisco would take about 35 minutes (Est. Distance: 21.06 miles). Therefore, if I loaded up all 170TB on a truck my effective bandwidth would be about 3.06e28 bps (or roughly 3e16 Tbps). Once again for huge data repositories there is no substitute for shipping physical media.

What?!?!

[Machina Fortuno]

Okay... not everything they do is shit. That, and CG doesn't make movies worse... only if it sucks. You can go watch claymation if you would like.

http://www.ilm.com/ilm_services.html

Look at all they have done. While some of the stuff on there may have sucked... there is some really fucking good stuff on there.

Also, if I remember correctly, they were some of the first to experiment with particle renders for CG (they used it in the Mask to create some of the storm/tornado transformations). Anyways... thats all aside from the point

Hey... more power to em. They get cooler stuff, they make more realistic CGs. And when all you nay-sayers are watching a movie, and don't notice a good CG... it has worked, and they have won. Don't fight CG now, soon it will just look like everything else.

Re:Hurray for Movie Technology!

[tyler_larson]

Unfortunately, all that storage can't provide decent acting, quality humor or plot lines without holes for their movies.

This statement is more true than you think. One of my high-school friends who went to work for ILM lamented that, as the most expensive special effects house in the business, they attract particulary the films that have nothing going for them but a high budget. No engaging plot, no spectacular acting, just a dumptruck full of money.

What they end up with, and why he was so upset, is that all of the films he's worked on (like Hulk, for example) were over-hyped under-performers. With such a huge advertising budget, the movie gets so much public attention that everybody has seen his handiwork. But nobody is terribly impressed because the movie itself was awful.