LoTR , Linux, and Database Management

minus23 writes: "Very interesting article over at Digitalanimators.com, talking about some of the challenges faced by the crew working on the second installment in the Lord of the Rings Trilogy. Interesting bits include managing an off-site database of 45TBs, Linux workstations from IBM, 1400 processors, and the animation methods to be used on Gollum. It's a good thing. :)"

Comments

Tolkein just used words

[saphena]

When I read LOTR many years ago, when computers were hard to come by and certainly not used for frivolity such as generating fairy tales, I had no trouble whatsoever "seeing" Gollum and all the other characters just from the textual descriptions.

Does all this computing power mean we've advanced?

Re:Tolkein just used words

[MilesBehind]

It's just for the kids these days. Can't push those few brain cells of theirs that are supposed to bring books to life, so they harness a few rooms of whirring computers to do it for them.

Now if we just used those things to do molecular interaction models for AIDS vaccines, maybe Tolkien wouldn't be spinning in his grave right now.

Re:Tolkein just used words

[palo0019]

Thanks grandpa! Tell me the story about how you walked to school 50 miles in the snow barefoot again!

Excuses to talk about OSS RDBMSs

[philovivero]

This story looks like a good excuse for me to share a little elation I have about Databases that are Free Software.

I've been a Database Administrator and Linux zealot for about 7 years now, and it always got under my skin that there are no good production-quality databases for Linux.

Then, a couple years back, Oracle, Sybase, IBM, and a few other giants made their RDBMSs available for Linux. So I upped the ante, and started complaining that there were no good Free Software databases that were production-quality for Linux.

Then, about nine months ago in New Zealand I started talking to a consultant who told me he'd successfully migrated a few clients off of Oracle onto Postgres. At the time, I was incredulous, because I'd previously reviewed Postgres and found it unsuitable for production systems.

Turns out, my information was outdated (things change FAST in the OSS arena).

Since then, I've been slowly, carefully, calmly trying to see if Postgres (and incidentally, MySQL) were ready for production databases.

Turns out, the answer is pretty much YES for Postgres and, sorry folks, still NO for MySQL.

Postgres is an amazing product. The version I'm running, which is fairly recent at 7.2.1 can create databases based on Oracle-complexity DDL, has good recoverability, stored procedures and triggers, and pretty much everything you'd expect in a full-fledged RDBMS.

They even have a few of those extra bits that aren't necessary but that some DBAs and DB developers like, such as a built-in language (PG/SQL I believe they call it) and ability to write stored procedures in esoteric and strange languages.

I've found their query tool (psql) to be the second-most powerful and useful query tool I've ever used (SQSH being the first).

Amazing product, this Postgres 7.2.1. And from reading the database administrators' mailing list, it's pretty obvious that there are some fairly large-size shops migrating from Oracle to Postgres or even just using Postgres as their main RDBMS.

Re:Excuses to talk about OSS RDBMSs

[philovivero]

I've never played with it so I'm not sure how well it works, but did you know that SAP Open sourced their database?

Hmmm. Do you mean to say you've never played with SAP DB either?

I'll admit, I'd never heard of SAP DB before your post. Going to the website you quote (sapdb.org) makes it look VERY promising.

Just a note, what I consider makes a production-quality RDBMS:

• Transaction support! Can't be stressed enough. Rollback and Rollforward recovery.
• Triggers to enforce referential integrity.
• Stored procedures (or another way to precompile queries and query plans).
• Commercial support. Since I'm paid ungodly sums of money, it's a waste of company resources for me to fix bugs. I need to be able to pay someone $2,000 per bug to fix it. No shit. On the other hand, Oracle, Microsoft, and Sybase have crap support. I have to spend as much time explaining their stupid bug to them as it'd take me to fix it my own damn self. I have met good commercial support before, but you can be guaranteed the Big Boys don't have it.
• Stable, Stable, Stable. If you think Microsoft makes a good RDBMS, I bet you've never supported it in a very large installation.
• Well-thought-out architecture. Well, okay, Oracle's architecture is like spaghetti code personified, so it doesn't meet this criteria. But everyone likes Oracle because it's got everything else, plus a kitchen sink...

SAP DB, from the site, looks damn good. I'll probably take a look at it to see if it's a good contender for Postgres.

It is unfortunate that Postgres is licensed by Berkely license, not GPL. It means that SAP DB can steal code from Postgres and get better, but not the other way around, since SAP DB has been licensed GPL/LGPL.

Re:Excuses to talk about OSS RDBMSs

[dodobh]

There is a postgresql port, that ships with cygwin.
It is as simple as clicking setup.exe, downloading the postgresql binary and starting it.
The big problem is that postgreSQL doesn't run properly on NT as a service by default, you need something like firedaemon to start it.
The PgAccess GUI is available on windows as well, but it lacks a few features that psql supports.
Pg doesn't run on Win 9x at all, AFAIK.

Just a thought

[cr@ckwhore]

Overall, the article was a good read. But, I must point to the following observation..

"... The problem with Linux is that it's an open source system, so if you are having issues or difficulties with its stability, it's like pushing on a rope; there's no single vendor to deal with. ..."

The very next paragraph...

"Weta had just taken delivery of 25 Linux workstations from IBM and Labrie reported that IBM and Hewlett Packard were the frontrunners for additional Linux workstation upgrades."

Alright, so... what am I missing here? You've got IBM behind your efforts. Whats the problem?

Perhaps the comment was referring to specific pieces of software, although my experience has been that dealing with a group of open developers is far more useful than dealing with a single inept vendor. When the vendor is full of crap, where else can you turn?

The first paragraph I mentioned continues...

"You have to be self-deterministic in terms of how things work. You have to make your own choices and do your own tests on motherboards, graphics cards, applications, operating system releases, all those kinds of things."

Again, I'm not buying this comment either... afterall, you have IBM behind you! Don't they test the motherboards, graphics cards, operating system releases, and all those kind of things?

Obviously Linux has been a good solution for them because they're using it. They're having success with it, and its saving them loads of $$ versus using an alternative proprietary system.

Can't wait to see this installment of LOTR!

Re:Just a thought

[zenyu]

Alright, so... what am I missing here? You've got IBM behind your efforts. Whats the problem?

He's probably comparing IBM service with SGI service. IBM will support your PC as well or better than Dell or Compaq, but SGI will send a guy in a cab with extra workstations if you have a problem. They charge for that type of service when you buy one of their PC's, but when they lend you an Origin on short notice you appreciate it.

SGI will gladly sell you Maya for your Linux box, but it's up to you to set up the scanner, find the right 1000Mbps network card, compile a custom kernel, pick the filesystem, etc.

nice, nice computerses

[gripdamage]

Interesting bits include managing an off-site database of 45TBs, Linux workstations from IBM, 1400 processors, and the animation methods to be used on Gollum. It's a good thing. :)

A precious thing, one might say...

Re:Its the problem?

[instinctdesign]

The problem with Linux is that it's an open source system, so if you are having issues or difficulties with its stability, it's like pushing on a rope; there's no single vendor to deal with.

I was going to comment on that too. Frankly, I'd rather have my choice of Red Hat, HP, IBM, etc... than no good choices (i. e. Microsoft).

Asset Management Issue?

[malakai]

I'm amazed in this day in age, they are having a problem with asset management/tracking. Although it's underplayed in the interview, it seems as though the Informix Media 360 was a complete bust.

I can't imagine it was beyond their programmers prowess to create plug-ins or custom scripts that could save the media to a server under some GUID of a filename, and insert a row into a table someplace with the meta-data for that asset. A homegrown content management system is really simple with todays scripting/filesystems/XML. Hell you could throw out the database insert, and just write a filename.xml in the same directory, then harvest the information later.

I'm amazed they stumbled on this, and even more amazed they payed for the Informix product (didn't IBM buy them, and drop that product anyhow?).

Also, is it just me or does it seem like this CTO was 'released' at an odd time?

-malakai

Re:Asset Management Issue?

[foobar104]

A homegrown content management system is really simple with todays scripting/filesystems/XML.

No offense, Malakai, but it's pretty clear that you've never worked on any kind of asset management system. It's a much harder problem than you give credit for. I write asset management systems for a living, so I've had a bit of experience here. A friend of mine, who now works with me, worked at ILM last year and this past spring; he was a compositor. I've talked to him for hours about ILM's asset management system. It's entirely home-grown. If anybody can do it right, you'd think ILM could. But my friend says that it's immensely frustrating in a lot of ways.

The things that were brought up in the article about Media 360 are not new; these are the same problems that all asset management system have to deal with. The biggest one being, of course, that, from the perspectives of the artists, it's easier not to use the system than it is to use it.

I'm amazed they stumbled on this, and even more amazed they payed for the Informix product (didn't IBM buy them, and drop that product anyhow?).

Informix spun the Media 360 product off into its own company, called Ascential. I've heard some ugly rumors about the health of that venture, but I probably shouldn't say anything specific.

Re:am i missing the point?

[Zeinfeld]

Call up any vendor. Tell them their systems are unstable out of the box. Think they're gonna say something like, "oh, yeah, just tweak this little setting...". I don't think the quote above is very logical; no vendor is going to be that helpful with stability issues. Maybe "stability issues" was just a poor choice of words?

Why is it that every time someone with real world experience of running Linux on a large scale talks of a problem the response is always that they must be either mistaken or stupid?

Fifteen years ago you could have made the same coment about running large scale UNIX clusters. Sure you could buy 64 RISC workstations and configure them in a farm, but you would end up rebooting a machine at least once an hour - I know because thats what I was doing fifteen years ago, only with rather more processors.

Experience of running a single machine or a small cluster of office or university machines is not applicable to running large scale systems. If you have a system that is using multiple processors in a single computational task you have to have both software that is designed for fault tolerance and a very high level of basic reliability. If you have a render wall of 256 processors and each one in standalone mode runs for a week without a crash you will end up dealling with a system crash every 40 minutes, most likely more frequently due to interactions between the machines.

This type of processing is the reason people used to pay a hefty premium for systems from folk like DEC who had lots of experience filling a room with machines and getting them to work reliably. Today that ability is the only thing keeping Sun afloat.

One token ring to rule them all....

[AntiTuX]

sorry, I just couldn't resist.

Re:Obligatory

[Provocateur]

Don't know whether to drool at

a. 25 Linux workstations from IBM, or
b. The Lord of the Rings sequel

Oh well, I'll drool anyway

Re:What software are they using?

[Oculus Habent]

I recall an article on WETA. Don't know about the entire rendering process, but they created a program called Massive - it allows each individual character to interact with the environment while still moving with others, i.e. soldiers shifting their weight over unever terrain while still marching.

They are also using Shake from Nothing Real for compositing. Not sure about anything else they use, though.

Re:Notice the last sentence: he left!

[rodgerd]

Weta Digital are doing a cool project. This does not make them a cool place to work.

Re:am i missing the point?

[Sircus]

This type of processing is the reason people used to pay a hefty premium for systems from folk like DEC who had lots of experience filling a room with machines and getting them to work reliably.

Perhaps you should tell that to Google, who seem to have realised you can make Linux work stably enough to run a cluster of 10,000 machines. I'm not saying there's no place in the world for commercial Unix, but the single vendor argument was always weak and remains so. If I pay Red Hat (for example) the same amount of money I pay DEC (Compaq/HP, whatever), there's no reason to expect I won't get the same level of support.

Separately, there's the consideration of whether I'm better off paying DEC/Sun/X this enormous chunk of change for their premium "we don't randomly close your tickets" support level vs. just supporting my large cluster in-house. Clusters are, oddly enough, the place where this comparison leans closest toward the in-house argument - hundreds/thousands of sets of identical hardware means you only have to solve the hardware/software compatibility issues once, only have to keep one type of replacement hardware around, etc.

If you have a system that is using multiple processors in a single computational task you have to have both software that is designed for fault tolerance and a very high level of basic reliability.

Actually, part of the point of clustering is that you don't need enormous levels of fault tolerance. You only need the systems to be as fault-tolerant as the rate at which you can replace them (though, sure, it's nice to have them quite a lot more fault-tolerant than that).

If you have a render wall of 256 processors and each one in standalone mode runs for a week without a crash ...then you have some incredibly unstable software. This isn't a "designed for fault-tolerance", it's not even normal - it's less stable than your average Windows-based desktop system. It's fallacious to use this example to attempt to support your arguments.