Third Largest Supercomputer... at Weta Digital

Designadrug writes "This story at the BBC details how the worlds third largest supercomputer (conditions apply) lives at Weta Digital - the company that provided CGI effects for The Lord of the Rings movies. The article also goes on to discuss the 500 TeraBytes of data generated for the films and how the epic Battle of Pelennor Fields almost defeated the film itself."

Comments

What platform?

[nbvb]

All the thing says is that IBM did the manufacturing ... were they xSeries, pSeries or zSeries? :-)

(I doubt the zSeries.... nobody buys 3300 processors' worth of mainframe :)

Re:What platform?

[OS24Ever]

It was the IBM eServer BladeCenter

Re:What platform?

[Chewie]

Damn, if I had just gone to Top500, I would have seen that it's running on IA32. Not sure whether they're 2-ways or 4-ways, but they're definitely Xeons.

conditions apply

[pbjones]

and for a limited time only... things change tooooo quickly these, todays supercomputer is tomorrows laptop

Re:conditions apply

[Bookcrosser]

things change tooooo quickly these, todays supercomputer is tomorrows laptop

In ten years time kids will be wearing that much power on their wrist and if they don't have a terabyte or two they'll feel left behind.

I was astonished to buy a handheld computer three years ago and realise that it had more computing and better graphics than the early Crays.

But the scary part is that in twenty years, they won't need computers to create orcs and oliphaunts.

Re:conditions apply

[SmackCrackandPot]

Too right ... but progress is good. My supervisor recently bought a laptop - the embedded video card does hardware bump-mapping as well as multi-texturing. Ten years ago you would have been lucky to afford a workstation with that capability.

Today's roomful of rack-mounted systems is tomorrow's server unit.
Today's server unit is tomorrow's workstation.
Today's workstation is tomorrow's desktop.
Todays's desktop to tomorrow's laptop.
Today's laptop is tomorrow's PDA.
Today's PDA is tomorrow's wristwatch.

Re:conditions apply

[cK-Gunslinger]

1) Tomorrow + Tomorrow + Tomorrow + Tomorrow + Tomorrow + Tomorrow = 6 days.

2) Longhorn + DNF!

(Score: +1, Obvious)

Re:conditions apply

[mi]

Second question, what could I use that much processing power on my wrist for?

Reliable and personalized weather prediction? Speech generation and recognition? Carry personal and global archives with you -- with quick searches. Detailed mapping of the surrounding area. Laser/hologram generated games, shows, and other entertaintment pointed at your eyes with quality sound aimed into your ears (so as not to disturb others)? Audio and video communication with anyone on the planet (and beyond)?

And last, but not least, the spare cycles can still be donated to SETI@Home, protein folding, and other worthy projects of choice -- those will always be able to use more...

Re:conditions apply

[LinuxHam]

Speech generation and recognition?

Close to one of my ideas.. instant translation between any two spoken languages. I sure could have used it in Paris. God help me if I have to go to the Tokyo office!

Re:conditions apply

[parksie]

Second question, what could I use that much processing power on my wrist for?

A handwarmer?

Re:conditions apply

[mi]

A heat-problem is not a "given". Modern PDAs are quite powerful, but don't run too hot, for example.

Re:conditions apply

[Anonymous Coward]

I would insert it in my anus.

Re:conditions apply

[Basehart]

"I started off in '81 with a ZX81 (they had a different name in the US, but I can't remember what it was now)which had 1Kb of onboard memory, or a 16Kb add-on pack."

You were lucky.

When I were a lad we'd have to carry over three thousand tons of coal around in our back pockets, heated to the required temperature by a two hundred ton blast furnace we had to grip between our legs, just to get the welcome screen to come on!

What processors?

[BorisZ]

Those numbers are nice and all, but what kind of processors are they? I doubt that they are x86...

Re:What processors?

[protohiro1]

IBM dual xeon blades and dell dual xeon racks. Very off the shelf.

File system ?

[kbsingh]

Be interesting to know what kind of a file system they use one something like this, and while they say loads and loads of data was generated - how and in what format was it actually stored.

MySQL is prolly not the best fit in this situation :)

Re:File system ?

[noelmarkham]

e interesting to know what kind of a file system they use one something like this...

My vote goes for FAT16.

Re:File system ?

[paitre]

Ummm, I've had DB files -much- larger than 2 GB on 32bit linux systems...3 years ago.
I don't find it at all implausible that they've got that much data lying around, at all.
I know of at -least- one research center with more data on disk than that. :)

Re:File system ?

[BiggestPOS]

I can have 9 gig files on NTFS in Windows XP routinely. All the linuxs and BSD are easily capable of this as well.

Re:File system ?

[neuroklinik]

HFS+ has no theoretical limit on file size. It is limited only by the size of the volume (Max 16TB in Panther).

See Apple Knowledgebase article 25557 for more.

Well..

[Anonymous Coward]

they don't have to go to 'Los Alamos National Lab' or 'Earth simulator' to shoot scene with cool super computer and hundreds of geeks (saves them on extras :-)

At least...

[Hangin10]

At least assembly isn't required...

This is just the beginning...

[thesaur]

Future films will use even more digital effects and will require even more data storage. If you consider Bill Gate's famous 640K quote, it won't be long until you'll have a 500 Exabyte keychain...

Re:This is just the beginning...

[dioscaido]

Yeah, famous made up quote.

http://www.wired.com/news/print/0,1294,1484,00.htm l

Still, your point about storage stands.

--

QUESTION: "I read in a newspaper that in l981 you said '640K of memory should be enough for anybody.' What did you mean when you said this?"

ANSWER: "I've said some stupid things and some wrong things, but not that. No one involved in computers would ever say that a certain amount of memory is enough for all time."

Gates goes on a bit about 16-bit computers and megabytes of logical address space, but the kid's question (will this boy never work at Microsoft?) clearly rankled the billionaire visionary.

"Meanwhile, I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again."

Silly quotations do have a way of floating like rumors.

Well, the truth starts here.

He never said it. No free software.

--

Post-project emotional crash

[NSash]

He is confident... "King Kong is covered in hair," he said, "we could be animating that."

Is it just me, or does that sound more desperate than confident?

Re:Post-project emotional crash

[LiquidCoooled]

Indeed he does.

He sounds like he's shitting himself at the prospect.

Sure, they CAN do hair - Monsters Inc was the proving ground for that, but when it comes down to it, the rendering difference between geometric wig units, and actual dynamic flowing furr is immense.
It is similar to the steps from real time game graphics to full on ray-tracing.

(Unless of course they have done some pretty damned nifty optimisations in the last few years)

Re:Post-project emotional crash

[malducin]

It wasn't really an independent animatronic, but a costumed performer. The practical gorilla was made by Rick Baker, though the face was animatronic (radio controlled). Lots of those shots were done either with perspective tricks or just composited in.

Both Dream Quest Images and ILM did CG version of Joe with digital hair. ILM did several shots when Joe is running evading capture, when he crosses the freeway at night with the heli on top and the final shot of Joe running, among others. DQI's Cg Joes was featured in the ferris wheel sequence at the end, when it's in Hollywood (the fun mirror and sitting on the car) and several others.

There are even previous examples than that. One of the first movie CG hair examples is from Jumanji for which they won a Sci-Tech Academy Award later. Other early examples were some commercials by PDI (I believe at least a few months earlier than Jumanji), and Island of Dr. Moreau by Digital Domain.

3rd Largest?

[binaryDigit]

If they're just counting the number of cpu's available to do a particular task, don't you then have to include things like Googles setup (10000+)?

Re:3rd Largest?

[EricWright]

I doubt Google has any one "host" with that many CPUs. As I understand it, they have thousands of machines that work (mostly) independently of each other. Google's goal is to perform a very large number of short tasks very quickly. Weta's goal is to perform one very large task as quickly as possible.

Re:3rd Largest?

[pe1rxq]

Weta's goal is to perform one very large task as quickly as possible.

Maybe not, if you render frame by frame you end up with lots of independ tasks.

Jeroen

Re:3rd Largest?

[John Harrison]

No, rendering each frame is a separate task. This more comparable to Google than to traditional supercomputing applications.

Re:3rd Largest?

[IWannaBeAnAC]

Correction, they currently come in 44th. This presumably doesn't include the extra 1000 processors mentioned in the article, doubling the performance would place it as 16 - although in practice for a distributed calculation you wouldn't get anything like double the performance by doubling the number of boxes.

Look at the difference in R_max (maximum achieved performance in a real LINPACK benchmark) and R_peak (theoretical maximum) - you can see that it is not a machine built for distributed calculations.

Re:3rd Largest?

[mmusson]

The rendering isn't as interesting to me as the generation of the content. Another article I read said that the system simulates every single person/thing/it on the battle field as discrete entities with their own limited form a intelligence in order to create a more realistic battlefield.

The funny thing was that their first attempts were spectacular failures because they attempted to model things as realistically as possible and the good guys fled the field.

So in the end they reduced the effective intelligence of the good guys until they wouldn't run and then they got what we see now.

Re:3rd Largest?

[Finuvir]

In fact the Orcs at the back of the fighting could hear the fighting, so they became "afraid" but they weren't close enough to become "frenzied". The net effect was a large-scale fleeing, starting at the back. Apparently it looked just like a real routing.

Re:3rd Largest?

[galen]

The system is called Massive. During one of the early runs they noticed the guys in the back (on both sides) were wandering off. The problem was that they couldn't 'see' the action so they wandered around randomly looking for opponents. The effect was that it looked like they were running away. The problem was solved by giving the agents something similar to the ability to hear. Thus they could sense the action over greater distances and act accordingly.

I've seen this misrepresented so many times it's begun to bug me. I believe all the information I just dropped is available on the Extended Edition DVDs special features (probably FotR). If not, then I'm sure a Google search will turn it up.

only number is processor number

[musikit]

the whole article only mentions the processor number to quantify it being a super computer. no tera-flops/seconds. nothing else. they may have the 3rd largest number of processors actively running at one time but a super computer that does not make.

Re:only number is processor number

[JAPrufrock]

No kidding. Especially since it sounds more like a renderfarm than a single "supercomputer". Then there's the data storage. That's nothing (though I say it myself). Go to high-energy physics for serious storage/processing centers. Little ol' me uses ~15 TB from 4 months of work, and I'm nothing compared to some people's requirements. I use up about 0.4% of our mass storage here (rough guess). WETA's cool - but not in supercomputer/HPC land.

Well..

[manavendra]

...third largest supercomputer on the planet if you count the number of processors: 3300.
..
The ones that beat Weta are the Japanese Earth Simulator (5120 processors) and Los Alamos National Laboratory's supercomputer (8192 processors).

Doesn't mention the actual processing power. Neither does it mention the individual processing power of each processor. Or what technologies were used to create the digital characters. Or how this "third-largest supercomputer" works...

Oh and it was fun to read Houston's comment: "We needed another 1,000 processors and we had nowhere to put them" - Someone must have surely commented "Houston, we have a problem!" :-)

Looking forward ...

[arhar]

I'm really looking forward to Weta doing more amazing special effects work in future projects.. without them, LOTR movies would be, of course, still good, but probably wouldn't have the amazing success among the mainstream, non-geek audience.

Actually, I'm drooling right now thinking what could they do with "The Lion, the Witch and the Wardrobe" - depending on how much their stuff would be used in the movie.

Re:Looking forward ...

[johnhennessy]

This brings up an interesting topic.

In my opinion what makes special effects add to a film like LOTRs is not how spectacular they look in isolation, but how the director integrates them into their film.

From what I gather, Mr. Jackson was heavily involved (and very knowledgeable) on what effects were being used where, and he was also very specific as to what he wanted.

A hallmark of a good effects film is where you can't tell where the effects begin and where the effects end. Regardless of how good your effects are, if the audience can identify them directly your not going to captivate them as completely.

Re:Looking forward ...

[ScottGant]

Indeed, the good effect is where it comes off as believable and not even noticed. It doesn't matter HOW they did the effect, the bottom line is if it works in the movie.

There are certainly scenes though in movies where it's an obvious effect and doesn't look real. But the ultimate moments in modern effects is where the ONLY way you could tell it's an effect is knowing in your head that such things don't exist.

To this day, one of the most convincing scene is still the original "Jurrasic ParK" where Jeff Goldblum strikes his flair and waves it at the T-Rex...the scene where the T-Rex is chasing him looks like they captured a real T-Rex and put it in the movie. I mean, my jaw just dropped.

There are certainly parts in the LOTR movies that are like this also, that you only know they are effects because they couldn't have done them in real life. Sure, there are parts of these movies where the effects fall short...with the amount of shots in this movie and the amount of time and limit of money (yes, 350 million for 3 huge movies isn't really that much these days), there will be some scenes that could have used more attention. For instance the warg scene in "The Two Towers" could have used more work (PJ even comments on this in the commentary for the DVD). But still, the scene works.

So anyone can build the largest?

[BondGamer]

If I collected 5121 computers (486s) and connected them all together I would have the largest supercomputer in the world?

That must be nice. . .

[noewun]

"Yeah, IBM? Yeah. . . Apple told me there was a four week wait for my G5. Could ya crank out a couple for me? Thanks. Hugs to all. . ."

Why King Kong

[R.Caley]

When he could do beowolf and bring down /. under a rush of nerds posting the same joke?

Not just graphics

[surgeonsmate]

Weta also did the model work. I was over in Wellington some months ago and the model for Barad-dur was on display in the foyer of the national museum. About twenty feet high and superbly detailed. Awesome.

They also built the models for the ships used in Master and Commander, but the computer graphics were handled elsewhere.

After seeing these films, I'm going to be very keen indeed to see what these Kiwis can come up with next!

Re:Not just graphics

[Coryoth]

They also enlisted the help of a sole Kiwi inventor, who'd made a handheld laser modeling gun. Held in the hand, with no external supports or machinery (like a barcode scanner at the shops), it can quickly scan a model into the computer with great detail. You "paint" it with the laser, and it draws it on screen.

It's a small company in New Zealand. They do a lot of interestign work, including some interesting non-mesh based methods for 3D models. Their website is here: http://www.aranz.co.nz

Jedidiah.

Top500

[FU_Fish]

*cough* Top500 List *cough*

More digital effects = less realism?

[dpbsmith]

"The Return of the King, which had more than 1500 special effects shots in it. By contrast the first movie had only 400 and the second 900."

The funny thing is that personally I lovedthe first movie, really liked the second, and... well... the third movie was pretty good too but seemed a little long.

Many factors could have contributed to this, but after hearing all of Jackson's encomia to model work and miniatures in the DVD "documentaries," I have to wonder whether the increased use of digital effects contributed in some subtle way to some loss of mood or atmosphere or reality in the third movie.

And not only...

[GeekDork]

... would the hair be animated, the hairs would be fighting each other!

Based solely on processor count

[jsin]

I'm sure that there are landfills that beat this setup...

Wetta@home

[WillRobinson]

Maybe something along the lines of paying for processing time, on millions of computers would get the work done like seti@home.

DVD set

[erroneous]

Personally, I'm looking forward to the 2005 release of the 100000-DVD-set with all 500TB of extra footage and of making of documentaries.

Look out! Here comes new zealand!

[nfabl]

2 posts ago they were building stonehenge, now they have the 3rd largest supercomputer in the world.

I for one...

Not quite true.

[anzha]

Saying the WETA render farm is the third largest machine in the world based on the number of processors is wrong. Just check the latest top 500 list and a quick skim points out that Lawrence Livermore National Lab's ASCI White (8192) and ASCI Blue Pacific (5808), Lawrence Berkeley National Lab/NERSC's seaborg (6656), Sandia National Lab's ASCI Red (9632), and Los Alamos National Lab's ASCI Blue Mountain (6144) all have more processors as well as the two already listed.

Also interesting that WETA Digital is listed as #44 on the list too, huh? They only listed a Xeon cluster though with 1080 processors. (prolly not be the same machine, but...).

I love technical articles from the popular press about technical subjects. They do soooo much in depth research. I hope that they don't hurt themselves.

*Disgusted look*

Not Third Largest

[turm]

"...third largest supercomputer on the planet if you count the number of processors, 3300, it can call on..."

From the Top500 List for November 2003:

Earth Simulator - 5120

LANL / ASCI Q - 8192

LLNL / ASCI White - 8192

NERSC / LBNL / Seaborg - 6656

Nice research, BBC.

It's not a supercomputer

[EnglishTim]

It's not a supercomputer, it's a render farm - there's a big difference. I'm sure there there are several VFX companies that have that number of processors in their renderfarm. The one I work for has somewhere between 2000 and 2500 processors in it, and I'm sure companies like PDI, ILM and Escape have more.

The way the render farms tend to work is this: you have a bunch of jobs, which get sent to the farm. You'll have a bit of software which allocates each job to a computer, or set of computers. Your job might be something like:

for frames 1 - 100
generate .rib file from scene for frame #
munge .rib
send .rib to PRMan

This would generate 300 jobs that will get allocated on the farm - obviously some of them are dependent on others being completed. However, each section of the job is a standard program on a fairly standard (normally) Linux install.

If all the computers were being combined into a supercomputer, they wouldn't all be running their own programs, they'd be combined into a huge 'virtual' computer, presumably with each processor running a virtual thread on that computer.

Re:It's not a supercomputer

[malducin]

ILM had about 2500 processors on the renderfarm as recent as last year, could ne more now with the work for Ep. 3. Pixar also recently upgraded. At Rhythm and Hues (and many other places) they incorporte the workstations at night for the render jobs so it also depends how you count "render processors".

Rumor has it...

[Ingolfke]

That the system is collectively known as "the precious".