Slashdot Mirror
Third Largest Supercomputer... at Weta Digital
Designadrug writes "This story at the BBC details how the worlds third largest supercomputer (conditions apply) lives at Weta Digital - the company that provided CGI effects for The Lord of the Rings movies. The article also goes on to discuss the 500 TeraBytes of data generated for the films and how the epic Battle of Pelennor Fields almost defeated the film itself."
awwwww yeah
(well, third place)
fp?
Batman would love me!
All the thing says is that IBM did the manufacturing ... were they xSeries, pSeries or zSeries? :-)
:)
(I doubt the zSeries.... nobody buys 3300 processors' worth of mainframe
and for a limited time only... things change tooooo quickly these, todays supercomputer is tomorrows laptop
There was an unknown error in the submission.
Those numbers are nice and all, but what kind of processors are they? I doubt that they are x86...
--- I hate my sig.
Be interesting to know what kind of a file system they use one something like this, and while they say loads and loads of data was generated - how and in what format was it actually stored.
:)
MySQL is prolly not the best fit in this situation
Haba-haba zout-zout
they don't have to go to 'Los Alamos National Lab' or 'Earth simulator' to shoot scene with cool super computer and hundreds of geeks (saves them on extras :-)
At least assembly isn't required...
Future films will use even more digital effects and will require even more data storage. If you consider Bill Gate's famous 640K quote, it won't be long until you'll have a 500 Exabyte keychain...
He is confident... "King Kong is covered in hair," he said, "we could be animating that."
Is it just me, or does that sound more desperate than confident?
If they're just counting the number of cpu's available to do a particular task, don't you then have to include things like Googles setup (10000+)?
the whole article only mentions the processor number to quantify it being a super computer. no tera-flops/seconds. nothing else. they may have the 3rd largest number of processors actively running at one time but a super computer that does not make.
Oh and it was fun to read Houston's comment: "We needed another 1,000 processors and we had nowhere to put them" - Someone must have surely commented "Houston, we have a problem!"
http://efil.blogspot.com/
I'm really looking forward to Weta doing more amazing special effects work in future projects.. without them, LOTR movies would be, of course, still good, but probably wouldn't have the amazing success among the mainstream, non-geek audience.
Actually, I'm drooling right now thinking what could they do with "The Lion, the Witch and the Wardrobe" - depending on how much their stuff would be used in the movie.
Whether you like the three Lord of the Rings films or not there is little doubt that they are a triumph for technology.
Running the huge technological resources behind the films was Weta Digital, a firm formed by Rings director Peter Jackson and others in 1993 to do the effects for the Heavenly Creatures movie.
Back then Weta had only one computer, which it leased, to do special effects work.
Now it runs the third largest supercomputer on the planet if you count the number of processors, 3300, it can call on, says Scott Houston, chief technical officer at Weta.
The ones that beat Weta are the Japanese Earth Simulator (5120 processors) and Los Alamos National Laboratory's supercomputer (8192 processors).
Big database, big problems
The reason Weta's data centre in Miramar New Zealand is not on the Top 500 supercomputer list, said Mr Houston, was because of the way computing power is calculated for those rankings.
The Weta executive was in London as part of a project to promote New Zealand as a place to find skilled technology workers and get programming jobs done.
Weta, said Mr Houston, was a standard bearer for what was possible.
"We have done some remarkable things," he said.
By any measure the amount of data prepared and processed for the movies is staggering.
Lord of the Rings has been a success worldwide
Weta's data store is 500 terabytes in size spread among 220 million files.
Some of that data is a few years old because some sequences, such as the Balrog from the Mines of Moria, appeared in more than one film.
Finding and moving the data for that sequence out of storage so it could be reworked for the second film took about three days.
A database of that size takes a lot of looking after.
Currently Weta is working on better ways to archive and organise it so that it can potentially be re-purposed for other projects.
This work is needed, said Mr Houston, because the current set up means that it would take months to recover from a double drive failure in one of its disk stores.
But most of the data is from the final movie, The Return of the King, which had more than 1500 special effects shots in it. By contrast the first movie had only 400 and the second 900.
Battle to build
One of the longest special effects sequences was the Battle of the Pelennor fields that took place in front of Minas Tirith.
The battle pitted 120,000 digital orcs, each with its own AI fighting style and armour against 8,000 digital Riders of Rohan.
Peter Jackson with friends called Oscar
It was also almost the sequence that derailed the entire picture.
In mid-August last year, Mr Houston had to work out how long it would take to prepare this sequence for the finished film with the resources they had.
He came back with an estimate of 347 days. Given that the immovable release date for the final film was 18 December, that was a few hundred days too long.
"There was never a question of putting any barriers in the way to Peter Jackson's vision," said Mr Houston.
Instead they just had to get more computers, even though Weta's four data centres were chock full of machines at that point.
"We needed another 1,000 processors and we had nowhere to put them," he said.
The answer was to build another data centre nearby, lay a 10gigabit per second cable to it to connect it to the other four and ask IBM to set up a production line just to build the computers Weta needed to stock it.
The result was that within two weeks the new data centre was finished and helping prepare the battle for the big screen.
Future projects
Since then it, and other Weta resources, have been used for effects in the forthcoming I, Robot movie and will help with future releases such as The Lion, The Witch and the Wardrobe.
Peter Jackson's next project is King Kong
For future productions that suddenly need more processing power Mr Hou
If I collected 5121 computers (486s) and connected them all together I would have the largest supercomputer in the world?
"Yeah, IBM? Yeah. . . Apple told me there was a four week wait for my G5. Could ya crank out a couple for me? Thanks. Hugs to all. . ."
I am a believer of momentum and curves.
When he could do beowolf and bring down /. under a rush of nerds posting the same joke?
_O_
.|< The named which can be named is not the true named
...In the 80's.
I am a T-1000 Model.
I have a neural-net processah in a metal alloy exo-skeleton, covered in living tissue.
Honestly now, these video rendering super-computer can't touch a T-1000!
-
They also built the models for the ships used in Master and Commander, but the computer graphics were handled elsewhere.
After seeing these films, I'm going to be very keen indeed to see what these Kiwis can come up with next!
*cough* Top500 List *cough*
And, as always, please show your work
-Patrick
"They never stop thinking about new ways to harm our country and our people, and neither do we."
I actually don't care too much for special FX (but the one's rendered in RealTime on my machine ;) or LOTR actually..
.. oh network-tv already did that..)
(I hope this wont modded 'not geeky' but I saw the first movie and it sortof pissed me off to have to wait for the next movies, which weren't yet released so I decided to not "go there again", and waste money on incomplete movies.. but that's another topic)
It seems kindof a waste to have the 3th 'biggest' computer working for media.
It's actually scary to have the 3th biggest computer 'be in control' of what most of the ppl will see.
(what if that machine becomes aware.. and using images to brainwash or manipulate the
But can someone give some insight on the "Japanese Earth Simulator" they mention?
I think they have a webpage here but it's very first thing I hear about the project...
I think we can keep recursing like this until someone returns 1
"The Return of the King, which had more than 1500 special effects shots in it. By contrast the first movie had only 400 and the second 900."
The funny thing is that personally I lovedthe first movie, really liked the second, and... well... the third movie was pretty good too but seemed a little long.
Many factors could have contributed to this, but after hearing all of Jackson's encomia to model work and miniatures in the DVD "documentaries," I have to wonder whether the increased use of digital effects contributed in some subtle way to some loss of mood or atmosphere or reality in the third movie.
"How to Do Nothing," kids activities, back in print!
Granted, this list http://www.top500.org/list/2003/11/ is a couple of months old but it shows WETA much lower in the rankings.
"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
... would the hair be animated, the hairs would be fighting each other!
Fight hunger. Filet a politician and send him to a 3rd world country of your choice.
I'm sure that there are landfills that beat this setup...
second society
Leela would love me.
Maybe something along the lines of paying for processing time, on millions of computers would get the work done like seti@home.
Personally, I'm looking forward to the 2005 release of the 100000-DVD-set with all 500TB of extra footage and of making of documentaries.
erroneous: look me up in a dictionary
Imagine a Beowulf cluster of these! :D
-- Look to the Rose that blows about us--"Lo, Laughing," she says, "into the World I blow..."
Third largest supercomputer is NOT at Weta Digital
2 posts ago they were building stonehenge, now they have the 3rd largest supercomputer in the world.
I for one...
You've got the terms swapped...
It's an ENDO -skeleton.
"Exo" means "outside".
"Endo" means "inside".
If he had an exo-skeleton, he'd look like Kryten.
That's going over the edge and swimming across the ocean right there.
Mix the failings of Usenet with the shortcomings of the World Wide Web and the result is slashdot.
Isn't it the point of supercomputing to get work done more quickly? Big deal if the gear that does it is big. Isn't it a loftier goal to get the most work done in the least amount of space? Or, in the case of this article, with the fewest number of processors?
do, and with any population as wel7 Practical purposes, documents like a
(general lamness to defeat the lame filter)
I can't wait until WETA does Evangelion. It literally cant be any lesser than 3 times the movie King Kong will be, because it's based around 3 giant mecha... When I find out that it's rated PG-13 however, I will commit suicide.
Insightful? IT'S PASSWORD PROTECTED.
(Damn checks. Yelling was the effect I was going for.)
Saying the WETA render farm is the third largest machine in the world based on the number of processors is wrong. Just check the latest top 500 list and a quick skim points out that Lawrence Livermore National Lab's ASCI White (8192) and ASCI Blue Pacific (5808), Lawrence Berkeley National Lab/NERSC's seaborg (6656), Sandia National Lab's ASCI Red (9632), and Los Alamos National Lab's ASCI Blue Mountain (6144) all have more processors as well as the two already listed.
Also interesting that WETA Digital is listed as #44 on the list too, huh? They only listed a Xeon cluster though with 1080 processors. (prolly not be the same machine, but...).
I love technical articles from the popular press about technical subjects. They do soooo much in depth research. I hope that they don't hurt themselves.
*Disgusted look*
Do you know why the road less traveled by is littered with the bones of the unwary?
and just as many people give a fuck.
From the Top500 List for November 2003:
Earth Simulator - 5120
LANL / ASCI Q - 8192
LLNL / ASCI White - 8192
NERSC / LBNL / Seaborg - 6656
Nice research, BBC.
The T-1000 was that liquid metal guy in Terminator 2. Arnold played a T-9something or whatever. And the other poster is right. It's an ENDOskeleton! Exo-skeletons are what crustaceans and insects have.
Jeeze, man, get your terminator facts straight!
Just for an added showoff of my knowledge of useless facts, the actual line mentioning the processor was: "My CPU is a neural net processah: a learning computer."
Much better would be to use a Grid-type approach that lets Weta call on processing power where-ever it is and when it needs it.
I'm sure that it's obvious, but so is every other great idea I've ever had. So, here's the business plan:
The cure for cancer is coming: Reovirus
They where an older version of the HS20 blades, I think the 2.8GHz version with only a 400MHz FSB.
Third largest super computer and for what? Entertainment. I don't know, it just seems like a skewed sense of priorities. Cancer research, fusion research, any number of possible alternative uses come to mind.
The article spoke of drawing additional computing power from other sources (à la grid computing). I didn't see a mention of it offering any computer power to anyone during their business "troughs".
"Consensus" in science is _always_ a political construct.
...that one of the largest computers in the world is used for entertainment...love the movies tho, so I guess I can't complain too much :-)
-c
did each haird have it's own AI and fighting style?
It's not a supercomputer, it's a render farm - there's a big difference. I'm sure there there are several VFX companies that have that number of processors in their renderfarm. The one I work for has somewhere between 2000 and 2500 processors in it, and I'm sure companies like PDI, ILM and Escape have more.
.rib file from scene for frame # .rib .rib to PRMan
The way the render farms tend to work is this: you have a bunch of jobs, which get sent to the farm. You'll have a bit of software which allocates each job to a computer, or set of computers. Your job might be something like:
for frames 1 - 100
generate
munge
send
This would generate 300 jobs that will get allocated on the farm - obviously some of them are dependent on others being completed. However, each section of the job is a standard program on a fairly standard (normally) Linux install.
If all the computers were being combined into a supercomputer, they wouldn't all be running their own programs, they'd be combined into a huge 'virtual' computer, presumably with each processor running a virtual thread on that computer.
Not even by their own silly measure of number of processors are they third.
He never said it, and even if he did, it was correct--in 1982, 640K was all anyone would ever need. The statement was not false.
The article says it is third by #of CPUs, not by Teraflops or by any computational measure.
Someone could come up with 10.000 dreamcasts linked up together for a few bucks these days, and while it would have quite some rendering power, that does not make it the first supercomputer in the world just because it has more chips in it!
(Score +1, Obvious)
--- "I didn't think anyone would understand it" -Prof. Bob Muller
Any one knows how to build a renderfarm?
I don't want to sound like some tin-foil hat wearing loony, but im sure that those comments will inevitably follow, but, do people really think this list is accurate? With our governments tendency to be very secretive about national security these days, it isn't unreasonable to assume that our most powerful computing resources are public knowledge. The government certainly doesn't want other nations to know even the theoretical limits of our weapons testing programs, etc.
"There's no way to rule innocent men. The only power any government has is the power to crack down on criminals."
From the article:
"The result was that within two weeks the new data centre was finished and helping prepare the battle for the big screen."
"Finding and moving the data for that sequence out of storage so it could be reworked for the second film took about three days."
They can coordinate with IBM to install 1,000 new PCs in a new datacenter with 10Gb from the telco in 2 weeks, but it takes them three days to find a few files?
"Only" 3300 processors? Isn't it said Google has more than 1000000?
If Google really cared they would fix Android Chrome to reflow text, instead of discriminating
Waste of computing resources. Those movies were awful. I'll leave this flame bait message with a quote from Mr. Tolkein himself, "My fans are deplorable..."
It is the Two Towers Extended Edition, actually. Very nice feature on Massive in it.
There is no excellent beauty that hath not some strangeness in the proportion. -- Francis Bacon
That the system is collectively known as "the precious".
But Nyarlethotep Invests - ask him how...
Sara
Designer, Gamer, Macgrrl in an XP World
It's supposedly in the works by WETA, Let's hope it get's the full attention it needs from weta's box of toys. http://eva.trivialbeing.net/
...He rested.
Just to temper a few geeks, yes the visual aspect of LOTR was quite a feet. The main things that pulled it off were : Actor chemestry, and having a predone source that had notoriety. Might i start hollywar three and note the lukewarm nature of Matrix 2b?
He did say it, you stupid idiot. It's the MS shills like you who go around trying to spread this false meme that he never said it--in spite of all the easily obtainable evidence to the contrary (much like the neo-nazis who try to say that the holocaust never happened).
It makes you look ignorant when you keep saying "HE NEVER SAID IT!!!!!" when, in fact, he did. Of course he denies it now, b-b-b-but he'd never lie about that would he? B-b-b-b-but he's never lied in the past, has he?
No, rest easy, he has never lied--and in his book "The Road Ahead" where he never went back and added in the Internet after the fact in order to keep himself from looking like a clueless idiot.
</sarcasm>