Slashdot Mirror
Weta Digital Supercomputer For Hire
sushi writes "NZ's Stuff news site is reporting: 'Peter Jackson's special effects shop Weta Digital has teamed ... to establish a world-class supercomputing facility in Wellington which will be rented out to clients worldwide.' Currently comprising 504 IBM blade servers, each of which contains two 2.8 Gigahertz Intel Xeon processors, 6 Gigabytes of memory and 40 Gigabytes of storage, and ranked 80th in the top 500 supercomputers, they are intending to upgrade into the top 10. Also covered at the Australian Financial Review."
Are we going to see a "Meet the Feebles part 2" or what?
Rent a Bit of Weta Digital
So now instead of curing diseases, finding large mersenne primes or discovering inteligent life forms we can get hyper realisitc renderings of Gollum's eyelash crust. Spectacular.
This is the top 500
fifteen jugglers, five believers
I'm holding out for "More Bad Taste".
6gigs *each*. Sounds ok to me.
DROS - Open-Source Robot Software
Only 6gig of memory? Is it me or that that seem a stupidly low amount when dealing with a amachine of this power?
That's 6GB for each of the 504 (not 512??) nodes, or 3GB per processor, almost 3TB total. Trust me, that's plenty for all but the most extreme uses.
GStreamer - The only way to stream!
Hmm... if you look at the top 500 it appears that 80th was their previous place and that they have since upgraded their cluster to become 77th.
Hrm...
How soon before the bad guys set up a dummy corporation and start running nuclear bomb or protein folding simulations on this cluster? I'd be very interested (probably along with some governments) in Weta's and Gen-i screening processes. Will anyone who can foot the bill get access?
I know, this is tinfoil hat stuff, but it's late and I get this "glass half full" visions when I'm sleep deprived.
Cheers,
E
http://eugeneciurana.com | http://ciurana.eu
my precious.....
Thanks to WETA, now I can run Doom 3!
Click here for a free picture of an iPod!
to other ppl that need to render stuff. Im sure they could figure out some reasonable pricing vs CPU time etc.
6GB per node - i.e. a hair under 3TB of main memory.
Where can I rent an analog one?
It's 6gigs of RAM per each of the 500+ blades. that's over 3 terabytes of RAM total
You know, that only has ~500 nodes right? How many people out there are either out of work, or sick of doing what they're doing? Maybe we ought to get about 1000 of us nerds together in some kind of co-op, cluster our machines then rent it out? My main box is dual opterons and I already have 6 dual P-pro 200s clustered...
Yeah, I know, the logistics of it, the devil would be in every detail... Neat to think about though.
Think for yourself, destroy your television.
Imagine Peter Jackson making a cluster of Beowulf movies using those...
a world in progress...
Missle trajectories: supercomputers can help design accurate missle systems, and missle defense systems. See 1 for why this should be restricted.
;)
Missile trajectories? All you need to calculate that stuff is a Playstation2... Quick! We need to keep these "supercomputers" inside honest nations too
Another movie will probably take too long to render before that renderfarm is needed again for one of Weta's own jobs. Hence hiring out CPU hours while it's sitting still doing very little (and helps pay the rent too, I suppose)
> almost 3TB total
.
.
Do remember that 90% of the time, it's not the size that matters , but the organization.
I've worked on a relatively small cluster processing experiment in college with 12 boxes on a 10 Mbps LAN with a combined memory of 1.5 Gb RAM . It might not look much , but with 32 MB of RAM on each box (each had 128 MB ram) being held by the home cooked shared memory daemon (this was waaay before memcached was born, Ok) , the boxes ran the number crunching beautifully
The operation needed was simple, to sort and process an amazingly HUGE chunk of data in almost realtime (in this case some wierd algorithm some Mech teacher wanted and did up in C).
Anyway in about 7 weeks and reusing a dozen of the college's vanilla PCs we did a LOT of interesting things
So my question is , how's the server connected memory wise (most of these tasks are highly memory bound or at least that's the major bottleneck to optimise).
Quidquid latine dictum sit, altum videtur
So this thingy can stream about 1000 pr0n videos streams per blade... With 500 blades it can stream 500k streams simultaneously. This must be more than enough pr0n for everyone.
How about using all those cpus as Distcc nodes!
(Go Gentoo!)
If someone paid to rent a supercomputer for a few hours to break your SHA1 SSL keys, you've got bigger problems than just your keys being broken :)
Wrong. ILM and Pixar's renderfarms have RAM in the terabytes. Rendering the complex shit they do is EXTREMELY memory intensive.
'Standards' in computing only impress those who are impressed by things like 'standards'.
Yep. Enough for Doom3, or Longhorn. Possibly even enough to run doom3 IN longhorn!!
455fe10422ca29c4933f95052b792ab2
with your budget you probably could play for excatly 1 second...
Whack!
Don't even say it!
Free Software: Like love, it grows best when given away.
Remember that those 6 gigs of ram are per node. Total combined ram is 6gigs * 504nodes = 3024 gigs ram. Grandparent propably meant to say that 6 gigs are enough per node.
We actually bought the old cluster Jackson used (for next to nothing I might add!) It's 62 PCs (no they run Windows) totalling 124 processors (2 racks full)
That's a total of 124GHz and 124GB of RAM. We're using it to render architectural fly-through movies.
Hmm... I'm all hot now... Need a cold shower!
Dude, Eniac computed artillery trajectories. A PS2 could probably do the onboard realtime guidance, nav and detonation sequence :-)
Calculating missile trajectories are less dangerous than making the warheads ..
... and then US wants others to quit simulations too ?. Talk about standing in the way of progress.
... Right now nukes on both sides are preventing India and Pakistan from starting a conventional war (terrorism is bad, but that's a lot less scarring than having tanks roll in onto NH 1 through Leh heading for Delhi).
On one side , real nuclear testing is banned
The US of A knows the importance of mutually assured destruction (@see{Cold War})
Anyway, maybe it's making India a LOT less dependent on commercial US tech for its military . All for the good, I hope.
Quidquid latine dictum sit, altum videtur
Are you kidding? You don't need a Playstation2 or any other kind of supercomputer to plot missile trajectories. Terrorists had no trouble doing that with archaic 6502 processors back in the early 1980s.
Breakfast served all day!
That's 40GB *each*... > 20TB total.
I thought I was going mad, and that stuff just appeared in my head.
Doom 3 Avalon: shoot the spyware and switch to the flashlight to read your documents!
Analogies don't equal equalities, they are merely somewhat analogous.
We will now truly be able to figure out how many New Zealanders it takes to change a light bulb.
Not per machine. Of course, it depends entirely on what kind of work is being done. A lot of these systems that require a huge amount of processing power don't produce a huge amount of data as the end result. This is esspecialy true when doing special effects for a film. The end result is just images/frames of a movie.
Actually they've decided not to use any digital effects for King Kong. Peter Jackson will be wearing a gorilla suit.
Why do you think they're renting out the cluster?
Game... blouses.
I guess with 6 GB RAM each they shouldn't have to do much (or ANY, if I was running this) swapping, and if the jobs are tweaked to not use the hard drives too intensively, they might be OK. If what you do uses the hard drives for much, they are sh*t, to put it mildly. If you could plug these into the blades, they's be very useful, quick machines. But you can't yet.
The really crap thing is, if you do want SCSI drives in the IBM blades, you connect a module ot the side of the blade which gives you a couple of proper SCSI drive bays. Which halves the number of blades-per-bladecenter to 7.
Given the bladecenter is 7U tall, you'd be better off with 7 1U servers with SCSI bays already in and better NIC options. The internal networking of the bladecenter is awful for everything but the simplest low-requirement setups - it's hideously expensive to give each blade a couple of gigabit connections.
Even these cheap little things are 1U, take 2 U320 SCSI drives, and have dual Gigabit connections built-in.
And I *still* can't get USB dongles to work with thes fscking blades, grumble grumble.
Having said all that, when can I play on this thing? My Folding@Home could do with a bit of a boost, and with Hyperthreading I could have 2016 units running simultaneously.... although it might get a little warm behind the racks, 1008 2.8 GHz Xeons pump out a good bit of heat!
You, and plenty of others, have failed to notice, that the stats they gave were *drumroll* PER NODE. And this machine has 502 NODES. So, multiply anything you saw by 502 (6 GB ram PER 502 NODES = 3.012 Teras of ram, 40 GB hds PER 502 NODES = 20 Teras!)
If you can't even read the stub correctly, I know you didn't RFTA...
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
Read this
:-P
"Weta Digital Supercomputer For Hire"
as
"Weta Digital Supercomputer On Fire"
Thought, whoa, finally some big news on Slashdot!
But no... Anyone willing to go with me to put them on fire for some hot Beowulf cluster action?
Beware: In C++, your friends can see your privates!
If they're going to market this capacity, they had better do it quick. The shelf life of computational power is not much greater than milk.
Do they use Tolkien Ring?
~D
I think you should run SETI for one day. I would love to see how many work units can be computed. It would probably set a record.
shit! so can a pencil and paper! damn we're in trouble now.
Theres no reference to WETA DIGITAL's second cluster, #77 on the list. It contains 588 computers as opposed to 504.
top 500 page for the cluser here
Why don't they just combine the two. That would surely grant them a top 10 spot...
Certainly seems like a decent idea. A super computer/cluster/whatever co-op. If you started with it widely distributed, you might make enough to take it to the single building type effort. Makes more sense and seems more likely to actually be useful and make money than any number of failed dot bombs.
They tried, but they discovered an error in the protocol whereby the one who held the token wouldn't let it go.
=)
Lost at C:>. Found at C.
You do realize most /. readers were in diapers last time anyone was using Token Ring, right? That joke probably went right over half their heads.
And I bet the majority on here haven't had to worry about taking a whole office network down by knocking off a terminator on a 10base2...
Damn kids today. Got everything easy.
**angry crowd chants** we want a hobbit movie!!!
On first glance I thought it was WETA, the Washington, DC - based PBS station.
The clearance system sounds logical. It is not. It is completely arbitrary. -- John Bolton
Mmmmmm.... terabyte...
'Standards' in computing only impress those who are impressed by things like 'standards'.
http://eva.trivialbeing.net/ wonder what they'll be using to make that...
450MB for 30 seconds isn't very much when you consider how much time and power it takes to produce those 720 images.
In other words, of the many calculations done in rendering (including ray-tracing, AI, etc.) very little those calculations will end up directly producing the final product, which in this case, is nothing but a simple image.
"Just" images/frames? They're BIG frames with lots of colour-space... which all draw on other frames.
If they're workign with 4K uncompressed film scans, which is becoming the requirement for DI work*, then you're talking data rates in excess of 540 megs per sec or 32 gigs per minute... thats around 1.2 TB for the finished product.
However, visual effects compositing joins together a number of effects elements. I've seen shots with over 90 different elements being combined into a single shot.
So you could have a one minute shot with, say, 10 elements all at 4:4:4 4k uncompressed RGB 10bit Log (people are pushing to 12 bit... and i've heard of 96bit files for some model work)... and that alone would push the storage required to 3.2 TB for the one shot.... a feature at that kind of data rate would be 384 TBs
*2K may be the standard, but its pretty grainy. 4K hides most these problems... 6K is great.
from this site
Our peak rate of
270,147,024 kkeys/sec is equivalent to 32,504 800MHz Apple PowerBook
G4 laptops or 45,998 2GHz AMD Athlon XP machines
ee wtf ? 800MHz PPC faster than 2GHz XP ? dont think so
Go grab those torrents.
>So my question is , how's the server connected
>memory wise (most of these tasks are highly
>memory bound or at least that's the major
>bottleneck to optimise).
it simply isnt, those are Xeons, they share one bus on the mobo, they cant do NUMA, they SUCK dog balls
Go grab those torrents.
I'm sorry. But they are just images. Even if they have a large colour-space and aren't compressed, they are nothing compared to the amount of data used during the processing of the image.
And don't forget, each server is only going to be rendering a very small amount of footage at a time. And these servers probably don't store all the video they create, it would be passed on to the storage network once it's finished rendering it's part.
ee wtf ? 800MHz PPC faster than 2GHz XP ? dont think so
If I remember correctly, the Altivec instructions in the DNet client were hand coded in assembly for the fasted possible execution on G4 processors. In just about every other comparison, the 800mhz G4 wouldn't be faster than a 2Ghz XP, but it's possible in this case.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
in other words they optimized PPC port, bu left AMD unoptimized ? sounds reasonable .. for a Microsoft, but not for such a project ! :/
Go grab those torrents.
It's just a higher level of optimization. Altivec instructions allow the computer to deal with 128 bit word sizes. So if the instructions are properly coded, it's kind of like performing 4 32 bit instructions at once.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
4 32bit sounds like mmx
Go grab those torrents.
There's not been a whole lot of recent music that I've been blown away by, but here's a few from the past couple years at least:
Four Tet - Rounds - I call it "thinking man's electronica." It's very organic sounding to me, and frequently throws in unexpected twists that I enjoy. Kind of a modern version of Brian Eno's early solo work/electronic experementation.
PJ Harvey - Uh Huh Her - She's not new by any means, but I think this is her most solid album in the past while.
Beth Orton - Daybreaker - I'm a huge fan of her earlier work, and this took a little while to grow on me, but it's got some great moments. If you've never been exposed to her, I recommend everything she's done. Also, the Two Lone Swordsmen remix of Anywhere was _excellent_.
The Walkmen - Dunno the album name - The overall album's OK, but I've really been digging the single "The Rat" that was big on college radio
The Rapture - Echoes - Yeah, they're too trendy to be cool to aging hipsters, and yeah, they're basically rehashing GOF, but this album found its way into my coding sessions when I was _almost_ there and needed a little extra push.
Delerium - Karma and Poem - perhaps cliched, but music that I've really loved. The remix collection was also common in coding sessions.
Tim Buckley - various - OK, so he's way past 20 years old, but there's been recent issues of some great material that had only existed in bootleg form before that's great, particularly the early versions of "Song to the Siren" and "Sing a Song For You." There's a medely of Hallucianations and Troubadour on one of the recent live releases that's breathtaking to me.
Richard Bona - various - a very nice blend of African, Jazz, World, and French music.
Tindersticks - Waiting for the Moon - not necessarily as cutting as some of their earlier works, but has some great moments in it.
Low - Things We Lost in the Fire - I've loved Low since the beginning, and at first the newer sound on this album didn't match my tastes, but it really grew on me in the end.
Nields - Play - Gorgeous folk rock.
But yeah, music's really been bad these past years. I listen to a lot of ethnic music and traditional music that I have hundreds CDs of, and get a lot of enjoyment out of those.
The first time I listened to Bitches Brew I had my headphones on and had a toke or two. I swear I saw the face of god. I am still waiting for any artists to invoke that kind of awe. Maybe one day...
evil is as evil does
Man. I wonder how many copies of the LOTR Deluxe DVD set could be stored in RAM?
Founder & COO, Hayai India (hayai.in) / USA (hayaibroadband.com)