Slashdot Mirror


Weta Digital Grows Cluster

Korgan writes "A little over 3 years after their last upgrade, Weta Digital has just added another 250 more blade servers to their render farm to help with the final renderings of King Kong. From the article: "The IBM Xeon blade servers, each with two 3.4 gigahertz processors and 8 gigabytes of memory, are housed at the New Zealand Supercomputing Centre in central Wellington. They have been added to the centre's existing bank of 1144 Intel 2.8GHz processors, boosting its power by 50 per cent to create a supercomputer with the equivalent power of nearly 15,000 PCs. The servers run the Red Hat version of the open-source Linux operating system. The purchase means the centre is back among the 100 largest supercomputing clusters in the world." And all that computing power is still available for hire when Peter Jackson isn't using it."

42 of 209 comments (clear)

  1. Export restrictions? by BWJones · · Score: 4, Interesting

    So, out of curiosity. What happened to the export restrictions of the US government on CPU's beyond a certain MIPS range? I remember that the old PowerMac 9600/300 eclipsed this federally mandated figure and now we have home game consoles that easily eclipse that performance range. Certainly the advent of cluster computing with commodity hardware made many of these issues moot, but what is the status of the law? Was it repealed or is it just commonly ignored?

    I know that historically, NeXT did quite a bit of work for TLA agencies and that Richard Crandall's program, zilla.app grabbed some attention from interested parties. Because of this work, NeXT had some cash infusion for their hardware even after shutting the line down for general commercial consumption. More recently, Apple has been selling Xserves to some of those same agencies, and contractors for work, but I do not know if they are selling any clusters outside the US?

    The history of course behind this law was that the CIA and NSA were concerned that foreign governments could use compute time to help design nuclear weapons as well as defeat cryptography that might compromise US secrets.

    --
    Visit Jonesblog and say hello.
    1. Re:Export restrictions? by 14erCleaner · · Score: 5, Informative
      They've been eased over the years, e.g. here is a list of articles.

      In 2002 it was upped to 195,000 million theoretical operations per second, and the limit goes up automatically every six months. A typical PC in 2002 was 2000 MTOPs, so this allows export of some rather big honking systems.

      --
      Have you read my blog lately?
    2. Re:Export restrictions? by AceCaseOR · · Score: 2, Informative

      IANAL, but, as I understand it, with the shipping restrictions, they only apply with shipping those computers to certain companies. If you were shipping to, say, North Korea, Iran, or any number of potentially third-world dictatorships, then you'd probably have problems. However, if you're shipping to Austraila or New Zealand, you probably wouldn't have any problems.

      --
      Zagreus sits inside your head, Zagreus lives among the dead, Zagreus sees you in your bed and eats you in your sleep.
    3. Re:Export restrictions? by Hey+Pope+Felcher+.+. · · Score: 3, Interesting

      . . . export restrictions are generally overcome by the all mighty dollar, and besides hasn't New Zealand merged with Hollywood now?

      More interestingly, can anyone see digital actors quickly surpassing their organic cousins, no matter what Peter Jackson says?

      And slightly more interestingly, when will New Zealand surpass California in flim making, it is the ideal location, with better light, more interesting geography, and (at the moment) far cheaper to work in. There are of course the problems with the remoteness of the location, but with the rapidly shrinking world cliché, this is surely no longer such a problem, especially with the work Mr. Jackson is putting in regarding the logistics.

    4. Re:Export restrictions? by Bitsy+Boffin · · Score: 2, Insightful

      New Zealand only has one director of any note, who's only done one film of any note.

      Eh? New Zealand has more than Peter Jackson. For example, Roger Donaldson who is about to premiere "The Fastest Indian" starring Anthony Hopkins, playing a New Zealand character, filmed in New Zealand and to be premiered in New Zealand. Or Niki Caro of "Whale Rider" fame. Or Jane Campion who directed "The Piano" amongst other films. Or Lee Tamahori who's directing credits include "Once were Warriors", an episode of "The Sopranos" and even the James Bond film "Die another Day". Or Andrew Adamson who directed Shrek, Shrek2 and the upcoming "The Lion, the Witch and the Wardrobe". All New Zealanders.

      Not only that but Peter Jackson is much more than "Lord of the Rings", his directing credits also include many films such as "The Frighteners" and the Oscar nominated "Heavenly Creatures".

      And those are just some of the big names I picked off the top of my head.

      New Zealand has a lot of directorial talent for it's paltry 4 million population.

      --
      NZ Electronics Enthusiasts: Check out my Trade Me Listings
  2. Clearly... by TooMuchEspressoGuy · · Score: 5, Funny

    Clearly we need a Beowulf cluster to slay this gigantic King Kong cluster!

    --
    Many Bothans died to bring you this sig.
    1. Re:Clearly... by WormholeFiend · · Score: 2, Funny

      Rumor has it that Peter Jackson is already considering a King Kong sequel, involving a MechaKingKong running Linux.

      He wanted to have a MechaGodzilla, but it was running a Sony proprietary OS.

    2. Re:Clearly... by Mercano · · Score: 4, Funny

      What about a MechaMozilla, the open source alternative?

      --
      #include <signature.h>
  3. Blah by Anonymous Coward · · Score: 4, Funny

    That's nothing. The people arrested in the story earlier today had the computing power of 100,000 personal computers. Beat that!

    1. Re:Blah by sleeper0 · · Score: 3, Interesting

      How do you combine 1644 server class ~3ghz CPUss and end up with the power of 15,000 PCs? Only in the marketing department...

  4. power draw by dreadlocks · · Score: 5, Funny

    that's why the lights dim in Wellington when the cluster is rendering

    1. Re:power draw by Farmer+Tim · · Score: 3, Funny

      You have no idea how hard it is for an Australian to resist making a joke about dim lights in Wellington having nothing to do with the power grid...oh, whoops...

      --
      Blank until /. makes another boneheaded UI decision.
    2. Re:power draw by shokk · · Score: 3, Interesting

      Seriously, I would love to see an article, not on what their render farm is like, but on what their power management for it is like. Do they use remote-managed power strips or are they all just popped in, hoping that they don't blow a fuse. How many UPSs do they have and what kind of on-battery runtime do they shoot for?

      What kind of power draw do the blade chassis have? What blades? What version of Red Hat?!?!?!

      Unfortunately TFA is very short on details and reads more like "Peter Jackson went out and bought 500 computers! Woo!"

      --
      "Beware of he who would deny you access to information, for in his heart, he dreams himself your master."
  5. Re:King Kong? by Capt+James+McCarthy · · Score: 2, Funny

    Sad to say that the pr0n industry has ruined this movie title forever.

    --
    There are no loopholes. It's either legal or it's not.
  6. Explain this "new" math to me... by pla · · Score: 5, Insightful

    They have been added to the centre's existing bank of 1144 Intel 2.8GHz processors, boosting its power by 50 per cent to create a supercomputer with the equivalent power of nearly 15,000 PCs

    Total processors: 1644.

    Now, the Xeons do a bit better than the run-of-the-mill P4, but 10x faster? No way.

    For that matter, they don't run faster at all. They just do somewhat better (as in, 10-25%, not 913%) on certain types of memory-heavy tasks.

    Someone either made a major typo or pulled numbers from their netherregion...

    1. Re:Explain this "new" math to me... by brunes69 · · Score: 4, Informative

      The 250 added blade servers each have two 3.4 GHz CPus, while the existing 1144 only had two 2.6 GHz CPUs.

      Every two new servers is approx. as powerful as three old servers. It is more like they are now running 1519 dual 2.4 Ghz machines, or 3038 2.4 GHz cores.

      Also, remember that a 2.4 GHz is faster than two 1.2 Ghz chips, because of instruction set improvements.

      So, I would not say it would be far off to say that this cluster is approx. the computing power of 15,000 1 - 1.5 GHz machines. This is probably what they are basing the numbers off of.

    2. Re:Explain this "new" math to me... by vertinox · · Score: 2, Interesting

      Now, the Xeons do a bit better than the run-of-the-mill P4, but 10x faster? No way.

      You have to understands this is rendering and not actual tasks of running a multi-threaded desktop environment. If they were using something like Maya3d or their own inhouse app... The answer is yes way.

      When you render to 3d it uses all of the cpu and every cpu you have and every register on the cpu and cache if the rendering software is up to snuff. So what you are looking for is raw computer horsepower. Each cpu can effectively reduce your render time by half (this is in theory because if one scene has more detail/polygons than another than the cpu that is given those frames to render will take longer, but most of the time the quality is the same), but with high end rendering you are looking as massive amounts of time spent rendering depending on how many frames and how high of quality (resolution) you are shooting for.

      Although I am only familiar with low end Maya3d setups. My hunch is they have customized their OS to only run minimal os and the maya rendering farm software. If they are using Maya3d or something equivalent it will take advantage of all cpu registers and cache and what not. If you have xeons with large amounts of cache on the cpu then you will see a benefit with rendering more so than just a regular p4. However, the P4 can usually outperform the Xeon when you need something with mult-threads like running a video game or an OS with a GUI.

      --
      "I am the king of the Romans, and am superior to rules of grammar!"
      -Sigismund, Holy Roman Emperor (1368-1437)
  7. Re:* sigh * by Average_Joe_Sixpack · · Score: 4, Funny

    Things have improved since TRON

    Except Jeff Bridges' acting

  8. Contact? by jridley · · Score: 3, Funny

    Hmm, how to get them on my SETI team....

  9. Re:Obligatory by frank378 · · Score: 2, Funny
    Yes, obligatory.

    In Soviet Russia, cluster grow you.

  10. Nonsense Statement by deadline · · Score: 2, Insightful
    boosting its power by 50 per cent to create a supercomputer with the equivalent power of nearly 15,000 PCs

    Such statements are utter nonsense. First, 15,000 PC's - what kind of PC? (dual core AMD, I think not). Second, how do you measure power ? (is this for their applications, or some other metric) If they ran the numbers they would find the cluster rather typical - unless there is more to the story.

    Yes they have a a lot of processors, however, lots-o-processors != supercomputer

    --
    HPC for Primates. Read Cluster Monkey
    1. Re:Nonsense Statement by deadline · · Score: 2, Interesting
      A good question. I believe at one point Gorden Bell said "anything with 6 zeros in the price".

      The lines have blurred due to clusters. My definition is "a collection of hardware that provides a non-trivial level of performance on a single problem" Of course, "non trivial" has various interpretations. And, working toward solving a single problem is important. Rendering is a trivial parallel application as it is really a bunch of small independent problems. Most supercomputer applications would probably run "sub-optimal" on this system (I assume it has GigE as an interconnect) because they require much more processor to processor communication. BTW, I run the ClusterMonkey site that talks about clusters and HPC if you want to learn more about clusters.

      --
      HPC for Primates. Read Cluster Monkey
  11. Extra-Beefy Power Supply perhaps??? by Anonymous Coward · · Score: 3, Funny

    Power of 1 PC: 250 Watts
    Power of 15,000 PCs: Enough to power a small town

    Ability to do math in your head: Priceless

    For everything else, there's xcalc.

  12. Re:And to think... by delire · · Score: 5, Funny

    Star Wars Episode III : Revenge of the Sith was processed on just a 140-processor Opteron AMD64 farm running Windows 64-bit beta.
    No wonder the film was so bad.
  13. New Zealand targeted by ONU !! by Dam's · · Score: 2, Funny

    In other news, New Zealand is accused by the ONU for not respecting the Kyoto treaty.
    It appears that New Zealand is now the World n1 heat producer, the origin of that heat is currently unknown.

    Well, joke aside, I hope for them than the clim won't break...

  14. The massive power of creating digital realism by digitaldc · · Score: 4, Informative

    The amount of power that is needed to create a realistic outdoor scene with multiple actors is simply astounding. King Kong will most likely be candy for the eyes when it is done. Halo, the next Peter Jackson movie, will probably just as amazing.

    An interesting article on building a digital animation studio (IBM) is here:
    http://www-128.ibm.com/developerworks/library/wa-a nimstudio1/

    --
    He who knows best knows how little he knows. - Thomas Jefferson
  15. Re:Netcraft confirms it by FishandChips · · Score: 2, Funny

    I think it's sheep-power in New Zealand. And hey, after that, Hollywood can reuse them to fleece movie-goers.

    --
    Las qué passoun
    tournoun pas maï
  16. Distributed computing... by Etherwalk · · Score: 2, Interesting

    Given the high degree of parallelism and the social aspects, you'd think that distributed computing would be ideal for hollywood rendering, given that you could implement sufficient security restrictions. (Security restrictions which should be perfectly managable.) How many people out there do you think would like to be able to say "I rendered part of this movie!"

    There are some issues, of course, but it strikes me as worth exploring.

    1. Re:Distributed computing... by Neoprofin · · Score: 2, Insightful

      Not many. The MPAA is already charging $7-12 for me to watch the movie plus another $10-30 if I want to buy it. The last thing they're getting from me is a single cycle of my CPU. If they send me a movie ticket they can use my network, otherwise they can put the billions of dollars they're making to good use.

  17. Sweet! by Black+Francis · · Score: 3, Funny

    The IBM Xeon blade servers, each with two 3.4 gigahertz processors and 8 gigabytes of memory...
    ... now they might even be able to run Vista!

  18. Re:And to think... by malducin · · Score: 4, Informative

    Those numbers are misleadingly wrong. Star Wars Episode III was rendered between the old ILM location and the new Presidio facility. The Presidio has about 4000 processors used for rendering, while old ILM had about 2,500 processors. The data center of the Presidio came online (I think) late last year. So frames from Ep. 3 and The Island were rendered both at the Presidio and old ILM. So surely ILM rendered Ep. 3 on a few thousand processors.

    I think the misleading part is that some articles stated that the initial order for AMD Opteron based machines for the data center was 140 processors. But their renderfarm is crtainly 4,000 procs which I think includes about 1,000 workstations that are used for overnight rendering.

    Data Center Gets Star Treatment

    Also while ILM does have an Opteron based renderfarm they run Linux on them, not Windows64 beta.

  19. The movie will still be a train wreck by kindbud · · Score: 2, Funny

    I mean, come on. Why King Kong? Was the world really lacking yet another King Kong adaptation? This movie will make Gozilla look like Independence Day.

    --
    Edith Keeler Must Die
  20. Re:* sigh * by Comboman · · Score: 3, Insightful

    The problem is that CG is used so indiscriminately now. In the begining, it was only used to do things that you couldn't do any other way (like the living water in The Abyss or the T1000 morphs in Terminator 2). Now-a-days CG is used as a cost cutting measure to do things that could be done in a more traditional way but would cost a little more. I was watching Raiders of the Lost Ark the other day (which still looks great after 20+ years) and thinking, if they shot that movie today they would have just used CG instead of finding 1000 trained snakes, and it would look like crap.

    --
    Support Right To Repair Legislation.
  21. xeons? by ruiner5000 · · Score: 3, Informative

    Did Intel do a massive discount? The standard is Opteron for render farms and for good reason. Much faster, much less power, and a superior upgrade path. 150 watts for the fake dual core Xeon, or 89 for a dual core Opteron. Hmm, tough choice there. Particularly when you look at the new Spec numbers for the dual core Xeon it is a massacre with Opteron well ahead still.

    --
    ignorance is bliss. googlefiberatx.com
    1. Re:xeons? by rodgerd · · Score: 2, Interesting

      Meaningless comment unless you know how they've optimised their code and practises. It may be they have a bunch of optimised render code that works well with the Xeon and would need to be re-written for the Opteron.

  22. Re:sheer power by Paul+Rose · · Score: 2, Funny

    They could reduce that with a little shear power

    Here all week, don't forget to tip your waitress.

  23. Re:And to think... by malducin · · Score: 2, Insightful

    Comparisons with Star Wars aren't helpful. King Kong has more fur. Rendering fur is hard work.

    There is no easy way to compare that. It's highly subjective, even when using the most basic comparison: number of shots. I believe Ep. 3 had 2400+ shots in the final film, although about 2800+ were rendered (about 400 were edited out). Don't remember how many shots King Kong has, though I think it's at least 1,500 though less than 2400. Then you get to the subjective part. for both films how many shots have major 3D work, how many it's mostly compositing, roto and paint, how many are miniature mostly, how many are mixed more or less equaly, how many minutes of digital character animation, etc. So when discussing those issues it's usuallyt a good idea to get as specific as possible and even then you can argue both sides.

    It's probably better to wait for the Cinefex issue in January.

    http://www.cinefex.com/magazine/next/next.html

  24. Re:* sigh * by Total_Wimp · · Score: 3, Insightful

    Am I the only one who prefers models and stop motion animation to the CGI garbage of the last 15 years?

    It depends on the CG. If I don't notice that it's CG I tend to like it. If it looks like CG I tend to groan.

    Jurassic park was extremely well done CG and I loved it. Spiderman was, well, cartoony at best (but a good story and Kirsten Dunst go a long way). In Gone in Sixty Seconds they should have just used real cars in all the scenes. There was no excuse for CG shenanegans. But the New York scene in AI was flawless and would have been impossible to film in scale models alone.

    Notice a trend? If the director is a master of visuals and refuses to accept compromise (just try to tell Spielberg "that's the best I can do") then your CG is gonna work. If your level of visual excellence is better exemplified by Xena the Warrior Princess then you may just be willing to settle.

    I don't mean to bash Raimi. I loved a lot of his stuff, including Spiderman. But did any of you really think Spiderman's level of CG excellence met the level of Spielberg? Directors and producers need to be more demanding of their digital special effects. They should reject mediocre work as readily as wire work with, well, visible wires.

    TW

  25. Excellent! by pyro+jackelope · · Score: 2, Funny

    Available for hire eh? Uh yeah...lemme just pull out my checkbook ;-).

    --
    28:06:42:12 - That is when the world will end...
  26. Re:tangent: "single-blade PCs" are also the future by jherekc · · Score: 2, Insightful

    "...small enough to fit into a keyboard..."

    Funny, my first computer fit in a keyboard (Atari 800XL) then the one after that (Amiga 500) then the one after that (Amiga 1200). wait a sec...

    --
    "lack of quality control is one of the pillars of slashdot"
  27. Re:All this to remake King Kong? by Dubwise · · Score: 2, Informative

    Given that there are about 35 people in the world who haven't seen one version or another yet, how worthwhile is any King Kong remake? I can't wait for Jackson to move onto another project like The Mote in God's Eye or the Foundation trilogy.

    It's not as if it's some desperate idea from the studio. Jackson has wanted to remake King Kong since he was a kid. He tried to make it before LOTR - but Miramax yanked it because there were too many monster movies that season. Now he gets to make it with a much bigger budget and a better cast.

    Do you yknow the original? It's campy as all hell, and I expect Jackson's will be too. It's not going to be just another monster movie.