Slashdot Mirror

← Back to Stories (view on slashdot.org)

HP & Commodity Computing

Posted by Hemos on from the the-grid-network-cometh dept.

Handpaper writes "The BBC has a story about HPs SE3D lab's pilot scheme to provide raw rendering power for smaller studios and amateurs. A sample movie is available.. " Yes, the long fabled "grid computing" may arrive soon on a massive scale.

64 comments

  1. Grid-Computing? by fembots · · Score: 4, Insightful

    Is this considered grid computing (which enables the sharing, selection, and aggregation of a wide variety of geographically distributed computational resources)? All processing seems to come from one source, more like lease-computing?

    One of the articles mentioned "All the animators are independently funded to make their films.", but none mentioned the cost to use 1 unit (however it's calculated) of the processing power. How do animators justify the expenses to sponsors?

    --
    Rock that crushes, Paper & Scissors that don't matter.
    1. Re:Grid-Computing? by Anonymous Coward · · Score: 0

      Imagine a beowulf cluster of

      oh, wait.

  2. Grid computing is a buzzword. by Staos · · Score: 1, Funny

    Like, if you fit in "grid computing" in your grant proposal, the probability that you'll get funding increases. Now, if in addition to "grid" you manage to fit in "nanotechnology", "bio-informatics" and "paradigm" you'll be funded with a probability very close to 100 %!

    --
    In Soviet russia, only old Koreans profit from pictures of Natalie Portman stored on Beowulf Clusters.
    1. Re:Grid computing is a buzzword. by Anonymous Coward · · Score: 0

      'Struth. We fit all but "nano-technology" into our last renewal, and got a really high score. Means I'll probably have a job for another 5 years.

  3. tragedy of the commons? by Anonymous Coward · · Score: 0

    How does grid computing keep people from mis/over-using the resource?

  4. Wonder what they do with their "idle" CPU cycles by xmas2003 · · Score: 4, Informative

    Since they have the hardware in place, I wonder what they do when they don't have films to make and/or work - i.e. would they consider contributing those idle CPU times to something like the Folding@HOME project ... the powder2glass team would love the work units! ;-)

    --
    Hulk SMASH Celiac Disease
  5. How about security by wheelbarrow · · Score: 4, Interesting

    I know that data security is a top concern in today's animated film business. This is why render farms for animated films are in secure office buildings rather than data centers. Additionally, the render farms are not networked to the internet.

    This makes me skeptical that a 'lease farm' model can work for anything more secure than things like TV Commercials.

    1. Re:How about security by bboy_doodles · · Score: 1

      Yes, security is an issue, but I don't buy your idea that this model can only work for less secure things like TV commercials.

      With the proper encryption, I think these render farms could be used for applications where high security is needed.

      Your argument is somewhat similar to saying that online commerce will never take off because the risk of snooping is too great.

    2. Re:How about security by Ignignot · · Score: 1

      The reason they aren't networked to the internet is to make things cheap, not to make things super secure. Think of it, which is cheaper - implementing good security on an entire render farm, or just disconnecting it from the outside world? These things have to run on limited budgets, and every cent matters. If it is disconnected, you don't even have to worry about security patches for your software! The less work the better.

      --
      I submitted this story last night, and it didn't get posted.
  6. OSS? by comwiz56 · · Score: 1

    Could someone not write something like this but open source and distributed. The animators could all put the client programs on their machines, and when they needed to render something, they could submit it to the central server for queueing up. Of course many rules would need to be imposed, such as 30 scene limits to allow everyone to cycle through the queue quickly, or 20 minute process time limits, for those really detailed scenes.

    Of course, security would be another issue. There would need to be some way to encrypt data, even when being processed on a machine. And bandwidth for transfering all of this would pose another issue.

    1. Re:OSS? by Anonymous Coward · · Score: 0

      anything distributed cannot guarantee when your scene would be started nor finished, which is the most needed for the types of people they are looking for.

    2. Re:OSS? by happyfrogcow · · Score: 1

      Kind of specialized, in that it is for POV-Ray, but IMP tries to acheive some of this.

    3. Re:OSS? by Doesn't_Comment_Code · · Score: 2, Informative

      Of course, security would be another issue. There would need to be some way to encrypt data, even when being processed on a machine.

      Renderman uses the concept of "buckets" to limit memory usage. It essentially renders on small part of the screen at a time. If you split the scene into many buckets (already done for you) and send each bucket to a different computer (like many renderfarms already do) you would minimize the security issue. You could "view" the image you rendered by hacking the software - but even then you'd only have a small fraction of the image.

      But before we even start talking about all this, we'd need to know HOW the software was going to communicate. Is it just sending raw scene information to be processed? Or is it sending computations to be processed at an abstract level like grid computing is supposed to? If you really are doing grid computing, your render nodes shouldn't even know they are rendering a picture, or be able to assemble their computations into a picture.

      Your biggest concern would be making the rendering software grid compliant in an efficient manner. It's all downhill from there.

      --

      Slashdot Syndrome: the sudden, extreme urge to correct someone in order to validate one's self.
    4. Re:OSS? by Doesn't_Comment_Code · · Score: 1

      You know, I just had another thought.

      The Experts Exchange website works by credits. You help someone (answer a question) and you get credits. You can spend your credits to get help from others (ask a question and have it answered).

      Implementing and OSS computing farm could run well using the same model. Offer your spare cycles for someone else's rendering and you earn some credits. You can spend your credits by running your own process on other computers. That should solve a lot of the issues with scheduling and demand, as well as give people a real incentive to put more into the pot rather than try to just take out of the pot.

      --

      Slashdot Syndrome: the sudden, extreme urge to correct someone in order to validate one's self.
    5. Re:OSS? by comwiz56 · · Score: 1

      One consideration that would need to be accounted for is security. A sandbox would obviously need to be made so malicious programs could not cause harm on the computer, but this would make the program lose efficiency.

  7. They have the power scotty by Ambient_Developer · · Score: 3, Insightful

    Guys, many schools already have the PC power for grid computing, my school (university of minnesota) has over 20,000 PC's! Now how would you like to tap into that type of power? It is just a matter of PR, trends, and software. Most Pc's are just sitting idle anyway, why not use them? I just cannot speak for what type of power bill the school will be footing then ;-).

    1. Re:They have the power scotty by Sir+Pallas · · Score: 1

      That's a supercluster, not a grid. The Grid is a type of organizational system which brings together resources; but does not necessarily imply high throughput or latency on computation.

    2. Re:They have the power scotty by hackstraw · · Score: 1

      Most Pc's are just sitting idle anyway, why not use them?

      Most PCs are not idle. They are riddled with spyware and viruses and do neet things to the rest of us like propogate viruses and send spam.

      My point, is that there needs to be a little more control over publically available resources on individual's computers. The naïvety of using open, basically read-only, technologies like email and www, have caused many problems in the recent past.

    3. Re:They have the power scotty by crimsun · · Score: 1

      My MSc thesis implemented a grid service for (non-computer science) computational scientists through Globus (V3.2) that harnessed a Condor (V6.6) backend for a heterogeneous computational grid. Here at the university, because funds are not exactly forthcoming, we utilize idle resources in computer labs and classrooms.

  8. Re:Yawn by Kenja · · Score: 1

    HP has long since declared that long term market share and profits come second to short term quarterly stock price boosts. Who cares if you piss off some of your largest customers, you've got boat payments dang it!

    --

    "Have you ever thought about just turning off the TV, sitting down with your kids, and hitting them?"
  9. Finally.... by Viceice · · Score: 2, Funny

    The thing that is going to save (slacking) animation students who finished animating their piece 1 week before deadline but need 1 month to render on their workstation.

    --
    Sometimes I wish I was a plumber, then I'd know how to deal with other people's shit.
    1. Re:Finally.... by Ironsides · · Score: 1

      On a more serious note this would help Architecture students who have to render their projects through AutoCAD. I have seen small frames without many effects take 10 minutes per frame. When they have 300 frames or more to render it takes a while.

      --
      Fly me to the moon Let me sing among those stars Let me see what spring is like On jupiter and mars
  10. Re:$$$money$$$ by Anonymous Coward · · Score: 0

    wtf???

  11. True Grid Computing by Doesn't_Comment_Code · · Score: 4, Interesting

    I just hope it is TRUE grid computing - as in following a standard communcation so that any application that is grid compliant can take advantage of the farm.

    More often than not renderfarms have a few formats that you can use (usually expensive ones) and that's it. PR-Renderman, Maya, 3dsMax, Lightwave, and a few other big ones are guaranteed to run about everywhere. Blender? Nice try, but "little" software projects like Blender don't have much of a chance at a renderfarm. Tell a renderfarm you have a Blender file to render, and that you have money in hand - they will tell you to go home.

    Grid computing is slightly less efficient than a direct rendering program since it has a little extra overhead. But the ability to run ANY program on a farm is quite enticing. That's what we should be aiming for. That is a good goal.

    --

    Slashdot Syndrome: the sudden, extreme urge to correct someone in order to validate one's self.
    1. Re:True Grid Computing by Dracolytch · · Score: 2, Interesting

      The ability to run any program on a farm doesn't make a lot of sense.

      I've studied and worked with distributed/parallel programming. There are a lot of things you need to take into account. Your primary troubles are how to divide up the work evenly and well, especially when you're working with multiple computers at different specifications.

      So, yeah, you have Blender, and you want to distribute it... But how do you tell blender to divide up the work? Do you have another program that manages multiple installations of blender?

      Other parts of the program, usually synchronization and organization, need to be run on one (or a few) places to prevent chaos.

      That's, honestly, the easy part. If you want a good system (versus getting by on the skin of your teeth), then you also need to deal with fault-tolerance (As the number of machines increases, so does the chance of failure).

      All of this assumes that the grid is designed for speed in mind (instead of fault tolerance).

      Writing for a distributed environment requires that you re-think the way that you approach the application at hand. You litterally have to program it differently.

      If you're into software, then I highly recommend that you do some research into distributed computing. It's a really interesting field.

      ~D

      --
      This sig has been enciphered with a one-time pad. It could say almost anything.
  12. We keep it busy by steve_l · · Score: 5, Informative

    We have things to do to keep the facility busy; its a flexible fabric and it can do interesting stuff. To be honest, pure CPU cycles is not its strength -many home computers can deliver more mips.

    What the farm(s) have is large amounts of storage near those mips, which is what you need for data-intensive computation. Large animation models is one use. data processing from things like the Large Hadron Collider another, though we wont have real data from the LHC yet.

    Now, if you have projects to run on such a fabric come talk your nearest HP sales rep.

    The stuff we do in HPLabs is focused on 'research on how to use these systems'; things like resource allocation, load balancing for thermal management, etc, etc. I'm working on distributed deployment and testing, using the datacentres and perhaps soon the PlanetLab facility, which has more distribution for interesting problems.

    If you want to play with the deployment tools, to http://smartfrog.org/ and download it. The technology is designed to make it possible to install and configure complex systems over a utility computing infrastructure.

  13. Flat-footed. by Anonymous Coward · · Score: 0

    "Yes, the long fabled "grid computing" may arrive soon on a massive scale."

    Stick it on the back of a flat-bed truck. Then you can say that it has arrived.

  14. Re:How about Shrek2? by steve_l · · Score: 1

    I see your point, but I should also note that shrek2 was rendered on an HP utility computing fabric.

    You can outsource rendering, if they are people you trust and you can lock down everything: VPN to the farm, encrypted HDD. Its like the old mainframe world -you dont want to trust the operators with your data.

  15. Alternatives?? by eclectro · · Score: 3, Interesting

    I have yet to see a service where you can just simply buy time anonymously, upload your program, crunch the data, download the results.

    I know some might say that this might be invitation for someone to unleash zombies, but the grid does not need a connection to the internet for this kind of work, and besides that outgoing traffic can be monitored.

    I did not notice what the process was for buying computing time on the SE3D website, nor sepcifications of the grid. Am I missing something? Is this a fluff/vaporware story?

    --
    Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
    1. Re:Alternatives?? by Anonymous Coward · · Score: 1, Informative

      Here's one: http://www.respower.com.

      Though they run a traditional render farm rather than a grid, they've beaten HP to the punch by several years.

      The advantage to NOT being on a grid is that they can run render nodes for software people use NOW, instead of what they'll be using years from now. Lightwave, Maya and 3DMax are all supported. You pay by the gigahertz/hour.

      I used them on my last job, and they definately delivered on the promise of speed and support.

  16. I hate to feed trolls, but... by Anonymous Coward · · Score: 0

    ...how is this insightful? How is it even factual? The above post has no basis of fact than the AC's own communist mindset.

  17. The deployment tools are OSS by steve_l · · Score: 2, Interesting
    If you point your web browser at
    SmartFrog you can read about and download the framework used to dynamically deploy the fabric management tools and the rendering apps.


    It's LGPL, Java based.


    What you are thinking about, community rendering, needs community rendering tools. Bandwidth is an issue with all these apps; the filesets are huge.

  18. Resource Allocation by steve_l · · Score: 2, Interesting

    Everyone in the project has virtual $ to buy render time; the cost of render time varies depending on demand. So, you get more rendering for your money if you time it right.

    Advanced resource allocation is one of the research projects that are keeping some people -AI and mathematicians- busy. That's things like options, auctions, etc, etc.

    1. Re:Resource Allocation by Anonymous Coward · · Score: 0

      This is the exact system we had for student time allocation on the computer at UC Irvine in 1972.

  19. Could work pretty well by ikekrull · · Score: 3, Interesting

    All the professional 3D apps e.g. Maya, Shake etc. are built to operate in this kind of 'decoupled rendering' environment, and render farms are probably the simplest type of 'distributed computing' - generally theyre just a bunch of independent computers on a LAN which see a shared disk on which the source materials (e.g. textures, models, motion data, imagery) is kept.

    Accounting is pretty simple too as a central dispatcher keeps track of which frame is submitted to which node, and how long it takes - it is uncommon to have 1 machine working on more than 1 frame simultaneously.

    Sun also offer this type of service, as well as others. I think it will be interesting to see if this model is adopted by hollywood etc. but I can see a lot of smaller shops taking advantage of it to acheive what would otherwise be impossible.

    I remember using a Compaq Test-Drive-Programme 4-CPU Alpha 21264 box to render a 4096x4096 cornell box using BMRT that would have taken my 128MB P3-500 an age to do.

    The model definitely works from my point of view, however I don't really think that Sun or HP will be in much of a position to actually make money out of this type of service since hardware depreciates so fast and the price people will be willing to pay would be pretty low I imagine - I guess they could simply use their unsold inventories to extract some kind of revenue from them instead of writing them off, but getting the 'volume' up to profitable levels will be a challenge.

    --
    I gots ta ding a ding dang my dang a long ling long
  20. It's utility computing by steve_l · · Score: 3, Interesting

    That is, CPU & storage is something you pay for on demand. It is distributed (two facilties, two continents), and the users are all scattered round the UK. One of the facilities is downstairs from me in Bristol; a large chilled datacentre that I dont have access to for either security or competence reasons :)

    Projects aren't charged real money for render time, do have virtual monety; cost of rendering varies depending on demand. Projects have their own balances. This lets money optimise the resource allocation. Get your design done early, render before the rest and you get more CPU time for your cash, hence better rendering.

    There are lots of other sponsors, as this is giving regional and up and coming artists/animators chances to do serious renderings on facilities they wouldn't normally get.

  21. uh-huh-huh-huh by stuffedmonkey · · Score: 1

    Hmmm. Amatures? Sample movies? excellent! Hey, wait a second, this article is about computers!

  22. Welcome! by Anonymous Coward · · Score: 0

    I, for one, welcome our new Beowulf Clustered overlords!

  23. Re:$$$money$$$ by pronobozo · · Score: 0

    "Pay-as-you-go computer power could soon be helping animation companies and students finish their projects more quickly."

    "The HP developers hope the service will make it easier for young, talented animators get a foothold in the movie industry."

    The school pay? or students?

    --
    ------
    insert sig here,here, and here
  24. Yeah... by KillaKen187 · · Score: 1

    but does it run Doom3?

  25. Re:$$$money$$$ by pronobozo · · Score: 0

    flamebait? valid concern no?

    would it be students paying for a service or the school??

    "Pay-as-you-go computer power could soon be helping animation companies and students finish their projects more quickly."

    "The HP developers hope the service will make it easier for young, talented animators get a foothold in the movie industry."

    --
    ------
    insert sig here,here, and here
  26. Re:$$$money$$$ by Anonymous Coward · · Score: 0

    You deserved that.

  27. Imagine by Anonymous Coward · · Score: 0

    Imagine a beowulf cluster of these!... oh wait

    DS Conga

  28. Utility, not Grid by steve_l · · Score: 2, Informative

    'grid' computing is such a vague term. The fabric here can be reconfigured to boot different virtual disks, with different stuff running on it. It's not scavenging spare CPU or anything fancy.

    But it could host Globus2.4, even Condor. Or a very large three tier Apache 2.0 HTTPD cluster. If you have the money, you get to choose.

  29. A Day Late and a Dollar Short by Anonymous Coward · · Score: 0

    http://www.respower.com

  30. I am by TheClassic · · Score: 1

    Could someone not write something like this...
    I'm not writing something like this currently, have been for awhile, and expect to continue for some time.

  31. Apple has them beat with XGrid by Anonymous Coward · · Score: 0

    What about Apple's XGrid? Seems to me that it does everthing this does, works on any apple computer running OSX 10.3 or later and requires almost no configuration to get it working.

    http://www.apple.com/acg/xgrid/

  32. Wow! by Anonymous Coward · · Score: 0

    >> The BBC has a story about HPs SE3D lab's pilot scheme to provide raw rendering power for smaller studios and amateurs.

    Imagine a beowulf of amateurs!

    *(looks to right, to the left, upwards and downwards)*

    Forget it...

  33. Did anybody actually pay for this? by Animats · · Score: 2, Insightful
    It sounds like HP ran a free demo. But has anybody actually paid for their service?

    There are commercial render farm services running right now. Over 400 machines. 440 frames are rendering right now. Over 6 million frames sold. On line. Self service. VISA/MC accepted. The going rate is about $1/GHz/hour, before discounts.

    And they never mention "grid computing".

  34. Mirrors Here - Pages and Videos by Kinetic · · Score: 1

    MirrorDot has mirrors of the links, including the movie. Just in case of the likely Slashdotting...

    --
    ~Jay
  35. Xgrid for UNIX by not_hylas(+) · · Score: 4, Informative

    "Could someone not write something like this but open source and distributed."

    Darwin:

    http://www.opensource.apple.com/darwinsource/ind ex .html

    Xgrid for UNIX:
    This article introduces the first working Xgrid agent for Linux and other Unix systems that can be integrated in any XGrid cluster (managed by OS X).

    http://unu.novajo.ca/simple/archives/000026.html

    http://developer.apple.com/hardware/hpc/xgrid_in tr o.html

    --
    ~hylas
  36. Set your Phasers to hype! by Duncan3 · · Score: 1

    The VC's are drooling, the press is desperate for a story, the big companies are all pitching the exact same technology claiming how new and shiney it is...

    This is OLD technology folks. Try the 1970s. Yes, really.

    Hear that sound? It's money being flushed down a dot-com toilet. :)

    --
    - Adam L. Beberg - The Cosm Project - http://www.mithral.com/