Slashdot Mirror


Sun Releases Grid 5.2 for Linux

Linux_ho writes "Grid 5.2 is a distributed processing engine that runs on Solaris, and now Linux. Apparently it has been released under an "an industry-accepted open source license" but I couldn't find out which one. The product was designed to make use of the spare cycles from any idle Solaris or Linux machines on your network. Sun mentioned in the press release that it can be used for frame rendering, but I bet you can come up with some other interesting applications. Here's the FAQ."

42 of 90 comments (clear)

  1. Message Passing Interface 6.3.2 by Wills · · Score: 3

    You might want to check out MPI-6.3.2 (aka LAM) which has been around for a lot longer than Grid. MPI is a library for writing parallel programs that execute on groups of workstations.

  2. Condor covers more platforms by mliggett · · Score: 3
    If you have a heterogeneous environment and are interested in harvesting spare cycles, then Condor might be a better bet than Sun's Grid. Condor is free as in beer, but the license does not cover source, and source is not freely available at this time. Condor works on more UNIX variants and also works (in limited fashion) on NT. I have done a little work with Condor and it is a very advanced and well-put-together system.

    It is very easy for each machine-owner to restrict or preference which jobs will run on his machine, for each job to preference certain machine attributes, and also for the queueing system to fairly distribute net CPU time across all active users of the system. All of this works using a very simple C-like language in which you express you desires.

  3. Re:This is great but by wiredog · · Score: 2

    Beowulf clusters might speed things up at a lower cost.

  4. Re:GPL? Doesn't look like it... by Dharzhak · · Score: 2

    > This doesn't look like the GPL I know...

    They never claimed it was under the GPL. The press release states, "Sun will distribute Sun Grid Engine software under an industry-accepted open source license in order to accelerate the adoption of the distributed computing model."

    Take note that "industry-accepted" is an adjective that means absolutely nothing other than there are other applications distributed under the same license. Also take note that, "in order to accelerate the adoption of the distributed computing model," roughly translates to, "so more people will use it."

    If you go to the "Download" page, you'll find the license statment as in all other Sun downloads. It says:

    Licensing

    The product is free and there is no software enforcement that restricts distribution. The product is issued under a standard Sun Binary Code License.

    As many of you already know, the SBCL is Sun's version of Open Source that allows them to control the modifications done to the product.

  5. Sounds like condor by leachj · · Score: 2

    It sounds in concept like condor being developed by the CS department at UW-Wisconsin Madison, unfortunately the condor folks aren't letting their source code out except for special special cases, and then you have to beg them for it.

  6. Re:More distributed computing... by cmason · · Score: 2

    I think this would be a great idea. Perl would be perfect for this. Consider:

    • Perl is widely ported to many many architectures.
    • Perl has a large library of existing code, including many high performance math libraries, many of which are already highly portable and fast, being written in X-Sub C or C++ code. (The fact that perl is slower than one would like for scientific computations is mitigated by the fact that all compute-intensive operations could be implemented by simply calling into C++ libraries.)
    • Perl has a number of schemes for executing scripts in secure "boxes," such as the Safe module, which limit the types of operations available to those which may not modify the host computer.
    • Perl allows one to combine program code and data into a single stream, by using the __DATA__ filehandle.

    Really this could be implemented very quickly, Clients could simply HTTP GET datablocks (complete perl scripts) and HTTP POST the results.

    The data integrity could be ensured by randomly testing clients by sending them data blocks for which answers were already known. The percentage of known answer test blocks to unknown answer work blocks could be varied with the perceived level of trust in the Hive.

    --
    "If you are an idealist it doesn't matter what you do or what goes on around you, because it isn't real anyway."-R.P.W.
  7. Not that much like Mosix.. by Dr.Dubious+DDQ · · Score: 3

    At least, it doesn't look that way to me, browsing Sun's documentation

    The nice thing about Mosix is that it automatically handles migrating any existing processes to wherever it thinks it will be finished the quickest. Last time I played with Mosix, I put it on an old P-100 Laptop with no L2 Cache and a K6-2/350. When not connected to the network, using LAME to generate MP3's on the P-100 went at about 15% of "real-time" speed. Connected to the network, it went at about 85% since the process could be automatically migrated to the K6-2. (It goes at about 105% or so when run "natively" directly on the K6-2/350). I suspect it would have been an even more dramatic difference had I been running on something faster than a 10BaseT Hub for the networking. Other than the kernel patch for Mosix, all of the software on both systems were standard "off the shelf" linux programs. (Compiling, I noticed, also went substantially faster, using plain old GCC, for example).

    It looks like with "gridware" you actually have to "submit a job" to the handler using a separate program. I can't tell for sure just browsing the documentation whether you can submit ANY process, but it does look like it has to be done 'manually' in any case.

    Gridware looks pretty neat, but I get the impression it'll be of more use to technical people who have need to distribute particular types of jobs, and have the resources to set up a "compute farm", and have technical enough users to make good use of it. Other than the installation, Mosix doesn't seem to have this limitation (but on the other hand, Mosix is Linux only [I think even ix86 Linux only, but don't quote me on that] and requires patching the kernel).

    Now, if Mosix would get a 2.4 patch out, I could get it set up again at home...


    ---
    "They have strategic air commands, nuclear submarines, and John Wayne. We have this"
  8. Re:More distributed computing... by volsung · · Score: 4
    I've had an interest in a distributed computing framework that would allow people to create a project like seti@home, distributed.net, and the like, without having to solve the same cross-platform issues every single time.

    The basic idea is to have virtual machine (of sorts) that provides an API friendly to algorithm implementation. (Lots of math and data manipulation functions) The virtual machine can limit both CPU utilization and memory/disk usage by the actual distributed program. The program is written in a scripting langage (grab your favorite one) that can be compiled on the fly. The API functions would be implemented in the fastest possible way for each platform.

    You could designed the virtual machine so that users could easily add programs to it for background execution. The client's security would be ensured by the resource limits enforced by the virtual machine and the lack of "dangerous" features in the scripting language.

    I never was able to solve the data integrity issue in a satisfactory way, though. Rogue clients in this scheme could always submit bogus results to the server. That's not catestrophic, but it means that the distributed platform could not be used in an uncontrolled environment like the Internet. If anyone has some ideas on how to solve this problem, feel free to post or email me. (Or you could go patent them and maybe make yourself some money.)

    Oh yeah, I also thought that "Hive" would be a cool name for such a program. :)

  9. "an industry-accepted open source license" by sharkey · · Score: 2

    How about "A community-accepted open source license" instead?

    --

    --

    --
    "Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
  10. Re:This is great but by sql*kitten · · Score: 3
    Beowulf clusters might speed things up at a lower cost.

    Which means buying a roomful of kit to build one out of. Grid is designed to run jobs on your existing hardware while it's idle - the rest of the time, they're all still general purpose, interactive workstations running regular applications. The Beowulfs of which I am aware use dedicated hardware.

  11. Re:More distributed computing... by volsung · · Score: 2
    I like the XML idea. Transparent conversion from structures (in the C sense) to XML would be a must for the Hive API.

    Are there any scripting languages that could be easily modified to serve as the basis for such a system? Candidates should be:

    • Securable - Memory management, package access, and resource access (disk and network) should be strictly controllable.
    • Fast - On-the-fly compilation that may cause a performance hit at the start of processing, but in the long term performance approaches that of compiled code.
    • Extensible - Ability to link to native C/assembly/whatever libraries that implement the API.
    • Nice to program - Syntax is functional and/or nice to use.
    • Modifiable - Can be modified to add any of the above features which are missing.

    I'm leaning toward Python simply because Guile will scare away potential users and Perl would be too hard to lock down and/or modify. Does anyone know about how Python scores on the above criteria?

  12. Further inspection by korpiq · · Score: 2

    You need trusted server(s) to control the jobs.

    You don't. You need knowledge of clients sufficient for your task. You can deal the job for them as necessary. Think - uuh, what's the name of that distributed "napster replacement" file sharing system again? A network of clients, each propagizing it's available resources (processing power, memory, storage; price for those?).

    two or more distant clients and compare the results.

    If there's a virus/worm rampant changing the way the clients compute, N results can be equal but still wrong :( Need workaround for this. Then again, the risk is always there even with "trusted" environments.

    Jobs should still be anonymous and abstract, even if the client owner should have right to see what kind of things are happening on its machine. An obvious job should be split into less obvious parts.

    "The Net is vast."

    WIth this I meant there's always more clients available elsewhere than you have on your network :)

    --

    I think, therefore thoughts exist. Ego is just an impression.
  13. do you really think Sun uses it? by 09876 · · Score: 2

    Gridware is definately an also-ran in the distributed environment at the moment. Companies who use this type of product generally go for LSF, a commercial product. Many big IC design houses use LSF to use the spare cycles from workstation cpu's + to run their compute farms. Sun is one of these companies. If you want a GPL equivalent go for GNQS.

  14. Re:More distributed computing... by volsung · · Score: 2

    As a follow-up: Python also has a feature for creating a "sandbox" for Python programs called RExec (for "restricted execution").

  15. More followup by volsung · · Score: 2
    I'm doing a lot of replying to myself, but I keep finding good info:

    This guy explains how to convert XML DOM objects to Python objects: A Closer Look at Python's [xml.dom] Module

  16. Re:More distributed computing... by regen · · Score: 2
    I never was able to solve the data integrity issue in a satisfactory way, though. Rogue clients in this scheme could always submit bogus results to the server. That's not catestrophic, but it means that the distributed platform could not be used in an uncontrolled environment like the Internet. If anyone has some ideas on how to solve this problem, feel free to post or email me. (Or you could go patent them and maybe make yourself some money.)

    One way to solve this is to submit the same work unit to multiple clients, each client chosen at random from the pool of availible clients. The results of all the clients are then compared and only if they all return the same result then do you accept the result. This way a small number of rogue clients will be unlikely to produce wrong results. (This of course assumes that the rouge clients are not wide spread, and doesn't ensure that bad data won't be returned.)

    Additionally, you could have the work unit itself perform computations which are unique to each work unit. This is mixed with the actual work you care about in such a way as to make it very difficult to perform one computation without the other. For example, you toss in some data, at random, which when processed returns a known result. This data is mixed with data on which you actually want to have some computation performed. Since the client (and the server for that matter) won't know what part of the data is what, if any of the computation is not performed correctly then it will be detected by the server.

  17. stability by bluelip · · Score: 3

    Yes, I know it's from Sun so it's probably stable. I'd hate to see it crash.

    Grid-Lock sucks.

    C'mon guys, it's a Friday

    --

    Yep, I never spell check.
    More incorrect spellings can be found he
  18. More distributed computing... by gus+goose · · Score: 4

    Grid is a push based system, monitoring the activity on a set of servers, and pushing work to the more idle ones.

    This is great, but I believe more in the Seti@home approach, let the idle servers pull work down.

    Everyone who has worked with distributed computing knows that the application really has to be designed paying carefult attention to the distribution model. How about a more generic solution, say an XML based data and programming unit (in a language with multi platform capabilities like Java or perl) queued on a controlling server, with a farm of slave servers pulling down a unit during an idle time. It sould be something similar to:

    nice -19 jobpoller --controlhost=control.server.com

    Picture this as a backend to a website processing CGI, etc.

    Anyone interested in forming a subscription based distributed computing project with me drop me a mail...

    --
    .. if only.
    1. Re:More distributed computing... by srichman · · Score: 2
      I used to work on a project at UCSB called Javelin: "Javelin is a Java-based infrastructure for global computing". It's presently a bit more academic than practical, but it seems to fit the bill of what you're looking for fairly precisely. It's a bit better than, for example, seti@home in that it supports more tightly coupled computations (e.g., branch-and-bound). Currently, Javelin supports:
      • piecework computations, where a large chunk of work can be split into smaller chunks, and
      • branch-and-bound computations, like the travelling salesman problem
      , but work is in progress on a version with a more general computational model that supports computations with arbitrary DAGs for task creation and data dependencies (a la Cilk).

      It's highly fault tolerant (uses eager scheduling) and load balanced (uses work stealing).

    2. Re:More distributed computing... by sql*kitten · · Score: 2
      This is great, but I believe more in the Seti@home approach, let the idle servers pull work down.

      That won't work in this case, because one thing Grid allows you to do is submit many jobs to the network service it manages, and assign a priority to each one, such that high priority tasks take precedence over low priority tasks. So the system as a whole needs to be able to package and distribute tasks based on criteria other than when they were submitted to the system, and other than their overall size. Seti-like approaches aren't flexible enough for this, because once a node has started a job, it will complete it before submitting its results and requesting more work.

  19. Distributed computing by jbert · · Score: 2

    Nice to see, especially the comments at the bottom about the road to widespread usage including not just open source release but also the standardisation of an API to allow compatible implementations.

    Amusingly, the biggest-horsepower compute farm in the company is likely to be the collection of people using office apps. You can't easily buy processors below around 700Mhz these days and desktop apps do little but wait for keypresses.

    The only real impact that people would notice would be in terms of memory footprint. Those high-res wallpapers and ten copies of Excel (or Staroffice, whatever) like to slurp the RAM and stay resident. So unfortunately, it is unlikely to be unnoticed by those donating their cycles...which is the real killer.

  20. Juno? by SEWilco · · Score: 3

    Well, there's no mention of an interface for Juno distributed processing...

  21. You can bet by BFOM · · Score: 2

    This so-called 'open-source' license will merely be another proprietary ownership by Sun deal. Just like Java. You know the old refrain, "It is open but Sun owns it, Sun controls it, Sun makes money from it, and Sun takes your development efforts and sells them back to you!"

  22. Re:Just like Mozix? by rufus+t+firefly · · Score: 2
    I was waiting for someone to make that comparison. MOSIX shares memory and CPU cycles over machines in a network, but as far as I know, Mosix is quite platform dependant (it is a Linux kernel patch right now). I'm also not sure if they have released a 2.4.x kernel patch yet, since I haven't seen any announcements on freshmeat or Linux Today about it.

    Does anyone know how cross-platform this is, or if the Linux and Solaris boxes aren't able to talk to each other? The main advantage over MOSIX that this seems to have is the cross-platform thing. Other than that, they (at least to the casual observer) seem to be in the same arena.

    (As an aside, I've been watching MOSIX to see if any major distributions are going to capitalize on it. Definately a killer feature, but I hope it is eventually brought into the kernel proper....)

    ---

    --
    "He may look like an idiot, and talk like an idiot, but don't let that fool you. He really is an idiot." - Duck Soup
  23. GPL? Doesn't look like it... by ThunderBucket · · Score: 2

    This doesn't look like the GPL I know...

    Sun Grid Engine 5.2.2

    Sun Microsystems, Inc.

    Binary Code License Agreement READ THE TERMS OF THIS
    AGREEMENT AND ANY PROVIDED SUPPLEMENTAL LICENSE TERMS
    (COLLECTIVELY "AGREEMENT") CAREFULLY BEFORE OPENING THE
    SOFTWARE MEDIA PACKAGE. BY OPENING THE SOFTWARE MEDIA
    PACKAGE, YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU
    ARE ACCESSING THE SOFTWARE ELECTRONICALLY, INDICATE YOUR
    ACCEPTANCE OF THESE TERMS BY SELECTING THE "ACCEPT" BUTTON
    AT THE END OF THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL
    THESE TERMS, PROMPTLY RETURN THE UNUSED SOFTWARE TO YOUR
    PLACE OF PURCHASE FOR A REFUND OR, IF THE SOFTWARE IS
    ACCESSED ELECTRONICALLY, SELECT THE "DECLINE" BUTTON AT THE
    END OF THIS AGREEMENT. 1. LICENSE TO USE. Sun grants you
    a non-exclusive and non-transferable license for the
    internal use only of the accompanying software and
    documentation and any error corrections provided by Sun
    (collectively "Software"), by the number of users and the
    class of computer hardware for which the corresponding fee
    has been paid.

    2. RESTRICTIONS Software is confidential and copyrighted.
    Title to Software and all associated intellectual property
    rights is retained by Sun and/or its licensors. Except as
    specifically authorized in any Supplemental License Terms,
    you may not make copies of Software, other than a single
    copy of Software for archival purposes. Unless enforcement
    is prohibited by applicable law, you may not modify,
    decompile, or reverse engineer Software. You acknowledge
    that Software is not designed, licensed or intended for use
    in the design, construction, operation or maintenance of any
    nuclear facility. Sun disclaims any express or implied
    warranty of fitness for such uses. No right, title or
    interest in or to any trademark, service mark, logo or trade
    name of Sun or its licensors is granted under this
    Agreement.

    3. LIMITED WARRANTY. Sun warrants to you that for a period
    of ninety (90) days from the date of purchase, as evidenced
    by a copy of the receipt, the media on which Software is
    furnished (if any) will be free of defects in materials and
    workmanship under normal use. Except for the foregoing,
    Software is provided "AS IS". Your exclusive remedy and
    Sun's entire liability under this limited warranty will be
    at Sun's option to replace Software media or refund the fee
    paid for Software.

    4. DISCLAIMER OF WARRANTY. UNLESS SPECIFIED IN THIS
    AGREEMENT, ALL EXPRESS OR IMPLIED CONDITIONS,
    REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED
    WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
    PURPOSE OR NON-INFRINGEMENT ARE DISCLAIMED, EXCEPT TO THE
    EXTENT THAT THESE DISCLAIMERS ARE HELD TO BE LEGALLY
    INVALID.

    5. LIMITATION OF LIABILITY. TO THE EXTENT NOT PROHIBITED
    BY LAW, IN NO EVENT WILL SUN OR ITS LICENSORS BE LIABLE FOR
    ANY LOST REVENUE, PROFIT OR DATA, OR FOR SPECIAL, INDIRECT,
    CONSEQUENTIAL, INCIDENTAL OR PUNITIVE DAMAGES, HOWEVER
    CAUSED REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF
    OR RELATED TO THE USE OF OR INABILITY TO USE SOFTWARE, EVEN
    IF SUN HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
    In no event will Sun's liability to you, whether in
    contract, tort (including negligence), or otherwise, exceed
    the amount paid by you for Software under this Agreement.
    The foregoing limitations will apply even if the above
    stated warranty fails of its essential purpose.

    6. TERMINATION. This Agreement is effective until
    terminated. You may terminate this Agreement at any time by
    destroying all copies of Software. This Agreement will
    terminate immediately without notice from Sun if you fail to
    comply with any provision of this Agreement. Upon
    Termination, you must destroy all copies of Software.

    7. EXPORT REGULATIONS. All Software and technical data
    delivered under this Agreement are subject to US export
    control laws and may be subject to export or import
    regulations in other countries. You agree to comply
    strictly with all such laws and regulations and acknowledge
    that you have the responsibility to obtain such licenses to
    export, re-export, or import as may be required after
    delivery to you.

    8. U.S. GOVERNMENT RESTRICTED RIGHTS. If Software is
    being acquired by or on behalf of the U.S. Government or by
    a U.S. Government prime contractor or subcontractor (at any
    tier), then the Government's rights in Software and
    accompanying documentation will be only as set forth in this
    Agreement; this is in accordance with 48 CFR 227.7201
    through 227.7202-4 (for Department of Defense (DOD)
    acquisitions) and with 48 CFR 2.101 and 12.212 (for non-DOD
    acquisitions).

    9. GOVERNING LAW. Any action related to this Agreement
    will be governed by California law and controlling U.S.
    federal law. No choice of law rules of any jurisdiction
    will apply.

    10. SEVERABILITY. If any provision of this Agreement is
    held to be unenforceable, this Agreement will remain in
    effect with the provision omitted, unless omission would
    frustrate the intent of the parties, in which case this
    Agreement will immediately terminate.

    11. INTEGRATION. This Agreement is the entire agreement
    between you and Sun relating to its subject matter. It
    supersedes all prior or contemporaneous oral or written
    communications, proposals, representations and warranties
    and prevails over any conflicting or additional terms of any
    quote, order, acknowledgment, or other communication between
    the parties relating to its subject matter during the term
    of this Agreement. No modification of this Agreement will
    be binding, unless in writing and signed by an authorized
    representative of each party.

    For inquiries please contact: Sun Microsystems, Inc. 901
    San Antonio Road, Palo Alto, California 94303

    SUPPLEMENTAL LICENSE TERMS

    These supplemental license terms ("Supplemental Terms") add
    to or modify the terms of the Binary Code License Agreement
    (Collectively, "the Agreement"). Capitalized terms not
    defined in these Supplemental Terms shall have the same
    meanings ascribed to them in the Agreement. These
    Supplemental Terms shall supersede any inconsistent or
    conflicting terms in the Agreement, or in any license
    contained within the Software.

    1. SEPARATE DISTRIBUTION LICENSE REQUIRED. You understand
    and agree that you must first obtain a separate license from
    Sun Prior to reproducing or modifying any portion of the
    Software.

    2. NO SUPPORT. Unless you have entered into a separate
    support agreement with Sun, Sun is under no obligation to
    support the Software or to provide to you any updates or
    error corrections (collectively referred to as "Software
    Updates"). If Sun provides any Software Updates, whether
    pursuant to a support agreement or otherwise, at its sole
    discretion, the Software Updates will be considered part of
    the Software and subject to the terms of this Agreement.

    3. BACK-UP. You have the sole responsibility to protect
    adequately and backup your data and/or equipment used in
    connection with the Software. Sun shall not be liable for
    any lost data, re-run time, inaccurate output, work delays
    or lost profits resulting from your use of the Software.

    4. TRADEMARKS AND LOGOS. You acknowledge and agree as
    between you and Sun that Sun owns the Sun, Solaris, and Sun
    Grid Engine 5.2.2 trademarks, and all Sun, Solaris and Sun
    Grid Engine 5.2.2 related trademarks, service marks, logos
    and other brand designations ("Sun Marks") and you agree to
    comply with the Sun Trademark and Logo Usage Requirements
    currently located at http://www.sun.com/policies trademarks.
    Any use you make of the Sun Marks insures to Sun's benefits.

    For inquiries please contact: Sun Microsystems, Inc. 901
    San Antonio Road, Palo Alto, California 94303

    --

    "All I do is eat and poop!" -- Bean
  24. Re:Soldier! by bluelip · · Score: 2

    Sorry sir!

    Won't happen again! Don't know what could possibly come over me. Must've been that liverwurst last night.

    --

    Yep, I never spell check.
    More incorrect spellings can be found he
  25. Re:This is great but by wiredog · · Score: 2

    True, but he said "we spend lots of money on the newest and fastest machines" so I'm assuming that it's at his business. Of course, he'd still have to rewrite the code to run distributed either way. Wonder if there's a distributed system that could leverage existing multi-threaded code? Then you'ld get the advantages of running distributed, without having to rewrite.

  26. Re:Can it run anything? by Pogie · · Score: 3

    The GridEngine system from Sun is an LSF-type batch-queueing/load-balancing system. Sun bought GridWare, which was an independent German company previously, and is looking to bundle the GridEngine with it's workstations, promoting the 'spare cycle' idea.

    In answer to your question, yes, GridEngine can run anything. It isn't an MPI-type implementation which requires you to modify your code. GridEngine allows you to set up multiple execute resources, based on processor type, OS, memory, disk, I/O, runqueue usage, or really any heuristic you want to implement. You submit your job, with whatever resource requirements you need, and then GridEngine runs it on the available resource which meets your requirements. There's also a Q3-ish available product called GRD, which allows you to further allocate resources on a more policy-style basis. This piece will be a licensed add-on, but it provides the enterprise with the ability to divvy-up compute farm resources on the basis of users and groups, etc.

    GridEngine also comes with a "grid-enabled" interactive tcsh, so you can have an interactive shell running which is actually spawing work all over the compute farm, as resources are available. There's also a "enabled" make, which does the same thing for builds.

    It's pretty neat, but I think it's more effective in a dedicated compute-farm type of installation than a "let's use spare desktop cycles" kind of installation.

  27. Re:This is great but by sql*kitten · · Score: 2
    Wonder if there's a distributed system that could leverage existing multi-threaded code?

    Threads are paths of execution through the same process context. It's very difficult to distribute them without a single system image, which is what Irix can do. In general, if you want to distribute a task across multiple machines, each part of the task must be able to run self-contained, and be incorporated within the overall result of the computation. This would very well for, say, rendering, because you can simply give each node a frame to do, then assemble all the frames into a movie. But it would be very difficult to break a scene apart into objects, render each one of those on a different machine and then incorporate them into a new image, because the different machines wouldn't be able to compute the effects of a shadow of one object on another.

  28. Yawn. Ho-hum. Sigh. by GoodFastCheap · · Score: 3

    I guess this is news... maybe not, though since the Condor Project has been available for a whole lot of platforms for quite some time now. (Yes, Linux is supported.)

    --
    "I know - let's make Quake...AGAIN! They just might be stupid enough to buy it..." (overheard at id)
  29. This isn't earth-shattering kids... by dsouth · · Score: 3
    Gridware isn't all that new, and it isn't a reaction to Mosix or SETI@home.

    Batch systems have been around a long time in the HPC world. Gridware was orginally developed by GENIAS Software GmbH. GENIAS produced a batch scheduler called Codine, which was a commercial version of DQS. In fact, Sun's Grid Engine FAQ even states that Sun Grid Engine is a new name for CODINE.

    Of course, DQS/Codine/Grid isn't the only batch-scheduling/cycle-scavenging game around. Other players are:

    • Condor
    • openPBS and it's commercial version PBS Pro
    • Load Leveler (which IIRC is IBM's commercial implementation derived from Condor)
    • LSF which is the product Sun was previously co-marketing until they purchased Gridware (probably because of the high per CPU cost of LSF).
    • and lots of others that I've forgotten, many based on the once-common NQS/NQE batch system.
    • There are also systems like Legion that represent a sort of ``next step'' computing enviroment.

    Many of these predate newcomers like SETI@home and Mosix by serveral years. Most also provide hooks into parallel computing APIs like MPI, PVM, openMP, or something similar.

    Batch scheduling and cycle-scavening are old concepts. Having wasted away my years as a graduate student submitting large quantum chem jobs to Crays, it's nice to see lots of groups continuing to squeeze every useful cycle out of existing hardware. Sun's recent annoucements are just the latest update to an old product---not a new idea, and not a Mosix/SETI rip-off.

  30. Re:Exactly by Arandir · · Score: 2

    Whose definition of "Open Source" is it?

    The definition belongs to the Open Source Initiative (OSI).

    If they can't decide on a known license, force them to post the license as "Unknown".

    That's putting the onus on Freshmeat to keep track of OSI's approvals and update their own list. It's far easier to just list the big five or six and group the rest under "Open Source". There's no need to worry. Freshmeat checks their submissions, and if anyone is lying it doesn't get put up.

    Even writing a custom license is better than using the generic term "Open Source".

    It's not open source unless it's uses an OSI approved license. It would have been nice if OSI had gotten their trademark on Open Source, but that changes nothing. "Open Source Software" is a much more specific term than "Free Software". While Microsoft can get away with calling IE "free software", it would be an outright lie for them to call it "open source software".

    I have confidence that if Sun is using an "industry accepted Open Source license" it will be one approved by the OSI.

    --
    A Government Is a Body of People, Usually Notably Ungoverned
  31. Can it run anything? by CAPSLOCK2000 · · Score: 2

    What I was wondering about, can this thing run anything, or do programma's need to be written for it. I couldn't find a clear answer to that skimming through the website. However a example job from the manual does suggest so:
    #!/bin/sh
    # This is a simple example of a CODINE batch script
    #
    # Print date and time
    date
    # Sleep for 20 seconds
    sleep 20
    # Print date and time again
    date
    # End of script file

    If it can truly run just any programm it would be really cool.

  32. Sun and the GPL by aburnsio.com · · Score: 2

    I'm not sure if it's GPL'd, but it may have to be. UT Austin did some work on distributed memory allocation that blew the current Solaris implementation out of the water. Fortunately, that work was GPL'd. Go Blum-Foe! ;-) The GPL Virus at Work.

  33. This is a good move for Sun. by hoegg · · Score: 4
    1. Sun Microsystems is a large Corporation.
    2. Microsoft is putting a lot into its .NET initiative.
    3. Sun has been nurturing Java for over 4 years.
    4. Both .NET and Java provide platforms for distributed processing in desktop applications.

    My opinion is that this is the beginning of an enterprise computing paradigm that Sun hopes will give Java an edge in the desktop market, after Microsoft's 15 year reign.

    Imagine an entire office of computers efficiently sharing resources. I get up for coffee, my cycles are used for my co-worker's application compile. He goes to lunch, his cycles are mine for Unreal Tournament.

    I think it's got potential.

  34. Re:This is great but by Omnifarious · · Score: 2

    Threaded applications use shared memory to communicate between threads. Distributed shared memory schemes that aren't carefully tailored to the application they're designed for are very slow right now.

    An application currently has to be split up in a such a way that the different parts need to communicate with eachother in the most minimal way because communication overhead is large.

    Even then, it helps if they communicate with eachother in a way in which the results arrive at the computer that needs them before they're asked for, which is even harder.

    SETI@Home works because there's virtually no communication between computers. The only thing that could make SETI@Home faster is to start downloading a new data set just before the current one is finished so you never have a break in doing the hard, CPU intensive mathematics.

  35. Controlling untrusted clients by korpiq · · Score: 2

    I never was able to solve the data integrity issue in a satisfactory way, though. Rogue clients in this scheme could always submit bogus results to the server. That's not catestrophic, but it means that the distributed platform could not be used in an uncontrolled environment like the Internet.

    Same solution as in HA problems: redundancy to avoid SPF. You need trusted server(s) to control the jobs. They can give the same job out to two or more distant clients and compare the results. Comparation can be done by producing MD5's in a second set of clients to reduce server load.

    The servers should not handle the programs or data, just schedule works and tell clients whom to ask for what when.

    I'm sorry: double, even quadruple work, but you can't have both open and trusted environment. Then again, who cares? "The Net is vast."

    Internet will sloowly turn into a big, amorphous of processing power and storage. I'm happy to see that other people are giving thought to this as well.

    A GNU project for job distribution VM would be essential in standardizing it.

    --

    I think, therefore thoughts exist. Ego is just an impression.
  36. Just after I got mosix working... by Bazman · · Score: 2

    Typical! got my mosix cluster going yesterday, and then Sun announce this!

    Mosix involves kernel patches - does this Grid thing?

    Mosix can migrate most processes, but there are some restrictions on threads, shared memory etc.

    There's other queueing systems with process migration, checkpointing, etc etc. I'm sure a search on the new-look freshmeat.net will find them for you. As usual Open Source was there first...

    Baz

  37. Re:Licence, and what about PVM,MPI and the like? by srichman · · Score: 2
    And the other thing - how does this releate to PVM, MPI and other parallell libraries?

    MPI, PVM, et al are libraries that parallelized programs can use for inter-node communication.

    Sun's Grid Engine is basically a queuing system. To quote Sun's website: "The basis for load management is the batch queuing mechanism. In the normal operation of a cluster, if the proper resources are not currently available to execute a job, then the job is queued until the resources are available. Load management further enhances batch queuing by monitoring host computers in the cluster for load conditions allowing additional utilization of resources."

    Grid Engine, however, seems to be designed to run single-node computations; the website mentions nothing about queuing a job for execution on N machines. This is a major distinction from queuing systems in traditional parallel machines and clusters, where you tell the queuer to run your job on N nodes, and when N nodes are available, it runs N copies of your program simultaneously on those nodes.

    So, your PVM/MPI/etc. programs won't be able to run parallelly on Grid Engine, because traditional parallel programs assume a tightly coupled network of processors (or processes) that are all running your code simultaneously, and Grid Engine doesn't provide for this. Grid Engine lends itself more to things that can be done by machines independently, like seti@home.

  38. Re:GPL? Doesn't look like it... by SquadBoy · · Score: 2

    Odds are there is a dual license with that one and this one . I would think that that one is for PHBs who don't want open source and the good one is for the rest of us. This is fairly common and it would make sense that once they have written a license in house and had some of the big names in OSS bless it that they will stick with it.

    --

    Cypherpunks: Civil Liberty Through Complex Mathematics. Those who live by the sword die by the arrow.
  39. This is great but by prisoner · · Score: 3

    do programs have to be written specificly to take advantage of this? I do lots of groundwater modeling and the models can take forever to run. Consequently, we spend lots of money on the newest and fastest machines. It would be nice to use something like this with our models "as-is"

  40. other info... by phantumstranger · · Score: 2
    can be found here

    --
    "From of old, there are not lacking things that have attained Oneness." - Lao Tzu