Slashdot Mirror

If Oracle can claim copyright on simply the function names of Java,

Because they've licensed it to the world, just like every other programming language worth using. Python APIs are just as copyright as Java's, but again, there is a license.

In the post-Oracle vs Google world, make sure you have a license for any APIs you are using.

I assume this is an epic troll, but am going to give an honest answer anyway, because there are some legitimate questions buried in there.

I work with a aggregate.org a university research group which has a decent claim to having built the very first Linux PC Cluster, set some records with them (KLAT2 and KASY0 were both ours), and still operates a number of Linux clusters, including some containing GPUs, so I feel like I have some idea of the lay of cluster technology. It is *way* overdue for an update (and one is in progress, we swear!), but we also maintain TLDP's widely circulated Parallel Processing HOWTO, which was the goto resource for this kind of question for some time.

In a cluster of any size, you do _not_ want to be handling nodes individually. There are several popular provisioning and administration systems for avoiding doing so, because every organization with a large number of machines needs such a tool. The clusters I deal with are mostly provisioned with Perceus with a few ROCKS holdovers, and I'm aware of a number of other solutions (xCat is the most popular that I've never tinkered with). Perceus can pass out pretty much any correctly-configured Linux image to the machines, although It is specifically tailored to work with Caos NSA (Redhat-like), or GravityOS (a Debian derivative) payloads. Infiscale, the company that supports Perceus, releases the basic tools and some sample modifiable OS images for free, and makes their money off support and custom images, so it is pretty flexible option in terms of required financial and/or personnel commitment. The various provisioning and administration tools are generally designed to interact with various monitoring tools (ex. Warewulf or Ganglia) and job management systems (see next paragraph).
Accounting and billing users is largely about your job management system. Our clusters aren't billed this way, so I can't claim to have be closely familiar with the tools, but most of the established job management systems like Slurm, and GridEngine (to name two of many) have accounting systems built in.
The "standard" images or image-building tools provided with the provisioning systems generally provide for a few nicely integrated combinations of tools, which make it remarkably easy to throw a functioning cluster stack together.

As for GPUs... be aware that the claimed performance for GPUs, especially in clusters, is virtually unattainable. You have to write code in their nasty domain-specific languages (CUDA or OpenCL for Nvidia, just OpenCL for AMD) and there isn't really any concept of IPC baked in to the tools to allow for distributed operations. Furthermore, GPUs are also generally extroridnarly memory and memory bandwidth starved (remember, the speed comes from there being hundreds of processing elements on the card, all sharing the same memory and interface), so simply keeping them fed with data is challenging. GPGPU is also an unstable area in both relevant senses: the GPGPU software itself has a nasty tendency to hang the host when something goes wrong (which is extra fun in clusters without BMCs), and the platforms are changing at an alarming clip. AMD is somewhat worse in the "moving target" regard - they recently deprecated all 4000 series cards from being supported by GPGPU tools, and have abandoned their CTM, CAL, and Brook+ environments before settling on OpenCL, and only OpenCL. Nvidia still supports both their C

As an admin of a cluster that has evolved and changed over the last 7 years, I think I can help a bit. That being said, you truly failed at defining your real needs. I understand liking to stay with what people know, but from an end user point of view with interactions with a cluster, the only interface that they use is the only thing that needs to stay similar, and you can almost certainly use the same interface on any other linux distribution. So that said, what are you currently using to submit, queue, and schedule jobs? There are a few proprietary solutions out there and several open ones. There is Grid Engine (or Sun/Oracle Grid Engine), PBS, Maui, Torque, and several others out there. That should be the only real interface that an end user should have to the cluster.

Now comes the second question I have. Why do they need to be running X? I can understand having the X server installed and all the libraries, but you absolutely should only be running your servers at run level 3 (i.e. command line). You can still run applications if you set your display to a remote X server as the output device, in this case one that you run on windows desktop like Cygwin, or Xming. All you do by running X.org on the cluster nodes is waste about 1 gig of memory and 5-10% CPU resources, which could be utilized by your end users' jobs/applications.

Third, what kind of applications are your end users running? Are they real parallel environment applications using some sort of MPI (LamMIP/OpenMPI), or off the shelf products like Clustered Matlab (which actually uses MPI, but it is built into the product already, you just need to configure it properly)? Or are you really just running lots of batch jobs which may or may not be multithreaded applications, but do not do any intra-node communication?

Fourthly, how are you monitoring your existing cluster? Are you using something like Ganglia?

Finally, what kinds of third party software do you need to be able run/use? Is there anything that is commercial which may have limited support to specific linux distributions?

All of those things are questions that you need to really answer in order to recommend a distro.

All things being equal, personally, I would deploy a cluster using "Rocks Cluster" distro. It is designed from the ground up to be a easy to maintain and deploy cluster distribution. There are plenty of HPC specific packages/application/libraries available to be deployed on the nodes. "Rolls" are available, which basically contain a group of packages/applications/tools which are typically used together, or otherwise easily configure/install software that is required on each system, possibly with some complex interactions (for instance there is a "Ganglia" roll, which easily installs the Ganglia cluster monitoring software and automatically sets it up based on your Rocks installation. There is a "BIO" roll, which contains many open source tools and librarys which are useful in doing biological research clusters, like ClustalW, Glimmer, NCBI BLAST, just to name a few. Then there is the HPC roll, which is just some basic things like MPICH, MPICH2, OpenMPI, iozone, iperf. There is also a roll for PVFS for setting up a quick Parallel Virtual File System cluster).

It is designed from the ground up to be a cluster, not just a bunch of nodes running linux with high speed interconnects. It has management utilities to deploy applications across all nodes at once, quickly install OS on all your cluster nodes via PXE booting the compute nodes. Flash/upgrade the BIOS of computer nodes remotely via PXE boot. Basically it is designed to be managed and maintained as a cluster, not "x" number of individual systems. Seriously consider something like it.

http://www.rocksclusters.org/wordpress/

NPACI Rocks is probably your best bet. http://rocksclusters.org/

one assumes 44.1KHz is ok to reproduce 20KHz audio... This is WAY too simplistic. Take a ~15KHz sine wave and apply a phase shift to it. Sample it at 44.1KHz. Look at the amplitude modulation you get on the output. 44.1Khz is terrible for audio. The distortion from 10KHz up when sampling waves with phase shifts, harmonics, modulations ie complex forms is very high. An example would be to mic up a cymbal with a mic perpendicular to the edge. Beat the cymbal. It makes high frequency sound but it also rocks (not http://www.rocksclusters.org/wordpress/ ). so on top of the complex wave with harmonics etc one gets phase shifts and Doppler effect. I have mathematically reproduced this to try to show the reasoning behind my understanding of how 44.1 just really is not enough and the effects were worse than I thought. 96Khz seems a good starting point to sample the audio spectrum up to 20Khz reasonably reliably.

I'm sure there has to be a way. Clustering suites, say Sun's Grid Engine, or ROCKS http://www.rocksclusters.org/wordpress/ have utilities for pushing applications off to their nodes, you might be able to use that functionality in your environment. Also, not sure it is possible, but could you use kickstart (PXE) but rather than having commands to format the harddrive and install you just have to post install bit which you'd want to rcp the app over, run the installer and clean up for example. It would require a reboot but I think it should work.

I suppose this is good news for those that don't want to get their hands dirty building their own cluster. You could just network several servers together and simply install Rocks or UniCluster or any number of other cluster packages.

Yes it does! Rocks Linux! http://www.rocksclusters.org/

I agree. openMosix with its fork-and-forget SSI model works well for certain tasks, but for those tasks a computer with lots of cores works better. First of all, the operating system automatically migrates processes to other cores, and second, even individual threads can migrate without experiencing horrible shared memory latency. With 8-core computers on the market and 16-core computers coming soon, there's just not as much need for openMosix.

Most clustering applications benefit some form of communication such as MPI or at least a scheduler such as PBS or SGE, so openMosix's fork-and-forget model doesn't work well. Those applications can use Rocks Cluster Distribution, the real mainstay of open-source clustering.

http://www.rocksclusters.org/

ROCKS cluster at our university's department of economics.
There is a couple of clusters registered on the site, too.

Wikipedia says:
"Rocks Cluster Distribution is a Linux distribution intended for computer clusters. Rocks is based on CentOS, but uses a modified anaconda installer that simplifies mass installation onto many computers."

http://www.rocksclusters.org/roll-documentation/vi z/4.1/

I'll admit, I've never tried this because I couldn't get anyone to spring for the displays, or the space to stack several monitors, but just follow their instructions, and you're off and running.

No, seriously, if you're setting up a cluster where your work can be batch-queued, or intend to run MPI, then Rocks http://www.rocksclusters.org/ is the way to go. It also comes with tools such as SGE (Sun Grid Engine) or OpenPBS pre-configured, Intel compilers and libraries ready for you to drop a license onto (but of course the entire GNU suite is there as well, including Ada), has more monitoring tools (plus some nice web-based interfaces) than you can shake a stick at, and runs on IA-32/AMD-64/IA-64 (Itanium). It also has a Roll to help build a tiled display wall, which would be a really cool use of a small cluster.

They're also really great guys.

On the other hand, Oscar is supposed to be good, and if you're not into the whole batch-mode thing, you can get OpenMosix up and running using http://clusterknoppix.sw.be// ClusterKnoppix, and just fire jobs off into space and let them find their own unburdened node.

But still, Rocks is really an elegant and clean way to go, plus it will scale up in case you're going to deploy a huge one of these for real after you get your feet wet.

beowulf: http://www.rocksclusters.org/wordpress/ or http://www.warewulf-cluster.org/cgi-bin/trac.cgi

i'd recommend looking at the ease of building diskless clusters w/ warewulf.

Rocks is a great tool to build a cluster. It includes lots of monitoring tools and such, so you can see the status of nodes, etc. However, I'm not quite sure how large you're planning on going... May be overkill for a 4-noder. =)

Why? Because it's hard to fit a normal sized system disk in a 3U server with 16 drive bays. There's a tiny sliver of space above the drives that can hold a laptop CD ROM, Floppy, and 2.5" Hard drive. I've built several of these as head nodes for clusters using dual 3ware SATA RAID controllers and quad AMD boards. The new Escalade cards use Infiniband wiring from the RAID cards to the SATA backplane, so there's only four cables instead of sixteen, which is much nicer than trying to fit 16 SATA cables, two IDE cables, a floppy cable and 8 power cables past the six fans that sit in the middle of the box.

Yes, yes I can picture a Beowulf cluster of those, though I actually use ROCKS.

Red Hat does not offer a competing product, so what is the problem? There are many "cluster distributions" out there, but neither Red Hat, Suse, or any other major vendor have a well integrated cluster version of Linux. There are things like Rocks, Oscar, Warewulf, and companies like Scalable Informatics, or Basement Supercomputing are there if you need help.

Why is it insane?


Because you'll waste plenty of time solving problems that the Rocks team has already solved. See this presentation. Especially slides 38 and 47-57.

Another, perhaps even more popular Linux cluster distro is Rocks Clusters.

While I don't have personal experience with OSCAR, Rocks is really good. These days, doing a cluster with a "normal" distro is insane. I think MS will have to think long and hard before they come up with something equally easy to install and manage as Rocks.

That being said, I think MS is not targeting Win CCS at academic supercomputing, which has a long history of using Unix/Linux, but rather they want to expand HPC to business customers who otherwise have a 100 % MS environment.

Presuming you're being truthful, then:

Thank you. Thank you (collective) for actually worrying about shipping a Fortran with your development tools for Solaris x86. For pushing Opterons, which, while not Itaniums, are still strong floating-point processors, and easy to cluster. I'd say nice things about the Sun Ultra60 I used to have as a desktop, but that's a different era.

Now, get together with the Rocks guys http://www.rocksclusters.org/, and get a Solaris/Rocks installation out there!

It is very easy to setup a Linux HPC cluster. There are several projects out there for example:

http://www.rocksclusters.org/ (Rocks guys do a great job of simplifying the process. I highly recommend this one if you are new to Linux Clusters)

http://warewulf.lbl.gov/pmwiki/ - Warewulf cluster toolkit. Not as easy to use as Rocks but some very cool features and very flexible for different kinds of Linux Clusters.

http://oscar.openclustergroup.org/ - Oscar Toolkit is popular as well. I can't comment on it much since I haven't tried it.

Rocks has a great system for making high-performance clusters from similar machines. A Rocks cluster consists of a front-end ("master") node and a bunch of compute nodes (and I think special-purpose nodes).

The master gets a full Linux (RedHat-based) install. It's a NFS/DHCP/Kickstart server for the compute nodes, and runs whatever other services you want the compute nodes to use. The master has two network cards and acts as a firewall (NAT optional).

The compute nodes boot via DHCP and Kickstart, downloading their kernel and whatever other OS files you want to their local disk. You decide how much NFS or local disk to use.

Job queueing is handled by, e.g., Sun Grid Engine (an Open Source queueing package) or some other queueing software.

Here's the neat thing: to make a change to a compute node setup, you change the Kickstart config and reboot all the compute nodes (as they finish whatever queued work they're doing, or immediately if you want). That makes the sysadmin's life easy, while still maintaining the speed of having the OS on the local disk.

But their links could at least have mentioned OSCAR http://oscar.openclustergroup.org/ or my personal favorite, ROCKS http://www.rocksclusters.org/, as these are more prevalent than xCat systems.

Personally, I like Rocks, as I ran three parallel architectures (i386/AMD64/IA64), on the same based distribution, just with each tuned to their particular processor. Comes with SGE and Myrinet support out of the box, and there are Rolls, i.e. custom software assemblages, for OpenPBS, for those who prefer it, as well as PVFS. It's easy to set up, and easy to administer, as the nodes are presumed to be interchangeable and disposable. When you reboot a node, it's obliterated and a fresh OS and supplementary package repository are laid down on a clean disk. No questions about version skew.

They now have a custom roll to help you build a visualization wall, but I never had a chance to try that one. (try convincing your boss that you want 4 digital projectors and a big room to play with)

The downside to the above distributions are that they presume batch-queue environments, which is appropriate for most of my work, but less so for many people trying to simulate owning an SMP, without paying SMP prices.

Other people assure me that the current version of OSCAR is solid as well, but they seem to lag in the multiple architecture support area (Itanium is always behind), and don't current support AMD64 natively. On the other hand, they build on top of several RedHatish linuces, as opposed to Rocks where you get Centos (RHEL), period.

I built my penultimate cluster from IBM 1U opteron systems, using Rocks (http://www.rocksclusters.org/ as the system software. That was a decent setup from a price-performance standpoint, ran well, and supported our legacy 32-bit linux apps rather well. Currently I'm using Apple XServe G5s, mostly because of the availability of XLF and the low power-draw. I also don't have the legacy binary issues in the new lab that I had when supporting a heterogeneous group.

Having started with white-boxes on bread-racks, passed through IA-64 and SP2, and ended up with racked 1U, I would say that what I care about is reliability and support first, speed second, and convenient software environment third. Google can afford to have a system fail, due to their distributed structure and design for failover, but tightly-coupled scientific codes tend not to tolerate this as well.

Therefore, my recommendation would be for scientific clustering, (2-way proc, 1U/2U rack enclosure from IBM, HP, Apple), G5/Opteron > Xeon. If I had the money, then swap G5/Opteron for Power4/IA-64(HP). Rack mounting saves more than just space; it's also easier to perform maintenance on than the bread-racked desktops had been.

Note, my opinion only applies to my environment; high-performance/high-throughput computing. High availability clusters are another beast entirely.

Although having 96 nodes in a single box makes it quite cute, from what I can interpret from the specs, you would get more bang for your $100K by getting what the beowulf crowd like to call MMCOTS (Mass-Market-Common-Off-The-Shelf, i.e. mass produced computers from Dell or the like), hooked toghether with a specialty high-bandwidth low-latency interconnect like Infiniband, Myrinet or SCI. Running a free beowulf cluster OS like for instance ROCKS would mean that a normal linux admin could maintain it quite effectively.

I expect this thing to be marketed towards scientists in small or medium businesses that aren't employing many/any IT staff, who use commercial computer models to do things like theoretical chemistry (Gaussian, ADF etc), bioinformatics (Phase, BLAS, Paralign etc), fluid dynamics, statistics, crypto, you name it. I don't expect to see any of these types of systems used in normal supercomputing sites, where people write their own (parallel) code and skilled staff maintain the cluster.

Certainly ... 1. RHEL like OS is being deployed and tested for free ... on much more equipment than it would otherwise be. This leads to more potential RHEL customers. We also supply bug reports to the RHEL bugzilla so they see and can fix many more problems.

2. There are 4 or 5 open source projects that are based on a CentOS base:

http://contribs.org/modules/phpwiki/index.php/Upco ming%20Releases%20page

http://www.openfiler.org/sponsors/

http://asteriskathome.sourceforge.net/

http://www.rocksclusters.org/Rocks/

http://www.visualmediatech.com/

3. There are many customers who want an Enterprise Linux that they don;t have to pay for ... we are providing them a service.

4. That is how the GPL works :)

Why does it take so long to build a super computer ...
It doesn't.

Rocks provides an easy way to build a Beowulf cluster. See http://www.rocksclusters.org/.

You can build a working cluster, starting with the hardware and installation CD-ROMs, in minutes; see http://servers.linux.com/servers/04/08/27/1943227. shtml?tid=29&tid=94 for one account.

Disclaimer: I work with the folks who created Rocks.

You might take a look at Rocks:
http://www.rocksclusters.org/.

Quite a few people have built Rocks clusters out of a bunch of old computers.

Disclaimer: I work with the folks who created this.

This is similar to what clusters try and do. It is important to maintain the same OS state on all nodes. Take a look at Rocks Clusters. Rocks will push the same OS image out to the nodes of the cluster. There is no reason the cluster nodes could not be workstations on a desk.

cluster webcam

I have been reading all the G5 benchmarks with mixed feelings lately. First of all I should clarify that I am a fan of Macs, and so I am glad to see that there is any interest in the platform at all.

However, I do most of my real computing on a home-grown linux cluster using Rocks. These intel machines are simply so cheap if you step slightly back from the bleeding edge, that I don't know if I could justify spending a significant amount on an equivilent Mac cluster (although I am watching V. Tech's apple cluster, just like everyone else apparently is...).

Is there really much need for so much desktop power? How many users will utilize the full potential of a dual G5? Keep in mind that if even slashdot users can't keep two procs going, the general public has little hope.

Of course, this will not stop me from buying one.. It's just so cool looking... I am just confessing that I realize it is wasteful...

We are actually using a derivative of rocks, called BioBrew. BioBrew also comes with software for biologists:

the NCBI toolkit, BLAST, mpiBLAST, HMMER, ClustalW, GROMACS, PHYLIP, WISE, FASTA, and EMBOSS.

• SCSI Support
• Easily reinstalled nodes
• Pre-installed queue software
• brain dead admin tools
• No French Government

Check it out: Rocksclusters