Slashdot Mirror
With Linux Clusters, Seeing Is Believing
Roland Piquepaille writes "As the recent release of the last Top500 list reminded us last month, the most powerful computers now are reaching speeds of dozens of teraflops. When these machines run a nuclear simulation or a global climate model for days or weeks, they produce datasets of tens of terabytes. How to visualize, analyze and understand such massive amounts of data? The answer is now obvious: using Linux clusters. In this very long article, "From Seeing to Understanding," Science & Technology Review looks at the technologies used at Lawrence Livermore National Laboratory (LLNL), which will host the IBM's BlueGene/L next year. Visualization will be handled by a 128- or 256-node Linux cluster. Each node contains two processors sharing one graphic card. Meanwhile, the EVEREST built by Oak Ridge National Laboratory (ORNL), has a 35 million pixels screen piloted by a 14-node dual Opteron cluster sending images to 27 projectors. Now that Linux superclusters have almost swallowed the high-end scientific computing market, they're building momentum in the high-end visualization one. The article linked above is 9-page long when printed and contains tons of information. This overview is more focusing on the hardware deployed at these two labs."
This is how we nerds measure our penises. ;)
Virginia Tech's "System X" cluster cost a total of $6M for the asset alone (i.e., not including buildings, infrastructure, etc.), for performance of 12.25 Tflops.
By contrast, NCSA's surprise entry in November 2003's list, Tungsten, achieved 9.82 Tflops for $12M asset cost.
Double the cost, for a Top 100 supercomputer's-worth lower performance.
And it wasn't because Virginia Tech had "free student labor": it doesn't take $6M in labor to assemble a cluster. Even if we give it an extremely, horrendously liberal $1M for systems integration and installation, System X is still ridiculously cheaper.
I know there will be a dozen predictable responses to this, deriding System X, Virginia Tech, Apple, Mac OS X, linpack, Top 500, and coming up with one excuse after another. But won't anyone consider the possibility that these Mac OS X clusters are worth something?
How to visualize, analyze and understand such massive amounts of data?
How to write complete sentences?
The article linked above is 9-page long when printed and contains tons of information.
I hope the poster doesn't actually expect any of us to post any meaningful comments based on having read that article, it's a lost cause.. At least on me.
... my computer could do that.
Damn! What kind of paper stock are you printing on?
--- Ban humanity.
Roland Piquepaille and Slashdot: Is there a connection?
I think most of you are aware of the controversy surrounding regular Slashdot article submitter Roland Piquepaille. For those of you who don't know, please allow me to bring forth all the facts. Roland Piquepaille has an online journal (I refuse to use the word "blog") located at www.primidi.com. It is titled "Roland Piquepaille's Technology Trends". It consists almost entirely of content, both text and pictures, taken from reputable news websites and online technical journals. He does give credit to the other websites, but it wasn't always so. Only after many complaints were raised by the Slashdot readership did he start giving credit where credit was due. However, this is not what the controversy is about.
Roland Piquepaille's Technology Trends serves online advertisements through a service called Blogads, located at www.blogads.com. Blogads is not your traditional online advertiser; rather than base payments on click-throughs, Blogads pays a flat fee based on the level of traffic your online journal generates. This way Blogads can guarantee that an advertisement on a particular online journal will reach a particular number of users. So advertisements on high traffic online journals are appropriately more expensive to buy, but the advertisement is guaranteed to be seen by a large amount of people. This, in turn, encourages people like Roland Piquepaille to try their best to increase traffic to their journals in order to increase the going rates for advertisements on their web pages. But advertisers do have some flexibility. Blogads serves two classes of advertisements. The premium ad space that is seen at the top of the web page by all viewers is reserved for "Special Advertisers"; it holds only one advertisement. The secondary ad space is located near the bottom half of the page, so that the user must scroll down the window to see it. This space can contain up to four advertisements and is reserved for regular advertisers, or just "Advertisers". Visit Roland Piquepaille's Technology Trends (www.primidi.com) to see it for yourself.
Before we talk about money, let's talk about the service that Roland Piquepaille provides in his journal. He goes out and looks for interesting articles about new and emerging technologies. He provides a very brief overview of the articles, then copies a few choice paragraphs and the occasional picture from each article and puts them up on his web page. Finally, he adds a minimal amount of original content between the copied-and-pasted text in an effort to make the journal entry coherent and appear to add value to the original articles. Nothing more, nothing less.
Now let's talk about money. Visit http://www.blogads.com/order_html?adstrip_category =tech&politics= to check the following facts for yourself. As of today, December XX 2004, the going rate for the premium advertisement space on Roland Piquepaille's Technology Trends is $375 for one month. One of the four standard advertisements costs $150 for one month. So, the maximum advertising space brings in $375 x 1 + $150 x 4 = $975 for one month. Obviously not all $975 will go directly to Roland Piquepaille, as Blogads gets a portion of that as a service fee, but he will receive the majority of it. According to the FAQ, Blogads takes 20%. So Roland Piquepaille gets 80% of $975, a maximum of $780 each month. www.primidi.com is hosted by clara.net (look it up at http://www.networksolutions.com/en_US/whois/index. jhtml). Browsing clara.net's hosting solutions, the most expensive hosting service is their Clarahost Advanced (http://ww
Supercomputers have become so advanced we need more supercomputers just to understand them.
oh, great, tell us about what the machines can do. i want pictures dammit!
- my userid is lower than yours
A 35 million pixel screen would rock for Half-Life 2. Where can I get me one? Looking at the picture, it's kind of like 3 monitors stuck together, so maybe I'll save some money and only get 1/3rd of the setup. How much can that cost? I mean, really.
Against stupidity the Gods themselves contend in vain.
eh?
42
With Linux Clusters, Seeing Is Believing
Does this mean that we don't have to just imagine a Beowulf cluster anymore?
It is easier to build strong children than to repair broken men. -Frederick Douglass
A machine that can compile a Stage1 Gentoo install in a reasonable amount of time.
"Joy is not in things; it is in us." Richard Wagner
That should be As of today, December 13 2004....
Also, it seems Roland had scaled back the number of ads on his page this month. Or maybe nobody has bought the ads. Last month, when I was researching this he had 1 premium ad and 4 regular ads available. I would have released this report back then, but I've been banned from posting for some time now. I'm only posting this now because I am not at my regular location.
Sigh... another jealous M$ fanboy who hates linux cause his career relies on running windows and clusterpatchupdate.exe.
n/t
So, if I've got this straight, Slashdot drives the banner ad traffic, real journalists write the content, and all Roland has to do is rip off a few articles, then sit in the middle and collect the checks. How do I get a sweet gig like that?
http://trackerwww.prq.to/download.php/3264787/11_0 4.2.pdf.torrent
Downloaded off the page.
10 years or so from now, you'll have this much power in a little 1" x 1" box (probably priced around $100 dollars, too).
To reaffirm what the article said building linux clusters is very simple. In fact certain distributions such as bccd and cluster knoppix specifically for that. Although configuring clustering softwares such as pvm mpi lam mosix etc wouldn't be a problem, I prefer something which has almost everything build into one package thats why I like the above distros. In fact I built a cluster (using BCCD) at home and used it to render images built from povray. I used pvmpov for the rendering on a cluster part. Although there were only four machines the speed difference was evident. And above all making clusters is extremely cool and shows the paradigm shift towards parallel computing.
So now Monsieur Piquepaille has been shamed by scornful posters into including a link to the actual article (instead of harvesting page views), but he'd still really, really like you to click through to his page....
grammar-lesson free since 1999. (rescinded - 2005)
There are a bunch of different viz techniques listed on http://www.tauceti.org/research.html#vhere.
Now that Linux superclusters have almost swallowed the high-end scientific computing market...
While some simulations parallelize very well to cluster environments, there are still plenty tasks that don't split up like that.
The reason clusters make up a lot of the Top 500 list is that they are relatively cheap and you can make them faster by adding more nodes - whereas traditional supercomputers need to be deisgned from the ground up.
Maybe they are building cool Linux clusters but they can't be that smart. They have their mail addresses just sitting here on the site for spammers to harvest!
Darl! Welcome back. We've not heard from you for a while.
Leave some market share for the big guys.
So these guys have some fancy computers displaying pretty pictures. Lots of computers = more detailed pics. But it's still viewed by humans in the end. I can't take in that much information. The art of simulation is to extract the bit you're interested in, and leave the rest. Aren't these systems just generating lots of trees and no wood?
Geez, I hope the youngsters can get through that without a break...
GridEngine is also a main component of Linux clusters. Without a batch system, you can only have one user using the cluster at a time...
e t
Opensource+free:
http://gridengine.sunsource.n
Look here.
The speed you quoted is the theoretical peak, not the actual maximum achieved in a real world calculation (like the Top 500 organization's use of Linpack).
System X's equivalent theoretical peak is 20.24 TFlops.
I'm also not indicting Linux clusters in the least; they've clearly shown they can outperform traditionally architected and constructed supercomputers for many tasks, with the benefit of using commodity parts - at commodity pricing. All I'm saying is that there's a new player here, and it's a real contender, and has done a lot for very little money...which was the whole goal of Linux clusters in this realm in the first place.
(Also, as I said, the volunteer labor model is irrelevant - let's just pretend it was professionally installed for an additional $1M, or even $2M if that would satisfy you. It's still several million dollars cheaper, and 3Tflops greater performance. These are BOTH rackmount clusters with similar amounts of nodes and processors, running a commodity OS with fast interconnects. There are differences, yes, and perhaps even differences in goals. But looking past that, price/performance for something like this is still an important metric.)
Once you have your visualization cluster, decided on the CPU, the interconnect, the OS, etc., you might ask what kind of application you can run on it.
"Provided by the management for your protection."
The only special thing they did for VT was *take back* the original G5 towers, and provide 2.3GHz G5 Xserves before they were otherwise available. The $600K upgrade did not reflect any significant discount or gift. A similar cluster could be built by anyone, now, for around that same total price of $6M.
And yes, they saved a lot of space, and heat/power (130nm chips to 90nm) and increased the performance by 2Tflops (by going from 2.0GHz processors to 2.3GHz). The major gain, though, was ECC memory.
Seeing 8 x 8 monitors are 12,800x11,200 pixels!!!
14.33 MegaPixels, wow!!!
open4free ©
Unless you change the settings so it is compiling mulible applications at the same time. The speed to install Stage 1 of Gentoo won't be much faster then a 2 maybe 4 CPU system. These super computers and clusters use a concept called Parallel Processing. It is a process where a task is broken up and are handled by many processors in parallel. Most applications are not designed to run in parallel. So unless you have a compiler that is designed with Parallel Processing the OS will give the compiling task to 1 processor to processes out. You may get a slight speed advantage because the OS resources are being handled by an other processor but you are not guanrenteed 2x performace with 2 processors. Espectially with most make scripts for application you compile one program when that is done you do the next one. There are some algroithms that can done very Well (orders of magintude less) on Parallel Processing and there are other algorithms that just cannot be parallelized. Having 2 1 Ghz Processors is not the same as having 1 2 Ghz Processor. The 2 1Ghz will probably handle load much better then the 1 2ghz processor but the 1 2ghz processor will probably run your game better.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
open4free ©
30 accepted stories since August 29th, 2004! Wtf is going on here?
E pluribus unum
...except get untold amounts of recognition, publicity, free advertising, news articles, and the capability to catapult themselves to the forefront of the supercomputing community overnight for a paltry sum of money, thus attracting millions of dollars of additional funding and grants to build clusters that WILL be doing real work, such as the one we're talking about now (which is more than capable now that it has ECC memory), and the several additional clusters they plan to build in the future, not to mention the benefit of proving that a new architecture, interconnect, and OS will perform well as a supercomputer, allowing more choice, competition, and innovation to enter the scene, which ultimately results in more and better choices for everyone.
10 years or so from now, you'll have this much power in a little 1" x 1" box (probably priced around $100 dollars, too)
yes...refer to subject line
-- haha i know it's not funny but i said it so i'm gonna pretend it's funny --
Clusters are proven to be cost effective, but they do require more labor to optimize code to get it to work in that environment. Its easier to have the system and the complier do the work for you in a single image system. This article address those issues and concerns. single image shared vs distributed memory in large Linux systems
I keep forgetting about Roland Piquepaille, and I click on his damn "overview" link.
/. post these damn things from him? The guy is a shameless shill.
Why does
There should be a highly visible disclaimer on everyone of his posts: "This link goes to an external site that is NOT the article's original site, and this external site is unendorsed by Slashdot. This external site profits from traffic generated by clicking on this link."
Someone needs to write a Firefox extension that filters any mention of his "overviews". Hmmmm....
122.88 MegaPixels, wow!!!
open4free ©
Yeah, and he even changed his URL. Maybe he was in too many spam blocklists. Does he spam other places too, or just Slashdot?
IMO, computer aided visualization is over rated. Sure, it's good in a production environment but the mental effort of visualization is a tremendous aid to imagination. There's no way to computerize epiphany.
Now I'm the grandest Tiger in the Jungle!
Skynet goes online unbeknownst to any human.
Yep - I'm sure Win 95 would do the job far better than any pinguin....... Or aaybe XPloder....... you don't need one of these pieces of kit to "visualise" the effect of running a MS system on it.......
The fastest rendered blue screen of death ever?
1)Whatever institution ran it would be bancrupt by the time the beta gets to them....
2)The soft will take 5 years and 2/4 major overhauls to fix/complete
By that time home pc's will be starting to overtake it performance wise......
Seriously - would you be able to dig around in a Windows systems code and customise it to the machine?
Shame my laptop is a linuxphobe....... at least my desktop is a proper piece ok kit.....
I *love* FreeBSD and I'd choose it over Linux any time.
But obviously, Linux *is not* garbage.
Since you're proving to be such a dickhead, would you please stop using any of the marvelous BSD systems? Please?.. Thanks.
--
Being able to read *other people's* source code is a nice thing, not a 'fundamental freedom'.
Blogads isn't based on ad views or clickthroughs. It is based on the number of hits the website gets.
Meanwhile, the EVEREST built by Oak Ridge National Laboratory (ORNL), has a 35 million pixels screen piloted by a 14-node dual Opteron cluster sending images to 27 projectors. Yes, but does it get a decent frame rate running Doom 3?
All I got from that was that she doesn't like slashdot. The rest just contradicted itself.
-- haha i know it's not funny but i said it so i'm gonna pretend it's funny --
I saw a demonstration of this a few months ago. It was a 3d simulation of a 10kiloton nuclear bomb going off on a street corner in what looked like a major city. It was a very high resolution rendering on a big widescreen display and it was pretty scary. I've seen a few documentaries on this but to see it in slow motion, in 3D, was just mind boggling. I had a nightmare a few days later in fact, almost wish I hadn't seen that.
Note: Feds, leave me alone, this was NOT a classified demonstration. Just happened to be in the right place at the right time.
What if Digg added local news and a Slashdot inspired comment karma system? ---
http://houndwire.com
Powerwalls, which are typically the size of a conference room wall, allow a group of scientists to study still images or watch a movie, frame by frame. "Researchers can freeze images, pan, zoom, move back and forth in time, and see details too subtle or small to discern on a desktop monitor," says electronics engineer Bob Howe, head of infrastructure and facilities for the Visual Interactive Environment for Weapons Simulation (VIEWS) Program. At the same time, because of the powerwall's sheer size, users can still view the global problem while keeping the details in perspective. Powerwall displays are especially useful for...
"I seem to have mastered a certain amount of control over physical reality."
Of course, the connectors may be a problem...
Lose essential liberties to get temporary safety = get only hassles and security theater.
Doesn't that assume that /. readers RTFA?
We all know that's not true.
PCI-E has symmetric bandwidth. Current generation graphics cards will undoubtedly not be able to take advantage of this feature, they've spent so long getting data to the graphics card that thats all they're optimized for, but in the long run this has some crucial implications.
Namely, it allows for graphics cards to operate better in situations exactly like this; clustered applications. As it stands, the graphics card can crunch an enormous amount of data, but is extremely poor at sending it back to the CPU & system. It's optimized for screen dumping only.
Sony's Cell is going to be absolutely crucial as a tech demo for this foresighted technology. We're heading towards a more distributed computer architecture where various specialized units pipe data between each other.
In summation,
Its my hope that eventually graphics cards will catch up and perform better bi-directionally. After that, we've got to wait another 5 years for PCI-E implementations to catch up and perform better switching (vis-a-vise multiple fully-switched x16 busses). We are moving away from the CPU for high performance computing; the cpu currently performs both control and data-processing. Graphics cards are just the first wave of the distributed architecture phenomena, Cell will be a light-year jump towards the future of computing in the intricate levels of hardware reconfigurability. there's a good powerpoint on the patents behind cell here.
Ultimately this will lead towards the tearing down of the computer as a monolithic device, and a rethinking of what exactly the network and os's roles are. Queue exo-kernel and DragonFly BSD debates.
Looking at Apple's website, I see that you can purchase XServe G5s w/ single or dual 2 Ghz processors. System X uses 2.3 Ghz processors (2200 of them), which are not availible for purchase.
So maybe creating a cluster of XServe G5s is cheap, and maybe a great deal all around, but if Apple is hand supplying the special 2.3 Ghz processors to VT, it is not unreasonable to assume they also cut them a nice price break. Now, maybe Apple will give the same price break on the same unavailible processors to anyone who asks, in which case, yeah, XServe G5 clusters are the way to go!
If I were Roland, I would do the world a favor and stab myself in the face. Seriously though, if I were Roland why would I post about what a hack I am and then link to my site? I only linked to his site because you should go there to see what I am talking about. I don't think that my post will generate any more traffic than he regularly gets from Slashdot.
...is that he has had ZERO rejections. Lifetime.
awesome combination of two classic slashdotisms
Virginia did not get any exceptional deal at all. Apple did provide consulting. But the hardware was ordered over the internet on the Apple Store. They just put 1,100 in the quantity field and checked out. No break was given and it was still far cheaper than anything else.
Click here or here.
SCO is going to make a killing on these things! Keep clustering, folks!
I have a redergarden (not quite a renderfarm ;) and I've used POV-Ray to make visualizations and animations of my supercell model data. See the Novermber 2004 Linux Journal (cover plus article) for what I did. What I did was get POV-Ray, which, note, is "free" (with restrictions especially on the latest version) and got it to recognize my model data format natively (using the source of course). Then I can fire up my 14 of my nodes, all NFS mounted to a terabyte RAID array, with a python script (using pyMPI) and they each read in their data and happily render frames. Then I make movies with the resulting PPM images using mjpegtools.
Note that this approach crashes and burns if you can't fit the 3D array of your data into core memory. This becomes an issue especially with very large datasets and if you are rendering lots of different isosurfaces at once, which I often do. You can always downsample, or just read in what the camera sees instead of the whole model domain, but you can still run into brick walls with very large data sets. Of course, if you are rendering to a 1024x768 screen, and you are looking at the entire domain which is 700x700x100 you probably can downsample significanly without losing visual detail.
Anyway, for those interested, Here is a link to a directory conttaing mpegs and a talk I gave earlier this year which contains 1024x768 mpeg files and the talk itself. NOTE: some of these files are BIG. I would recommend this 32 MB mpeg and this 73 MB mpeg for a sample of what can be done with open source tools. Some supplemental material to the LJ article can be found here
Leigh Orf
Professor of Atmospheric Sciences
Central Michigan University
A squid eating dough in a polyethylene bag is fast and bulbous, got me?
Obviously a pseudonym. I mean, come on, 'Roland Piquepaille'? Where do they come up with this stuff? I tried to say the name aloud and my mouth damn near exploded.
Cue 'old Korean' joke. Can't have anyone feeling left out, can we?
Ironically enough, each and every one of those is professionally-developed. Linux? Pcha ... keep on wishing.
:)
Hmmm, I wish I was paid half what Linus is paid for kernel development. Not to mention Alan Cox, and most of the @ibm.com, @redhat.com, @sun.com or @suse.com in the kernel changelogs.
What were we talking about again? Ah, yes. You're an ignorant asshole
ROLAND PIQUEPAILLE
is an anagram for:
A LIQUOR LIPPED NEAL
"Roland Piquepaille" is the aptly chosen pseudonym that CowboyNeal uses to generate his drinking money. $650 a month will buy a lot of hooch. Of course the editors aren't going to reject a story from one of their own alter egos!
Here are a few other favorite anagrams:
A DOLLAR PENILE QUIP
A LIQUOR PEE LIP LAND
I PILL A POLAND QUEER
A LO PENILE LARD QUIP
Each of these visualization clusters boast huge resolutions, but little is said of geometry performance. Asking Chromium to distribute OpenGL calls from a head node to N graphics nodes for such a large scene will surely eat bandwidth in enormous quantities. Naturally, throwing Myrinet or Infiniband at the problem will help. Just like the old IBM mainframes, you might not want to trade in that SGI Onyx just yet. There may be some problems for which it can excel (and accel).
I'd like to see some geometry perf specs for these hi-res screens and comparisons of various interconnects. Anyone got a link?
On 07 October 2004:
"Nvidia 6800 Ultra as rare as hens' Doc Marten boots"
http://www.theinquirer.net/?article=18932
and on 05 December 2004 : still no 6800 Ultra available!! :
"6800 Ultra hardly available in EU $740 for the card that you can't buy"
http://www.theinquirer.net/?article=20055
Robert