Building The MareNostrum COTS Supercomputer
karvind writes "IBM Power Architecture Community Newsletter has a story about making a supercomputer (Number 4 on top 500 list) from easily available components (like BladeCenter and TotalStorage servers, 970FX PowerPC processors, and Linux 2.6). A joint venture between IBM and the Spanish government, it is named MareNostrum: the Latin term meaning 'our sea.' Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers."
Mare Nostrum refers to the Mediterranean Sea.
Mare Nostrum literally means "our sea". It is what the Romans called the Mediterranean Sea during the Empire. As you can see, it was an apt name.
Rhymes that keep their secrets will unfold behind the clouds.There upon the rainbow is the answer to a neverending story
I am an African Grey parrot, and I can tell you that while you humans are celebrating this achievement, I and my fellow Greys are laughing at you. Supercomputers are old news to us; in fact, one of my friends solved the halting problem while taking a crap the other day. Seriously, people, we like you 'cause you feed us, but leave this kind of stuff to us.
/. thought my user name was too long)
(I tried to register an account but
but does it run Linux? Oh crap, never mind.
Couldn't they just "me too" it and run xserves >_
Now im going to cry!
(Number 4 on top 500 list)
...while being Number 6 on top 300 list, and Number 65 on top 2000 list.
This is like those CDs that have 'best of the Top40' and not contain the top10 list of that
If an experiment works, something has gone wrong.
and after about 16 (or 32) years we'll have that power in our desktops...
-- and if life has failed you leave the cross you're nailed to
Isn't this basically a Beowulf-type cluster with just many many nodes? Exactly where lies the innovation? The fact that a cluster of many processors have a lot of computing power is not exactly new.
see a Text Widget
I love the first line in the article, which ends, "is constructed of such totally off-the-shelf parts as IBM BladeCenter JS20 servers, 64-bit 970FX PowerPC processors, TotalStorage DS4100 storage servers, and Linux 2.6. This is its story."
Right, like I regularly go to Fry's to stock up on some DS4100s and Bladecenters. I'd love to be the geek for whom that stuff is "off-the-shelf". Can you even buy bare PPC CPUs and mobos?
...about it still not being sufficient to catalogue ones porn collection.
I happened to look at the Top 500 supercomputers site and I couln't help noticing out of the top 5 supercomputers almost half are in non-US countries like Spain and Japan. This is not to beat some kind of patriot act drum. Instead, it got me to thinking.
With supercomputing powers now avaible to any country or group with a few readily available components, it is only a matter of time before these supercomputing powers may be used by a rogue state or radical group to cause havoc among electronic communications using methods like denial of service attacks, spyware, and crapflooding message boards.
I think it is high time the nations of the world put their heads together and addressed this issue. For example, I don't think the US Federal Government even has any cabinet-level position like Secretary of Information Technology or something like that. When are they going to get with the times? It will probably take another terrorist attack or something.
Of course, the *squawk* was only inserted to conform with human stereotypes about parrot speech. We're actually much more sophisticated than that; in fact you have probably heard one of us talking and not realized it was a bird!
This is all about timely and focused execution. The speed at which this project was realized is important. Consider: from the initial concept in late December of 2003 to assembling the computer in Madrid took less than a year. Normally, this kind of supercomputer projects take years.
Lame!
SGI had NASA AMES' Columbia online in 120 days, and landed #2 on the Top500.
Brandon D. Valentine
w00t
yes now I can make my own amazing super computer made of... BOOKS!
Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers.
Sounds like the specs of Microsoft's Xbox 3...
Don't take life so seriously. No one makes it out alive.
I'm sorry, but why are we giving IBM free press again? For god's sake, the very first sentence of the freaking article is utter rubbish:
The MareNostrum supercomputer at the Barcelona Supercomputing Center, ranked number four in the world in speed in November 2004, is constructed of such totally off-the-shelf parts as IBM BladeCenter JS20 servers, 64-bit 970FX PowerPC processors, TotalStorage DS4100 storage servers, and Linux 2.6
That's it - the first sentence of the article, if you exclude the title, the credits and the date, that is.
And we can already call advertising bullshit. I'm sorry, but how is the MareNostrum system made of any more "totally off-the-shelf" parts than the number two system on the Top500, NASA's very own Columbia? In fact, the 64-bit 970FX PowerPC processor is NOT an off-the-shelf part: unlike the Itanium 2 CPUs in Columbia, you can NOT buy such CPUs individually (for good reason: unlike the Itanium CPUs in Columbia, MareNostrum's CPUs are not socketed but soldered to the JS20 system board, so there goes upgradeability...)
Seriously though, why should we give a rat's ass about MareNostrum? Columbia is faster, more efficient, really is made from off-the-shelf parts and also runs Linux.
What's that? Oh, IBM are a good company and SGI aren't? For fuck's sake, SGI are better friends of Linux than IBM are. What did IBM do for linux? Nothing compared to SGI. IBM ported JFS, a crappy journaling filesystem, oh and they ported it to their own POWER/PowerPC architecture systems. W00p!
SGI:
- Gave Linux XFS, one of the fastest filesystems around, with _many_ advanced features (just look in your kernel config sometime)
- Scaled Linux beyond 64 CPUs for the first time (and indeed, they hold the record at 2,048CPUs): they fixed a _ton_ of scalability problems, and continue to do this on a daily basis (just look at this week's archive of the linux-ia64 mailing list to see what I mean!)
- Open sourced their Itanium compiler
- Created OpenGL (notice carefully the Open in OpenGL. You can bet your bottom dollar if IBM created a funky new graphics API it would _only_ work on PowerPC machines with IBM video hardware!)
Yada yada. All I'm trying to say is there are other companies out there who have really taken Linux to heart and have made open source development in their best interest, not just IBM. SGI is just one example, there are many others.
... a beowulf cluster of these.
This was a political decision, HP already complained because there was not a sellers competition.
Last government of Jose Maria Ansar aka 'Estamos trabajado en elloooooo' brought this computer to try get the ITER.
PD: Africa comienza en los Pirineos.
Screw easily available. Anyone with a budget to buy 2,282 servers and 4,564 processors can afford custom parts.
Call me when it's also easily affordable and I can pick up the parts at my local Fry's, or better yet, my local supermarket.
The only real item of interest was that it was made with all IBM parts and Linux.
...
Come to think of it, if Fry's also sold groceries I won't have to shop anywhere else...
It's just the thing to find SHA1 collisions of ISO images in 2^56 operations...
The spanish are going to use this system to calculate how many litres of ice cream are required to satisfy the tourist season, and other important problems faced by the economy.
NCH
The literal translation of 'Mare Nostrum' is 'Our Sea' it however referred to the Mediterranean, which was controlled by the Roman Empire.
In the same way Americans (well USIANS) today refer to Earth as 'Our Planet' (because Earth is controlled and policed by the US Government) and 'Pax Romana' is equivalent to 'Pax Americana.
Ok, for me as Spanish is not a proble is Africa frontier start in the Pirineos, Africa was the kind of the Egyptians and other brilliand cultures. The problem is that some people that live in Spain should be deported because are so much stupid. I'm Spanish I like to be Spanish and for SPAIN is the better contry. I'm a republican man, porn in Barcelona, with all my parents from Barcelona, Ripoll for surname. I love SPAIN.
god damn it
Is it Madrid or Sevill where Black Pete drops off all the naughty children?
...but how fast can it open Photoshop CS?
lol
fuckin' gooks
but I bet Windows still runs slow on it.
Coding Monkey.org - Spanging the heavy spade of truth into t
REALY, DOES IT????
Given enough money how is that impressive anymore? What's the best single-thread-performance machine today?
ken livingstone!
I just love how every time someone writes about another grid or supercomputer or beowulf cluster they always say "easily available components" as if I could find most of them in a standard IT closet or just run down to the local computer shop and pick them up with my corp. AMEX.
In what world is 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers easily available??? Maybe if you are Big Blue, but then, why would it be more difficult for them to throw together a fully proprietary supercomputer?
...does it have an AGP slot?
Need an ISP in South Africa?
I'm a republican man, porn in Barcelona ...
... you mean there's Republican porn over there? yuck! ... with all my parents from Barcelona
... well, there goes one more failed genetic experiment. Better luck next time!
umm
Really? ALL of them, no kidding
Get a load of Xserves, install Xgrid or something else, plug in, process (it seems it takes about no time to assemble and it runs osX).
Add more Xserves to get more power...
But I thought it was Ouray Easay!
What's going on?
This is all so confusing! I need to take a nap.
which is why you're seeing so many IBM articles lately.
There's trouble brewing, friends -- Sims was laid off for financial reasons, not fired.
it's really simple -- there are serious financial issues at /. Sims was laid off for financial reasons, not fired, and the rest of the publishing group has been warned to increase revenue and decrease costs. Hence running articles like this to please IBM.
This looks like a lot of brainpower concentrated in one small area. Who knows? They may even get USB broadband working. Ok I know it's a long shot.
They must love mares and have sex with them daily!
:)
....nasza szkapa?
Am I the only one finding it curious that nowhere in the specs do they mention how much RAM there is per node or in the aggregate?
It mentions how many nodes, how many CPUs, how many racks, how much storage, but not how much RAM.
500GB of disk, 5TB of transfer, $5.95/mo
Some one get these guys one of these computers, see how long it takes for their virii to evolve enough to take over the world!
The scary thing is, that the CELL with its potential for 25 GFlops of double precision floating point, could rival this system with just 1600 8 SPE units.
Granted, the CELL isn't exactly off the shelf, and I'm willing to bet 4,564 970FXs will be cheaper than 1600 CELLs for quite some time, so the project still has merit.
Anyway, HP can complain as much as they want. I find that surprising. The Mare Nostrum is a result of many years of colaboration between the UPC and IBM in the form of the CIRI (CEPBA-IBM Research Institute), a center where research on parallel applications and tools is conducted. IBM had the intention to build a large supercomputer out of PowerPCs and blades and they were considering even places outside of the US. People at UPC and particularly the CIRI and the Computer Architecture department started talking with the government to bring the machine here, and along with it, start a new research institute on supercomputing (the BSC, btw my new employer). well that's at least what I've heard here at upc.
So HP would have wanted to compete??? As far as I know, all research that HP does in barcelona is dedicated to printers ...worse, it could be 4500 itaniums, can you imagine? ;-)
a beowulf cluster of these...
http://www.coattails.net
niggers
Africa start in your mind and end over the ground you are ....
If we consider how much of international law it misinterprets, breaks, or simply ignores ( the list is waaaay too long), the US is certainly the primary rogue state.
Sounds to me like building the biggest supercomputer today is nothing more than throwing more money at it than your compeition. No real technical skill needed beyond beyond that of wiring up your new home entertainment center a few hundred times over.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
Bit of hint there that the grandpappy was sending up the 'terrorists are everywhere' idiots ;-)
J.
You're only jealous cos the little penguins are talking to me.
please, when blade servers are considered off the shelf items, let me know. but at the present, most blades run around $2,000 dollars plus, more than what I paid to build my pc from scratch. Still it could probably run doom 3 well enough I suppose..
Yes dammit crap flooding message boards is a very grave and serious internatinal warcrime covered under the Geneva Convention. It is against human rights.
So damn, you are right. Our founding fathers like John Wayne and Gary Cooper would not have stood for it. Damn. We need to get a pre-emptive strike in against rogue states like Europe planning to flood message boards because if we don't it could give the terrorists an advantage.
damn
Actually, the cluster began with dual G5 powermac towers (looking kinda strange in racks). They later changed out to the current off the shelf G5 xserves with ECC memory. You probably mis-remembered the reason for the changes as G4 vs G5 rather than non-ECC vs ECC memory.
I think there was an intermediate update to non-ECC G5 xserves, but I'm not sure about that and I'm too lazy to look it up.
I will chip in 1 slightly used 3 pronged US power cable, black.
1 IBM is 2 IBM's are Of course, I sometimes are wrong mod me up scotty
The thing that most people don't understand, especially when they talk about the parts of a high end cluster being expensive, is the cost of the alternative. I admin a cluster which comes in at .9 Tflops, and cost ~3/4 of a mil. There's a X1 which comes in considerably lower, and cost considerably more. So yes, cluster is still commodity when compared to CRAY or SGI, even if you buy high end components.
Anyway, the Feds had a bunch of export control laws to prevent Commies from getting Big Computers that they could use to design better Nukes, as well as laws to prevent Commmies (and Americans) from getting crypto.
Fundamentally, computers do keep getting faster, but they've been Fast Enough for Government Work for a long time - Moore's Law says that just about anything you can buy at WalMart is faster than a high-end niche-market governments-and-big-corporations computer of a decade ago. The ideal computer for terrorism isn't some supercomputer - it's three pounds of wetware that's really pissed off. The way to deal with it is not to clamp down on exports of technology or circle the wagons into a laager to keep the enemy hordes out or declare anybody with interesting oil reserves to be a Rogue State - it's to stop acting like an Evil Empire and pissing people off by supporting oppressive governments around the world.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks