Slashdot Mirror
Debian Cluster Replaces Supercomputer For Weather Forecasting
wazza brings us a story about the Philippine government's weather service (PAGASA), which has recently used an eight-PC Debian cluster to replace an SGI supercomputer. The system processes data from local sources and the Global Telecommunication System, and it has reduced monthly operational costs by a factor of 20. Quoting:
"'We tried several Linux flavours, including Red Hat, Mandrake, Fedora etc,' said Alan Pineda, head of ICT and flood forecasting at PAGASA. 'It doesn't make a dent in our budget; it's very negligible.' Pineda said PAGASA also wanted to implement a system which is very scalable. All of the equipment used for PICWIN's data gathering comes off-the-shelf, including laptops and mobile phones to transmit weather data such as temperature, humidity, rainfall, cloud formation and atmospheric pressure from field stations via SMS into PAGASA's central database."
that would be HURD
Snowden and Manning are heroes.
it's a cluster, in other words a herd of computers :-)
i would suppose that debian is quite a versatile distro for any purpose...
-- from a debian user... who actually started quite late with potato....
Actually, Debian is intended for servers and runs on more architectures than any other distro. The whole reason for the long testing cycle on Debian is to make sure it's as stable as possible so it can sit on a server and need little or no attention for days, weeks, or even months at a time.
I hated Debian at first because it wasn't friendly, but I looked into it more and realized it was the best choice I could make for my production servers. I can set them up and check once a week or so and they're still chugging along without need of intervention.
I wouldn't use Debian on my desktop (I use Kubuntu), but it can't be beat for servers.
It's NOT a desktop distro. Especially compared to Mandr* or Ubuntu or many others out there.
Debian, like most distros, is what you want it to be. Debian is used regularly as a server OS.
I doubt they have X installed on these machines.
I won't join Slashcott. OTOH, If Beta goes live, I just won't be back until it's fixed. Sorry Dice.
Sure. Add in paying for tech support or the cost in man-hours it takes to keep it running. Both can make a serious dent where nobody expected to see one.
How different can Debian really be compared to RedHat in terms of stability? They both use the Linux kernel, and GNU tools, and follow the LSB, no?
"Thanks for all the money you paid to us. We've used it to buy off ISO among other things" -Microsoft
I'm sure that if they can manage an ancient SGI supercomputer, they can easily manage Debian. I've been using it since woody, and I must say, compared to many other distros, Debian is easy to manage. Not only that, its reliability is second to none on the Linux platform. I have a machine that's been running the same Debian install since the days of woody, all up-to-date with Etch. Not a single problem with it, runs a lot better than an XP desktop I have, which has needed 2 reinstalls in the past year, or Gentoo, which frequently breaks when packages fail to compile.
Actually I don't like Debian much as a desktop machine, but I love it as a number cruncher OS. I've had a 10 machine openmosix cluster going for several years now, problem free.
Stability is a major thing with Debian, and my experience has been that this is quite true.
What was the age and the specs of the SGI being replaced?
Going by Moore's law, a factor of 20 performance improvement takes about 6 to 8 years. If the SGI was at least that old, this isn't news -- it's just the state of the art these days. In other words, small clusters capable of weather forcasting are relatively run-of-the-mill.
Of course, props to linux for being the enabler in this case.
Most weather prediction centers have adapted their weather forecast models to use Linux clusters. By running an operational forecast model on a cluster, it's easy for forecasters to scale the models so that they can be run (albeit slowly) on desktop machines, and are easily worked on by real meteorologists (versus IT professionals). At my university, we use a large cluster of machines on a RedHat enterprise system, and then able to scale the models and run them on multiple processors using MPICH compilers and batch jobs. Really, using a Debian cluster is no different then using a RedHat cluster. My colleague has access to the NOAA machine, which has more processors then you can shake a stick at... he talks about some code that takes 3 days to run on his personal workstation that takes 2 minutes on 40 processors. With the relatively low cost of a linux cluster, weather forecasting models can be run quickly and efficiently on numerous processors at a local level. With the ease of use of a Linux machine versus some of the supercomputers, it puts the power in the meteorologists to make those changes to the model so that it can improve forecasts.
Debian works out just fine for these kind of tasks. Here in the Netherlands the national compute cluster Lisa runs on Debian (http://www.sara.nl/userinfo/lisa/description/index.html) with 800~ to a 1000 nodes (I think the page needs updating by now).
All I can say is that I enjoy running Debian servers and RHEL clients at my work... and you're a douche...
oh, wait...
Do not mock my vision of impractical footwear
More importantly, is it Vista capable?
Debian is sure not a desktop distribution. Ubuntu would be one. I run Debian on all my servers.
"Freiheit ist immer auch die Freiheit des Andersdenkenden" - Rosa Luxemburg, 1871 - 1919
I tried: /lib/libearthquake-2.3.so.0 is not a symbolic link"
apt-get -f -y install gweather
But it failed with something about "ldconfig:
Is libearthquake in unstable?
Interactive Visual Medical Dictionary
The binary package management really says it all.. you shouldn't be running anything but compiled source on a performance cluster.
/bin/ls though? I don't think it matters to anyone on a high performance cluster. Just so long as the cluster apps are optimised then the rest is just noise - better to have a system that's less work for your administrators so they can concentrate on what's important.
Wow - how many performance clusters do you run again?
Not that I run a "performance cluster" as such - but I do run a bunch of machines that are very busy, all on Debian.
You know what? We compile the couple of programs where CPU is the bottleneck from source. We also compile Cyrus IMAP from source because we apply a pile of patches, but if someone else was packaging up all those patches in upstream, I'd be happy for them to be compiled there. Disk IO is the issue with Cyrus, and a custom compile won't help with that.
Yeah, we build our own kernels as well - that's another point that's worth the effort to customise.
Not sure why you call Debian a desktop distro, it's much more useful as a server.
You obviously doesn't have a clue about Debian. Debian has been a fine server since 1995, and I still choose it before RHEL anyday. I have always found it strange that everyone went for RedHat, when they could have Debian. Mark S. saw the advantages of Debian and based Ubuntu on it, Ubuntu is a server and a desktop distro, based on Debian. It has made more people realize the strength of the Debian approach.
Exactly, My job is running high performance computing clusters. You don't need to put much effort into your cluster distribution at all. As long as it's stable, and gets the job done, why mess with it.
/bin/ls don't matter a bit.
The things I (and my co-workers) put a lot of optimization effort into is the kernel and our apps. You're exactly right.. 99.9% of our CPU cycles go into getting work done, and that 0.1% used by
I don't get why every one is saying it's not for desktop. It runs great on my T60. Everything works fine: sleep, mouse, red nub, wireless, sound, screen brightness, Blue-tooth. I guess I have never used the media buttons...
Money is the root of all evil?
It is difficult to get a man to understand something when his job depends on not understanding it.
They never said it can't be used for desktop. They are just saying that it isn't primarily aiming for desktop use and works very well on servers, debating the original posters claim.
Perhaps they mean it is scalable in the sense that one could simply add more machines to the cluster, rather than adding more cores to the machines already in place.
MacBook Pro. Worst name since the Bicycle
From TFA What was even surprising to us is that Intel FORTRAN is also free of charge ...
I bet Intel are surprised too. Their compilers are not that free of charge. The people at the Philippine government's official weather service are hardly "not getting compensated in any form"
http://www.intel.com/cd/software/products/asmo-na/eng/219771.htm
I guess he isn't referring to licensing costs for the various distributions not putting dents in his budget but rather the operating costs of their new cluster as a whole. The SGI machine seems to have dented the budget pretty badly while replacing it with a Linux (beowulf?) cluster polished those dents out to the extent of them being able to test various setups in addition to their cluster and still have a smoother budget.
Also, their supercomputer may just be outdated, not necessarily because of bloated software. I don't know how well SGI's products and support survived their recent bankrupcy, but I'd imagine not too well (though they seem to have built the Xeon-based #3 from the Top 500 recently).
Debian isn't - and never has been - a desktop distro. If you want a desktop distro built on Debian architecture, you get Ubuntu, or Knoppix, or one of a dozen others. Debian's unique selling proposition is a combination of stability, which is very important to production servers, and a rigorous commitment to free software. Packages don't make it into Debian Stable until they have been thoroughly tested. Debian also has the best package management system in the industry.
Frankly, I wouldn't run a server with anything else.
I'm old enough to remember when discussions on Slashdot were well informed.
Hmmm... 0 to Gentoo/Slack fanboyism in like... 4 comments.
I know people who know a fair amount about running clusters. None of them want the headache of dealing with the random-ass unexpected conflicts that arise out of having the explosion of possibilities for custom compiling for each server. Also, nobody wants to use their precious "performance cluster" cycles compiling every update. If you really need to compile tweaks (for the important stuff only), you do it offline, once, and then build a *binary* package to distribute to your nodes.
I never said it doesn't work on a desktop. I use it here on mine right now. It's just not as easy to setup as Ubunutu I think.
"Freiheit ist immer auch die Freiheit des Andersdenkenden" - Rosa Luxemburg, 1871 - 1919
Nope, it's one big, happy family :)
Every time I want to install Ubuntu on some random machine, it fails. I always have to go back to Debian.
I have Debian currently installed for my father and my sister. Spares me the headaches of Windows problems. The only support I need to deliver to them is giving information about performing tasks.
since it is a cluster that would be correct, a herd of hurd...
Politics is Treachery, Religion is Brainwashing
That's the big attraction for Debian. For a production system, support tasks drop to almost nothing. It's there. It runs. If and when a patch is needed, it is just that - a patch - and not any weird licensing changes or mutations in functionality.
Of the linux distros, it's an excellent choice for servers, perhaps the best. Given the rock-solid nature, it can be good for enterprise desktops, if you are willing to plan. However, Kubuntu LTS meets that need.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
How fortunate that apt/dpkg handles source packages so well then... Punching in 'apt-get -b source <whatever>' is not a whole lot harder than 'port install <whatever>' or whatever you prefer, is it? I know, I know... Don't feed the troll... Sorry.
"Live free or don't."
You customise absolutely everything yourself in Debain too, but when you have more than 50 servers running, thats not what you want. In that situation you want something dependable, stable that let you do other things. In Debian you don't ever have to reinstall, you just upgrade. I started out with Slackware, if I remember right, but since I started using Debian in 1995, I have not seen a better server,and during that time I have used quite a few. No one has served me better than Debian.
Give the guy a break. He just wanted to see Gentoo compile in less than 2 weeks; he had a cluster handy...
Censorship is telling a man he can't have a steak just because a baby can't chew it. --Mark Twain
While Ubuntu might be friendlier, and be more polished, I'd like to say that Debian is perfectly workable as a desktop os. (Started with a base install that didn't even appear to come with less, moved onto fluxbox when I wanted a gui, moved back to kde because I missed it).
It just takes a little more effort if you do something pointless like start out with just the min install.
Um, your unfamiliarity with Debian is painfully showing. apt-get update doesn't destroy servers. apt-get upgrade might, if you're running testing or unstable. I'd recommend you use neither for production servers, and stick with stable.
And consistency? Like how the entire Debian repository is cross checked every day to ensure consistency?
I'm also intrigued by your reference to updates destroying servers. Do you get this behaviour with Red Hat? Makes me glad I'm not using it, then.
And managing an arbitrary number of nodes under Debian is easy with clusterssh.
No tyrant thrives when every subject says no.
Max.
I agree. I used to run some clusters for the UCLA Chemistry department and the only real customizations we did was to install a custom kernel in Redhat 9 to handle the huge amount of memory we had installed. And even that wasn't in all the clusters. But yeah, the code the clusters was actually running was pretty much always compiled by hand.
the code the clusters was actually running was pretty much always compiled by hand.
You really oughta use a compiler for that.
Wow, sent an e-mail as suggested when clicking on "use classic" banner, and got a fast response that addressed my msg