We built and mantain a Beowulf with about 70 nodes. We use Debian GNU/Linux.
I agree with you, in principle, it *is* easy to do but the problems increase with the number of nodes. IMHO, the main problems are:
-Administration effort per node has to be almost zero. Beyond a number of nodes you definitely need things like fully automatic instalation, automatic power control, automatic diagnostic tools, a batch system, etc. All these tools already exist but you need some know-how to put all them together.
-You need a large enough room with a cooling system that gives at least 100 W per node, 7kW in our case. Room temperature has to be about 20oC.
-Low cost PC hardware is not allways reliable enough for this application. If you have codes that run 24x7 for months in a large number of processors, the probability to have a hardware problem is very high.
We have found that our hardware suppliers do not carry out extensive tests on the systems they sell. This is because "normal" users run low quality OSs and they assume that it is normal that the computers just hang from time to time. Therefore, they do not allways detect failures in critical components such as RAM.
-Of course, your application has to be suitable for parallel computing, specially if your cluster uses a low cost 100Mb/s network. In this case, compared to a "conventional" parallel computer (eg Cray T3E), the processors are roughly equivalent but the network is about 10 times slower and is easily the bottleneck of the system.
Having said that, despite all the problems, I love Beowulfs. They have totally changed high performance computing, and they are definitely here to stay.
All this has been possible thanks to free software, so thanks Mr. Stallman/Torvalds and many others...
For an outsider like me, it seems that the risk of ending with a totally broken camera is high. How often does it happen ?
I've browsed the pages and they dont seem to mention digital cameras. Is it because they are too expensive to risk them ? (my aim is NOT to reopen the passionate discussion about digital/conventional cameras !)
>But note that they are at least using SMP (2-way Xeon) nodes: that probably helps if their algorithms are smart.
Usually, parallel algorithms assume that each processor is equidistant (in terms of network) to all the rest. In some cases, yes it may be possible to take advantadge of having them in groups of two or four.. but in general this is difficult.
>When you have an 8-way SGI, you have 2, 4-cpu nodes with their own local memories
Yes but the problem is that the complexity of the bus increases with the square of the number of nodes, so you cannot scale that to any number of processors even if you are very rich. This is why really large computers use distributed memory. Well, maybe on each box there is more than one processor but overall they are distributed memory systems.
>I know there are starting to be NUMA Linux machine (...) so any hop from one machine to the next will have enormously more latency
NUMA is indeed a promising technology. However, as far as I know, it can in principle be implemented on top of any networking device, even conventional 100 Mbits/s stuff, so it won't solve the latency problem without special hardware. The main advantadge will be to be able to use shared memory programming model that is accepted to be easier than distributed memory (ie MPI)
>So yes, maybe bandwidth will catch up, but I don't see how latency really can
I agree with you. This is a crucial aspect for the future of low cost parallel computers. The problem as I see it is that high bandwith is important for many internet applications such as video. But not low lateny as for a human poit of view there is no difference between 10^-3 or 10^-5 seconds. The only application that I know where low latency is important is scientific computing. Therefore, the cost of low latency communications will remain high.
Please correct/complement my point of view.
>Or maybe you were just talking about really old supercomputer technology
The first parallel computer I programmed was a SGI. I'm 34:)
The distributed memory Crays (T3D, T3E) are just the same: boards and network cards. The processors they use are not faster than the last generation PC processors. The difference are the NICs, that have about 10 times more bandwith and 10 times less latency (compared with standard fast ethernet cards).
There is the difference. As you say, for certain problems, this means that the whole machine is about 10 times faster than a Beowulf.
However, if/when conventional NICs are fast enough, specially in terms of latency, both systems can be equivalent again. In the meantime, a lot of people are trying to develop parallel algorithms that minimize the number and size of the messages, allowing to use cheap PCs as supercomputers.
You are right, the network AND the algorithms are the the key in many cases, not just FLOPS.
A good example is solving large linear equation systems, with say 10^7 unkowns or more. This is a central problem in many fields of scientific computing. In our CFD simulations we need to solve 10^6 linear systems with 3x10^6 unkowns each to obtain the final answer.
It is difficult to use large number of processors to do it efficiently, specially if you use a conventional 100 Mbits/s network with high latency. Currently we are using 36 processors to do so, and the solution of each system takes about 4 seconds. Just multiply to have an idea of the total processing time !
But without Beowulf clusters (and GNU/Linux is a central part of them) this kind of problems would requiere conventional, very expensive supercomputers.
Congratulations to Juan Luis Arsuaga and coworkers for their excelent work and also for this book.
For a number of reasons it is hard to do good science in Spain. In short, funding is scarce and it is almost impossible to get a decent contract in a University or research institution. The personal situation of many Ph.D and post-doc students is difficult. It is a truly vocational job.
Therefore, their success is a little bit the success of many others. Again, congratulations !
I did exactly the same installation but just for fun. I never upgraded the RAM so the system works terribly slow... but runs and it is nice to see Debian on it. It is a 386 SX 16.
My pentium MMX 166 laptop with 32 Mb RAM runs woody with X windows. The only trick is NOT use KDE or gnome but a small window manager. There are several to choose in the distribution. I have used the machine for presentations with acroread, playing games with xmame, music playing with xmms, even parallel code development with xemacs, gcc, and MPI (LAM).
As far as I know, Debian is the best distribution for small systems. I think it would have been perfectly possible to install it in the thinkpad described in the paper.
Well, it would be more exact I would need 6Gbytes, but I have to do with 4x0.5=2Gbytes + swap on dual K7 ASUS boards. This is why I was interested in the swap RAM disk.
But, if I understood your comments, I cannot go beyond 4GB per process using Intel: It doesn't matter if it's RAM or swap.
In Spanish Linex sounds good. They want to
emphasize the importantance of Lin-Ex as a "new"
thing developed by them... and win the regional elections again.
I'm in favour of switching from MS to Linux but
in my opiniosn the main emphasis of Linex is political propaganda of Mr. Ibarra, the president of Extremadura state.
First, I'm writing this with Debian 3.0 and I think that the overall idea of replacing MS products with open source alternatives is great.
But, why was it necessary to develop a new distribution, based on Debian, call it Lin-Ex (for Extremadura), and spend 180.000 $ in it ?
Is it that expensive to pack a number of.deb files on a CD ? What will be the cost of maintaining Linex ? Should every body else in the world create their own distributions ? What's the point of that ?
I suspect a politic intention behind Linex.
To me, it would have been much better to use a standard distro in spanish and spend the money in something else such as formation or support.
Slashdot Mirror
We built and mantain a Beowulf with about 70 nodes. We use Debian GNU/Linux.
I agree with you, in principle, it *is* easy to do but the problems increase with the number of nodes. IMHO, the main problems are:
-Administration effort per node has to be almost zero. Beyond a number of nodes you definitely need things like fully automatic instalation, automatic power control, automatic diagnostic tools, a batch system, etc. All these tools already exist but you need some know-how to put all them together.
-You need a large enough room with a cooling system that gives at least 100 W per node, 7kW in our case. Room temperature has to be about 20oC.
-Low cost PC hardware is not allways reliable enough for this application. If you have codes that run 24x7 for months in a large number of processors, the probability to have a hardware problem is very high.
We have found that our hardware suppliers do not carry out extensive tests on the systems they sell. This is because "normal" users run low quality OSs and they assume that it is normal that the computers just hang from time to time. Therefore, they do not allways detect failures in critical components such as RAM.
-Of course, your application has to be suitable for parallel computing, specially if your cluster uses a low cost 100Mb/s network. In this case, compared to a "conventional" parallel computer (eg Cray T3E), the processors are roughly equivalent but the network is about 10 times slower and is easily the bottleneck of the system.
Having said that, despite all the problems, I love Beowulfs. They have totally changed high performance computing, and they are definitely here to stay.
All this has been possible thanks to free software, so thanks Mr. Stallman/Torvalds and many others...
One of the pictures is today's "Astronomy Picture of the Day":
http://antwrp.gsfc.nasa.gov/apod/ap021114.html
For another surprinsing view of the sun, but very different, take a look (in the same site) at:
http://antwrp.gsfc.nasa.gov/apod/ap021110.html
I
For an outsider like me, it seems that the risk of ending with a totally broken camera is high. How often does it happen ?
I've browsed the pages and they dont seem to mention digital cameras. Is it because they are too expensive to risk them ? (my aim is NOT to reopen the passionate discussion about digital/conventional cameras !)
>But note that they are at least using SMP (2-way Xeon) nodes: that probably helps if their algorithms are smart.
:)
Usually, parallel algorithms assume that each processor is equidistant (in terms of network) to all the rest. In some cases, yes it may be possible to take advantadge of having them in groups of two or four.. but in general this is difficult.
>When you have an 8-way SGI, you have 2, 4-cpu nodes with their own local memories
Yes but the problem is that the complexity of the bus increases with the square of the number of nodes, so you cannot scale that to any number of processors even if you are very rich. This is why really large computers use distributed memory. Well, maybe on each box there is more than one processor but overall they are distributed memory systems.
>I know there are starting to be NUMA Linux machine (...) so any hop from one machine to the next will have enormously more latency
NUMA is indeed a promising technology. However, as far as I know, it can in principle be implemented on top of any networking device, even conventional 100 Mbits/s stuff, so it won't solve the latency problem without special hardware. The main advantadge will be to be able to use shared memory programming model that is accepted to be easier than distributed memory (ie MPI)
>So yes, maybe bandwidth will catch up, but I don't see how latency really can
I agree with you. This is a crucial aspect for the future of low cost parallel computers. The problem as I see it is that high bandwith is important for many internet applications such as video. But not low lateny as for a human poit of view there is no difference between 10^-3 or 10^-5 seconds. The only application that I know where low latency is important is scientific computing. Therefore, the cost of low latency communications will remain high.
Please correct/complement my point of view.
>Or maybe you were just talking about really old supercomputer technology
The first parallel computer I programmed was a SGI. I'm 34
The distributed memory Crays (T3D, T3E) are just the same: boards and network cards. The processors they use are not faster than the last generation PC processors. The difference are the NICs, that have about 10 times more bandwith and 10 times less latency (compared with standard fast ethernet cards).
There is the difference. As you say, for certain problems, this means that the whole machine is about 10 times faster than a Beowulf.
However, if/when conventional NICs are fast enough, specially in terms of latency, both systems can be equivalent again. In the meantime, a lot of people are trying to develop parallel algorithms that minimize the number and size of the messages, allowing to use cheap PCs as supercomputers.
You are right, the network AND the algorithms are the the key in many cases, not just FLOPS.
A good example is solving large linear equation systems, with say 10^7 unkowns or more. This is a central problem in many fields of scientific computing. In our CFD simulations we need to solve 10^6 linear systems with 3x10^6 unkowns each to obtain the final answer.
It is difficult to use large number of processors to do it efficiently, specially if you use a conventional 100 Mbits/s network with high latency. Currently we are using 36 processors to do so, and the solution of each system takes about 4 seconds. Just multiply to have an idea of the total processing time !
But without Beowulf clusters (and GNU/Linux is a central part of them) this kind of problems would requiere conventional, very expensive supercomputers.
Congratulations to Juan Luis Arsuaga and coworkers for their excelent work and also for this book.
For a number of reasons it is hard to do good science in Spain. In short, funding is scarce and it is almost impossible to get a decent contract in a University or research institution. The personal situation of many Ph.D and post-doc students is difficult. It is a truly vocational job.
Therefore, their success is a little bit the success of many others. Again, congratulations !
I did exactly the same installation but just for fun. I never upgraded the RAM so the system works terribly slow... but runs and it is nice to see Debian on it. It is a 386 SX 16.
My pentium MMX 166 laptop with 32 Mb RAM runs woody with X windows. The only trick is NOT use KDE or gnome but a small window manager. There are several to choose in the distribution. I have used the machine for presentations with acroread, playing games with xmame, music playing with xmms, even parallel code development with xemacs, gcc, and MPI (LAM).
As far as I know, Debian is the best distribution for small systems. I think it would have been perfectly possible to install it in the thinkpad described in the paper.
Well, it would be more exact I would need 6Gbytes, but I have to do with 4x0.5=2Gbytes + swap on dual K7 ASUS boards. This is why I was interested in the swap RAM disk.
But, if I understood your comments, I cannot go beyond 4GB per process using Intel: It doesn't matter if it's RAM or swap.
Is there a PC motherboard that supports 64 GB ?
I don't think the idea of using it as swap area is nonsense. What if you need 6 Gbytes of RAM (as I do) ?
As far as I know, normal motherboards don't suport more than 3 Gbytes, even using 1 Gbyte modules that are very expensive.
For me the main question (apart of price) is if this device runs in Linux.
In Spanish Linex sounds good. They want to emphasize the importantance of Lin-Ex as a "new" thing developed by them ... and win the regional elections again.
I'm in favour of switching from MS to Linux but
in my opiniosn the main emphasis of Linex is political propaganda of Mr. Ibarra, the president of Extremadura state.
First, I'm writing this with Debian 3.0 and I think that the overall idea of replacing MS products with open source alternatives is great. But, why was it necessary to develop a new distribution, based on Debian, call it Lin-Ex (for Extremadura), and spend 180.000 $ in it ? Is it that expensive to pack a number of .deb files on a CD ? What will be the cost of maintaining Linex ? Should every body else in the world create their own distributions ? What's the point of that ?
I suspect a politic intention behind Linex.
To me, it would have been much better to use a standard distro in spanish and spend the money in something else such as formation or support.