Slashdot Mirror
23 Second Kernel Compiles
b-side.org writes "As a fine testament to how quickly linux is absorbing technology formerly available only to the computing elite, an LKML member posted a
23 second kernel compile time to the list this morning as a result of building a 16-way NUMA cluster. The NUMA technology comes gifted from IBM and SGI. Just one year ago, a
Sequent NUMA-Q would have cost you about USD $100,000. These days, you can probably build a 16-way Xeon (4X 4-way SMP) system off of ebay for two grand, and the NUMA comes free of charge!"
2 grand? i think not :(
ok..I'm NOT about to start the perverbial deluge of people wanting to know about a beowulf cluster of these things. But what I will ask is this: if it can do that for a kernel, I wonder how long it will take to do Mozilla, or XFree? It'd be interesting to see those stats.
JoeLinux
Is that "wow, sweet" because of that sexy kernel compile time or because you got first post?
Someone had to ask.
No way. Just a no-CPU, no-memory case and
motherboard costs $500. More like $2000
to $3000 for an old quad.
23 seconds is impressive. I, personally, have seen a 42 second compile time of a 2.2 series kernel on a Intel 8-way system (8GB ram, 8 550Mhz PIII Xeons w/ 1mb L2). It was in the 1 minute range with a 2.4 kernel.
Definately the most impressive x86 system I have ever seen.
Ok, it compiles a kernel hella fast. But can it be applied to other stuff. Obviously these fast compiling machines are best for places where you have tons of users compiling all over the place, but will other compilers be as fast?
For example, just a few years ago my main computer was a Intel 486 running at 100 Mhz. Just think how long it would take to compile the kernel on that system. From that I went to a Pentium 90, then a Pentium 233, a Celeron 533, a Pentium III 533, a Pentium III 800, and finally a Pentium III 933...
A good portion of my life is lived, if you'll allow me the word, on-line. It might even be said that I live a double life, part of it with family, friends, and co-workers in the immediate, material world, and another part with a different (although sometimes overlapping) circle of family, friends, and co-workers on the net. Not only am I connected to other people, but via the Internet I am also connected to a collection of tools and resources that help me in my day-to-day life. For example, I use hypertext as a thinking tool for producing and developing my ideas; I use the World Wide Web as a canvas for mapping and presenting myself and my work; and I use email not only to keep in touch, but also to manage my time and organize my projects.
I will not argue: our culture is indeed defined by our formulations of foundational concepts such as religion, family, and truth. But, with time, meaning changes. The defining aspects of a culture are not mystically immutable. On the contrary, they experience drastic changes, causing jihads and holocausts, but also rebirths and renaissances. At this turn of the century it is clear that computing and related technology is pivotal to our culture's progression, for better or for worse. And my response to the "neo-Luddites" is to ask them to pay attention to the "or." There is no evidence or sound argument that our redefinition of basic cultural terms spells out doom. Changing, evolving society-wide conceptions are a routine part of any culture's movement through time, no matter how "unnatural" the cause of the evolution might appear to be. Just as it is too soon to be heedless and inattentive to the consequences of computing and the Internet, it is also too soon to be so afraid.
In the end, the arguments of the "neo-Luddites" do not withstand deeper inquiry. Neil Postman, Clifford Stoll, and Stephen Talbott repeatedly attribute to the Internet qualities which actually describe our use of, or attitude toward, the Internet, not the Internet itself. It is clear that the technology is neither intrinsically good nor evil: it is neutral. We can have meaningful experiences with it, or waste our time. We can be lost in its flashy colors and informational floods, or we can take control. We can lose our identities, or hold on to them. There is no goodness or badness intrinsic to the Internet any more than there is goodness or badness intrinsic to a block of marble. The marble may tend toward cleaving one way rather than another, as a medium is inclined to, but its nature -- its feeling, its attitude, its content -- is left to us, not to it. With this awareness we can shape the on-line future to our best ideals. "It's all a question of what we do with it," Talbott wisely observes of the Internet. And despite the fact we differ on many points, on this point we agree. He writes:
...who wondered, "I didn't know that Clive Cussler had gotten into cluster design?
Remember, this was in 1996. Now, how much did we progress in the last five-six years? :)
"Ten years from now, they could do it in a few seconds." -- The Racketeer of the Hellfire Club, 1993, Phrack 42
Maybe this is a silly question.. but why would you want to compile a kernel in 23 seconds? I mean, ok, it's cool and everything, but is there some hidden application of this that I'm not seeing? Or are people really devoting hardcore time to this just because they can?
6 years ago, a kernel compile for me took about 3 hours. These days, it takes less than 3 minutes, which is more than fast enough for my needs. So, you can push it down to 23 seconds with a few thousand $ - what's the point? Someone help me out here!
Mod this one +1, funny. I havn't laughed so hard in ages.
But, does anyone know how NUMA compares with, say, a beowulf cluster? Does NUMA allow you to 'bind' multiple systems into one, so that I wouldn't need to rewrite my software? Did these guys use a stock GCC or something special? I know you would need to use MPI or similar for beowulf. Is NUMA as scalable as Beowulf in terms of building huge-ass machines (of course if I was going to expend the effort to do that, I might as well want to write custom software).
If this type of system would allow 'supercomputer' performance on regular programs... well... that would be really nice. How much work is it to setup?
autopr0n is like, down and stuff.
You can also get 23-second kernel compiles in software using Compilercache :-).
-- Ed Avis ed@membled.com
This may be good news, but what the heck! They should have at least included the .config that they used so that we can know what drivers/modules that are compiled with it, or maybe this is just bare-bones kernel enough to run the basic. We need to know the complexity of the configuration before we could really say it's fast.
Take-off every
But where can I get a NUMA cluster for $80? Should I Ask Slashdot?
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
That's on a "16 way NUMA-Q, 700MHz P3's, 4Gb RAM".
I've been following that thread wondering if anybody would post better results with a dual Athlon or similar. Any lucky soals with really cool hardware who want to post benchmarks? In fact, it would be interesting to know how quickly the kernel compiles on single P3/700, just to get an idea of how it scales.
It is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail. - Abraham Maslow
I'll tell you what annoys me about building a kernel...
The damn NETLINK option isn't ticked by default. The amount of times I've forgotten to choose it, and when I boot the new kernel, it moans about not understanding IPv4.
WTF.
Change the default option.
I don't get it! What's supposed to be widening? It wraps nicely, with spaces before the periods. PLEASE EXPLAIN!!!
So if I could compile a new kernel in less time than it takes to boot-up, then a new kernel would be ready before the boot process was finished. So I'd have to restart with the new kernel, and if I start a new kernel compile too then that boot wouldn't be able to finish before there was another new kernel, so I restart with the new new kernel and begin another compile...
Maybe naming this box 'Zeno' wasn't such a good idea after all.
(PS. You can now compile a kernel faster than Nautilus opens a folder. Go fig.)
Any sufficiently advanced civilization is indistinguishable from Gods.
... just so long as I never have to program on Sequent iron, and that it's insidious operating system ever again. Of course, that was 10 years ago when Dynix, trying to be the best of both worlds, was really neither ATT nor Berkly !
... hence, making it a REAL-world bargain.
... of coure the other problem was indeed the expense, leaving us in situations where we had to program at odd hours and off-days because the client couldn't affort a "development" machine.
... two issues which I would hope are sovled a 16-way Xeon for $2K
healyourchurchwebsite.com - WWJB?
You can't build a NUMA cluster worth a crap without a fast, low-latency interconnect.
Sequent's NUMA Boxen use a flavor of SCI (Scalable Coherent Interface) which is integrated into the memory controller.
While you can use some sort of PCI-based interconnect, the results are just plain not worth it.
Infiniband should be better, though I've heared the latency is too high to make this a marketable solution.
Keep your eyes on IBM's Summit chipset based systems. These are quads tied together with a "scalability port" and go up to 16-way. They should go to 32 or higher by 2003. That's when NUMA will -finally- be inevitable...
Unless it's two years old I don't believe the price of that cluster is $2k. The cheapest quad-xeon motherboard on pricewatch is $500. If you cut that in half for being used, that's still $1k for just the motherboards. Add 16 processors, ram, cases, NICs, drives, power supplies, and other parts, and there's no WAY you're coming in under $5k, and $10k would be more realistic.
On the other hand, if someone IS selling such a beast and I can win the bidding with a $2k bid, I might be tempted...
... with the advent of this new technology and raw speed, you should actually be able to use them!
[this is actually a joke]
chris at darkrock dot co dot uk
http colon slash slash www dot darkrock dot co dot uk
I went and looked at the email and noticed that the very first patch he mentions was from the woman who came and gave a talk to EUGLUG last spring. For one of our Demo Days we emailed IBM and asked them if they would send down someone to talk about IBM's Linux effort. We were kind of worried that they would send a marketing type in a suit who would tell us all about how much money they were going to spend, etc., etc. But we were very pleasantly surprised when they sent down a hardcore engineer who had been with Sequent until they were swallowed by IBM.
She did a pretty broadranging overview of the linux projects currently in place at IBM, and then dived into the NUMA/Q stuff that she had been working on. The main gist of which is that Sequent had these 16-way fault-tolerant redundant servers that needed linux because the number of applications that ran on the native OS was small and getting smaller. Turned out that even the SMP code that was in the current tree at the time did not quite do it. She had some fairly hairy debugging stories, apparently sprinkling print statements through the code doesn't work too well when you're dealing with boot time on a multiprocessor system because it causes the kernel to serialize when in normal circumstances it wouldn't...
I think the end result of all this progress with multiprocessor systems is that we'll be able to go down to the hardware store and buy more nodes plug 'em into the bus; and compute away.
What was the point in this? Some cryptographical use? Whaaaat?
You might give Galeon a try. It uses the same rendering engine supplied with Mozilla (actually you must install Mozilla to run Galeon) but the interface is *much* faster and more elegant.
But the reserve for this machine is $3850. The article says 16 way, which would be four of these four-way SMP systems. That also doesn't take into account the need for a high-bandwidth, low latency interconnect (like SGI's NumaLink.) If you aren't expecting more than 16-way SMP, then you can probably get away with switched Gigabit Ethernet, as long as it is kept distinct from the nornal network connectivity. If the Gigabit upgrade is still dual portm then you are set. If not, you'll neet another NIC - though you will only really need one for the whole cluster.
Maybe instead of two grand, the poster meant twenty-grand. Either way, $20 grand is better than $100K!
You have 26 rows.
Russian Russian Russian RussianDollSig DollSig DollSig DollSig
... a beowulf cluster of these ?
sorry I couldn't resist
NUMA means Non-Uniform Memory Access. It is a kind of computer where you have shared memory but you dont have the same access time for every processor to every memory position. Therefore, every processor will have access to all memory but sometimes it will take longer or shorter (if the memory belongs to another processor).
In a Beowulf cluster, you dont have shared memory (unless inside a node, if you have a SMP machine) and you must use message passing to communicate (unless you are using DSM--Distributed Shared Memory--, maybe with SCI).
Having read the site for compilercache, I fail to understand why the parent is (by some) moderated as funny.
It may be my lack of understanding, but it seems rather wasteful to recompile everything when only a few files are changed. Same goes for changed comments.
Ok, a NUMA achitecture is nice to have for compiling, but it's probably a lot more useful for things that cannot be cached at all (rendering, simulations, etc. they've been mentioned already).
I would moderate the parent as interesting or informative....
OK so it can compile the kernal in 23 seconds but the real question is how long does it take to launch an app in kde?
"Madness is something rare in individuals - but in groups, parties, peoples, ages it is the rule." -- Nietzsche
Reason? Not enough information as to the options.
Never the less:
I WANT ONE
www.eFax.com are spammers
http://samba.org/~anton/e10000/maketime_24
Wheeeeeee!
And seriously, I saw some comments about needed a really fast interconnect... check out Sun's Wildcat.
--NBVB
Very nice. :)
my 386-dx40 with weitek coprocessor and 8M ram,
at 1.36 bogomips, will compile a 2.2 kernel in only 27 hours 13 minutes.
and if not, why not?
How well would Firewire, Fibre Channel, or SCA work as NUMA interconnects? How would these guys compare, pricewise and in effectiveness, to 1000baseT?
That's good, but compilation is awfully parallelizable: You could (almost) just assign a computer to compile each individual source file; the total time would be the time to compile the slowest file plus link time. You could accomplish this with a shell script and a network file system -- what's the benefit of doing it with a shared-memory system like NUMA?
i said ebay, not pricewatch. you can pick up (and i have) quad xeon mobo's for less than $150, and xeon 400's for around $15.
Who would have guessed that technology would progress? Most people alive this century. There wa at least one guy at the beginning of this century who thought that no more technical advances were possible, and the patent office would have to be closed. He was the exception. Even "Moore's (so called) Law" postulates that the transistors on a CPU would double in number every 18 months, which mostly explains your progression along the Intel product line. That is insightful?
You don't get the point at all. This is about NUMA technology escaping the proprietary systems of the past and becoming feasible outside of Government funded Nuclear Detonation Simulations and corporate data-mining. This is about Free Software enabling this once proprietary technology due to the generous donations by SGI and IBM, who want Linux to bring its buzz to fruition on their hardware and services.
Now who would have guessed that somebody could use this as an opportunity to talk about email, the web, and not just "jihads and holocausts, but also rebirths and renaissances", not to mention neo-luddites. This is the poorest mismoderation I've seen for a while, and I wish that AC who claimed to have mod points had actually had the balls to use them. This obvious display of crap-flooding doesent even end with a proper sentence.
And despite the fact we differ on many points, on this point we agree. He writes:
God, this is disgusting. What a fucking rediculous troll. If anybody w/mod points actually possesses an ounce of intelligence, maybe I wouldn't have to puke right now.
-castlan
It would be quite easy to configure the kernel build process for several machines to each make a .o file, and them send them to a master machine for a final link.
There are about 540 object files for (my) kernel build. Given say 20 Pentium Pro's, each would have to build 27 object files. That's not too bad. I don't have a pentium to test the speed on, or a cluster of 20 of them to do this, but it seems alot easier than NUMA.
Beowolf clusters are considered horizontal scaling, while NUMA clusters are considered vertical scaling. From my experience (SGI CC-NUMA) a NUMA cluster looks like a single computer system, with a single shared memory. (SGI systems are even Cache-Coherent, so that there is minimal performance loss if your data is in the RAM of the most distant CPU.. a significant issue with 256 nodes). This means that you don't have to deal with MPI or other systems to deal with disparate memory of seperate machines, so you can mostly code as if it were a single supercomputer. In fact, that is how SGI actually makes their supercomputers.
NUMA clusters tend to have scalability problems related to the cache coherence issue, so for a vertically scalable CC-NUMA box, you have to pay SGI the big bux. I haven't looked at IBMs NUMA technology, but if they own sequent, then they probably have similar capability.
As for the work to set one up, SGI's 3000 line is fairly trivial, the hardware is designed to handle it, and I think you only need NUMA link cables to scale beyond what fits inside a deskside case, if not a full height case. Now if you have a wall of these systems, you will need the NUMAlink (nee CrayLink) lovin'. As for an Intel based system, I suspect it wouldn't be nearly as easy... unless your vendor provides the setup for you. On your own, you would need to futz with cabling the systems together, just like in a beowolf. Except that your performance depends on finding a reasonably priced, high bandwidth, low-latency interconnect. Gigabit Ethernet wont scale very far, so going past 16 CPUs would be very unpleasant. If you expend the effort, you will end up with a cluster of machines that behave very much like a "supercomputer" though. Good luck!
Yes, and if the Kernel were written in Borland Object Pascal, you could get a 23 second compile on a Celeron 500.
I still remember my first kernel compile, it was way back in the early ninties, don't exactly remember the year. I had installed linux on my pride and joy 486slc/40 with 4 megabytes of ram. After reading how to do it I started at around 6pm, around 3am I grew bored at staring at the console and went to sleep.
When I awoke, it was finally done but I wasn't aware of how lilo worked at the time, so I just erased the old kernel and copied the new one into my / directory and rebooted. But the system never came back up. Frustrated I installed DOS back onto the system.
absorbing technology formerly available only to the computing elite but we are l33t!1111111 d0nt j0 c4ll m3 c0mpu3t3r3 l3tma4m3@!!11111 1 l33t!11111 Absorbing technology? Extend and embrace? :p
Do you like German cars?
haha great troll
B-E-R-K-E-L-E-Y
"bourgious"? ROFL...rhymes with "gorgeous"
Don't forget to add about $10,000 per quad for the custom interconnect, which is what really makes this machine work
Oh, and BTW, yes this hardware is about 2 years old.
Let me get this right... you buy a used computer, and then go straight to the manufacturer for replacement parts??? (Surely you know 'accessories' are one of their higher-margin profit centers!)
Still... if you're in the Seattle, WA area, stop in the Boeing Surplus Retail Store. I was there last week, and they had a bucket of what looked like 80-pin 2.1GB Compaq hot-plug drives. They were just sitting there next to a cash register like candy would be at a supermarket. I don't remember the price, but I want to say they wanted ~$5 each for 'em.
They were also selling an Indigo ($50), lots of PCs (mostly old Dell OptiPlex models, $20 - $300, Pentium MMXs to P-IIs), and even a Barco data-grade projector ($2500). Fun place to go and blow half a day poking around.
"...America's great minds of today, teaching America's great minds of tomorrow. Poor bastards." -- A Beautiful Min
The poster was thinking of "Borgia" and not the bourgeoisie ... obviously ;-P
it is in 2.5.
1. There exists a MOSIX implmented as a linux patch.
2. With MOSIX, the migration is largely a function of network. 100BaseT is low, but Ethernet over PCI, Infniband and friends can make this work well.
3. The MOSIX advantage over NUMA is that it is linearily scalable in the number of machines, NUMA can't go beyond a certain limit.
Gilad.
I have a dual athalon 1500+ (1.3 GHz), 512 RAM, and 2 uw scsi drives soft raid(0)ed. I issue the following command "time for c in dep clean dep clean dep clean bzImage; do make -j 4 $c; done" after I do a make distclean on a 2.4.18 and a make menuconfig for the defaults. Compile time, around 41 - 43 seconds.
-G.T.
You're having a laugh, right? My 18 month old dual 1 GHz P3 compiles the 2.2 kernel in about a minute. I would have thought that a dual Athlon MP 2000+ can easily match the 23 seconds. What is all the fuss about?
http://www.wulfkit.com/scibenchmarks/latency.html
http://www.wulfkit.com/scibenchmarks/latency.html These are used in the bewoulf community already. Almost any other solution has much higher latency. Ive never seen anything less than 7us and I think that was myrinet (similar price). Ethernet has a MUCH higher latency.
Well, a 23-second kernel compile is impressive and all, but the most important question I would have of such a machine is: How fast can it run Quake-3?
If it can do 1280 * 1024 * 32bpp at 300 frames/second, then I'm getting one.
Schwab
Editor, A1-AAA AmeriCaptions
You can buy the bits needed to build your own NUMA hardware system out of separate boxes relatively[1] cheaply. The speed depends on how you manage the memory and I/O. You'd need Linux to support it as a coherent whole though and I'm not sure that it does.
[1] For large values of relatively.
Government of the people, by corporate executives, for corporate profits.
But that's still just the mb and cpus. As the other guy who replied to me pointed out, you also need networking hardware to connect the machines, and the result is still 2 years old. $2k is not realistic.
Ooh, for 5 years or so.
They've since been bought by EMC and closed down but they had it working *and* scaling to 32 CPUs and on the market. 64 CPU systems were well on the way but I don't recall if they finished them.
Government of the people, by corporate executives, for corporate profits.
Female Prison Rape in NY
While I may not recall my first kernel compile (late 96 or early 97), I wanted to share a small kernel compile trick. When compiling on my P75's, I usually
:)
nice -n -20 screen
make dep clean bzImage modules modules_install ; echo "Kernel done" | Mail myname@skytel.com
I started doing that in '98 when compiling on a 486 I commandeered in the server room. Saves you from doing "are we there yet?" with your box. Also lets me know when I need to head home from running my errands or eat a little faster at the dinner table.
Intelligent Life on Earth
1. 2.4.18, and I also told you what patches I was using (though some of them won't be published until next week).
2. OK, I just posted the config file. http://lse.sourceforge.net/numa/config.mem
3. I did five kernel compiles in a row (though I omitted to mention that).
Hi Martin!
--
Daniel
Life's a bitch but somebody's gotta do it.
In fact, somebody please go and mod up all oxfletch's posts on this article, he's Martin Bligh, the guy who did this.
Life's a bitch but somebody's gotta do it.
This is definitely out of line with the intended benchmark, and perhaps I don't know better, but I am just curious... With that amount of RAM (4Gig), maybe they can setup to do an "in memory" build, and cut the build time by half ?!
2 damn funny!
iiiiiiii cnt ss-stop laugn... ha ahaaaa!
You pointed this out in another thread.
While it's true that these cards are mighty useful for Beowulf type clustering, nobody's gonna built a NUMA machine out of them anytime soon. Even if these cards are 64bit 66Mhz PCI, that's not a great deal of bandwidth. Without a lot of bandwidth, you have to really hack the OS to keep memory accesses as local as possible. Linux can't do that right now, but it might make a nice development platform to put NUMA awareness into Linux...
However, all the developers working on Linux NUMA awareness have access to NUMA machines already, so the point is moot.
Menuet compiles in 0.3s on my laptop :o)
The machine he's running in is a Sequent/IBM NUMA-Q, not just a bunch of PC servers...
I don't know what was meant by the submitter or the editor or whomever it was that was ranting about $500 machines, but this is not what it's about.
First of all, it's a porting project, making linux run on the platform. Only then, it's a feature project, making linux make good use of the NUMA capabilities.
So there. It's a work in progress, and no, you can't afford one of these for home. Maybe, in time, the lessons learned from here will help us build generic Fast-NOW clusters and have software-based NUMA on linux. Until then, keep dreaming, or start working on a project you can contribute to. Remember, the difference with open-source is that you can always do more than just complain.
free the mallocs!
My SGI indy (8 years old?) made it in 26 seconds.
yep, I've submitted, but I don't think it's gonna be accepted. See here for more details:l /0203.2/0009.html
http://www.uwsg.indiana.edu/hypermail/linux/kerne