2.5.65 On 32-way NUMA-Q with Preempt Enabled

_iris writes "I think the subject speaks for itself. Here is the link to the story on KernelTrap." In case you have a spare 32-processor machine munching grass in the back 40.

Comments

Congratulations to the Linux Developers

[Talez]

Good to see the Linux kernel making such leaps and bounds.

Keep at it guys!

Re:Congratulations to the Linux Developers

[Eunuchswear]

Such leaps and bounds, ah yes. A quote from one of the followups to the article:

These machines have been in production since something like 1996 and were EOL'd around 2001. i.e. this is not just 5 years old, its entire product line is 2 or 3 years dead.

Yup, Linux, so up to date it's just beginning to suport hardware that hasn't been built for 2 years.

Re:Congratulations to the Linux Developers

[Hellkitten]

Yup, Linux, so up to date it's just beginning to suport hardware that hasn't been built for 2 years

The point is that 2.5.65 booted with preemtion patches on a 32 processor machine

That is preemtion of kernel threads. If there is a deadlock or race condition it would be more likely to show up un a beast like that than in your average dual athlon. So this is really not about supporting 32 processors (which is old news) but about the quality of the work that has gone into kernel preemtion

I have no idea if any other OSes out there support preemtion of kernel threads running on multiple cpus. Anyone care to enlighten me?

Re:Congratulations to the Linux Developers

[julesh]

I have no idea if any other OSes out there support preemtion of kernel threads running on multiple cpus. Anyone care to enlighten me?

Solaris. I believe Mach does as well. There are probably others that aren't as well known.

Beowoulf

[rf0]

I would like a ... oh wait it is :)

Rus

Re:Beowoulf

[ePhil_One]

No, no it's not. A multiple processor machine != a multiple machine cluster.

But a NUMA is not purely a SMP machine either, its a little of both. The Sequent NUMA-Q is a series of Quad-proccessor systems (Quads) linked via a high speed bus. Each Quad has its own memory pool, but on a virtual level its also one big memory pool. Hell, I was ready to be certified on Dynix/ptx and I don't fully understand it beyond knowing its not Parrallel computing, and ints not SMP. Its NUMA (Non-uniform memory architecture!).

And its not really dead. Sequent was bought out by IBM, who stopped producing the Intel based systems, but is continuing to produce NUMA systems based on PowerPC systems. (or at least were; there is some embarassment here in that DB2 doesn't run right on NUMAs, so they have to benchmark with Oracle :^)

I also suspect the NUMA technology will be very important for upcoming SMP on a chip systems, if you have 4 CPU's on a chip, how do you combine them into a MP machine?

32 Proc ?

[MosesJones]

Sheesh, I'm sitting here with a 64 Way and two 32-way boxes just waiting for decent to run on them.

Does this mean that FINALLY I can shift Quake Server off the clustered S80s in the basement ?

Munching grass?

[Eric Ass Raymond]

In case you have a spare 32-processor machine munching grass

I'd rather have a girlfriend who is also into muching carpet.

Cool

[rf0]

Now this is cool. I know that SGI can scale the Altix to 64 CPU's running 2.4 with their own additions in an SSI. However not sure about. 2.5. Its nice to see it in the main kernel anyway and the only way is up

Rus

Man and i though i had it good

[tokaok]

i thought i my daily 3-ways were good but it seems ive been missing out :(

Re:Man and i though i had it good

[Shishak]

You and your two hands does NOT make a 3-way....

Complete article

[blackcat]

Here is the complete article, the fscking lameness filter made it quite a struggle to get it posted here. Anyway:

Zwane Mwaikambo announced today on the lkml that he's successfully boot the 2.5.65 development kernel on a 32-way NUMA-Q server with -preempt enabled. Speaking to Robert Love [interview], the kernel preemption maintainer, he began his announcement saying, "Robert, I suppose you can add another notch on your erm.. bedpost(?) and congratulations to all the kernel developers!" NUMA awareness in the scheduler was added into the development kernel in late January [story].

William Lee Irwin III [interview] explained the significance of this achievement:

"This has had a hard time historically. I'm really glad NUMA-Q's are now immune (in the sense of correctness) to this config; previously it was believed that preemption points in printk(linux_banner) would take out the machine early in boot if preemption was enabled. Congratulations rml! If you're booting without issues on these things, you are a _very_ long way toward being race-free. This is incredibly good news, both for the preemption support, and for the general stability of the i386 bootstrap."

Read on for the full thread.

From: Zwane Mwaikambo
Subject: 2.5.65-preempt booting on 32way NUMAQ
Date: Sun, 6 Apr 2003 06:48:33 -0400 (EDT)

Robert i suppose you can add another notch on your erm.. bedpost(?) and congratulations to all the kernel developers! It survived some local networking stress tests, but there is more fun stuff like tty layer to completely obliterate ;)

(Hardware courtesy of OSDL)
Running configuration
32 Processors, PIII 500
32G RAM

Patches required:
2.5.65 (only because isp1020 decided to get huffy)
Purge assign_irq_vector panic - Zwane Mwaikambo

[boot messages]

From: Robert Love
Subject: Re: 2.5.65-preempt booting on 32way NUMAQ
Date: 06 Apr 2003 14:28:42 -0400

On Sun, 2003-04-06 at 06:48, Zwane Mwaikambo wrote:
> Robert i suppose you can add another notch on our erm.. bedpost(?)
> and congratulations to all the kernel developers! It survived some
> local networking stress tests, but there is more fun stuff like tty
> layer to completely obliterate ;)

Excellent, Zwane.

Congratulations! Good work.

Robert Love

From: William Lee Irwin III
Subject: Re: 2.5.65-preempt booting on 32way NUMAQ
Date: Sun, 6 Apr 2003 04:23:40 -0700

On Sun, Apr 06, 2003 at 06:48:33AM -0400, Zwane Mwaikambo wrote:
> Robert i suppose you can add another notch on our erm.. bedpost(?)
> and congratulations to all the kernel developers! It survived some
> local networking stress tests, but there is more fun stuff like tty
> layer to completely obliterate ;)

Wow!

This has had a hard time historically. I'm really glad NUMA-Q's are now immune (in the sense of correctness) to this config; previously it was
believed that preemption points in printk(linux_banner) would take out the machine early in boot if preemption was enabled.

Congratulations rml!

If you're booting without issues on these things, you are a _very_ long way toward being race-free. This is incredibly good news, both for the preemption support, and for the general stability of the i386 bootstrap.

All that's really left is driver and non-i386 arch coverage if I'm right.

-- wli

From: Zwane Mwaikambo
Subject: Re: 2.5.65-preempt booting on 32way NUMAQ
Date: Sun, 6 Apr 2003 07:25:09 -0400 (EDT)

On Sun, 6 Apr 2003, William Lee Irwin III wrote:

> This has had a hard time historically. I'm really glad NUMA-Q's are now
> immune (in the sense of correctness) to this config; previously it was
> believed that preemption points in printk(linux_banner) would take out
> the machine early in boot if preemption was enabled.

Which kernel version was that from

Re:Complete article

[BJH]

Oh bugger...

Sylpheed

Re:Complete article

[platypus]

I really hope this was a joke. If not, try to find a meaning of the sentence "sort out individual threads" which doesn't imply a missing feature of his mail client, there really is one.

Re:Now where'd I put that 32 processer machine ??

[ergo98]

Microsoft just set the #2 TPC-C result in the non-clustered category using Windows Server 2003 and a 32-way Itanium 2 machine. They did this, of course, because Oracle publicly derides clustered results as not counting (and really setting up horizontally partitioned views across a huge federation of serves is not the easiest thing, and it's far from transparent for the database developer: You have to specifically design around it), so now there's a SQL Server 2000 result higher than any Oracle result.

So there you have it: A 32-way machine that's actually useful (when available on 2003-06-30).

That sounds really cool!

[wheany]

What the hell are you guys talking about?

Superb

[vesamies]

Yes, it is now possible to launch 32 preemptive NUMA-Q missiles strikes simultaneously using the Linux kernel. Excelent!!!

Hidden Meaning

[evilviper]

I think the subject speaks for itself.

Translation: I'm going to bed, and the editors are lazy.

what does that mean.

[leuk_he]

NUMA, ok, that i understand.

(Instead of one big shared memory pool it uses processors that each have their own pool, and can access other memory with a timing penalty)

but what does "-preempt " have to do with this. what does this option do? Int unix always preemtive?

Re:what does that mean.

[platypus]

Preemt means preemtive in kernel space, you are talking about userspace. kerneltrap has an interview IIRC with Robert Love where the ins and outs are explained, if not, try google.

Stop stealing ideas from SCO!

[Billly Gates]

We all know that SCO invented NUMA and SMP. Jeesh.

Inevitable

[Ed Avis]

Taco's Law: any story about massive scalability will be posted on a web server which craps out due to 'too many connections'.

Anyone got a mirror?

Re:Inevitable

[brejc8]

Try this Mirror

Correct URL

[lemmen]

It seems the URL isn't working because of the session ID. Use this link instead if you get a "to many connections" error.

http://www.kerneltrap.org/node.php?id=628

Have fun!

Re:Now where'd I put that 32 processer machine ??

[gazbo]

Interesting that the 3 MS solutions (SQL server on Windows Server 2003) all also offer the best price/performance ratio too. Just something to think about.

*sigh*

[Anonymous Coward]

Would you have all been as interested in this story if you'd known:

• It's been on KernelTrap for a *couple days* now
• The machine in question is about 5 years old
• Linux already has been booting on large-number-of-CPU machines for awhile. This story is about the CONFIG_PREEMPT subsubfeature.
• This has very little to do with Linux's scalability. If I booted Linux 2.5.x on a bazillion CPU monster tomorrow, it wouldn't be a bazillion times faster than my single-CPU desktop system.

*sigh*

Kernel 2.6 release

[Anonymous Coward]

when?

The greatest thing about Slashdot...

[mofolotopo]

Is that occasionally there are headlines like this that I can read, re-read, and still have no clue what the article's actually about. I don't know what ANY of that stuff means.

Re:Now where'd I put that 32 processer machine ??

[khuber]

Interesting that the 3 MS solutions (SQL server on Windows Server 2003) all also offer the best price/performance ratio too

McDonalds makes hamburgers with the best price/performance. Just something to think about.

./ server

[mad27]

Would be nice if /. mirrored the stories it links to. This way only news >1 day old is accessible :(

More info

[mdw162]

It refers to preemptable work on BSD, but here is a good general description of kernel preemption.

Stability vs Features

[the-dude-man]

Its good they finally got it to boot...but still...i think there are far to many bugs in printk. I've had the 2.5's barf on me quite a bit because of this, and it only seems to get worse as it spans out over more proccessors. I think we need to proritize here. The kernel devlopers should be focusing more on stablilzing the 2.5.x kernels rather than adding loads and loads of new features. The recent benchmarks show the 2.5.x kernels are lagging way behind 2.4 and even 2.3 kernels. I think we need to stop loaded all the pretty new features for a minute and focus on getting what we have right now to work. I still have problems with ntfs writing out malformed blocks :|

There is alot of cool stuff in the new 2.5.x kernels i will admit, and i look forward to using it, but as it stands i cant put a 2.5 kernel anyweres but on my home machine because once it hits a production envoirnment it craps itself. I know its just a devlopment release, but lets get it speed up a little before we start working on features for distrubted systems :)