Slashdot Mirror
Linux May Need a Rewrite Beyond 48 Cores
An anonymous reader writes "There is interesting new research coming out of MIT which suggests current operating systems are struggling with the addition of more cores to the CPU. It appears that the problem, which affects the available memory in a chip when multiple cores are working on the same chunks of data, is getting worse and may be hitting a peak somewhere in the neighborhood of 48 cores, when entirely new operating systems will be needed, the report says. Luckily, we aren't anywhere near 48 cores and there is some time left to come up with a new Linux (Windows?)."
It appears that the problem, that affect the available memory in a chip when multiple cores are working on the same chunks of data, is getting worse and may be hitting a peak somewhere in the neighborhood of 48 cores, when entirely new operating systems will be needed, the report says.
Seriously? You picked that over my submission?
I submitted this earlier this morning I guess my submission was lacking. But if you're interested in the original MIT article and the actual paper (PDF):
eldavojohn writes "Multicore (think tens or hundreds of cores) will come at a price for current operating systems. A team at MIT found that as they approached 48 cores their operating system slowed down. After activating more and more cores in their simulation, a sort of memory leak occurred whereby data had to remain in memory as long as a core might need it in its calculations. But the good news is that in their paper (PDF), they showed that for at least several years Linux should be able to keep up with chip enhancements in the multicore realm. To handle multiple cores, Linux keeps a counter of which cores are working on the data. As a core starts to work on a piece of data, Linux increments the number. When the core is done, Linux decrements the number. As the core count approached 48, the amount of actual work decreased and Linux spent more time managing counters. But the team found that 'Slightly rewriting the Linux code so that each core kept a local count, which was only occasionally synchronized with those of the other cores, greatly improved the system's overall performance.' The researchers caution that as the number of cores skyrockets, operating systems will have to be completely redesigned to handle managing these cores and SMP. After reviewing the paper, one researcher is confident Linux will remain viable for five to eight years without need for a major redesign."
I don't know, guess I picked a bad title or something?
Luckily we aren't anywhere near 48 cores and there is some time left to come up with a new Linux (Windows?).
Again, seriously? What does "(Windows?)" even mean? As you pass a certain number of cores, modern operating systems will need to be redesigned to handle extreme SMP. It's going to differ from OS to OS but we won't know about Windows until somebody takes the time to test it.
My work here is dung.
SGI has some awfully big single-system-image linux boxes.
I saw a comment on the kernel mailing list about someone running into problems with 16 terabytes of RAM.
They have an one-off error in their math, it's actually 9 times a 6 core CPU. So, at 42 cores a rewrite is needed.
640 cores ought to be enough for anybody . . .
We are Dead Stars looking back Up at the Sky
Can somebody please explain what the fuck they are actually talking about? They've dumbed down the terminology to the point I have no idea what they are saying. Is this some kind of cache-related issue? Inefficient bouncing of processes between cores? What?
It looks like TFS was written by a Windows fanboy; why mention Linux specifically when it is a general problem? Why try to half-assedly imply that Windows is more advanced than Linux?
Yet Another Tech Blog
(but so much more, including game and movie reviews)
http://yanteb.peasantoid.org
At my last job we had a bunch of Sun T5120s which housed 64 cores. So yeah, we are "anywhere near 48".
Reviewing just the first hour of video games.
Cray seems to have addressed this problem, yes?
I'm still waiting for Windows to work well on ONE.
No kidding. SGI's Altix is a huge box full of multi-core IA-64 processors. 512 to 2048 cores is more normal, but they were reaching 10240 last I checked. This is SMP (NUMA of course), not a cluster. I won't say things work just lovely at that level, but it does run.
48 cores is nothing.
UNIX and C were great in their days. But perhaps not in the meg-core era.
So, what is better in your opinion? Java? Or maybe even ruby? Oh yes, that would be great. Run-time OS reflection through kernel drivers implemented as ruby modules.
Too bad CPU's don't come with built-in ruby interpreters.
http://xkcd.com/619/
Do they have support for smooth full-screen flash video yet?
My Ubuntu 10.04 system still can't play embedded youtube videos. At least Adobe provided a work-around by adding a "play on youtube" option in the right click context menu.
Hahaha. Oh arrogances from ignorance, how I loath you.
The Kruger Dunning explains most post on
I'm not affiliated with Supermicro in any way, but they have four 1U serverboards designed for the 12 core opterons, so that's 48 cores in a 1U server. I'm guessing that Supermicro is not the only vendor of quad opteron boards supporting the latest chips. There are most likely quite a few of these in use by real people. Anyone want to speak up?
I know from personal experience that the socket F opterons performed very poorly in an 8 way configuration compared to the previous generation (socket 940 gen). I ran multiple tests on dual core chips (885s, I think), back in 2006 or 7 where I'd get nearly double the performance in going from a quad configuration to an 8 way configuration, but with the socket F breed of chips, there was no performance boost at all, it was like the clock speed was being cut in half and all the threads took twice as long to complete. I saw this behavior again and again, and the motherboard manufacturer that I was testing the chips with told me that it was an issue with the chips themselves. I think this is the reason why 8-way opteron systems are very rare now.
Nobody's every going to need more than 640 cores
--
Stay tuned for some shock and awe coming right up after this messages!
We've known about this problem for ... well, as long as we've had more than one core - actually as long as we've had SMP... You increase the number of cores/CPUs, you decrease available memory thruput per core, which was already the bottleneck anyway. Am I missing something here?
I'm trying to understand the point of this article..Do we really need a new paper to say that centralized memory bandwidth is at some point a limiting problem in an SMP environment? Isn't this why we have NUMA?
If you want to go after linux internals like the BKL more power to you but that horse left the stable a long long time ago as well.
You could talk about the software problem in dealing with decentralized memory access, synchronization, scalable algorithms...etc but this is all likely something needing to be addressed in application space rather than at the kernel where this paper seems to focus.
There are no shortage of huge single system image linux systems with thousands of processor cores and not a single one of them use SMP architecture. They are all NUMA based (decentralized memory access).
So, they found scalability problems in some microbenchmarks. Well, some of the scalability paths cited in the paper will be fixed when Nick Piggin's VFS scalability patchset gets merged. But it's not like you need to rewrite every operative system to scale beyond 48 cores, it's just the typical scalability stuff, and the kind of scalability issues found these days are mostly corner cases (Piggin's VFS being an exception).
What they're saying is basically two things:
First, there's a bottleneck in the on-chip caches. When a core's working on data it needs to have it in it's cache. And if two cores are working on the same block of memory (block size being determined by cache line size), they need to keep their copies of the cache synchronized. When you get a lot of cores working on the same block of memory, the overhead of keeping the caches in sync starts to exceed the performance gains from the additional cores. That's not new, we've known that in multi-threaded programming for decades: when you've got a lot of threads dependent on the same data items, the locking overhead's going to be the killer. And we've known the solution for just as long: code to avoid lock contention. The easiest is to make it so you don't have multiple threads (cores) working on the same (non-read-only) memory at the same time, that just requires some thinking on the part of the developers.
Second, you only gain from additional cores if there's workload to spread to them usefully. If you've got 8 threads of execution actually running at any given time, you won't gain from having more than 8 cores. And on modern computers often we don't have more than a few threads actually using CPU time at any given moment. The rest are waiting on something and don't need the CPU and, as long as we aren't thrashing execution contexts too badly, they can be ignore from a performance standpoint. To take advantage of truly large numbers of cores, we need to change the applications themselves to parallelize things more. But often applications aren't inherently multi-threaded. Games, yes. Computation, yes. But your average word processor or spreadsheet? It's 99% waiting on the human at the keyboard. You can do a few things in the background, file auto-save and such, but not enough to take advantage of a large number of cores. The things that really take advantage of lots of cores are things like Web servers where you can assign each request to it's own core. And no, browsers don't benefit the same way. On the client side there are so (relatively) few requests and network I/O's so slow relative to CPU speed that you can handle dozens of requests on a single core and still have cycles free assuming you use an efficient I/O model. But it all boils down to the developers actually thinking about parallel programming, and I've noticed a lot of courses of study these days don't go into the brain-bending skull-sweat details of juggling large numbers of threads in parallel.
An E10K is a glorified network computing cluster.
It's not what's being discussed at all.
A Pirate and a Puritan look the same on a balance sheet.
Linux supposedly scales to 1024 or something like that. This is not what they supposedly scale to, but the performance impact of actually trying to use that many cores.
The K42 project at IBM Research investigated the benefit of a complete OS rewrite with scalability to very large SMP systems in mind. This is an open source operating system supporting Linux-compatible API and ABI.
Their target systems, "next generation SMP systems", back in 2003 seems to have become the current generation of SMP/multi-core systems in the meantime.
OpenSolaris is dead. Solaris sucks to use without GNU userland anyway and being sued by oracle is no fun. Besides you troll, this would not need a new linux, just some small changes to the current one.
Tilera Corp. already has CPU architecture with 16-100 cores per chip.
TILE-Gx family
Support for these is already being included in the mainline kernel.
...there is some time left to come up with a new Linux (Windows?).
Windows, the new Linux.
You read it here first...
XKCD:Xeric Knowledge Comically Dispen
Possibly, but they still have tons of work to do. I recently installed Arch Hurd http://www.archhurd.org/ just to get some hands on time with the state of the OS, and was kind of surprised at the status. Many things are in place and work correctly, but it's nowhere near something I could say I'd actually want to use on a daily basis.
Using "cat /proc/cpuinfo" as a benchmark, I can see that my quad core is several times slower with an SMP kernel compared to a non-SMP kernel.
(But a non-proprietary NVIDIA driver will still not play your Flash movies smoothly. :P)
Lets drive the greenhorn OUT! No filthy high UID's with their spelling and gramar and solid well researched non-sensationlist writing. I want my editors to rape the language (bonus points if it is several languages at once) and sent my heart racing by raising my bile and fear of the unknown and known.
Headlines sell adverts. Truth, accuracy, honesty do not. Accept it, you are reading slashdot, it works.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
PER CPU. As was pointed out in many other comments. Linux has already scaled to thousands of cores across many cpus.
"Who is the Journal of Quantum Physics going to believe?" --Stephen Hawking
If BeOS had survived this wouldn't be an issue. Cores and threads everywhere! But noooooooo...
I have 34 systems which have 48 cores already in the server room. These are quad socket systems with 4 AMD 12-core CPU's. So I call BS to the guys who think we have plenty of time, because there are plenty of people deploying these things already.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
However, posting your own post in your own post is a bit excessive, and there could have been better ways to do this than just repost your entire freakin story as the first comment.
Yo dawg, I heard you liked my post so I put a post inside my post so you could enjoy it while you're enjoying my post!
My work here is dung.
A language can change nearly overnight to add mechanism for threading.
Is that why the C and C++ people have spent so long at trying to come up with a memory model that will actually work correctly under concurrent execution? Is that why Java got it wrong the first time?
The point isn't that NT Scales to 256 cores, the point is how efficient it is when scaling to this many processors. The NT Kernel in Win7 was adjusted so that systems with 64 or 256 CPUs have a very low overhead handling the extra processors.
Linux in theory (just like NT in theory) can support several thousand processors, but there is a level that this becomes inefficient as the overhead of managing the additional processors saturates a single system. (Hence other multi-SMP models are often used instead of a single 'system')
Just simply Google/Bing: windows7 256 Mark Russinovich
You can find nice articles and even videos of Mark talking about this in everyday terms to make it easy to understand.
The point is the article dealing with a simulated theoretical cpu with 48+ cores on a single die with shared l2 cache.
The changes made are incremental and I imagine will be dealt with long before this actually becomes an issue when (or if) we get cpus with that many cores on a single die.
multi socket systems are already immune to this the way it is setup, you could have an 8 socket system with each cpu having 8 cores and it would not show the problems shown in the article.
In other words, business as usual, the kernel gets optimized for hardware that actually exists or will exist in the near future. 48 core single cpus are a few years away, and the changes to accomodate them don't require anything significant so I'm sure it will be dealt with at the time.