Slashdot Mirror
Hyper-Threading Speeds Linux
developerWorks writes "The Intel Xeon processor introduces a new technology called Hyper-Threading (HT) that makes a single processor behave like two logical processors. The technology allows the processor to execute multiple threads simultaneously, which can yield significant performance improvement. But, exactly how much improvement can you expect to see? This article gives the results the investigation into the effects of Hyper-Threading (HT) on the Linux SMP kernel. It compares the performance of a Linux SMP kernel that was aware of Hyper-Threading to one that was not." Ah, the joys of high performance.
Xeon folks arent having the only fun. The 3 Ghz Pentium 4 is also hyperthreaded for that crunchy flavor and great taste.
We've used XEON's on our DB server for a few months now. The performance has been outstanding. You also see 4 processors when you run top.
At first we thought this was an error, and got in touch with Dell's tech support. But the geeks there said this is normal behavior.
Newsfollow.com
>was aware of Hyper-Threading to one that was not."
But if you aren't going to use hyper threading you would use a UP (non-SMP) kernel, which would gain you considerable performance. The benefits are not so clear cut as many of the benchmarks show limited benefit from hyperthreading and would perform faster on a uniprocessor kernel.
All operating on a single chip!
Does SMP support automatically allow benefits from Hyperthreading, or does that require special support all it's own?
"Everything you know is wrong. (And stupid.)"
Moderation Totals: Wrong=2, Stupid=3, Total=5.
The results on Linux kernel 2.4.19 show Hyper-Threading technology could improve multithreaded applications by 30%. Current work on Linux kernel 2.5.32 may provide performance speed-up as much as 51%.
while it may not be very useful for a single-user box(it actually looks like it would be a detriment), integrating it into client-server situations would give us some nice boosts in performance. web servers ought to see some real gains with this.
The World's Worst Webcomic!
Of course multi-threaded applications are going to improve. What's your point?
For those who didn't RTFA:
Simple syscall 1.10 1.10 0%
Simple read 1.49 1.49 0%
Simple write 1.40 1.40 0%
Simple stat 5.12 5.14 0%
Simple fstat 1.50 1.50 0%
Simple open/close 7.38 7.38 0%
Select on 10 fd's 5.41 5.41 0%
Select on 10 tcp fd's 5.69 5.70 0%
Signal handler installation 1.56 1.55 0%
Signal handler overhead 4.29 4.27 0%
Pipe latency 11.16 11.31 -1%
Process fork+exit 190.75 198.84 -4%
Process fork+execve 581.55 617.11 -6%
Process fork+/bin/sh -c 3051.28 3118.08 -2%
is it just me? or does the linux kernel not perform so much better in SMP HT?
I know, there might be many places where it has been discussed before, but could someone please tell me if HT is only for threading or can it be used for precesses, too. ...
And I know, they are essentially the same syscall under linux, and might be faster, b/c of synchronization issues wrt to the memory access IIRC
"Conclusion
Intel Xeon Hyper-Threading is definitely having a positive impact on Linux kernel and multithreaded applications. The speed-up from Hyper-Threading could be as high as 30% in stock kernel 2.4.19, to 51% in kernel 2.5.32 due to drastic changes in the scheduler run queue's support and Hyper-Threading awareness."
My questions: What's the downside? Is AMD doing anything similar?
Fight with computer brings SWAT team
The pretty detailed (for me anyway) article on Ars Technica concludes that performance on a HyperThreaded CPU will be very much dependant on the application mix. While research like this is useful it will probably always be a try and see scenario.
Tested HT running couple large jobs on a 2 CPU box with each process using over a GB of RAM. Performance went down.
Also HT can play havoc with a openMosix cluster since processes can start being migrated around to CPU's that do not really exist and appear to have no load, yet the physical CPU may be 100% loaded in reality.
It is not all peaches and cream.
Like most development shops, we do a great deal of development for multiprocessor machines so we write a lot of multithreaded code. Multithreaded code creates a whole host of new debugging pitfalls that don't show up if the developer is debugging on a single processor workstation. As John Robbins says in his terrific Debugging Applications book, if you are developing a multithreaded application, you better be certain you are doing your debugging in a multiprocessor environment.
From a development standpoint, will a hyperthreaded chip provide an adequate environment in duplicating the behavior of a multi-processor PC well enough that shops can buy cheaper, one CPU machines for development and still be confident in their results? I'm guessing nothing will replace the real thing but I'd be interested in any commentary.
if you had read the article, you would have seen that the kernel doesn't show too many signs of superb HT usage. In fact, performance degrades in many places.
Also, if you knew just an itsy bit about kernels, you would know that Microsoft has done some pretty good advancements and achievements in the SMP realm.
Well if I must say something, it' this: that's really going to put a fancy how-do-you-do in the knickers of all those pay-per-processor software types. I mean Oracle, for heaven's sake, is going to have to go absolutely bonkers trying to figure out how to screw the light-bulb into that buffalo (if you pardon my french). I mean what's a meglomanic to do? I mean I've got expenses! I've got tricarbonfiberalloy yacht hulls to pay for! Can have people going around trying to process code in a processor without us getting some slice of that monkey, I'll tell you right here and now sir! No sir! Maybe it's your not patriotic enough. Trying to cut corners, eh gov'nor? Now I'm gonna have to go and rewrite all the contracts stating explicitly that "processor" is defined as a virtual space for processing. Yes that ought to do it. But I'll still have to have the lawyers check it, just to make sure they aren't any loopies. Drats those laywers! Taking all my money too!
"This isn't a study in computer science, its a study in human behavior"
Yeah
SMP support has existed since NT 4.
If you use NT 4 MP edition, 2k Pro or XP pro, HT just works if you have the hardware.
Linux had to change to accomodate it, as it bypasses the original system BIOS with it's own code.
So what you meant to say was "once again Linux plays catchup to MicroSoft, but only about a year or so later this time, and not 5-10."
I don't need no instructions to know how to rock!!!!
If you overclock the Xeons (And newer P4 CPUs) too high...
"Prepare to go to HyperThread."
"Go to HyperThread!"
*WHOOSH*
"My God, they've gone plaid!"
(Just to keep on topic, this is a very informative shootout between HT/non-HT Intel and AMD SMP processors setups here.)
Just couldn't resist the Spaceballs reference, tho!
On moft platforms, CL.exe goes file per file, and outputs. It's a linear opertation. So HT for compiling makes no difference.
However, the NT DDK has a cool feature that allows you to spawn as many instances of CL as there are processors. Which you guessed it, is only of any use if you are compiling tens/hundreds of files.
Sorry, I do not really know of compiler internals for *NIX. Maybe someone can back me up? or clear it up?
Hyperthread support vs not.
Standard API calls (w/ hyper thread) Increase (a bad thing (tm)) of latency of calls by 1-6%.
STD workload (w/ hyper thread) Increase in throughput an average of 5-10%. Disk writes decreased throughput by 30%.
Client network perf: "Chat room" test, increase of throughput 22-28%.
Server network perf: File serving, increase of 9-31%.
Kernal 2.5.24 roughly doubles the above benefits.
Looks like no real downfalls... (How often are you running a single thread? Me either.)
Hyper-Threading Speeds Windows
FoundNews.com - get paid to blog.,
It's not so great, if you need SMP you still cant beat two or more physical CPUs.
In this scheme, the pipeline is split into two and two concurrent threads run in it. Which is pretty neat, but hurts performance in some situations.
- Cache latency is basically doubled, as two VCPUs now fight over access to the cache
- Pipeline depth is shortened for either given VCPU, which hurts code that was optimized for the longer pipelines (lots of matrix math, MMX stuff).
It's a cool development in CPU design, but it has a ways to go, and the OS needs to be aware of it. You should be able to 'shut it off' in code on the fly, if you want to dedicate 100% real CPU to a given task.
I don't need no instructions to know how to rock!!!!
Sorry, I do not really know of compiler internals for *NIX. Maybe someone can back me up? or clear it up?
With gcc, the -j will setup gcc to utilize SMP. You specify the number of processors you physical have. I do not know how it would work with HT, and I didn't RTFA to see if they covered it. There is native support inside of gcc for SMP-based compiling though.
Dacels Jewelers can't be trusted.
...I don't understand how this helps. I'm typing this on a Dual 1.4 GHz system -- even if a process is multi-threaded, it's still not as fast as a 2800 MHz processor. In addition, many programs can't take advantage of SMP, rendering dual processors 'useless' (for any single process; Linux distributes processes across processors.)
So if 2*1400 1400? Shouldn't taking, say, the 3 GHz P4 and 'emulating' SMP actually slow things down slightly? I don't understand how it can help, and am actually surprised that it doesn't *hurt* speedwise.
________________________________________________
suwain_2
* 128-byte lock alignment
u zzword-cipher-reallignment too.
That's something new that you guys haven't heard of yet ;)
* Spin-wait loop optimization
* Non-execution based delay loops
* Detection of Hyper-Threading enabled processor and starting the logical processor as if machine was SMP
* Serialization in MTRR and Microcode Update driver as they affect shared state
* Optimization to scheduler when system is idle to prioritize scheduling on a physical processor before scheduling on logical processor
* Offset user stack to avoid 64K aliasing
Is that all?! I hoped it'd do the post-integer-supercooled-re-automation-longterm-b
Get your own free personal location tracker
Faster clock speed processors speed up Linux.
As a rock-in-roll Physicist once said, No matter where you go, there you are.
-j is an option for GNU make, not gcc. And there is no rule that says you must specify the number of processors you physically have; for big compiles, you'll get a somewhat better time if you say -j2 on a single-processor machine. This is because, when two gcc's run in parallel, one can take the processor while the other is waiting for disk.
There is no native support inside of gcc for SMP-based compiling. gcc itself is completely sequential. You are perhaps thinking of parallel makes.
As if there wasn't enough already...
processor : 0
bogomips : 3191.60
processor : 1
bogomips : 3198.15
According to that the logical processor is actually faster than the physical one! Just think of what you could wind up with if you instantiated a logical CPU on the logical CPU!
Um, no, not quite. You pass the -j option to make. make will then go through your makefile, and assuming you wrote it right, run specified commands (like gcc) in parallel. You have to be careful about target dependencies when doing this, though. And this parallelization is even useful on uniprocessor machines, as if you use make -j2 you will get some gain in time in a big compile because while one gcc is doing I/O, the other can be using the CPU and compiling.
Just to be a pedant,
-chris
WHat you've conveniently snipped out in your trollish post is all of the applications benchmarks showing improvements. If you're not going to run any application code, you might as well shut the machine off and save the marginal stress on the environment.
Most of us have our computers do work and those applications, running on an OS which has *barely* slowed, will be able to do more work in the same amount of time under the HT-aware OS than under one which does not utilize the second, virtual processor.
I have discovered a truly marvelous sig, unfortunately the sig limit is too small to contain i
As opposed to MS OSes that are also sped up by this? If anything, it's more of a threat to other UNIX vendors where Linux and other x86 based free Unices get better performance/cost ratios.
Maybe this is a feature of newer releases of gcc, but I've never heard of -j doing auto SMP. There is a -j option for parallel makes with gnu make, but this is only for the compilation and not runtime.
The portland group compiler and the intel compiler. Do support some auto-parallalization via openmp and threads.
Typo, I assume you meant GNUmake -j, not gcc.
Gcc has no -j option. Make has a -j option.
Which has nothing to do with SMP, it's simply how many jobs make will run simultaneously, which of course is a wise thing to use in a multi processor environment, but also a good thing to use in places where the CPU waits for IO (ie. if your code and output is stored on a disk).....
Anyone know of any details around SMP versions of HT CPUs. It's not a very google friendly set of search terms.
I expect that there would be a performance difference if the scheduler knew which were real cpus and which were half of an HT pair.
Even flags to fork concerning which processor to fork to. i.e. --this_cpu_but_different_HT_CPU
Because you might want the freedom to attempt to reduce the in-CPU cache misses and the like.
Likewise the the implmentation of Process Groups - setpgid() warrants investigation.
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
Wow. Its posts like this that really make me feel old at 25.
They had 486 SMP systems. In fact there was an awesome upgrade that came out like ten years ago that let you put two 486 processors in one socket. Of course you needed the clearance for it. SMP was actually all the rage ten years ago for the same reason PowerPC was all the rage. Intel had a hard time scaling, so one of the solutions was to us multithreading and divide up the work.
OS/2 2.11 SMP was out in 1993, and NT 3.1 came out shortly thereafter. Both supported SMP. The Pentium Pro, which came out in early 1995 was highly optimized for 32-bit code and multiprocessing. 4 and 8 way Pentium Pro boards existed. And were somewhat common.
If anything, SMP is LESS common today. When was the last time you saw a 4-way SMP board for sale anywhere? You could easily get them back then. The reason its less common today is processors really are a lot faster. Intel is doing this Hyperthreading crap because they know that orders of magnitude performance gains are a thing of the past, so multithreading is the key.
Of course, us old OS/2 fanatics were saying this ten years ago.
I don't read or respond to AC posts
If you're running code that's efficient on a P4 (few mis-predicted branches, low cache miss rate, good parallelism, etc.) then HT is pretty much useless.
If you're running code that's inefficient on a P4 (which pays for its high GHz with long pipelines, large latencies, a slow decode stage, and several other drawbacks), then HT can usually paper over a fair percentage of these problems. But remember that HT requires OS support, may require application support, and "your mileage will vary".
It's easy to make up & spread cool- and credible-sounding stuff. Finding & checking hard facts is hard work.
In Europe P4 3.0 with HT costs ~745 euro (+tax)
An Asus A7M for dual Athlon costs ~260 euro (+tax)
Two Athlon XP 2200+ cost ~340 euro (+tax).
Alternatively you can get two Athlon MP 2000+ for
roughly the same money (if you don't trust the
XPs).
Now, please explain to me why would someone
with real SMP needs in mind (and NOT games)
consider the P4 with HT.
P.
P.S. I understand that the prices in the US are
different, but still, it is VERY expensive.
Holy intellectual dishonesty, Batman!
NT and Windows 2000 do not support HT and never will. NT will not becuase it's been end-of-lifed, and Windows 2000 will not because of Microsft policy. On a 2-CPU system with HyperThreading, NT and Windows 2000 will think they have real 4 CPUs (unsurprisingly, this is what a pre-HT version of Linux will see as well). HT support means the OS knows that it has, in this example, 2 real CPUS and 2 fakes, and the scheduler will weight the real CPUs accordingly.
XPPro SP1 is the first, and only shipping version of Windows to support HT.
Simply put, you'll need two or more processes consuming all available CPU power before you'll see some real benefits from HT. If you're severely IO-bound, running a high-end FC SAN solution on an old P2 server will outperform a 5ghz machine with a mediocre disk.
So - yes, not all people and applications will benefit from this. But no - it is not try and see.
Stop the brainwash
While Win2K will see a hyper-threaded CPU as 2 physical CPUs, WinXP is smart enough to see it as a CPU and a Virtual CPU. At the last Intel conference I attended they made sure to emphasize that while XP Home doesn't support 2 phyisical prosessors it will properly recognize a hyper-threaded CPU and allocate resources accordingly. Do you think Intel would enable the technology in the P4 3Ghz(a desktop CPU) without making sure Microsoft supported it in their desktop operating systems?
So, in a nutshell, what MS says is: Windows 2000 counts processors in a broken way and requires you to buy licenses for every logical processor, even though you won't get nearly as much processing power as you would if you really had that many physical processors. But rather than fix this bug, we're going to solve the problem by making you buy .NET, which counts processors correctly. So either way, if you're going to use hyperthreading, expect to send us more money.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
This is fine, I guess, if you're going to run a processor as slow (!) as this. Point being that a hyperthreaded system will place greater demands on the ram bandwidth.
... say 3GHz, where HT is enabled in vanilla P4's ... and we can expect to see the memory bandwidth being toasted continuously. Under these conditions I doubt we would see a speedup at all, and quite possibly the reduced cache efficiency would reduce it.
With a slow processor they may be using 80% of the available bandwidth instead of 60% with HT switched off. Upping to processor speed to
Executive Summary: Can we do this again with a non-Xeon P4 3GHz?
Dave
I write a blog now, you should be afraid.
If the results are similar to running SMP with two processors (and they look roughly similar), isn't a system with 2 Athlon-MPs still cheaper for a given performance level?
"with their freedom lost all virtue lose" - Milton
Obviously that depends.
If your web server is just doing static content, then probably not as a 486 can saturate a T1.
If your web server is doing dynamic content, then possibly.
> Cache latency is basically doubled, as two VCPUs now fight over access to the cache
I'm pretty sure this is wrong -- cache latency isn't doubled; the SIZE is HALVED. The two threads access two different virtual caches. Trying to get them to contend for a single cache would be an architectural nightmare.
Though I believe it's still one physical cache -- which means that the latency is going to be higher than what you'd expect for a cache of its apparent size.
> Pipeline depth is shortened for either given VCPU, which hurts code that was optimized for the longer pipelines (lots of matrix math, MMX stuff).
I don't actually know about the pipeline, but I suspect this is wrong too: shortening the pipeline (reducing the number of stages) is a fundamental change in the architecture; a pipeline isn't something you can cut in half and give the front end to one process and the back end to another. Each stage is quite unique.
Now if you mean that the latest Pentiums have a shorter pipeline than previous incarnations, then maybe that's right (though I'd doubt it -- they're always *lengthening* the pipeline to get those higher GHz numbers). But that would have nothing to do with Hyperthreading.
"Orthodoxy is unconsciousness" - Orwell
Don't companies like (guessing) Oracle charge by how many processors you use with their software? I know for solaris (even intel) you are licensed by how many cpus you can use. (Just like windows I guess, 1, 2, 4, 8+ cpus)
Also since XP Home is only single processor capable where does that leave the home users that buy 3.x Ghz computers? Surely it wouldn't be long before someone figures out how to swap a multiprocessor HAL into XP Home...
To simplify greatly, if the CPU has separate units for integer and floating-point math (for example), Hyperthreading means you can use these units in parallel. Therefore, HT will not speed up pure integer or pure FP math, like SMP would. It will only speed up things if you run different kinds of process simultaneously.
Also, many people have noted that HT sometimes slows things down a bit. I don't find this very surprising because the OS needs more work to organize things for HT, but it may not have more CPU resources than a non-HT version.
Personally, I think HT is a good idea because it's using the existing hardware more efficiently in a true hacker spirit. However, it's nowhere near proper SMP.
Escher was the first MC and Giger invented the HR department.
One of the major impediments to increasing CPU performance has been increasing memory latency. Memory latency has grown worse as CPUs have gotten faster. Accessing RAM will now cause a >150 cycle latency, during which the processor sits IDLE.
Cache only partly mitigates this problem. Some applications, such as databases and OLTP, are heavily dependent on repeatedly accessing non-cached RAM. There is no way to cache all the relevant data, since virtually all databases are larger than can fit in any present cache, no matter how large, and there is sometimes no way to predict which data will be accessed. ALL of these applications have CPUs that spend much of their time being IDLE, waiting for memory to be returned.
SMT (hyperthreading) allows the processor to perform useful work during these otherwise idle periods, by allowing the cpu to switch to a thread that is not blocked on memory access. The "idle bubbles" in the execution pipeline can therefore be "filled in" by useful work that advances the state of relevant programs.
SMT can cause a degredation in performance beceause it can lead to "cache thrashing." In an SMT-naive kernel, two unrelated threads could be scheduled for the same physical CPU. These unrelated threads will likely share very little code or data. The two threads will therefore "compete" for the single shared cache, with each thread's data being repeatedly displaced by the other's.
This difficulty can be substantially mitigated by making the kernel aware of "virtual processors," and by implementing scheduleing algorithms to minimize the impact. The performance of hyperthreading will likely improve as kernels are better able to exploit it.
Incorrect, OS/2 was SMP since 2.1. The OS/2 SMP model is still known to be one of the best SMP models to have ever been written. Click on this link http://www.byte.com/art/9406/sec11/art2.htm and learn something about OS/2 SMP (oh geez, it's 1994) and SMP in general.
I have seen screenshots of Windows task manager showing (2) CPU performance graphs.
Since the "Professional" line of NT/2K/XP kernels only support two processors, does this mean you can only use one HT CPU?
It really bugs me when I see benchmark numbers relied upon when they have not been presented as statistically significant.
Whenever you run a benchmark, you MUST run it multiple times and do the proper statistical calculations for standard deviation.
It is NOT VALID to do one run, and it is NOT VALID to average a bunch of runs without knowing what the deviation is.
Some times a benchmark's time will vary by more than 100%. Sometimes the reasons are valid, sometimes they are because of an error in the benchmark.
Without this sort of validation, the numbers presented should not be trusted.
SMT (hyperthreading) will become increasingly important when processors are able to execute more than 2 threads simultaneously.
This development is inevitable. Previously, each new processor generation was faster than the prior one at a given clock rate, because each new processor core had more execution units, and was therefore able to perform more work in parallel. This trend abruptly ended recently, for one reason: there is no more instruction-level parallelism (ILP) to exploit. It is impossible for a processor to look at a thead of execution and find more than a few instructions to execute in parallel.
The only parallelism left to exploit is THREAD-LEVEL parallelism (TLP). Therefore the only way to continually increase performance is to increase the number of threads that a CPU can execute in parallel. This requires two modifications to CPU cores: first, increase the number of thread contexts per CPU, and second, increase the number of pipelines to which those threads can be dispatched.
With the P4, it would be pointless to have more than 2 thread contexts, because there aren't enough CPU resources lying idle to execute more than 2 threads. But future CPUs could make use of more than 2 thread contexts by having enough CPU resources to execute all of them. Future CPUs could have 20 execution units or more, which would be enough to execute several threads. Remember that the number of transistors per CPU continues to increase exponentially.
It's easy to forsee a time when processors have 20 execution units (10 integer, 10 fp) and 4 thread contexts, offering more than triple the performance of a non-SMT cpu. In the future, non-SMT CPUs will make as little sense as a non-superscalar CPU would today.
Any SMP capable operating system supports HT. On the other hand license issues make combining true SMP and HT a pain on non server version of Windows.
You're totally missing that part of the beauty of HT is the transparency.
On the other hand you can write things SPECIFICALLY for HT to deal with things such as cache issues, but saying that windows doesn't support it at all is rather misleading and makes it seem like people wouldn't see any improvements at all.
Keep in mind, a 30% gain (for the 2.4 series) in a 2GHz machine would equate to a machine that performed server-oriented functions at an effective 2.6GHz.
When they benchmarked 2.5.32, they showed a 51% increase, which would boost your effective server performance to 3GHz.
Granted, the way I understand it, the actual coordination of core components for the two threads is hard-wired or in firmware. That means Intel can still improve HT, to get a better performance boost. To further that line, consider if Intel were to add additional core sections of their CPUs, to be allocated dynamically by the firmware. That means you're increasing your per-clock performance without the major overhead of developing a whole new CPU core.
I can't see Microsoft standing for it. Intel could put all the pieces for two CPUs on the same die, and call it HT. You might have all the functionality of a dual-CPU setup, with less latency, and still have it show up as a single HT-enabled processor.
With the way Microsoft's handling SMP machines (with CPU licenses), in addition to their statement that they are developing a 64-bit version of Windows based on the Hammer architecture, I think AMD's future looks pretty bright.
What's this Submit thingy do?
It all starts with the long pipelines and being able to dispatch several instructions at once. The problem was that some of the execution units would go idle due to instructions that couldn't be reordered effectively enough.
The idea behind Hyperthreading is to have an additional context also dispatching instructions which need not be in any particular order with respect to the first thread. This allows the CPU to actually run at closer to 100% capacity.
The upside is that when things work out well, less execution units sit idle and waste cycles. The downside is that if one thread does manage to fully utilize the CPU, you don't benefit, and will likely pay a penelty for the extra scheduling.
From what I've seen, many apps benefit. Heavy and well optomized floating point computation can lose on HT. Some of that can be helped by a more aware scheduler that tries to pair up primarily integer threads with threads doing a lot of floating point.
Microsoft has done some pretty good advancements and achievements in the SMP realm
Hmmm. Given that SMP has been around an awfully long time, I find this a little hard to believe. And I also remember talking to a senioir DB guy at Microsoft where he was explaining how they'd just started to do SMP optimization in the OS - this was for an NT SP in 1998 or 99.
Also, with DDR or Rambus costing nearly triple what SDRAM costs, I wonder if some enterprising company will develop a chipset that can interleave access to two SDRAM DIMM's for performance similar to DDR. Even if the two DIMM's have to be on completely separate electrical busses, I would think that it would result in a lower total cost for most popular combinations of performance levels and memory capacity.
The SMP in 2.11 wasn't necessarily the best, but I was just talking about when it really first occurred, so I may have misworded it. They actually had to write the SMP for some pretty interesting hardware (486 SMP) and needed to do some amazing wizardry to make it all work right. After OS/2 Warp came out, the SMP was amazing and NT couldn't touch it. I remember reading an article about the SMP software engineer (name escapes me right now) who he was a programming savaunt and IBM by all means had hired the best of the best. A lot of his work was put on hold for OS/2 Warp as IBM was making OS/2 into a microkernel OS for the then new PowerPC architecture (which was a ultra amazing OS, but never truly released to the public). This is what basically killed OS/2 in the end and held up the SMP implementation that today is currently one the best in the world (on Intel hardware).