Slashdot Mirror
Hyperthreading Hurts Server Performance?
sebFlyte writes "ZDNet is reporting that enabling Intel's new Hyperthreading Technology on your servers could lead to markedly decreased performance, according to some developers who have been looking into problems that have been occurring since HT has been shipping automatically activated. One MS developer from the SQL server team put it simply: 'Our customers observed very interesting behaviour on high-end HT-enabled hardware. They noticed that in some cases when high load is applied SQL Server CPU usage increases significantly but SQL Server performance degrades.' Another developer, this time from Citrix, was just as blunt. 'It's ironic. Intel had sold hyperthreading as something that gave performance gains to heavily threaded software. SQL Server is very thread-intensive, but it suffers. In fact, I've never seen performance improvement on server software with hyperthreading enabled. We recommend customers disable it.'"
indeed has once again proved it is expensive to be poor.
Question I find more interesting: What is the performance gap between dual CPU vs Dual-core?
If you mod me down, I *will* introduce you to my sister!
Well, AFAIK, the HTT thing only allows for the processor to sort of split execution units (FPU, ALU, etc) so that one can work on one thread, the other on another one. If an application resorts heavily to one of those units -- and my somewhat uninformed feeling is that software like SQL probably works mostly on the ALU, it, can't possibly GAIN performance. On the other hand, I can see the effort of thrying to pigeonhole the idle threads on the wrong execution unit (will it even try that?) completely borking performance. So yeah, no surprises here.
As someone who commented above pointed out intel openly acknowledges performance can be hurt. I don't know what you mean about not being acceptable to notice this as I've seen this sort of issue mentioned in pretty much every article I've read on HT starting quite far back.
HT is just another chip technology like any other. It is only in the rarest circumstances that a new technology will be better/faster for everything. These things all have tradeoffs and the question is whether the benefits are enough to exceed the disadvantages.
I really think you are being a little unfair to intel. If you had evidence that it decreased performance for most systems even when the software was compiled taking HT into account then you might have a point. However, as it is this is no different than IBM touting its RISC technology or AMD talking about their SIMD capabilities. For each of these technologies you could find some code which would actually run slower. If you happen to be running code which makes heavy use of some hardware optimized string instructions a RISC system can actually make things worse not to mention a whole other host of issues. The SIMD capabilities of most x86 processors required switching the FPU state which took time as well.
It's only reasonable that companies want to publisize their newest fancy technology and they are hardly unsavory because they don't put the potential disadvantages centrally in their advertisements/PR material. When you go on a first date do you tell the girl about your loud snoring, how you cheated on your ex or other bad qualities about yourself. Of course not, one doesn't lie about these things but it is only natural to want to put the best face forward and it seems ridiculous to hold intel to a higher standard than an individual in these matters.
If you liked this thought maybe you would find my blog nice too:
HyperThrottling
I have had an ATI all in wonder 9800 for close to more than a year now. I never really used the tuner part until a few weeks a go when I took delivery of several new LCD's and decided that I could be watching a little tv on one while working.
The 9800 sits on my XP box, which rarely gets rebooted. Games, browsing etc. My mac mini and linux boxes sit in their places with a KVM
Well after using the tuner part, it looks great with my digital cable. But the box would lock, couldnt kill the process of the ATI software MMC. A few times an hour sometimes at least once a day. Well I was on the point of sticking an old haupage in there. Or using another MMC.
Well after much digging I found a thread on how HT could cause issues with the software. I disabled it in the bios, do not really need it for anything. And ran the Tuner 48 hours solid without a lockup.
Now perhaps ATI is at fault for the software, but then again HT caused the incompatibility in my book.
Puto
The Revolution Will Not Be Televised
Is it two complete cores? Front Side Bus speed? Memroy Speed? etc.
The IBM 970MP that Apple is using for the dual core PowerMacs was designed right. And due to the cache snooping (among other things), a dual core 970MP can be slightly faster than a dual processor setu at the same clock and bus speeds.
Another multicore chip to look at for being done right is the Sun UltraSPARC T1 processor. Up to 8 cores with 4 threads per core. Sun's threading model in this processor doesn't have the faults that Intel's HyperThreading does.
Intel HT technology seems as bad a patch on the architecture much like Microsoft's updates to Windows.
I remember early discussions from LKML where developers realized that if you were to run a high-priority thread on one virtual processor and a low-priority thread on the other VP, you'd have a priority imbalance and a situation that you'd want to avoid. The developers solved the problem by adding a tunable parameter that indicated the assumed amount of "extra" performance you could get out of the CPU from HT. In other words, with 1 CPU, max load is 100%; with two physical CPU's, max load is 200%; with one HT CPU, max load would be set to something on the order of 115% to 130%. So, when your hi-pri thread is running and the lo-pri thread wants to run, we let the low-pri thread only run 15% of the time (or something like that), resulting in only a modest impact on the hi-pri thread but an improvement in over-all system throughput.
That being said, I infer from the article that Windows does not do any such priority fairness checking. Consider the example they gave in the article. The DB is running, and then some disk-cache cleaner process comes along and competes for CPU cache. If the OS were SMART, it would recognize that the system task is of a MUCH lower priority and either not run it or only run it for a small portion of the time.
As said by others commenting on this article, the complainers are being stupid for two reasons. One, Intel already admitted that there are lots of cases where HT can hurt performance, so shut up. And Two, there are ways to ameliorate the problem in the OS, but since Windows isn't doing it, they should be complaining to Microsoft, not misdirecting the blame at Intel, so shut up.
(Note that I don't like Intel too terribly much either. Hey, we all hate Microsoft, but when someone is an idiot and blames them for something they're not responsible for, it doesn't help anyone.)
Mysql on linux with a 10gig DB for me definetely benefits my server's performance.
In fact turning it off results in a 20+ percent query time, especially with multiple fulltime queries.
Of course differently written queries and different systems/sql engines might behave differently.
In fact I am so happy with HT, that I am going to change my desktop to one, as it is a linux machine with lots of running apps at the same time. Not mentioning that it is also a devel station with SQL+apache that benefited with HT according to my experience.
(well it is time to upgrade anyway, and I choose HT over non HT).
I don't have a HT-capable proc (AMD Athlon XP 1700), so I don't know anything from personal experience.
I decided to check out how PostgreSQL did with HT.
The first link (1) was suggesting to someone--who was having performance problems under FreeBSD--to turn off HT. Of course, that may not be related to PostgreSQL itself, but rather FreeBSD. I really don't know.
The next thing I found showed some mixed results with ext2 under Linux (2). Somethings showed gain with HT, but not others.
Another link (3) commented that HT with Java requires special consideration when coding.
I didn't come up with anything useful under PostgreSQL, so I checked out Linux.
According to Linux Electrons, Linux performance can drop without proper setup.
// file: mice.h
#include "frickin_lasers.h"
I use Nuendo for professional music recording and even though their latest version says it's HT aware, the performance is poor. In fact in several instances it only takes a few instruments loaded for it to peak CPU, change it back to basic CPU with HT off and it works fine.
MY understanding is it's this way with Cubase as well.
"If any question why we died, Tell them because our fathers lied."
Anybody who understands HT has been saying this since chips supported it
People also have to trouble themselves to configure things properly which isn't the obvious or the default. HT pretends to Windows that its another processor but as you know it isn't. So you have to set SQL Server's '# of processors for parallel processing' setting to the number of real processors, not virtual. We changed ours to this spec and performance went up markedly. SQL Server defaults to what Win tells it the number of procs are and tries to run a full CPU's worth of load on the HT. Not gonna happen.
Ah, but with Intel, now you have to choose between dual cores and HT (or pay a lot for the super gaming processor). And choose 2M over 1M cache over 2 processors with 1M each cache, et cetra. Even in the medium priced processors.
Experience here shows the servers I deal with running Linux 2.6 kernel/Apache/MySQL and dual Xeons up to 6GB is that turning HT on as well reduces performance. When a CPU fan failed and one CPU had to be temporarily removed, however, there was a clear benefit turning it on with the single processor.
Can anyone explain to me the exact difference between HT and CMT ? I'm wondering if these same issues would plague Sun's new Niagra prcessor.
Hmm interesting.
:)
I was talking about a single proc and HT,
I imagine that with dual + HT it is different. I do not see why it is happening. Actually if I bought an expensive server and experienced that, I might try to get some official explanation
for the problem.
I wonder If you tried BSD or Windows on the same or similar hardware, that might be some OS specific problem as well.
Hmm, Google on it I will.
If people want multiple of threads of execution on the same processor then they should get one with two cores.
If you read the article / summary you'd see what its talking about are servers that come with HT enabled by default. Thinking off the top of my head I can't come up with a single Intel processor still being sold and used in servers today that doesn't have HT technology built in. We're not talking about people specifically buying HT processors looking to get a performance boost, we're talking about every single individual who is buying or has bought an Intel server certainly within the last year or two. I've certainly noticed HT being enabled by default on our servers over the past couple of years. Novell has always strongly advised against it, so its always been turned off not long after they're powered up for the first time.
From the way the article reads, the software companies are expressing concern that HT is being enabled by default on server, rather than that they're baffled why its causing slower performance. It even goes so far as to point out that the shared L1 and L2 cache is the problem.
"Joy is not in things; it is in us." Richard Wagner
Quite interesting. So SQL Server spawns processes as opposed to threads when it finds a second processor? I can't imagine thats true. What exactly do you mean by a 'full CPU's worth of load'?
The only situation I can imagine is if SQLServer spawns say, 2 threads per CPU for performance. But this is a cheating way to get more CPU time and I wouldn't expect a _server_ class program to do such a thing when such a program would tend to expect its getting dedicated CPU anyway.
Ah, but with Intel, now you have to choose between dual cores and HT (or pay a lot for the super gaming processor). And choose 2M over 1M cache over 2 processors with 1M each cache, et cetra. Even in the medium priced processors.
The above (as you actually imply) is about single socket uniproc DESKTOP systems, not the servers (generally at least 2, 4+ socket) and server apps we're talking about.
As a matter of fact, both Intel's current dual socket, dual core cpu (Paxville DP) and the follow-on dual core Dempsey HAVE Hyperthreading. Please don't contaminate a discussion about servers with irrelevant desktop technology observations.
Software shouldn't be expected to handle hardware quirks. It's up to the hardware to run the software efficiently.
Seems to me a hardware fix would be to partition the cache into two pieces when HT is enabled and running -- use the whole cache for the processor otherwise.
With 2MB caches per processor now becoming available, would this be such a bad thing? IIRC once you're up to 256KB of cache you've already got a hit rate near 90%. That severely limits your possible improvement to less than 10% regardless of how much more cache you add. And yes I am aware that increasing the processor multiplier does make every cache miss worse in proportion, but still having HT run more efficiently in the bargain could make this tradeoff worth it. And that's even before you consider uneven partitioning if the OS can determine that one thread needs more cache than the other.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
is still a kludge. HT was a cheap hack to get extra performance under certain scenarios. Looks like their getting called out for it. Dual-core is the right answer, HT wasn't.
Quack, quack.
(erons). But the price makes them a hard sell. I'll definately be keeping my eye on these things, as soon as the price points start to line up. I want to see AMD suceed in the server market, but for now (aside from Sun and a few HP systems) Xeon is still the dominant player.
Quack, quack.
I have two identical high-end dual cpu desktops, both with HT enabled sitting on my desk. One runs win-xp, the other a 64 bit Linux. The thing I observe every day is how windows scheduler sucks. I don't know for how long marketing dept. of MSFT knows about HT, but their OS definitely doesn't know about it yet (start update in subversion or compilation in VC -- go to drink some coffee, as computer is unusable). On Linux, on the other hand, HT really improves both responsiveness and throughput. I'm waiting to test quad- dual-core box with HT enabled ;)