Slashdot Mirror
Benchmark Program Rewritten to Favor Intel?
BrookHarty writes "Interesting article over at Van's Hardware, that BAPCo the maker of the SysMark benchmarking program, has re-written its SysMark 2002 benchmark program in favor of Intels P4. AMD joined BAPCo in order to "correct" these "broken" results. AMD reports that BAPCo's SysMark 2002 (written by Intel Engineers) is a collection of tasks to summarize "Real World" performance. Interestingly, these tasks are selected for Intel's favored performance, while removing certain tasks that favor AMD. Vans Hardware has additional information on BAPCo's Shady history."
Obviously, the best bet for cpu benchmarks would be an open-source one compiled using a standard compiler. This is a case where open-source really shines.
GoatPigSheep, the 3 most important food groups
I don't think there is much motivation on the part of compiler writers to optimize for this particular implementation of the x86-32 ISA. This isn't like previous chips, where new cache handling opcodes were added, which compilers could use if available. I've talked to people much better versed in compiler writing than myself, and they all seem to agree, when it comes to "optimizing for P4", their answer is going to be "don't hold your breath".
I've had enough abrasive sigs. Kittens are cute and fuzzy.
it's plausible that the Athlon's performance is going to lag quite a bit more than it already does.
It doesn't. The fastest X86 processor is currently made by AMD. See a non-BAPCo performance comparison.
AMD has always been the value chip company. You can't expect them to keep up with Intel forever
Are you an Intel employee? Intel isn't keeping up with AMD. The P4 is underperforming, as well as overpriced. Take a look at the web page referenced in the lead story.
Intel has used the "once compilers catch up..." scam for years, and every time people find themselves with a long obsolete processor by the time the software the theoretically exploits it arrives.
My general practice is to ignore any synthetic benchmarks because they represent no real world value whatsoever: Instead I look to application benchmarks, like compressing divx movies or rendering 3D scenes, if that was the use that I had in plan for my PC.
For a year and a half AMD managed to blow Intel out of the water in the performance arena, from the time they both reached 1.0GHz with the P3 and Athlon, to the time Intel started releasing Northwood core P4s with the extra 256KB L2 cache. The longer pipeline of the P4 has nothing to do with performance. All it does is enable the processor to ramp in clock frequencies easily. A general rule of thumb is the longer the pipeline, the lower the IPC (instructions per clock/cycle) and the LOWER the actual number-crunching performance.
------- "From bored to fanboy in 3.8 asian girls" ----------
What this says is that SySMark is really poorly coded, not "optimized" to favour Intel silicon. Incompetancy isn't evil. It's just...incompetant. This would explain why most serious benchmark runs seem to lack SysMark these days...
------- "From bored to fanboy in 3.8 asian girls" ----------
I hate to say this, I am a big AMD fan, but it is partialy AMD's fault that SysMark favors Intel. They have refused to work with the BABCo people in the past knowing the Intel people have. Is it any suprise that they end up favofing the company that works with them over the one that ignores them? AMD is now supposeidly working with them to make the next version do a more fair job testing their proccesors. So hopefully this will be a non issue in future realeses. It would prob be most fair if AMD and Intel would both let the benchmarking programs be written with out either of their interfiernce, but if one is going to get invloved, then they both realy need to.
"Whenever you find that you are on the side of the majority, it is time to reform." -- Mark Twain
Coming from the Unix world, I'm used to comparing machines based on their SPECint and SPECfp performance...
/ cpu2000-20020506-01357.html
/ cpu2000-20020701-01441.html
In general the SPEC people have done a better job being platform agnostic than some of the "miscellaneous" PC benchmarks.
Current benchmarks for Intel http://www.spec.org/osg/cpu2000/results/res2002q2
and AMD http://www.spec.org/osg/cpu2000/results/res2002q3
Keep in mind that results for more recent AMD CPUs are not shown. If you compare the AMD 2200 with a 2.2G P4 you'll have 734 v's 784, which gives some credence to AMD's claimed rating.
html4me!
As compilers become tuned to exploit this, it's plausible that the Athlon's performance is going to lag quite a bit more than it already does. That there is some benchmark out there that is specifically designed to show off this strength of the P4 is no real surprise to anyone, is it?
That's not the complaint at all. Read the linked article. The complaint is that Sysmark 2002 has been systematically altered relative to Sysmark 2001 so as to favour the P4 over Athlon.
For example, the PhotoShop test in Sysmark 2001 had 13 filters, of which 8 run faster on the Athlon and 5 faster on P4. The Sysmark 2002 PhotoShop test has 6 filters, of which 3 are filters from Sysmark 2001 on which P4 wins and the other 3 are additions on which the P4 also wins. The 8 filters on which the Athlon does better have all been removed.
There are several other examples in the article. Read the article
BTW, an interesting point is that this whole thing is basically an AMD publication that AMD have chosen to proxy via Van's. Van is at least open about it. The AMD presentation containing all the information in that article is linked at the end and is available here
Here's Kyle's 4th Edition post from yesterday. Excerpts from Van's comments are in italics.
VansHardware & AMD: There is a report on VansHardware this morning that visits the differences between BAPCo's SysMark 2001 and SysMark 2002. The report's basic theme is that SysMark 2002 is skewed towards making the Intel Pentium 4 results look better than the AMD CPU results could have looked. It basically shows examples of things that were changed in SysMark 2002 that cherry pick areas in certain programs that the Pentium 4 excels at. While the article might seem to be work done by VansHardware there is something you need to know. All of the data shown in that article has been put together by AMD and not VansHardware. Take note of this one statement in the article.
However, AMD has been able to "pick the lock" on SysMark to gain a much keener understanding into the internal workings of these tests.
VansHardware is not the one with the "keener understanding", AMD is.
The original PDF document from AMD is linked for download so the fact that this data is not Van's is not exactly hidden either.
Also their opening paragraphs state this.
At this moment we will pause from the long march through our benchmark results to revisit the significant issues regarding BAPCo's SysMark 2002 brought up by AMD during our recent meeting with representatives from that chipmaker.
We must state up front that despite the condemning information divulged to us, the AMD spokesmen repeatedly expressed support and guarded optimism for the reformation of BAPCo.
The "significant issues" and "condemming information" shown were not harvested by VansHardware, actually all they do is interject a little bit of commentary.
AMD has verified to me this morning that all of the graphed and tabled data shown on the VansHardware report is data that has been mined by AMD. Does this make the data inaccurate? Of course not, but I am sure that it hardly shows both sides of the story. AMD is not going to supply VansHardware with information that makes Intel look good. VansHardware represents to me, nothing more than an AMD fansite that takes shots at Intel every chance they get. I think they are far from what anyone could consider objective journalist and reporters. Them doing a cut and paste job with AMD's data goes to show that as true in my opinion. Websites get fed information all the time, trust us, we know. It is our jobs to go back and prove data and claims in our labs on our own time, not to repost corporate data, that can be considered far from objective. Independent sites in our hardware community should not be reposting PR spin in such a way as this. There is a fine line here but I think this is stepping across it.
VansHardware does not exactly hide the fact that the data shown is not theirs but rather AMD's, but they certainly did not seem to represent that in an upfront manner so the reader sees the information for being exactly what it is...data released by the AMD PR machine.
I am a huge AMD fan but I just don't like big companies being able to pump their corporate data into our community when it is not presented as such. I think AMD should have the balls to post information like this on their own website and not try and "slip it in" through a back door. In fact, I would consider the information to be much more credible if it were posted on AMD's own website as AMD research.
I know Van has gotten upset here recently with his past employer removing his name from articles he has written. It seems to me that Van has done little to deserve his name being on this article and it should show authored by AMD.
(ED NOTE - This is referring to some allegedly plagiarised articles that Tom's Hardware published after removing Van's name from them)
Also worthy of mentioning is that AMD is now fully working with BAPCo, which they have not done in the past. AMD has had the ability to work with BAPCo for a long time now to make sure their products get represented properly and we are certainly happy to finally see AMD join the party to give the boat a more even keel.
Lastly, another tidbit worth throwing into the mix is that Van Smith, owner of VansHardware, possibly either works for or is contracted to VIA as a CPU validation tester. We are working on a confirmation of this from VIA now. Do we need hardware websites that do work for the companies they end up reporting on? Just another thing to consider when objectivity is in question.
"Mod, mod, mod...and another troll bites the dust."
Yeah random PC hardware fans can do "meta-benchmarking" and decide if the benchmarks are accurately portrayed.... now if they could only do something like that to the mods here....
When I dig through reviews on the latest CPU and/or mainboard, I initially groaned at the increasing number of benchmarks folks would put out. It is more than just increasing click-through rates (well maybe not for some, but...) - it lets me see applications that I use. Synthetic benchmarks and politician's promises garner then same level of trust from me.
Anyhow, I game and code but use games to judge where my cash goes. When the P4 came out, I saw it did great job with Quake and I started to get excited about the CPU. Then I saw the benchmarks on the games I actually play - UT, CS, and a few others - and it was not black and white. After the ATI fiasco, Quake is up there with synthetic benchmarks IMHO. As for Photoshop, you can pick what platform you want to 'win' by tuning the filters. Apple does it, their dually box wipes out the competition, the other do it and the tables are turned.
There are great graphs out there that show benchmarks using different sizes of data. Its like comparing a small turbo charged engine to a larger normally aspirated one - so what RPM were you at when you ran your test? BMW's M5 feels slower than an Audi S4 at the start, but get the RPM's up there and it is a different story. Even pickup trucks can beat a Ferrari if you tune the test to take advantage of a sweet spot.
I've done my homework, and my personal cluster is mostly AMD today. Still have one celeron 566@800 as a CS server, but my workstation (Intel Xeon box) was replaced by AMD MP chips. Secondary boxes are all XP chips, but they use to be PII&III's when Citrix and the K5 sucked. They run Oracle, Weblogic, LDAP, and other stuff quite well when I'm working, and one swap of a hard drive later I'm getting some solid fragging in on the same box. In another year or so, if Intel really hold the crown , the price is right, and my boxes are 'only fast enough for web browsing and email', I'll chose them.
+++ UGUCAUCGUAUUUCU
A couple of days after some Lawyers get together for a class-action suit alleging that Pentium IVs are slower than the AMD competition, new BAPCO tests 'prove' that the Pentium IV was quicker all along.
Nice one Intel. At the very least, this should muddy the waters with respect to which one is quicker being a matter of opinion.
I use a 7 Watt Via C3 as opposed to one of the 60 Watt P4/Athlone and do not really care either way.
Mielipiteet omiani - Opinions personal, facts suspect.
for non-commercial purposes, anyway
/Styx
I'm actually partly serious here, I think wider publication of more 'real-world' performance figures is in order. The people who frequent sites like Tom's Hardware and Anand are the only ones who really care about raw benchmark numbers. The rest of the world is more interested in getting their work done more quickly.
___
Cogito cogito, ergo cogito sum.
Nice try, but the link in question reveals a page which does not even have the name AMD anywhere on it. What it says is:
Van's Hardware Journal is a site dedicated to providing honest, accurate, hard-hitting, and thorough reviews and analysis of the latest in computer hardware and related high technology.
So you would trust a benchmark from a webpage that is in the midst of bashing another benchmark? A little biased, aren't we...
"We" aren't, but apparently you are. When a web page takes a critical look at benchmarks and exposes those that are biased, then I would tend to trust that site to choose an appropriate, unbiased benchmark for CPU comparisons.
Also, the story didn't imply this was a big deal. It only remids us of all the dirty tricks Intel is forced to resort to when they try to maintain a market lead with a grossly inferior product. As long as people know this, benchmark-cooking is really no big issue.
HardOCP notes that Vans got their info from AMD so it may be a bit biased. a quote from HardOCP:
" AMD has verified to me this morning that all of the graphed and tabled data shown on the VansHardware report is data that has been mined by AMD"
"AMD is not going to supply VansHardware with information that makes Intel look good. VansHardware represents to me, nothing more than an AMD fansite that takes shots at Intel every chance they get. I think they are far from what anyone could consider objective journalist and reporters."
http://www.kubuntu.org/
However, AMD has been able to "pick the lock" on SysMark to gain a much keener understanding into the internal workings of these tests.
Isn't that a violation of the DMCA?
Yes, filtering web access is just like raping and killing children...
Not exactly. There's more screaming when you filter web access.
Actually, both Tom's Hardware and Anandtech sites clearly mention during their testing that certain benchmarks favor the Pentium 4 due to things like SSE2 multimedia extensions support and better support in the test program for the way the Pentium 4 processes CPU instructions.
Hopefully, this will force BAPCo to do a benchmark from the ground up that will test functionality common on both the Pentium 4 and Athlon CPU's (like just MMX and SSE support only for now until the new Barton and Hammer core CPU's from AMD get full SSE2 support).
If AMD would stick to making totally Intel compatible chips instead of trying to infuse their own personality, we wouldn't have this problem. Hint: my software shouldn't need to know it's running on an AMD chip.
This is so wrong on so many counts...
1. Intel's chips aren't "totally Intel compatible". The Pentium 4 contains instructions that were not present in the Pentium, P2, and P3. Why should your software have to "know it's running on a" Pentium 4 rather than a P3, P2, or Pentium? Hell, there was even a Pentium and a Pentium MMX (the latter adding the MMX instructions).
2. Intel tries every trick possible to patent their instructions to prevent people from implementing them. They do it with hardware, too. Remember when you could plug a K6-2 in place of an Intel Socket 7 CPU? Starting with Slot 1, intel used patents to prevent others from making compatible CPUs, which is why AMD and Intel motherboards are now incompatible.
3. Why should AMD not provide useful processor extensions that improve on Intel's base instructions? That's what provides useful competition and makes the industry grow.
4. What interest do you have in seeing AMD in a constant catch-up mode? In your scenario, Intel gets an advantage every time they release new instructions -- that will take AMD months to implement in silicon. Do you own Intel stock?
5. Why doesn't Intel just stick to providing processors that are 'totally AMD compatible'?
Besides, AMD has always been the value chip company. You can't expect them to keep up with Intel forever.
AMD has had a superior (in design) processor architecture to Intel since the K6 was released (though the K6 had mediocre FPU performance, the design was still more elegant--ask any x86 assembly programmer). The Athlon has given the P2, P3, AND P4 a run for its money, and early benchmarks of the hammer would seem to indicate that the expensive Itanium 2, which almost nobody actually uses, is going to be outrun as well.
The Pentium IV's really looong pipeline does allow the P4 to run at higher clockspeeds, but the branch prediction you mentioned is instant death. Branch mispredictions happen VERY frequently in any CPU (note the K6 had the most sophisticated branch prediction unit up until the "XP" series of Athlons) but with the Pentium IV, a single branch prediction requires up to 20 full clock cycles of work to be discarded.
The Pentium IV has other questionable design desisions that hurt performance as well. It has 8K of L1 cache, the same amount found in the ancient 486 processor, whereas the Athlon has that amount squared and doubled (128K). Current P4's have more L2 cache, but L2 cache is less important and slower. (Note though that the P4's L2 cache is particularly fast L2 cache)
The P4 has buffers to remember a series of decoded x86 instructions so that it does not have to decode them again--these are almost required because of the terribly long pipeline--but it doesn't have enough to speed things up in server environments. Most servers execute a wide variety of instructions such that the buffered instructions get very little use before being replaced by new instructions. This is even more a problem on systems that run many different applications at once, but this problem can be demonstrated just with DB servers (which use plenty of instructions) as the P4 tends to not scale as well as the Athlon MP when a second or third task is added (such as mail serving, web serving, etc.)
One dissapointment that I had with the Athlon is that AMD never used the excellent EV6 bus to its fullest. Athlons are superior in multiprocessor capabilities because different processors needn't share access to the memory bus. On Intel SMP setups, even on P4 Xeons (Which, IMO, are inferior to P3 Tualatin chips by the same company) when one CPU accesses main memory, it locks main memory for the other CPUs. All other CPUs have to sit and twiddle their transistors while the main memory is on use by only one CPU.
On AMD SMP setups, ALL processors can simultaneous access memory, merely sharing the bandwidth simultaneously. So, if one CPU is only using 100MB of memory bandwidth, the rest can be used by other CPUs at that time.
Unfortunately, this doesn't really matter much with only two CPUs, which is the largest AMD configuration you can get. You can, of course, see it in action with 8+ CPUs on EV6 Alpha setups (AMD licensed the bus from DEC's Alpha team) but Alpha setups are expensive as hell and are a dying breed.
If AMD had created a quad or 8-way setup, we would see the true power of a good design.
Fortunately, the Hammer has an even better design (one made by AMD no less) on an even better CPU. I fully expect the Hammer series to wipe the floor with all Xeons and possibly the Itanium 2 because of its design. An integrated memory controller that will tremendously drop memory latency, twice as many general-purpose registers of twice the size (Much less pushing and popping, for those that know some assembly) and, unlike the big vendor 64-bit processors, the ability to split half of the general purpose registers into chunks of 16 and 32 bits when huge numbers (2^64) are not needed. (On an Alpha/SPARC/R12000, if you want to store the number "42" you must use all of a register that can hold values up to 18,446,744,073,709,551,615. A bit wasteful)
Computer Science is no more about computers than astronomy is about telescopes. --E. W. Dijkstra
Besides SPEC Benchmarks I don't trust any other ones.
They need a SPECquake.
> Benchmarking the various "Benchmarking Programs"
:)
Yes but, "Quis benchmarkiet ipsos benchmarkiem ?":
Who benchmarks the benchmarks ?
(s/benchmark/custod/g and Google for the original quote
-- don't discount flying pigs until you have good air defense
Wouldn't a better CPU benchmarks be taken by using the chipmakers' own compilers?
... almost the antithes of what Intel is trying to do. Has this been a longstanding strategy on AMD's part?).
No.
The chipmaker would simply then optimize their compiler for the benchmark(s) in question, rather than for code more generally. In other words, what you suggest would still allow the chipmaker to cheat.
In order to have complete transparency in the benchmarking, both the benchmarks and the compiler should be open source (ideally free software, so that anyone can run and verify the benchmarks as well, allowing repeatable experimentation in the broadest scientific sense). If the chip maker wishes to submit optimizations to such a compiler they would be free to do so, since any such optimizations would in turn be open source (or free software) and subject to peer review.
A good candidate would be gcc, which runs on numerous platforms, and on several operating systems on AMD and Intel hardware.
Cheating would be much harder in this case, perhaps even impossible, something we need given the sordid history of benchmarking by all parties involved (except perhaps AMD? Can anyone recall an instance where AMD has cooked results? I ask because their current chip rating system is extremely conservative
The Future of Human Evolution: Autonomy
does it means that sysmark`s benchmarks is to intel as arthur andersen`s audits were to enron?
It otherwise might, but this possibility may be somewhat mitigated by the reality that being 'in bed with finance executives' registers as just slightly less repugnant than the prospect of being 'in bed with benchmark coders.'
I don't think there is much motivation on the part of compiler writers to optimize for this particular implementation of the x86-32 ISA.
Over 80% of new computers being shipped are P4 based. Therefore, applications will be optimized for P4. Not optimizing your application for P4 would be look a hardware vendor not releasing a Windows driver for their device.
The source should be published, and compiled with a standard (GCC) compiler.
The CPU vendor, or any other compiler vendor can compile the same code, and publish the benchmark as well.
This then opens up the competitive market in compilers.
What I'd like to see is the same code, compled with cross cpu options, IE what happens when the code optomizes for Intel but runs on AMD. How much does that penalize you, useing the wrong optimizations?
As long as the compiler, and options used are disclosed, I don't see a problem.
The Pentium IV has other questionable design desisions that hurt performance as well. It has 8K of L1 cache, the same amount found in the ancient 486 processor, whereas the Athlon has that amount squared and doubled (128K).
Obviously you flunked your freshman-level computer architecture course. The P4 8K L1's 2-cycle load-use latency is 50% better than Athlon 128k L1's 3-cycle load-use latency (not even accounting for P4's clock speed advantage). The difference in hit rate between 8k and 128k is only about 5% meaning that it is substantially faster to go with the small/fast cache than the big/slow cache. Do the math - even an infinitely large 3-cycle load-use cache is slower than an 8k 2-cycle load-use cache.
Cache size comparisons are more meaningless than megahertz comparisons. Whenever somebody tries to justify a big cache size without looking at performance, just walk away. AMD is playing marketing games with their slow-as-molasses (but massive) L1 cache.
I won't bother to address the rest of the technical errors in your post...
Since the P4 creams Athlon in SPEC (both Int and FP), that means it's not good enough for Van/AMD.
Obviously you flunked your freshman-level computer architecture course. The P4 8K L1's 2-cycle load-use latency is 50% better than Athlon 128k L1's 3-cycle load-use latency (not even accounting for P4's clock speed advantage).Obviously you are imagining things, as I never said that was not the case. Latency is important, but it doesn't matter if the cache size isn't large enough to fit enough code in to enjoy the low latency.
The difference in hit rate between 8k and 128k is only about 5% meaning that it is substantially faster to go with the small/fast cache than the big/slow cache.
Really? That's interesting, and here's me wondering why both AMD and, other than in the P4, Intel have wasted so much money adding more cache memory.
Because you seem to be such an expert, so why don't you go ahead and list a few common programs for me that have a working set of less than 8K--the size that will fit into the tiny L1 cache. Can't find any? Gee, I guess that makes the size of the cache pretty important then. When a program's working set has to be swapped in and out between L1 and L2 cache, suddenly that latency doesn't much matter. Of course, you may feel free to prove to me that the P4 can run addition loops faster. Those will fit into about 8k.
Do the math - even an infinitely large 3-cycle load-use cache is slower than an 8k 2-cycle load-use cache.
Who was it again flunked their freshman computer architecture course? You're saying that if the Athlon had 512MB of L1 cache that the system would be slower than the P4 and it's 8K of lower latency cache?
What math is it that I should do? Do you know what the working set of a program is?
Having a tiny amount of cache is analogous to having a tiny amount of RAM. Put 32MB of low-latency RAM in your system. Overclock some DDR SDRAM to 200MHz (AKA "400MHz" by people that don't understand clock speeds) and set it to CAS2. Tell me how your system performs. Just as your system will have to swap just about all running code to disk, the Pentium IV will not be able to contain the core loops of the various running programs in L1 cache. The vast majority will have to be dropped to L2, which is significantly slower and higher latency, kinda defeating the purpose of that 8k of fast memory, no?
Working sets that cannot be fit into the P4's 256k or 512k or L2 will then be relegated to main memory and moved to L2 then L1 when the data is executed, and anything that won't fit in main memory (very rarely which includes the working set of a program) will be swapped to disk if the platform supports virtualizing memory.
In closing, your comment was surprisingly brash and conceited, not to mention rude and totally innacurate. Thankyou.
Computer Science is no more about computers than astronomy is about telescopes. --E. W. Dijkstra
Apples to apples please - when you clock the Athlon at the same clockspeed as the Intel chip, (which are possible with the new chips that AMD just released) the FPU is far faster on the Athlon chips.
This is what indicates a superior FPU design, not a comparison based on a ~700mhz difference in clockspeed.
"Nothing strengthens authority so much as silence." - Charles de Gaulle
All benchmarks favor whoever requested they be written. So its a crap shoot, choose the one you want to believe in, then go do you own tests, Your own testing is the only tests that matter.
According to Hennesy & Patterson, 2nd Edition, page 391, the total miss rate (for SPEC92) of a 8k 4-way set associative cache (like the P4's) is 2.9%. The miss rate of a 128k 4-way set associative cache (like Athlon's) is 0.6%.
The hit time for P4 is 2 cycles, and for Athlon it's 3 cycles. The L2 hit / L1 miss is ~10 cycles for both. Everything further out is approximately the same so we can ignore it for simplicity.
So, the average memory access time for P4 is (0.971 * 2) + (0.029 * 10) = 2.2 Cycles. The average memory access time for Athlon is (0.994 * 3) + (0.006 * 10) = a little over 3 cycles.
Suppose Athlon had an infinite size L1 cache (or 512 MB if you like to use numbers). The highest hit rate it could ever achieve is 100% (actually slightly less, since you cannot eliminate complulsory misses). The average memory access time would then be 3 cycles - which is higher than P4's 2.2 cycles!
BTW, Paul DeMone wrote a pretty good article about P4's L1 cache.
"impeccable"?! I dunno about that!
SPEC CFP2000 is by far the most widely supported benchmark, but it has it's share of flaws, just look at Sun's latest SPEC CFP2000 scores and you might notice one of them. For the Sun Blade 1900 (900MHz USIII), Sun had a rather abysmal CFP score, with the Sun Blade 1900 Cu (900MHz USIII), they were suddenly quite competative. Hugely improved compiler right?! Wrong. Actually they only improved by 0-15% on most benchmarks (about what could be expected with a slightly better compiler, ie what you could expect in the real world), however their CFP score improved by over 50%. Why is that? Well ONE single sub-bench of CFP (179.art) was increased by a remakable 560%! The end result is that the USIII looks like a rather fast chip at floating point, when it's actually butt-slow (WAY slower then either the Athlon or the P4) at everything except for one particular Fortran application. This isn't to say that Sun's score is invalid, simply that it doesn't take much to really skew the scores of ANY benchmark, including SPEC CFP.
I've also critized SPEC in the past for being a bit to focused on large datasets. For example, the 171.swim benchmark is essentially a memory bandwidth test taken from some shallow water modeling code. Now, that isn't to say that memory bandwidth isn't important for floating point and scientific applications (if it weren't IBM wouldn't have thrown a whole boatload of bandwidth at their Power4), simply that SPEC seems to have over-emphasized this aspect. I think the reason for this is that SPEC CPU95 was just the opposite, and often critized for emphasizing SMALL datasets too much, and therefore become almost completely a test of the cache architecture of the processor in many cases. For CPU2000, it seems like they over-corrected in my mind.
Long story short, if your floating point work involves small datasets, the Athlons MUCH larger L1 cache and 3 floating point execution units will crunch the code much faster. On large datasets, the P4's higher bus and memory bandwidth and larger L2 cache will crunch the code faster.
You're quite correct in that saying that the P4 has a weak FPU is definitely wrong. The chip actually is VERY good at floating point calculations, better then just about anything out there. Only the Power4 and the Itanium2 are decidely faster FP number crunchers then the P4.
Actaully AMD had the performance lead for longer then that. They took the performance crown away from Intel the day that the Athlon was first released, since it came out at 650MHz when the fastest PIII was only 600MHz, and the Athlon was, at that time, just slightly faster clock for clock then the PIII. AMD kept the clock speed lead and increased their clock-for-clock performance over the PIII up for the next few years. Intel only just started to catch up with the P4 2.0GHz (which was released just a little bit before the AthlonXP, ie when the fastest Athlon was only at 1.4GHz).
Basically for the last 3 years (since the release of the Athlon), AMD has had the fastest x86 chips for about 2 years. Intel has had the fastest x86 chips for about 6 months, and for the remaining 6 months it's been too close to tell which was faster.
As for the P4s long pipeline, I'd say that it WAS largely responsible for increasing the performance because it allowed Intel to clock the chips so damn high. They clocked the P4 up to 2.0GHz easily on a 180nm fab process. Compare this to the PIII which they struggled to get up to 1.13GHz on the exact same 180nm process (and that took them until 1 year after their first attempt failed miserably and had to be recalled completely). AMD did slightly better with their Athlon design, but it still was only able to clock up to 1.73GHz on a 180nm process, and they had a more advanced process then Intel did in some ways (ie they were using copper interconnects).
Long story short, performance is determined (in an overly simplified way) by IPC * clock speed. With the P4, Intel looked to sacrafice IPC slightly to dramatically increase the clock speed with the goal of overall faster performance. When compared to the PIII at least, they definitely succeeded. Compare the fastest 180nm process PIII (1.13GHz) to the fastest 180nm process P4 (2.0GHz) and which do you think is faster?
Thankyou, that was a much more informative post than your previous one, and probably more informative than mine as well. :)
/slower/ on the P4? Eh?
Regarding the comment about infinite cache, your message seemed to imply that latency was all that mattered, and that an Athlon with a huge (say, 512MB) cache would be slower than a P4 with it's 8K of faster cache. Seeing as how hust about everything would be run entirely from L1 cache on the Athlon, that seemed rather silly. Perhaps I misinterpreted what you were trying to say.
Indeed, the L1 cache of the Pentium IV is extremely fast. The benchmarks of handling data sizes of 8K are astonishing, but size does matter as has been seen by increasing the L2 cache of the exact same Pentium IV.
As I understand it, Intel chose an 8K cache both to reduce transistor count and to reduce L1 cache latency, which worked, but is of dubious real value. (it is difficult to tell for sure unless Intel makes a P4 with a larger, slower cache, which I doubt)
The purpose of my original post wasn't really to focus on cache size vs. speed, but to highlight questionable design decisions made with the P4. There are other problems (what I consider problems) with the core as well, which I didn't highlight. For example, why is bit shifting so incredibly slow on the Pentium IV? It's faster on just about any other Intel or AMD processor and has long been recommended as an optimization for doing multiplication/division of numbers whose values neatly fell on bit boundries (256, 65,536, etc.) and now this optimization can actually make code
Looking at the execution time of various instructions, the P4 has taken quite a few other steps backwards as well.
I remember when people were criticizing AMD for calling the Athlon a 7th generation processor. The rationale was that the P2 was about 30% faster per clock than the Pentium Classic, and the Pentium was about 30% percent faster (this all depends on the code of course) than the 486, yet the Athlon was not 30% faster per clock than the Pentium 2 or K6.
I wonder where all of these people are now that Intel's "7th generation part" is not only not 30% faster, but is actually 30% slower in most tasks, on average.
Strange world we live in.
Computer Science is no more about computers than astronomy is about telescopes. --E. W. Dijkstra
Apples to apples please - when you clock the Athlon at the same clockspeed as the Intel chip, (which are possible with the new chips that AMD just released) the FPU is far faster on the Athlon chips.
Not that it matters, but P4 and Athlon performance are approximately equivalent on a per-clock basis: Athlon does 624 / 1800 MHz (0.346) "SPECfp points per megaherz" (whatever that means), and P4 does 861 / 2533 MHz (0.399). Of course since P4 can clock much faster it has better absolute FP performance.
This is what indicates a superior FPU design, not a comparison based on a ~700mhz difference in clockspeed.
Let's throw Itanium 2 into the picture, which does a whopping 1356 SPECfp at an equally astonishing 1.0 GHz, which puts it at this silly "SPECfp points per megaherz" of 1.356 - quadruple that of Athlon/P4.
Does that mean Itanium 2's FPU is four times better than Athlon?
I could swear I've heard all this before, like maybe here: http://www.apple.com/powermac/processor.html
While I will admit that apple does skew their tests in their favor, they have been saying this for years, AMD realized it a while back when they switched to an alternative numbering system. Only intel continues with the idea that Mhz is the only measure of a processor.
Personaly speaking, I think processors, in theri current 32-bit forms have essentialy hit a brick wall. But Intel doesn't want to admit that yet becasue they've invested so much into making people believe in Mhz, that to try to convince people that 1ghz is as fast as the 2ghz they bought last year would be nearly impossible.
T Money
World Domination with a plastic spoon since 1984
Both of these companies spend *billions* of dollars on producing these processors. Both companies run lots of simulations to determine what design choices best fit with the rest of their design. When you're spending that much time and money developing these CPUs, you can't afford NOT to consider every option.
When it comes down to it, both AMD and Intel have really good engineers, and both companies listen to them when figuring out how to build cpus.
So consider that the P4's 8KB L1 trace cache is so small because that's as big as they could make it while keeping the latency down to 2 cycles-- something that was critical to keeping their double pumped ALUs busy (and thus their IPC up as much as they can)-- and that they could compensate by working a bit harder on a fast L2.
Perhaps AMD decided that they could live with an extr cycle of latency in the L1 because they have enough instrucions in-flight that blocking on a cache access wouldn't hurt them as much as a low hit rate would.
Or, perhaps there are multiple sweet spots in size/hitrate, especially when you factor in die size and cost. Honestly, I don't know the reasons why they made these decisions, and I'd love to find out why-- but I have 100% confidence that all the options were carefully considered.
When it comes down to it, both architectures are performing really well! And for YEARS, they have been competitive with each other. So while you may have your favorite (I, for example, think the P4 SMT and trace cache stuff is pretty neat), you've got to realize that zealously promoting one over another just makes you look silly.
Cheers!
-Ed
I never have and I never will trust benchmarks. Just like I will never trust any specs that a company puts out untill I try the machine for myself. It's one of the reasons I use Apple computers. As I type this, I'm on a custom built PC, with an AMD Athlon XP 2000 processor. It has 512 MB of memory, an Abit KR7A-133 mo-bo and plenty of other bells and wistles. Now theoreticaly speaking, this computer should be a hell of a lot faster than my 300Mhz iBook. Yet for everything except playing UT, thsi computer seems either equivilent to or sometimes (especialy right after startup) slower than the iBook. My point is, don't trust the numbers, use it yourself and decide for yourself. You're the one using the computer, and 90% o fusing a computer is how it feels to you.
T Money
World Domination with a plastic spoon since 1984
point 1.
;-)
/dislikes for both companies...but CURRENTLY, i'm going to recommend amd 90% of the time. What most Intel FANS (read: biased) DON'T UNDERSTAND, is while they are bashing AMD, if it were not for AMD they would be paying a lot more for their precious little Intel processors. Very likely around $800-$900 for a dinky 1 Gigahertz P3 right about now. How can you read slashdot, and know how important competition is and still be an Intel cheerleader? They must be ignorant or stupid.
amd is, mhz for mhz, dollar for dollar, pound for pound (the currency & weight
is THE FASTEST x86 cpu out there.
If I give you $300 and tell you to go buy the fastest x86 cpu you can for the money....it's an AMD cpu... period.
point 2. i own 2 amd based systems, and about 7 intel based systems. i'm not biased, i have likes
Point 3. it's simple. a competitor like AMD, simply means that if you prefer AMD, you are getting a damn good processor with a LOT of value. If you prefer Intel, while not quite the value of AMD,you are still getting a very good cpu, with good value. Because of AMD, we the consumers are enjoying greater value.
Without AMD, I can guarantee you that the current situation would not exist.
Intel would definitely be the equivalent of Microsoft....a monopoly. Trying to control all aspects of the industry, charging rediculous prices, trying to build their own 40 billion in liquidity.
The P4 still has quite a performance lead. I just took your advice and checked the 2600+ benchmarks (on tomshardware). The Athlon 2600+ only beat the 2.8 GHz P4 in 3 of the 28 tests. The Athlon beat the P4 in just under 10% of the test- that doesn't sound like the P4 is a "grossly inferior product" to me. And BTW- those benchmarks don't include the SysMark tests that are in question here.
Intel's performance lead is even more impressive when you realize that the 2.8 GHz P4 is available to customers starting Monday, but the Athlon 2600+ won't be availble to the OEM's for another month (and even longer before customers have it).
"The defense of freedom requires the advance of freedom" - George W Bush
I've been saying that as software gets optimized for the P4, it'll start seeming to be the faster processor - Look at Lighwave 7, 3dsmax 4.2sp1, and now this. :)
if(cputype==AMD) {
sleep(1);
}
I bought 3 athlonXP motherboards and an athlonXP1800 last spring. I threw out every single motherboard and the cpu is in my closet. I have downgraded back to my pentiumIII. Why?
I had major stability problems and the athlon burns too hot. First I relized my old powersupply did not have the power to even boot these babies(300 w), I first had to buy a new powersupply, then new ram, then a new case. After this my first motherboard would boot and then crash after it was done posting. I returned it for an abit which was extremely buggy, and my netgear nics wouldn't work, it blue screened multiple times whenever I used my geforce(well known bug), and the apic controller did weird things in linux and sometimes would not shut down properly. It then died totally 2 weeks later. It was exchanged for an msi board that supprisingly worked as expected, well at first. The system would not shutdown properly under linux but I didn't care that much. I then noticed the cpu kept reaching above 45c which could damage the chip over time. A third problem was whenever I enabled the nvidia opengl patch the system would crash quite often. The problem went away if I disabled it. I built the drivers from scratch so it was the right version for my kernel compilied from source. I then spent $60 for the top of the line cpu cooler. I had to use some force to get it to close on the cpu and my screwdriver flew off the cooler and damaged a chip on the board!
The guys at the computer shop said they would only exchange my msi for another exact ones and they were becoming agravaited at me for obvious reasons. I was so angry I just said f*ck this sh*t and didn't bother to replace my motherboard. Even if I get the board replaced, I am faced with the same problems and bugs! I admit I broke the last motherboard and it was totally my fault and not AMD's but after a month and $750 dollars later, I did not care. DDR ram only worked with athlons at the time and I now had two cases and powersupplies which I did not need. I felt like a sucker who just flused $750 dollars down the toilet when I downgraded back. Why should I have to put up with that crap? Anyway the first board was probably defective and the second should of not even left the manufactoring plant. I did look up my bugs online and there were many pissed off consumers who had the same problems with the same exact sets of hardware, so the abit one is a piece of sh*t. Many early chipsets for the athlon processors are buggy. Especially VIA's. There is even a well known linux/nvidia/amd bug that can crash your system if you do any opengl which plauged my msi system.
Are there stable bugfree wonderfull athlon boards out there? Yes I am sure there are. I am not trying to start a flamewar here but rather just share my experience with them. I am thinking about finally getting rid of my pentiumIII. This time with a guinine intel processor.
Is it the fastest or cheapset? No. Do I care? no. I want something close to the fastest that will work with my case, work with my power supply, work with all of my perihperals, and be reliable. All the benchmarking websites do is show how fast the chip is. I want to know how reliable it is. I can find some vendor references to overall reliablity but the boards vary from chipset to chipset. If I put down big bucks to upgrade my system it better well work and keep working for a long time! There is a similiar arguement about buying an expensive sun box over a cheap lintel one. You get what you pay for. This is why many bussinesses choose intel overwhemingly over AMD.
Most of the reliablity problems which plagued the first generations of athlon processors are gone and I admit the first intel 810 chipsets were terrible but there are less bugs with intel chipsets overall. I am willing to spend $250 more this time and feel at ease and look forward to use it for a long time.
http://saveie6.com/
Which was the compiler company that wrote into it's compiler the ability to recognize a common benchmark that didn't require output, and just converted it to NOOPs? Wow, did that computer ever chomp on those NOOPs fast...
Benchmarks measure speed on benchmark code. It's like horsepower in a car - a car with 300 horsepower isn't necessarily faster than one with 280, or even 200. It just depends, man.
What I meant was in terms of the itanium. LIke I said, current processors have hit a brick wall. There isn't much farther they can go without some new technology. So we have new tech, AMD is working on 64 bit chips, IBM has the new Power4 chips (I think that's what they were, I'm tired right now) for the macs and pcs and then there's the itanium stuff from intel. One of the reasons intel hasn't and won't for a while pushed the itaniums is that it would be nearly impossible for them to convince their consumers that the chip with a lower Mhz is actualy better than the ones they have now. It's the same battle that Apple and AMD are fighting, except Intel has to fight it against themselves
T Money
World Domination with a plastic spoon since 1984
There's a reason it's called benchmarketing.
There's also a reason that the credible organisations (TPC, SPEC, etc) have to regularly tune thier benchmarks to work around manufacturer cheats.
Is Intel forcing anyone to use BapCo benchmarks? Aren't there dozens of websites out there with their own? Just look at any chip review on Tom's Hardware or Anandtech, Sysmark is only one of many benchmarks used.
Is this a surprise? Are people actually "outraged"? Please. Are you trying to tell me AMD has never hand-picked their own benchmarks?
Any CPU manufacturer will pick the benchmark that makes them look best but try to play it down, and then use that score extensively as propaganda.
Just look at when Apple used Bytemark to claim a 400% performance boost over intel chips -- of course, they used the 486-compiled version on a P6 core, but they didn't let that little nugget of disniformation get in their way.
Intel, AMD, Apple -- all whores.
https://www.accountkiller.com/removal-requested
SPECfp benchmark is highly dependent on the memory bandwidth. P4, with two channels of RDRAM, has more memory bandwidth than Athlon with a single channel of DDR. Thus, P4 gets a free ride here. Show me an apples to apples comparison here: P4 and Athlon with the same memory (be it SDRAM or DDR). I have yet to see one.
For a truly FPU-intensive benchmark that does not depend on memory bandwidth much, check out 3D Sudio rendering (tomshardware & others have it). Athlon simply smokes P4 on that one. Incidentally, on this benchmark P3 1000MHz outperformed P4 1500MHz.
___
If you think big enough, you'll never have to do it.
he's just trolling for his employer. He always does that.
___
If you think big enough, you'll never have to do it.
The situation is not quite as dire due to P4's trace cache (you actually addressed that later in your post). Nevertheless, your point stands.
On Intel SMP setups, even on P4 Xeons (Which, IMO, are inferior to P3 Tualatin chips by the same company) when one CPU accesses main memory, it locks main memory for the other CPUs. All other CPUs have to sit and twiddle their transistors while the main memory is on use by only one CPU. On AMD SMP setups, ALL processors can simultaneous access memory, merely sharing the bandwidth simultaneously. So, if one CPU is only using 100MB of memory bandwidth, the rest can be used by other CPUs at that time.
P4 Xeons (as well as P3s) have a shared memory bus. That is, multiple CPUs share the bandwidth of the 400MHz or 533MHz bus when accessing memory. However, Athlon has a point-to-point channel for each CPU. That is, each Athlon CPU has the full bandwidth of the 266MHz (soon to be 333MHz) memory bus, regardless of how many CPUs there are in the system. This means that beyond 2-way SMP systems, Athlon has a significant advantage in memory bandwidth over P4.
___
If you think big enough, you'll never have to do it.
Hoser, the first Athlon was a 500 Mhz version of the Slot A. I know because mine is still running sweetly after all this time. I do have a new Athlon 1400 though and it is mighty nice too.
Heh, I see no reason to pay any premium to Intel for a processor with no immediate benefits.
Anyway, nice post.
Codifex Maximus ~ In search of... a shorter sig.
Here are part 1 , part 2 , part 3 , part 4 and part 5 . Pabst's accusation is that Smith and Bennett have both written articles where they claimed to have discovered flaws in the benchmarks that make one manufacturer's product look good, when they were really being coached by the that manufacturer's rivals.
Here is Smith's rebuttal .
Van Smith used to work for Tom Pabst. In my opinion the quality and utility of tomshardware.com has gone down since Van Smith departed.
And, about this fight, I would say that Dr Pabst (he is an MD) hasn't learned the value of civility. In my opinion, in a fight like this one, people can't really follow the details, so they base their assessment of who is right, by looking to see who remains more civil.