Slashdot Mirror
AMD Rips 'Biased and Unreliable' Intel-Optimized SYSmark Benchmark (hothardware.com)
MojoKid writes: AMD is making a stink about SYSMark, a popular benchmarking program that's been around for many years, and one the chip designer says is not reliable. Rather than provide meaningful results and information, AMD claims SYSMark unfairly favors Intel products and puts too much emphasis on strict CPU performance above all else. John Hampton, director of AMD's client computing products, explained in a video why SYSMark itself is an unreliable metric of performance. He even brought up the "recent debacle" involving Volkswagen as proof that "information provided by even the most established organizations can be misleading." Salinas says SYSMark's focus on the CPU is so "excessive" that it's really only evaluating the processor, not the system as a whole. In comparison, PCMark 8 probes not only the CPU, but graphics and subsystems as well. In an attempt to drive the point home, AMD ran a set of custom scripts it developed based on Microsoft Office and timed how long it took each system to complete them. The Intel system took 61 seconds to finish the benchmark versus 64 seconds for the AMD platform, a difference of about 6-7 percent and in line with what PCMark 8 indicated, though Sysmark shows a stark delta of 50 percent in favor of Intel with comparable CPUs.
Intel's a member of BAPco, the SYSmark organization, and AMD isn't.
https://bapco.com/about/
On the other hand, if it's really a big deal to AMD, they should be able to find $100K or so to join BAPco and tilt the deck in their favor - total annual budget only seems to be $400K.
http://www.faqs.org/tax-exempt...
SYSMark has been known to be not particularly representative of actual performance for quite some time. In particular, they seem weirdly sensitive to memory latency, way beyond its actual impact, yet they deliberately evade caching even in benchmarks measuring something where caching is normally useful. And they do seem to unreasonably penalize AMD chips, although I'm not sure if that's malice or simple incompetence.
The review sites I frequent tend to use PCMark for the general-overview synthetic benchmarks, along with some actual-program benchmarks (usually compression, crypto, and video encode). I of course prefer the latter - nobody runs synthetic benchmarks in production, it's always some actual application. The closer you can get to benchmarking that actual app, the better.
The real outrage should be that operations in Microsoft Office are measured in seconds and minutes instead of nanoseconds and milliseconds.
I agree with AMD that most of the market is slanted towards Intel.
But I don't want a benchmark score that is dictated by a graphics card and it's driver set. I want a cpu score that is based on CPU performance, only CPU performance and perhaps taking into account the effects of memory memory bandwidth. Plenty of tools on the internet for that they could have showed instead.
What I want from AMD is a cpu in the 150$~ range with a performance equal or exceeding my old overclockedi5-2500k (2011 vintage) and be capable of gnarly overclocking. If they can deliver that, they have me for an entire socket generation.
The reality is for desktop and gaming workloads AMD's upper shelf products really are a better value. You will get a lot more value putting the savings on the processor/chipset into other system components like a little faster SSD or another 4GB of memory. The performance delta between Intel and AMD looks bigger in some of the benchmarks than it is in real world applications. The vast majority of desktop/laptop PC users don't benefit from Intel's premiums and lets call it superior "strait line performance" as for most applications the situation is I/O bound.
Which is NOT at all to say that if you have more specific workloads like you do a lot video processing, certain types of simulations, etc not to pick Intel.
If you are just getting a PC to do "generically everything and anything" with the A-series systems are a great value. If you have little more to spend FX + discrete video is still probably a better value than an equivalent i3/5 + video on the Intel side. If you really need an i7 for something than by all means get an i7.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
If I benchmark a common office task, like typing War and Peace into Microsoft Word, I find that these new processors run no faster than an old IBM PC/XT with a 4.77 MHz Intel 8088.
"National Security is the chief cause of national insecurity." - Celine's First Law
I hate to break it to you, but Intel's been cheating on benchmarks since the late 70's. There have been countless stories through the years about how their compilers and processors have detected that they're running Whetstone, or SPECmark or whatever, and done something out-of-band as a result. There have also been numerous attempts by them to game the benchmark standards themselves. Think the Volkswagon diesel cheat on steroids -- Intel is famous for it.
When you say you're buying Intel for performance, you're saying that you're really buying Intel for brand-name recognition.
I used to run AMD's consumer benchmark group during the K6, K7, K8 days. I'm not sure what you mean by "unbiased reports", but I can tell you that the process the company went through to create and execute benchmarks that were unbiased was remarkably fair. In the time I was there, the company ran benchmark results for any application that met three key requirements:
1) repeatable results
2) relevant software
3) practical to benchmark
So this meant that using canned benchmarks from applications such as Winstone for MS Office applications was a great option to look at office productivity software. We spent a lot of time trying to figure out how PC Magazine was weighting the application between the various MS Office applications, and I hit upon a way to do this by changing core frequency during benchmark runs so that we could create a multi-dimensional array of scores vs. frequencies to determine that Word was x%, Excel was x+5%, etc. We came up with a likely weighting scheme, although I don't recall what became of that work. In the consumer space, the other big hitter is obviously games. At the time of my tenure, AMD used many or most of the same gaming applications that were en vogue with Firing Squad, Toms Hardware, Anand Tech, Sharkey Extreme, etc. There was nothing nefarious about the work we did, nothing unbiased. We looked at these applications with equal weighting and determined that for a given frequency of relevant, competing Intel CPUs, there was an AMD offering that on balance, performed equally or better at a lower frequency. This processor was then given a model name such as 1800+ that was meant to convey it compared favorably to an Intel 1.8GHz CPU. In the days that my group did this work, AMD made a point of publicizing this process and went so far as to have the process vetted via direct supervision of a 3rd party auditing company who was one of the big-4 industry auditors. It was painstaking work to demonstrate that software load order and procedure was identical for AMD and Intel parts. When a benchmark completed, we showed the score to the auditor. Sometimes benchmarks returned imperfect scores because of a stray hard disk latency event and would throw the score off for either product. We would work with the auditor to show that the result of the otherwise repeatable values was an outlier and subsequently toss it in favor of another run.
Others in this Slashdot post have complained of heat dissipation. My team was solely concerned with instructions per second and performance per watt was not a concern for us. I do vaguely recall that this may have been a factor for the server team. My guess is that based on reading the occasional tech article here and there, AMD has made some important progress on power management.
Well not really, since Intel has put cheap dual core, dual thread CPU on the market that have an incredibly high single thread performance. e.g. Celeron G1820 is cheaper than A6-6400K, and Pentium is cheaper than A6-7400K (or same price)
AMD is good somewhere at the mid range, and if you're not gaming.
My work desktop is AMD, my home fileserver is AMD, and both my parent's desktops are AMD. That's because in those use cases, AMD is "good enough". Web browsing and email don't require a lot of horsepower.
That said, my gaming/transcoding PC is an Intel i5-4690, because AMD's top line CPU can barely compete with Intel's I3 line. CMT didn't pan out, and they've been held hostage by TSMC/GloFo's failure to produce a sub-28nm lithographic process.
I love AMD's engineers, they have some impressively smart people working for them, and I hope Zen + 16nm heralds a new beginning for them. But today, they aren't "competitive", merely "good enough".
I like AMD and all, but we're talking about a company that ships water coolers with their i7 competitors ( the 9xxx line). My brother's running his i7 with a stock fan... I'd love to be proven wrong but right now AMD just doesn't seem like they can hang. My A10-5800k is nice and all but in games it's about the equivalent of a mid range i3, but I can replace that i3 with a 5 or a 7... With the AMD the best I could do is an 8350...
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Celeron G1840 here. Never seen a slowdown in the games I'm playing. The limit is always the GPU and the HDD.
I watched a lot of videos and perused a lot of benchmarks before I went ahead and went AMD anyway, but I have to say that while I agree with you in principle, the gap between AMD and Intel is actually pretty big if you measure pessimistically. If you're counting dollars per maximum FPS, then AMD comes out well ahead. If instead you count dollars per minimum FPS, look at the worst cases instead of the best-case or the average, then Intel doesn't look so bad. The minimum frame rates are where you get your ass kicked because you can't react quickly, and that's where the AMD chips do most poorly.
With that said, I have an octocore AMD system with 16GB of RAM and it can do a whole bunch of stuff at once without choking, which was my plan. If I want to keep a VM running in the background while I play a game or something, the system doesn't even notice. For my purposes, this was the right move. It was the first machine I've built out of more or less all new parts in ages, and this FX-8350 is ticking along wholly reliably at 4.4 GHz with a $25 air cooler (up from the $20 air cooler in my 1045T system... fancy.) I'm not hating on AMD. But if all you care about is gaming performance, then Intel is probably a better choice.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
While an AMD chip can run x86-64 code compiled for an Intel processor, it isn't surprising that the code doesn't perform as well since a lot of optimisations relate to features of the specific chip. You can't use a precompiled binary across all chips and expect them to be useful other than to say one chip can run that binary quicker than another. I remember years back having some code that was optimised for the Intel PIII and when that same code ran on the AMD Opteron it was slower despite the Intel running at a clock speed of 1.4Ghz and the AMD running at 1.7. Once I went in and had a look at the ASM I could see why - the AMD had a 64 bit bus and the code was using instructions which weren't as efficient on AMD's chip as a result of this. Once I realised that, I rewrote that section of code to account for this and the AMD ended up being 30% quicker than the old code when I rewrote four lines of C. Compiler optimisations only go so far but you still have to be aware of the underlying chip if you really want to get the most out of it.
"I have the attention span of a strobe lit goldfish, please get to the point quickly!"
I read an article lately where they tested a whole pile of games against different CPU's. The results showed beyond your average 2-3 year old 4 core i5 there is little if any advantage for gamers for spending more money on a CPU. Your average gamer would be better of buying a better graphics card (or two), SSD or motherboard than buying an expensive/new cpu.
Rule 1 (good to a first approximation): All benchmarks are meaningless.
Rule 2 (for experts only): Every benchmark measures something very specific. A benchmark is only meaningful if you know exactly what it is measuring, and the thing it measures is something you actually care about.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
First of all, I said for anything BUT gaming AMD is a better processor. And the main reason for that is that while the each individual core is slower, it's true 8 cores (vs Intel's 6 cores on 5930k or even 4 cores which was all Intel could do for you when AMD's 8350 came out). The only way to actually use performance this high is to run poorly designed algorithms. Otherwise, the IO is much more of a bottleneck then the processing. The main reason Intel is better for gaming is not that it has faster cores. It's asymmetric L1 cache. AMD has symmetric (same input and output amount of L1 cache). So for most things that people use computers for (multitasking rather than really fast performance of small-thread-count tasks) AMD is a smoother experience.
Any guest worker system is indistinguishable from indentured servitude.
Gee, I wonder why AMD waited so long to attack that evil evil Sysmark?
Your sentiment would be something more people could get behind if AMD hadn't been bitching about Sysmark for the best part of the last 10 years.