Slashdot Mirror
AMD Rips 'Biased and Unreliable' Intel-Optimized SYSmark Benchmark (hothardware.com)
MojoKid writes: AMD is making a stink about SYSMark, a popular benchmarking program that's been around for many years, and one the chip designer says is not reliable. Rather than provide meaningful results and information, AMD claims SYSMark unfairly favors Intel products and puts too much emphasis on strict CPU performance above all else. John Hampton, director of AMD's client computing products, explained in a video why SYSMark itself is an unreliable metric of performance. He even brought up the "recent debacle" involving Volkswagen as proof that "information provided by even the most established organizations can be misleading." Salinas says SYSMark's focus on the CPU is so "excessive" that it's really only evaluating the processor, not the system as a whole. In comparison, PCMark 8 probes not only the CPU, but graphics and subsystems as well. In an attempt to drive the point home, AMD ran a set of custom scripts it developed based on Microsoft Office and timed how long it took each system to complete them. The Intel system took 61 seconds to finish the benchmark versus 64 seconds for the AMD platform, a difference of about 6-7 percent and in line with what PCMark 8 indicated, though Sysmark shows a stark delta of 50 percent in favor of Intel with comparable CPUs.
SYSMark has been known to be not particularly representative of actual performance for quite some time. In particular, they seem weirdly sensitive to memory latency, way beyond its actual impact, yet they deliberately evade caching even in benchmarks measuring something where caching is normally useful. And they do seem to unreasonably penalize AMD chips, although I'm not sure if that's malice or simple incompetence.
The review sites I frequent tend to use PCMark for the general-overview synthetic benchmarks, along with some actual-program benchmarks (usually compression, crypto, and video encode). I of course prefer the latter - nobody runs synthetic benchmarks in production, it's always some actual application. The closer you can get to benchmarking that actual app, the better.
The real outrage should be that operations in Microsoft Office are measured in seconds and minutes instead of nanoseconds and milliseconds.
I agree with AMD that most of the market is slanted towards Intel.
But I don't want a benchmark score that is dictated by a graphics card and it's driver set. I want a cpu score that is based on CPU performance, only CPU performance and perhaps taking into account the effects of memory memory bandwidth. Plenty of tools on the internet for that they could have showed instead.
What I want from AMD is a cpu in the 150$~ range with a performance equal or exceeding my old overclockedi5-2500k (2011 vintage) and be capable of gnarly overclocking. If they can deliver that, they have me for an entire socket generation.
The reality is for desktop and gaming workloads AMD's upper shelf products really are a better value. You will get a lot more value putting the savings on the processor/chipset into other system components like a little faster SSD or another 4GB of memory. The performance delta between Intel and AMD looks bigger in some of the benchmarks than it is in real world applications. The vast majority of desktop/laptop PC users don't benefit from Intel's premiums and lets call it superior "strait line performance" as for most applications the situation is I/O bound.
Which is NOT at all to say that if you have more specific workloads like you do a lot video processing, certain types of simulations, etc not to pick Intel.
If you are just getting a PC to do "generically everything and anything" with the A-series systems are a great value. If you have little more to spend FX + discrete video is still probably a better value than an equivalent i3/5 + video on the Intel side. If you really need an i7 for something than by all means get an i7.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
Having to pay money to remove corruption in an established organization is not the definition of ethical business in the first place.
That's called the old extortion/thug plan "pay up for protection".
BAPco deserves $0 from anyone, especially if they have a problem discriminating against people who don't give them omney.
AMD actually left BAPCo in 2011, for this very reason: http://hothardware.com/news/am...
If I benchmark a common office task, like typing War and Peace into Microsoft Word, I find that these new processors run no faster than an old IBM PC/XT with a 4.77 MHz Intel 8088.
"National Security is the chief cause of national insecurity." - Celine's First Law
I hate to break it to you, but Intel's been cheating on benchmarks since the late 70's. There have been countless stories through the years about how their compilers and processors have detected that they're running Whetstone, or SPECmark or whatever, and done something out-of-band as a result. There have also been numerous attempts by them to game the benchmark standards themselves. Think the Volkswagon diesel cheat on steroids -- Intel is famous for it.
When you say you're buying Intel for performance, you're saying that you're really buying Intel for brand-name recognition.
Apparently, AMD was pissed off about SYSMark even when it used to be a member, and quit the organization in protest.
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
I used to run AMD's consumer benchmark group during the K6, K7, K8 days. I'm not sure what you mean by "unbiased reports", but I can tell you that the process the company went through to create and execute benchmarks that were unbiased was remarkably fair. In the time I was there, the company ran benchmark results for any application that met three key requirements:
1) repeatable results
2) relevant software
3) practical to benchmark
So this meant that using canned benchmarks from applications such as Winstone for MS Office applications was a great option to look at office productivity software. We spent a lot of time trying to figure out how PC Magazine was weighting the application between the various MS Office applications, and I hit upon a way to do this by changing core frequency during benchmark runs so that we could create a multi-dimensional array of scores vs. frequencies to determine that Word was x%, Excel was x+5%, etc. We came up with a likely weighting scheme, although I don't recall what became of that work. In the consumer space, the other big hitter is obviously games. At the time of my tenure, AMD used many or most of the same gaming applications that were en vogue with Firing Squad, Toms Hardware, Anand Tech, Sharkey Extreme, etc. There was nothing nefarious about the work we did, nothing unbiased. We looked at these applications with equal weighting and determined that for a given frequency of relevant, competing Intel CPUs, there was an AMD offering that on balance, performed equally or better at a lower frequency. This processor was then given a model name such as 1800+ that was meant to convey it compared favorably to an Intel 1.8GHz CPU. In the days that my group did this work, AMD made a point of publicizing this process and went so far as to have the process vetted via direct supervision of a 3rd party auditing company who was one of the big-4 industry auditors. It was painstaking work to demonstrate that software load order and procedure was identical for AMD and Intel parts. When a benchmark completed, we showed the score to the auditor. Sometimes benchmarks returned imperfect scores because of a stray hard disk latency event and would throw the score off for either product. We would work with the auditor to show that the result of the otherwise repeatable values was an outlier and subsequently toss it in favor of another run.
Others in this Slashdot post have complained of heat dissipation. My team was solely concerned with instructions per second and performance per watt was not a concern for us. I do vaguely recall that this may have been a factor for the server team. My guess is that based on reading the occasional tech article here and there, AMD has made some important progress on power management.
My work desktop is AMD, my home fileserver is AMD, and both my parent's desktops are AMD. That's because in those use cases, AMD is "good enough". Web browsing and email don't require a lot of horsepower.
That said, my gaming/transcoding PC is an Intel i5-4690, because AMD's top line CPU can barely compete with Intel's I3 line. CMT didn't pan out, and they've been held hostage by TSMC/GloFo's failure to produce a sub-28nm lithographic process.
I love AMD's engineers, they have some impressively smart people working for them, and I hope Zen + 16nm heralds a new beginning for them. But today, they aren't "competitive", merely "good enough".
To some degree that's because most of AMD's CPUs are still being fabbed on an older 28nm (or even 32nm) node, which Intel moved beyond almost four years ago. Right now Intel has such a leg up on them due to process differences that there's no way AMD can legitimately compete at the high-end. The best they can do is crank up the clock speed and use a more powerful cooler, but Intel is shipping chips on a 14nm node right now. That comes out to a 4x difference in transistor density.
AMD still provides really good value for a regular consumer machine, and with them finally having some CPUs on a newer process node (14nm supposedly based on Samsung tech) later this year they might even be able to compete with Intel at the higher end of the charts, though I doubt they'll nab any performance crowns.
Rule 1 (good to a first approximation): All benchmarks are meaningless.
Rule 2 (for experts only): Every benchmark measures something very specific. A benchmark is only meaningful if you know exactly what it is measuring, and the thing it measures is something you actually care about.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."