Slashdot Mirror

← Back to Stories (view on slashdot.org)

Your Chance To Influence CPU Benchmarking

Posted by timothy on from the how-many-gears-can-you-spin dept.

John Henning writes "When comparing CPUs, is it enough to look at MHz? Of course not; architecture matters, as do caches, memory systems, and compilers. Perhaps the best-known vendor-neutral CPU performance comparison is from SPEC, but SPEC plans to retire its current CPU benchmarks. If you would like to influence the benchmarks that will replace the current set, time is running out: SPEC Search Program entries are due by midnight, June 30."

48 comments

  1. the new spec: by Tumbleweed · · Score: 3, Funny

    BogoMIPS! :)

  2. BogusMark 1.0 by Anonymous Coward · · Score: 0

    Feh! I don't care what benchmarks are chosen, just so that I know far enough in advance so that I can add some code to the board level drivers to optimize it for the benchmark. You know, like what Nvidia did with their latest video card. Great idea, saving time on rendering the scene correctly.

    I can at least double a CPU's MFLOPS rating just by always returning a random value if the operation can't complete in one cycle. Brilliant! Maybe I can get a job at Nvidia!

  3. The best benchmark - circa 1981 by Anonymous Coward · · Score: 0

    10 PRINT TIME$
    20 FOR I=1 TO 9999
    30 NEXT I
    40 PRINT TIME$

    Huh? You mean there's some *other* way to benchmark computers other than using BASIC ROM or Cassette BASIC?
    Oh Well...

    Thomas

    1. Re:The best benchmark - circa 1981 by gfody · · Score: 0

      10 PRINT TIME$
      20 FOR I=1 TO 9999
      30 NEXT I
      40 PRINT TIME$

      compiling...
      warnings (1) empty for loop removed

      --

      bite my glorious golden ass.
  4. Quake 3 is the best benchmark by Anonymous Coward · · Score: 3, Funny

    It is also platform neutral - if it can't run on that platform then I don't buy it.

  5. But I don't know anything about CPUs... by hkon · · Score: 2, Funny

    wouldn't it be better to have the people who actually make CPUs to decide how...

    oh, wait...

    :-)

    1. Re:But I don't know anything about CPUs... by Wesley+Felter · · Score: 2, Insightful

      You joke, but SPEC is a consortium made up of IBM, Intel, Sun, HP, etc.

  6. Who cares? by kawika · · Score: 5, Insightful

    I think we're at the point where it doesn't matter what a synthetic benchmark says about the performance of a CPU. The top end of today's processors have plenty of power for what 95% of people use them to do. The workloads of the remaining 5% are specialized enough that a synthetic benchmark is unlikely to be a good predictor.

    I would rather have a really big and fast RAID array, 2GB of RAM, or a 2Mbps Internet connection than a faster CPU.

    1. Re:Who cares? by amorsen · · Score: 4, Interesting
      The top end of today's processors have plenty of power for what 95% of people use them to do.

      This has been true for me since the Pentium 100MHz or so. However, demands change. My current computer has a 1.8GHz PIV. While I would gladly trade that for a 1GHz PIII, I would not go for anything less. In a few years this computer too will seem impossibly slow and useless.

      The only thing that is new is that high-end gamers now spend more on their graphics cards than on their CPUs. That is truly a change, and it would scare me a lot if I was Intel or AMD. The inside joke at nVidia is that GPU is short for General Processor Unit, while CPU is short for Compatible Processor Unit. Imagine a day when all performance critical software runs on the GPU, while the CPU is reduced to handling I/O and legacy applications...

      --
      Finally! A year of moderation! Ready for 2019?
    2. Re:Who cares? by be-fan · · Score: 1

      Do you people do any *work* with your CPUs? Scientific computing, programming, gaming, graphics, mathematics, simulations, engineering, among other fields, all take more CPU power than is currently available. I'd be willing to bet that the number of people who use those types of applications make up more than 5%, and more importantly, make up far more than 5% in terms of how much revenue they generate for computer companies.

      --
      A deep unwavering belief is a sure sign you're missing something...
    3. Re:Who cares? by Randolpho · · Score: 1

      Um.... what's programming doing on that list? IME, unless you're compiling a huge swath of code all at once (which is extremely rare in the real world), effective programming can be still be done with, oh, say, a p2 350 w/ 32MB. Or maybe even lower; that just happens to be the box I code on. :)

      --
      "Times have not become more violent. They have just become more televised."
      -Marilyn Manson
    4. Re:Who cares? by eelke_klein · · Score: 1

      It wont scare Intel and AMD a bit those GPU's cannot replace the CPU. They may be fast but they simple can't run a preemptive multitasking operating system and make sure processes don't mess up each other memory. The reason they are so fast is that they are designed to do one thing (graphics) and do it good and everything which isn't needed for that one thing has been thrown out. This also applies to other specialized processors like DSP's.

    5. Re:Who cares? by Colin+Douglas+Howell · · Score: 5, Informative
      Just to clarify, SPEC's CPU benchmarks aren't synthetic benchmarks. A synthetic benchmark is a program written to test performance that doesn't do any real useful work (for example, Dhrystone). SPEC's CPU benchmarks are real applications performing real application workloads (for example, running a particle accelerator simulation, or executing Perl scripts), so they actually provide some indication of how fast a computer system with a certain compiler can perform those kind of tasks.

      The biggest problems with SPEC's CPU benchmarks is that they tend to concentrate on technical applications and that people only talk about the average SPECint and SPECfp scores, neglecting the individual benchmark scores that correspond to real tasks. But you can always find the individual benchmark scores on SPEC's website.

    6. Re:Who cares? by archeopterix · · Score: 1
      Um.... what's programming doing on that list? IME, unless you're compiling a huge swath of code all at once (which is extremely rare in the real world), effective programming can be still be done with, oh, say, a p2 350 w/ 32MB. Or maybe even lower; that just happens to be the box I code on. :)
      That depends on the environment. Sure, if you do C programming with command line tools and don't keep all your code in one huge file, then a 100 MHz Pentium is enough, but try Websphere Application Developer - Pentium III 800 MHz with a half gig of RAM and it goes between "annoyingly slow" and "unusable".
    7. Re:Who cares? by amorsen · · Score: 1
      What you say used to be true. However, the latest graphics chipsets are basically vector-FPU's. The only annoying limitation right now is that they only do single precision floating point. Hopefully this will be remedied in coming generations.

      Now you may say that integer performance matters. Personally I do not think it does. The only time I run the integer units flat out is when compiling or encrypting/compressing. Compilation is a niche market, Intel or AMD cannot survive off of that. Encryption and compression is already being pushed to vector units in the CPU. Soon it will make sense to push it all the way to the GPU.

      --
      Finally! A year of moderation! Ready for 2019?
    8. Re:Who cares? by eelke_klein · · Score: 1

      If they do not have a memory managemnt unit and facilities for multiple tasks they will not be able to replace the CPU.

      Adding these parts will make those GPU's much more complex, which will slow them down.

    9. Re:Who cares? by duggy_92127 · · Score: 1
      The workloads of the remaining 5% are specialized enough that a synthetic benchmark is unlikely to be a good predictor.

      This is correct, and thus the CPU benchmarks should be timed runs of various things that high-CPU people actually do. Encode a range of WAVs to MP3, encode a range of MPGs to DivX, render a range of scenes in a 3D engine or something, and so on.

      They key would be to have a standard set of WAVs and MPGs and whatnot, and run the same version of the encoders with the same run-time flags every time. Thus you'd get a proper comparison between CPUs in any given test, and also proper relative performance to previous tests. Every year or so, publish a new standard version of the encoders with the proper flags to use, and you've got updated benchmarks.

      Doug

    10. Re:Who cares? by p3d0 · · Score: 1
      Oh good grief. Someone says this every time CPU speeds are mentioned. How you ever got to +5 Informative I'll never know.

      So, Mr. Smarty Pants, suppose you're trying to evaluate different CPU architectures to decide which one will give the most bang for the buck. How exactly would you evaluate them for performance without benchmarks?

      --
      Patrick Doyle
      I mod down every jackass who puts his moderation policy in his sig. Oh, wait a sec....
    11. Re:Who cares? by p3d0 · · Score: 1
      Um.... what's programming doing on that list? IME, unless you're compiling a huge swath of code all at once (which is extremely rare in the real world), effective programming can be still be done with, oh, say, a p2 350 w/ 32MB.
      Nonsense. Every time I run "make" at work on my 1.5GHz Opteron, it takes about 20 seconds to compile my changes and link the binary, using nearly 100% CPU the whole time. If that were 10 seconds, I'd be happier. If it were 2 seconds, I'd be happier still. That means I'm interested in at least a 15GHz CPU before I stop appreciating the improvement.

      If you're happy with your P2, then good for you, but don't assume everyone would be happy with your P2. (At home I have a dual Celeron-400 which suits me just fine.)

      --
      Patrick Doyle
      I mod down every jackass who puts his moderation policy in his sig. Oh, wait a sec....
    12. Re:Who cares? by amorsen · · Score: 1
      They will not replace the CPU's. The CPU will still be there. It will just not be a performance critical part, and as such its price will fall. Look at C3 prices for an example of how cheap a CPU can be if performance is less important.

      Eventually the CPU will be so small and unimportant that the GPU will swallow it and stick it in some corner. Or maybe the CPU will be integrated into the motherboard chipset instead.

      --
      Finally! A year of moderation! Ready for 2019?
    13. Re:Who cares? by be-fan · · Score: 1

      I'm not saying it can't be done, but it isn't pleasent. C++ compilers are *slow*! And god help you if you change a header file and invoke recompilation of a couple of dozen template-heavy source files!

      --
      A deep unwavering belief is a sure sign you're missing something...
    14. Re:Who cares? by turgid · · Score: 1
      today's processors have plenty of power for what 95% of people use them to do

      True, but the other 5% are designing cars, running large databases (on many-way boxes) and simulating neutron stars and black hole formation. For these people, accurate and meaningful measurement of all aspects of a computer's performance are relevant. I'd like to see you try designing an anti-cancer drug on your piddling little single-processor Pentium IV 2.8GHz.

      --
      Stick Men
  7. Well, compiler writers care by Tom7 · · Score: 3, Interesting

    Researchers in compiler optimizations usually use SPEC benchmarks to test how their optimizations do. This keeps them from cooking up programs that their optimizations do really well on ("Our optimization results in a bajillion percent increase on this program!!"), though of course it encourages them to cook up optimizations that do really well on SPEC benchmarks. That's why the benchmarks are supposed to be "real world" programs, as much as possible.

  8. Deal! (or: "Why ./ != eBay") by jfisherwa · · Score: 1

    "My current computer has a 1.8GHz PIV. While I would gladly trade that for a 1GHz PIII, I would not go for anything less."

    Damn. All I have is this 800MHz Pentium 3.. would you take that and the difference towards the 1GHz in cash?

    While we're at it, my girlfriend is getting a couple of generations behind. Someone have a newer model that they want to trade "up?"

    1. Re:Deal! (or: "Why ./ != eBay") by amorsen · · Score: 1

      If you can fit it into my notebook and actually make it work, the deal is on. I would have bought a PIII-based machine at the time, but I could not get the same graphics and the same screen.

      --
      Finally! A year of moderation! Ready for 2019?
  9. Send All Ebay complaint letters here... It works! by Anonymous Coward · · Score: 0

    24 Edge Rd
    Atherton, CA 94027-2226

    AND

    187 Seminary Dr
    Menlo Park, CA 94025

  10. Meh. by Daleks · · Score: 1

    The best benchmark is to see how well something runs doing exactly what it will be doing when you buy it. When you can't do that just know that the size of your penis increases proportionally with the Ghz rating of your CPU and how many buzzwords are associated with the system you run: HT, DDR, 8x AGP, etc.

  11. WB/s by jrpascucci · · Score: 4, Insightful

    Hi,

    The most effective benchmark I can think of for typical use is Windows Boots per second (WB/s).

    First of all, restarting is the single most used feature of Windows. :-)

    But beyond that, what's funny is I'm not kidding: it does more or less everything you want it to do - lots of disk IO, lots of processing, lots of memory access.

    WB/s should be measured from power on to 'quiescence' - that is, when the services have finished initializing and are 'ready for action'. This goes beyond gina-time login to actually being able to, for instance, start up an IE and connect to yourself.

    This figure has stayed nearly constant for 5ish years, at about 0.005 WB/s (i.e. about 2 and a half minutes between power on and being able to really do stuff). Even 'hibernate' (the ultimate fake optimization for WB/s), is only .06 WB/s.

    Ultimately, I'm waiting for a 10 WB/s CPU. Then, I'll be happy. BSOD? Who cares.

    J

    1. Re:WB/s by vadim_t · · Score: 1

      Linux boot time would be easier to measure. You can measure it more accurately. How do you exactly know when Windows finished loading? With what drivers/software?

      With Linux you can just measure the time spent on booting say, the minimum Debian install. It'd be easy to make a CD for benchmarking purposes that'd quicky make that miminim install on the hard disk.

    2. Re:WB/s by DigiShaman · · Score: 1

      This is a joke right? I mean, just what VERSION of Windows are we talking about? I'm willing to bet that booting into Windows 3.1 (with DOS 6.22) will blow the cyber doors off of WinXP. WB/s...gimmie a break *sigh*

      --
      Life is not for the lazy.
    3. Re:WB/s by packeteer · · Score: 1

      CPU has almost nothing to do with boot time. Boot time is basically a measurement of how fast the data can get from the hard drive/other bottable media to the ram. The CPU becauswe unimportant long ago in doing this. You want fast boot time? Get DDR RAM and/or a SCSI hard drive. When i got my Western Digital JB series hard drive (8mb cache series) my boot time went down a lot. Although i dont have SCSI myself my system boots plenty fat with a fast IDE drive.

      --
      unzip; strip; touch; finger; mount; fsck; more; yes; unmount; sleep
  12. A spec with multiple marks by mnmn · · Score: 2, Insightful

    Various SPEC benchmarks should emulate a desktop, CAD workstation, server, game etc computers, and the SPEC results should always be summarized as SPEC(x,y,z,w...) where each of those variables corresponds to the different applications emulated. A game machine uses the CPU in a very different way than a server.. more IO, less task switching. While CAD users compare CPUs for their own applications and do not need other numbers that show the performance for games, servers etc, these values cannot be united either because that will be too general then.

    --
    "Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
    1. Re:A spec with multiple marks by p3d0 · · Score: 1

      You will never get people to stop reporting average benchmark scores. Better get used to it.

      --
      Patrick Doyle
      I mod down every jackass who puts his moderation policy in his sig. Oh, wait a sec....
  13. iDCT by skookum · · Score: 1

    I think a component of it should involve the inverse discrete cosine transform (iDCT.) This algorithm is used in all kinds of lossy compression methods (jpeg, mp3, mpeg-4 [DivX, Xvid, 3ivx, etc]) and it seems like something that's pretty common. I'm pretty sure those 12-hour DVD-to-AVI transcodes are running a zillion iDCTs, and probably a lot of other stuff too.

  14. Perfect Benchmark by d_insley · · Score: 1

    Calculate how long until DOOM 3 will be released.. heh.

    1. Re:Perfect Benchmark by DigiShaman · · Score: 1

      No no no.... that would be Duke Nukem Forever. Get it right man. Geez

      --
      Life is not for the lazy.
  15. cpu should learn from the gpu by gfody · · Score: 0

    Instead of a cpu crunching thru simple instructions we should have seperate dedicated processing units that implement standard apis.

    Just like a gpu handles opengl/directx maybe have an spu that is optimized for string processing (search replace sort concat etc) and an api designed to expose those common string functions.. say opensl or directsex. similar units for math, ui elements, misc utility stuff etc etc

    what do you ever use the instructions directly for anyways? if your coding in asm chances are your just trying to optimize some more complicated method. I'd rather use an api and leave it to the cpu manufacturer to optimize their implementation.

    --

    bite my glorious golden ass.
    1. Re:cpu should learn from the gpu by gfody · · Score: 0

      this would make benchmarking pretty simple too. instead of crap like "3dnow optimized read modify store blah blah executes in x milliseconds" vs. "sse2 optimized blah shit fuck executed in y milliseconds" you'd just have amd executes 10000 specific_api_function per second vs. intel executes 8000 specific_api_function per second

      --

      bite my glorious golden ass.
    2. Re:cpu should learn from the gpu by temojen · · Score: 1

      hmmm... sounds like a VAX. Oh wait... those went out of production.

    3. Re:cpu should learn from the gpu by Anonymous Coward · · Score: 0

      Instead of a cpu crunching thru simple instructions we should have seperate dedicated processing units that implement standard apis.

      Imagine the hell of synchronizing them all across/within processes...

      maybe have an spu that is optimized for string processing

      rep stosb?

      similar units for math

      They have that, it's called a CPU. Unless you mean floating-point, then it's called an FPU.

      ui elements

      aka "2d accelerator"

      misc utility stuff

      Like what, hard drive controller?

      what do you ever use the instructions directly for anyways?

      Everything, although I usually let a compiler arrange them.

    4. Re:cpu should learn from the gpu by gfody · · Score: 0

      the idea is that we're not married to an isntruction set the way we have been for so long with x86. and not so much that us as developers being constrained to the instruction set, but think of the cpu manufacturers.. really how much faster can they make an fdiv, sqrt and how much faster are computers expected to get when all they can do is make a faster x86 executer.

      now if we programmed with apis instead of instructions, a cpu maker could come up with really insainly fast ways to do more specific things. kind of like the concept behing VLIW architecture but spread across multiple units instead of one instruction based cpu.

      they could even be interchangeable. imagine upgrading your tpu for faster text processing. or perhaps a better way to scale down a computer for the people who only need word and email.

      I enjoy optimizing with a specific instruction set as much as the next guy, but I can't help but feel like I'll never to as good a job as someone with more insight, say for instance the creators of the isntruction set.

      --

      bite my glorious golden ass.
  16. GCC is not a synthetic benchmark by Per+Abrahamsen · · Score: 2, Interesting

    GCC is one of the SPEC benchmarks, and the speed of running GCC matter a lot, judging by the flames about slow compile times after GCC 3.0 was released.

    All the SPEC programs are supposed to be real applications which represent classes of problems real people cares about, allthough some of the floating point benchmarks may fall short of that.

    Basides who cares about what 95% of the population needs. And who, beside you, cares about what you want. SPEC was never intended for "95% of the population", it was intended for high end enginering and scientific users. It was never intended for people whose main use of their computer is to write letters to their mom, surf for porno, and occasionally edit a spreadsheet.

    And this is development section of site called "news for nerds", not the leisure section of a luser magazine. We are supposed to be either developers ourself, or at least technofiles enough to realise that faster cmputers open up for ebitrely new applications areas.

  17. Hmm.... by JHelgie · · Score: 1

    Hows about we see how long it takes to calculate the answer to life, the universe, and everything...

    Oh wait, some company would probably just go and "Optimize" their systems for it, and mess it all up.

  18. I know! by 42forty-two42 · · Score: 1

    Here's my entry

  19. DDR? by yerricde · · Score: 1

    just know that the size of your penis increases proportionally with ... how many buzzwords are associated with the system you run: HT, DDR, 8x AGP, etc.

    So if I can pass "Max 300" on heavy, I can do better in bed? I better practice for an hour a day! Or should I just buy a white box manufactured in former East Germany?

    </double-data-rate-pun>

    --
    Will I retire or break 10K?
  20. LINPACK, the one true benchmark by Animats · · Score: 1
    For decades, the number-crunching community has used the same benchmark, LINPACK. The standard problem is to solve a 100x100 system of linear equations. The latest results for hundreds of machines, updated through June 3, 2003, are here. Some highlights:
    • IBM eServer pSeries 690 Turbo: 1462 MFlops/sec.
    • Intel Pentium 4, 3.06GHz: 1414 MFlops/sec.
    • Cray T94: 1129 MFlops/sec.
    • Cray Y-MP EL: 41 MFlops/sec.
    • Pentium Pro 200MHz: 38 MFlops/sec.
    • Apple Macintosh: 0.0038 MFlops/sec.
    • Palm Pilot III: 0.00081 MFlops/sec.

    That's how much floating point work the CPU can do per unit time. That's the no-hype benchmark for CPUs.

    1. Re:LINPACK, the one true benchmark by Anonymous Coward · · Score: 0

      If only you hadn't ruined your post by including
      Apple Macintosh: 0.0038 MFlops/sec.
      And not specifying that it was an original Macintosh. The way you wrote it implied that it was a modern Mac, even if that wasn't your intention.

    2. Re:LINPACK, the one true benchmark by Animats · · Score: 1
      Yes, that was an original Apple Macintosh. That's what Apple called it at the time, so that's how it is listed.

      The LINPACK benchmark results go back for decades, so they use whatever name the machine was called at the time. "IBM PC w/8087" is also listed, at 0.0069. That's the original IBM PC.