Slashdot Mirror

← Back to Stories (view on slashdot.org)

Your Chance To Influence CPU Benchmarking

Posted by timothy on from the how-many-gears-can-you-spin dept.

John Henning writes "When comparing CPUs, is it enough to look at MHz? Of course not; architecture matters, as do caches, memory systems, and compilers. Perhaps the best-known vendor-neutral CPU performance comparison is from SPEC, but SPEC plans to retire its current CPU benchmarks. If you would like to influence the benchmarks that will replace the current set, time is running out: SPEC Search Program entries are due by midnight, June 30."

11 of 48 comments (clear)

  1. the new spec: by Tumbleweed · · Score: 3, Funny

    BogoMIPS! :)

  2. Quake 3 is the best benchmark by Anonymous Coward · · Score: 3, Funny

    It is also platform neutral - if it can't run on that platform then I don't buy it.

  3. But I don't know anything about CPUs... by hkon · · Score: 2, Funny

    wouldn't it be better to have the people who actually make CPUs to decide how...

    oh, wait...

    :-)

    1. Re:But I don't know anything about CPUs... by Wesley+Felter · · Score: 2, Insightful

      You joke, but SPEC is a consortium made up of IBM, Intel, Sun, HP, etc.

  4. Who cares? by kawika · · Score: 5, Insightful

    I think we're at the point where it doesn't matter what a synthetic benchmark says about the performance of a CPU. The top end of today's processors have plenty of power for what 95% of people use them to do. The workloads of the remaining 5% are specialized enough that a synthetic benchmark is unlikely to be a good predictor.

    I would rather have a really big and fast RAID array, 2GB of RAM, or a 2Mbps Internet connection than a faster CPU.

    1. Re:Who cares? by amorsen · · Score: 4, Interesting
      The top end of today's processors have plenty of power for what 95% of people use them to do.

      This has been true for me since the Pentium 100MHz or so. However, demands change. My current computer has a 1.8GHz PIV. While I would gladly trade that for a 1GHz PIII, I would not go for anything less. In a few years this computer too will seem impossibly slow and useless.

      The only thing that is new is that high-end gamers now spend more on their graphics cards than on their CPUs. That is truly a change, and it would scare me a lot if I was Intel or AMD. The inside joke at nVidia is that GPU is short for General Processor Unit, while CPU is short for Compatible Processor Unit. Imagine a day when all performance critical software runs on the GPU, while the CPU is reduced to handling I/O and legacy applications...

      --
      Finally! A year of moderation! Ready for 2019?
    2. Re:Who cares? by Colin+Douglas+Howell · · Score: 5, Informative
      Just to clarify, SPEC's CPU benchmarks aren't synthetic benchmarks. A synthetic benchmark is a program written to test performance that doesn't do any real useful work (for example, Dhrystone). SPEC's CPU benchmarks are real applications performing real application workloads (for example, running a particle accelerator simulation, or executing Perl scripts), so they actually provide some indication of how fast a computer system with a certain compiler can perform those kind of tasks.

      The biggest problems with SPEC's CPU benchmarks is that they tend to concentrate on technical applications and that people only talk about the average SPECint and SPECfp scores, neglecting the individual benchmark scores that correspond to real tasks. But you can always find the individual benchmark scores on SPEC's website.

  5. Well, compiler writers care by Tom7 · · Score: 3, Interesting

    Researchers in compiler optimizations usually use SPEC benchmarks to test how their optimizations do. This keeps them from cooking up programs that their optimizations do really well on ("Our optimization results in a bajillion percent increase on this program!!"), though of course it encourages them to cook up optimizations that do really well on SPEC benchmarks. That's why the benchmarks are supposed to be "real world" programs, as much as possible.

  6. WB/s by jrpascucci · · Score: 4, Insightful

    Hi,

    The most effective benchmark I can think of for typical use is Windows Boots per second (WB/s).

    First of all, restarting is the single most used feature of Windows. :-)

    But beyond that, what's funny is I'm not kidding: it does more or less everything you want it to do - lots of disk IO, lots of processing, lots of memory access.

    WB/s should be measured from power on to 'quiescence' - that is, when the services have finished initializing and are 'ready for action'. This goes beyond gina-time login to actually being able to, for instance, start up an IE and connect to yourself.

    This figure has stayed nearly constant for 5ish years, at about 0.005 WB/s (i.e. about 2 and a half minutes between power on and being able to really do stuff). Even 'hibernate' (the ultimate fake optimization for WB/s), is only .06 WB/s.

    Ultimately, I'm waiting for a 10 WB/s CPU. Then, I'll be happy. BSOD? Who cares.

    J

  7. A spec with multiple marks by mnmn · · Score: 2, Insightful

    Various SPEC benchmarks should emulate a desktop, CAD workstation, server, game etc computers, and the SPEC results should always be summarized as SPEC(x,y,z,w...) where each of those variables corresponds to the different applications emulated. A game machine uses the CPU in a very different way than a server.. more IO, less task switching. While CAD users compare CPUs for their own applications and do not need other numbers that show the performance for games, servers etc, these values cannot be united either because that will be too general then.

    --
    "Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
  8. GCC is not a synthetic benchmark by Per+Abrahamsen · · Score: 2, Interesting

    GCC is one of the SPEC benchmarks, and the speed of running GCC matter a lot, judging by the flames about slow compile times after GCC 3.0 was released.

    All the SPEC programs are supposed to be real applications which represent classes of problems real people cares about, allthough some of the floating point benchmarks may fall short of that.

    Basides who cares about what 95% of the population needs. And who, beside you, cares about what you want. SPEC was never intended for "95% of the population", it was intended for high end enginering and scientific users. It was never intended for people whose main use of their computer is to write letters to their mom, surf for porno, and occasionally edit a spreadsheet.

    And this is development section of site called "news for nerds", not the leisure section of a luser magazine. We are supposed to be either developers ourself, or at least technofiles enough to realise that faster cmputers open up for ebitrely new applications areas.