Slashdot Mirror
Crusoe and Benchmarks
duffbeer703 wrote to us with a ZD story regarding Transmeta's Crusoe speed and benchmarking. As we've heard the benchmarks haven't overwhemled people - but are we measuring things the wrong way? Of course emulation is slower then native chipsets - that's a given - but are the other elements of Crusoe enough to make up for it?
Sigh. It's a troll, but I'll bite...
You may not run the same program twice in a row. (I do. Often. But let's ignore that for now.) But... Within a single program of any complexity, there will be code that gets executed repeatedly. It might be the menu-redraw routines, or the file load/file save routines, or the search routine, or the "EMACS has consumed all available memory" error dialog... whatever; there is code that will be executed repeatedly.
Crusoe will most likely perform better than the benchmark shows in these real-life instances. Benchmarks are an artifical environment... how many benchmarks incorprated floating point calculations before FPUs became common? Given that Crusoe is the first chip of it's kind, I'm not surprised that it does poorly on current benchmarks, which were written assuming that performance on the benchmark would not change significantly from one run to another. If Crusoe (or other code-morphing chips) gain in importance, we'll probably see new benchmarks designed to test the new capabilities of these processors.
"Great men are not always wise: neither do the aged understand judgement." Job 32:9
We need a good set of common use benchmarks. Things that run the spellchecker in Word while trying to play MP3s. Then the tests need to be done, on these mundane tasks, with non-virgin crusoe chips. Chips that have been run through these routines and have adjusted themselves to those tasks.
WARNING: there is a trojan on your
We're still not measuring the correct quantities: consumer value and whether the damn thing will have to be recalled within the first week of hitting the market because of some stupid FOOF error like Intel's have been known to do.
You make a correct point, but you don't take it far enough: what's important is not whether Crusoe is clock-for-clock equivalent to Intel's and AMD's offerings. The question is whether a device can be put together using Crusoe at a reasonable price that taken as a whole, fulfills consumers' needs.
The issue is not so much that Crusoe cannot compete in markets currently dominated by Intel and AMD. The question is whether Intel and AMD are the appropriate choices for those markets to begin with. For most consumer devices, they're overkill, both in computing power and electrical power dissapated. Crusoe has a bright future in these markets, and if it were publicly traded, I'd consider picking up a couple shares.
-- Anne Marie
There are three ways to lie: lies, damned lies and statistics. Benchmarks are, have been, and always will be about politics.
Every time I see a benchmark or other metric in the computer world, I have to constantly ask myself, "Is that Linux worse than NT article comparing a Linux box on a 386 vs NT on a 8 processor SMP Alpha?", excuse the exaggerating to make a point.
So what if the benchmarks suck. Is getting the highest numbers the aim of the game? If so, then the person who wins hands-down is going to be the guy with the 5000W power supply that has to be plugged into a 220V outlet, with a liquid-nitrogen cooled non-conducting fluid bath containing a 20 PowerPC SMP Beowulf cluster with attached tachyon cache. The thing'll weigh 250 lbs, be totally impractical for practical use and burst into flames within five hours, but by God, it gets some serious BogoMips before heat death.
A processor that runs really cool, with low power consumption and the capacity to adapt to whatever's thrown at it is pretty cool in my book.
And given that most people do email and write documents, really a 8088 would do. I think we should get over the whole comparing antler or genital size thing that men do. RRR RRR RRR Thog have 200 MhZ! Ug have 240 MhZ! GRAR! Ug get girl!
Sheesh.
--- Jump!! Fire!! Bullet time!! - Lego version of the Matrix
You mean you don't understand the marketing, or you don't understand the bragging rights? ;)
Getting serious: I don't compile kernels on an airplane, but I do serious scientific and engineering work on 'em -- often with a dozen or so heavy-hitting apps running at once -- and sometimes I even attempt as much CAD as I can manage, without having my tablet available.
Some people need the power: my notebook has to do most of what my desktop (a dual PIII 800) does, while I'm on the road. So I have a Dell Inspiron with a PIII 650 (SpeedStep is usually OFF!), 320MB RAM and a 1400x1050 screen... and it's barely enough to get me by.
All that said, I agree with you that 90% of the notebook users don't need more than a 300 MHz processor for what they do. At least, not this year...
---
---
Politics is about making compromises. Religion isn't. --Michael Horton
AFAICT the Crusoe is far more interesting for its design and the possibilities for emulation that for its raw processing power. If I wanted to have a fast IA32 machine I'd go buy one. But if I wanted a processor that had the potential possibilty of letting me design my very own processor to simulate atop it, and at near-hardware speeds, then I'd buy a Crusoe. Battery life is nice too, but I'm much more swayed by the possibility that I could even conceivably use the processor as a nice emulation engine for some other existing architecture, or one of my own design.
Ok, so how does that relate to real world computing then? How often does an application do the same thing over and over again?
Yes, this is a good thing for the Crusoe, but just how many people sit there and run their favorite program twice in a row with the exact same data both times? Seems kinda redundant.
Something I see over and over again is that emulation has to be slower than the original, yadda yadda blah blah.
m o-1.html for more information.
:-)
This is not true.
There are a lot of advantages to being able to modify the code at run time. Most of them involve the optimiser being able to check its initial guess about how to optimise the relevant code. Think of it as a sort of runtime profiling followed by regeneration of code segments.
One example is HP's Dynamo, a PA-RISC CPU emulator that runs code faster on a PA-RISC than the same code running natively. Try:
http://arstechnica.com/reviews/1q00/dynamo/dyna
Another example would be Dr Michael Franz's work on Dynamic Optimisation. Try: http://www.ics.uci.edu/~franz/DynamicOpt.html for more information.
Right. Now stop making assumptions.
---
I'm sitting here using an old pentium pro 200 with 256 megs of ram. It does everything I basicly need it to do. Sure it won't do four VMWare sessions, but it can run netscape 4.75, a dozen eterms and a ssh session for each term, dia, gimp, xmms, apache-ssl, and X11 all at the same time. That's all I really need. I mean really, do I care that this thing can only do quake at 200 fps? No.
/. , reading Nealz Newz, and IRC'ing becuase grep can't find 'life' anywhere in /proc/sys/marnuke/. What I do need in a laptop is something that can last for hours even days without losing power, not a damn supercomputer.
I do the same thing day in day out on this machine. Every now then, I might compile a kernel or some other package on this machine, but I have a cluster of dual 600's to build packages on. Of course I wouldn't do compiles on my laptop, and for the love of god not on this machine. Why would I?
When I'm not on my desktop, what will I be doing? Most writing a letter, checking email, ssh'ing into a server, posting on
If I have a choice between four days uptime on a laptop and being able to compile a kernel in a minute, I'll take the four days uptime.
MarNuke
Okay fine, let's run the benchmark program 3 times in a row. Are we getting similar results?
Besides, benchmarks should be run at least 3 times and the result should be the average of the runs, assuming there's no large discrepancy.
But what REALLY isn't being taken into account is that the benchmark suites run everything ONCE... but the Transmeta chip gets faster with subsequent runs...
Check this article on Wired for more info.
According to something else I read (sorry no link on this one) performance improves an average of 30% on subsequent runs... so take the benchmarks with a grain of salt.
.technomancer
.technomancer
They're just questioning the usefulness of a processor that adjusts it's speed. "Is the decrease in speed worth the battery life?" It's the same mindless drivel as usual. If you can run [insert office app here] for that much longer when you need it, should the speed make a difference? I doubt the speed will drop to a rate where you type faster than the screen refresh can keep up. Just a thought...
Crusoe simply doesn't perform as well. In fact, it performs much more poorly. It does consume less power, but doesn't lower the power consumption of the hard drive, memory and display, resulting in less bang for the buck without a huge gain in battery life.
That's what the press is saying, and it's true. The Crusoe has an interesting design, and will clearly be a useful chip, but it is not competative against Intel and AMD. Perhaps when they get the speed up and let it run code frm several ISAs at the same time (say, PowerPC and x86 at once), it'll be cool.
I think Transmeta gets a lot of hype-factor from employing Linus.
________________________________________
Napster-to-go says "Fill and refill your compatible MP3 player", which is a lie. It's not MP3. It's WMA with DRM.
We can't forget that the goal of Transmeta wasn't to create amazingly high speed processors compared to today's standard, but to make comparable (and slightly better) processors for mobile Internet-connected computers (or web slates, laptops, etc.). It's ashame that people are "dissing" the company, so to speak, because the benchmarks (very old-school traditional ones, mind you) don't amaze them. Give Transmeta time, and I think you'll be incredibly pleased at what they produce. After all, working with code-morphing technology is just a bit challenging ;-D
______________________________
Eric Krout
If you celebrate Xmas, befriend me (538
Usability, however, has never really featured as a big item on Slashdot. For example, in coverage of Gnome and KDE, most of the articles seem to be more about the politics, personalities involved, and so on than the actual usability of the environments.
The overall usefulness of computer products is difficult to quantify, and is one of the reasons why companies like Apple have had a tough time. Issues of user-friendliness are thought of as the whinings of computer-illiterates.
Interestingly enough, now we arrive at the Transmeta speed reviews, and it appears that many of the same people who decry anything "pretty" are offering up excuses as to why their favorite chip doesn't run as fast as the others. "But it's more *useful*" they complain.
Hmm... usefulness rather than pure numbers as a benchmark. What an interesting concept.
Read the EFF's Fair Use FAQ
After all, you don't expect your laptop to have a 19" screen? Four speaker Dolby (tm) surround sound? In a laptop, you want it to be light, easy to carry, fast *enough* and to have *enough* battery life for your needs.
As has been said in previous articles, many people would be THRILLED with a Pentium 200 based laptop, if only it would have a reasonable battery life. The Crusoe chips appear to give FAR better performance than that, with lower power usage ( gotta love intigrated memory controllers! ).
Who REALLY needs a P-III Coppermine 800 in their laptop? And if you do, don't bitch about battery life, you're getting other benefits.
The fact that no one understands you doesn't mean you're an artist.