Crusoe and Benchmarks
duffbeer703 wrote to us with a ZD story regarding Transmeta's Crusoe speed and benchmarking. As we've heard the benchmarks haven't overwhemled people - but are we measuring things the wrong way? Of course emulation is slower then native chipsets - that's a given - but are the other elements of Crusoe enough to make up for it?
Exactly... if I want a mini-laptop like one of those slick VAIO's that's based on the crusoe, or a cool webpad, or something similar, I don't care how well it will run Quake, 3DStudio, or how many Linux kernels it will compile per second. I'd want to be able to run a browser, a word processor, an email client, and maybe a DVD player. My old K6-III/400 handles that beautifully. Price, heat dissipation, size, and power consumption are all one should worry about in this case.
"Evil beware: I'm armed to the teeth and packing a hampster!"
Lex orandi, lex credendi.
Sigh. It's a troll, but I'll bite...
You may not run the same program twice in a row. (I do. Often. But let's ignore that for now.) But... Within a single program of any complexity, there will be code that gets executed repeatedly. It might be the menu-redraw routines, or the file load/file save routines, or the search routine, or the "EMACS has consumed all available memory" error dialog... whatever; there is code that will be executed repeatedly.
Crusoe will most likely perform better than the benchmark shows in these real-life instances. Benchmarks are an artifical environment... how many benchmarks incorprated floating point calculations before FPUs became common? Given that Crusoe is the first chip of it's kind, I'm not surprised that it does poorly on current benchmarks, which were written assuming that performance on the benchmark would not change significantly from one run to another. If Crusoe (or other code-morphing chips) gain in importance, we'll probably see new benchmarks designed to test the new capabilities of these processors.
"Great men are not always wise: neither do the aged understand judgement." Job 32:9
We need a good set of common use benchmarks. Things that run the spellchecker in Word while trying to play MP3s. Then the tests need to be done, on these mundane tasks, with non-virgin crusoe chips. Chips that have been run through these routines and have adjusted themselves to those tasks.
WARNING: there is a trojan on your
We're still not measuring the correct quantities: consumer value and whether the damn thing will have to be recalled within the first week of hitting the market because of some stupid FOOF error like Intel's have been known to do.
You make a correct point, but you don't take it far enough: what's important is not whether Crusoe is clock-for-clock equivalent to Intel's and AMD's offerings. The question is whether a device can be put together using Crusoe at a reasonable price that taken as a whole, fulfills consumers' needs.
The issue is not so much that Crusoe cannot compete in markets currently dominated by Intel and AMD. The question is whether Intel and AMD are the appropriate choices for those markets to begin with. For most consumer devices, they're overkill, both in computing power and electrical power dissapated. Crusoe has a bright future in these markets, and if it were publicly traded, I'd consider picking up a couple shares.
-- Anne Marie
There are three ways to lie: lies, damned lies and statistics. Benchmarks are, have been, and always will be about politics.
Every time I see a benchmark or other metric in the computer world, I have to constantly ask myself, "Is that Linux worse than NT article comparing a Linux box on a 386 vs NT on a 8 processor SMP Alpha?", excuse the exaggerating to make a point.
So what if the benchmarks suck. Is getting the highest numbers the aim of the game? If so, then the person who wins hands-down is going to be the guy with the 5000W power supply that has to be plugged into a 220V outlet, with a liquid-nitrogen cooled non-conducting fluid bath containing a 20 PowerPC SMP Beowulf cluster with attached tachyon cache. The thing'll weigh 250 lbs, be totally impractical for practical use and burst into flames within five hours, but by God, it gets some serious BogoMips before heat death.
A processor that runs really cool, with low power consumption and the capacity to adapt to whatever's thrown at it is pretty cool in my book.
And given that most people do email and write documents, really a 8088 would do. I think we should get over the whole comparing antler or genital size thing that men do. RRR RRR RRR Thog have 200 MhZ! Ug have 240 MhZ! GRAR! Ug get girl!
Sheesh.
--- Jump!! Fire!! Bullet time!! - Lego version of the Matrix
You mean you don't understand the marketing, or you don't understand the bragging rights? ;)
Getting serious: I don't compile kernels on an airplane, but I do serious scientific and engineering work on 'em -- often with a dozen or so heavy-hitting apps running at once -- and sometimes I even attempt as much CAD as I can manage, without having my tablet available.
Some people need the power: my notebook has to do most of what my desktop (a dual PIII 800) does, while I'm on the road. So I have a Dell Inspiron with a PIII 650 (SpeedStep is usually OFF!), 320MB RAM and a 1400x1050 screen... and it's barely enough to get me by.
All that said, I agree with you that 90% of the notebook users don't need more than a 300 MHz processor for what they do. At least, not this year...
---
---
Politics is about making compromises. Religion isn't. --Michael Horton
AFAICT the Crusoe is far more interesting for its design and the possibilities for emulation that for its raw processing power. If I wanted to have a fast IA32 machine I'd go buy one. But if I wanted a processor that had the potential possibilty of letting me design my very own processor to simulate atop it, and at near-hardware speeds, then I'd buy a Crusoe. Battery life is nice too, but I'm much more swayed by the possibility that I could even conceivably use the processor as a nice emulation engine for some other existing architecture, or one of my own design.
Ok, so how does that relate to real world computing then? How often does an application do the same thing over and over again?
Yes, this is a good thing for the Crusoe, but just how many people sit there and run their favorite program twice in a row with the exact same data both times? Seems kinda redundant.
What am I missing here?
Dims screen to save power every 5 seconds.
Spins down HD every 10-15 seconds of no disk activity.
Runs much slower on battery, in low power mode.
Is basically dead weight and bulk when it's dead.
I'd like to try something that runs moderately faster with considerably more battery life. Extra battery packs are just plain annoying and expensive.
--
A feeling of having made the same mistake before: Deja Foobar
I still don't understand this rush for ultra faster processors in notebooks, while more than 90% of the people use them only for text processing and power point presentations. I've never seen a hacker compiling a kernel in a plane. Humm... may be Eric Raymond?
--ricardo
sgis ddo ekil t'nod i
It wasn't meant to be a troll, I apologize. I honestly wondered how much of an effect this had. You talk about actions repeating, and I thought of stuff like that too. But what's the 'memory' on the chip? How far back does it remember to optomize?
I know that programs do the same things over and over and over, but what I wanted to ask was if this actually made a large impact in relation to the Crusoe itself.
The Register also has an interesting editorial from Wednesday about Transmeta's benchmarks.
--
Have fun: Join D.N.A. (National Dyslexics Association)
Similarly, I suspect (because it seems obvious) that the processor keeps track of how often translated code is used so it can discard the less-used translations when it starts to run out of space. It's like virtual memory, but instead of getting paged out to disk the less-needed translations are discarded since they can be redone if they're needed again. It sounds bad to repeat work, but the impact of re-translating is likely to be fairly small, because the code that needs it doesn't get called as often as the code that's kept.
One other thing that might be interesting is what sort of changes might be made to the code morphing software to improve the optimization. Perhaps it doesn't do a good least-recently-used discard of optimized code, but a processor firmware update adds that capability. What kind of impact will that have on system speed? I don't know if updates like that would be free, but would you pay $20 for a 10-15% improvement processor speed on your system?
It's interesting that Transmeta has four areas of potential improvement where Intel and AMD have only three. The first is improving processor speed, the second is improving the cache size and/or speed, the third is architecture improvements, and the third is better code morphing software (which is a form of architecture improvement). Intel has its BIOS update to let it address fixes in the same way that Transmeta could, but that's not going to make a difference in the overall performance of a processor.
I suspect that internally Transmeta is continuing to test processors to find out things like how much cache memory is needed to keep frequently-used translations from being discarded. I'm sure we'll never publicly see those, but eventually it'll probably be clear from the point at which Transmeta stops increasing the amount - when the amount of local cache stops growing it means they've either hit the point where most of what gets discarded wasn't going to be used again anyway or they've hit the point where indexing into the cache becomes a problem.
-- fencepost
fencepost
just a little off
Something I see over and over again is that emulation has to be slower than the original, yadda yadda blah blah.
m o-1.html for more information.
:-)
This is not true.
There are a lot of advantages to being able to modify the code at run time. Most of them involve the optimiser being able to check its initial guess about how to optimise the relevant code. Think of it as a sort of runtime profiling followed by regeneration of code segments.
One example is HP's Dynamo, a PA-RISC CPU emulator that runs code faster on a PA-RISC than the same code running natively. Try:
http://arstechnica.com/reviews/1q00/dynamo/dyna
Another example would be Dr Michael Franz's work on Dynamic Optimisation. Try: http://www.ics.uci.edu/~franz/DynamicOpt.html for more information.
Right. Now stop making assumptions.
---
Trying to compete with the price/performance ratio of the x86 architecture is a lost cause. The crusoe architecture might have a bright future if it could run Java byte code instead of x86 code.
I'm sitting here using an old pentium pro 200 with 256 megs of ram. It does everything I basicly need it to do. Sure it won't do four VMWare sessions, but it can run netscape 4.75, a dozen eterms and a ssh session for each term, dia, gimp, xmms, apache-ssl, and X11 all at the same time. That's all I really need. I mean really, do I care that this thing can only do quake at 200 fps? No.
/. , reading Nealz Newz, and IRC'ing becuase grep can't find 'life' anywhere in /proc/sys/marnuke/. What I do need in a laptop is something that can last for hours even days without losing power, not a damn supercomputer.
I do the same thing day in day out on this machine. Every now then, I might compile a kernel or some other package on this machine, but I have a cluster of dual 600's to build packages on. Of course I wouldn't do compiles on my laptop, and for the love of god not on this machine. Why would I?
When I'm not on my desktop, what will I be doing? Most writing a letter, checking email, ssh'ing into a server, posting on
If I have a choice between four days uptime on a laptop and being able to compile a kernel in a minute, I'll take the four days uptime.
MarNuke
I just hope this is being fair, and not whining. After all, emulation is slower, but Linus does work for these guys...
-----------------------------
1,2,3,4 Moderation has to Go!
By allowing the instruction set presented to the world to be set by software the chip has a lot of advantages in terms of customizability and turnaround time for fixing bugs. Software is also more compact that the corresponding hardware so the chip is overall less complex and presumably more reliable, once debugged.
There is another whole category of applications that may become possible thru use of the native instructions, or better, by custom design of a logical instruction set for a given use.
Four revisions ago, the transmeta chip took half an hour to boot Windows. Consider where they'll be four revisions from now before discounting them on speed alone.
This information and more can be found in a good Technology Review article I found on Linux Today here.
Okay fine, let's run the benchmark program 3 times in a row. Are we getting similar results?
Besides, benchmarks should be run at least 3 times and the result should be the average of the runs, assuming there's no large discrepancy.
All I use my latpop for is browsing code and typing. I am currently using a 266 AMD, it works find but the battery life is horriable. I have a 900 AMD for compiling my work on, so all they really need to do is, for most users, give decent speed at running word, so even if their 600 is only comparable to a intel 400 or even a 300 it will still do what most consumers use laptops for. Besides it sucks flying from Californa to North Carolina and have to cary 3 extra batteries just to use my laptop while I am on the plane.
.sigs suck, thus nothing here.
But what REALLY isn't being taken into account is that the benchmark suites run everything ONCE... but the Transmeta chip gets faster with subsequent runs...
Check this article on Wired for more info.
According to something else I read (sorry no link on this one) performance improves an average of 30% on subsequent runs... so take the benchmarks with a grain of salt.
.technomancer
.technomancer
They're just questioning the usefulness of a processor that adjusts it's speed. "Is the decrease in speed worth the battery life?" It's the same mindless drivel as usual. If you can run [insert office app here] for that much longer when you need it, should the speed make a difference? I doubt the speed will drop to a rate where you type faster than the screen refresh can keep up. Just a thought...
Crusoe simply doesn't perform as well. In fact, it performs much more poorly. It does consume less power, but doesn't lower the power consumption of the hard drive, memory and display, resulting in less bang for the buck without a huge gain in battery life.
That's what the press is saying, and it's true. The Crusoe has an interesting design, and will clearly be a useful chip, but it is not competative against Intel and AMD. Perhaps when they get the speed up and let it run code frm several ISAs at the same time (say, PowerPC and x86 at once), it'll be cool.
I think Transmeta gets a lot of hype-factor from employing Linus.
________________________________________
Napster-to-go says "Fill and refill your compatible MP3 player", which is a lie. It's not MP3. It's WMA with DRM.
for a portable.
No one complains when their Palm won't run Quake Arena. (well they do, but they shouldn't...or should they???) This chip is not meant to be a full fleged asskicker. It's not competing with the Athlon or PIII. It's in a market all by itself.
If this chip is not noticably slower when I surf the web or read email or such from my laptop, and does noticably extend battery life, then that's enough for me.
So, according to these benchmarks, at the same core frequency, a Crusoe is 30% slower than a Pentium III, right? Well, I think it's pretty good. I was expecting this code morphing stuff and this very low consumption architecture to hurt performance by a factor 2 or 3, and things are obviously much better than that. I'm especially pleased with the fact that the FPU is not ridiculous (which is definitely not the case for StrongARM and most ARM CPUs).
As far as I'm concerned, the speed of a Crusoe is waaayyyy enough for my needs, especially on a laptop, and I think it is true for most users. Besides, apparently a 1Ghz version is expected to ship by the end of the year, so there really is very few to complain about.
SPEC works for everyone. If Crusoe doesn't perform well on SPEC, why do we need to design a new SPEC?
And yeah, it runs longer on a laptop's batteries. So do Speedstep chips. I just received a new laptop (a screamer from Dell) that runs beautifully in both Windows and Linux. It uses a Speedstep chip, and the battery power I save is noticable (I've run the battery sometimes up to 4.5 hours).
- I don't care if they globalize against free speech. All my best free thoughts are done in my head.
I would use a crusoe laptop/webpad for serial consoles on servers, browsing the net with some stripped down mozilla, or otherwise just using a terminal for daily tasks. I don't need a huge PIII system, cuz I won't use the thing for games, photo editting, or ray tracing. That's what my desktop is for. But I do want either a huge 15"+ XGA TFT display or a nice sized webpad with no fan that runs cool enough to hold comfortably.
I would also hope that transmeta is not going to be another stuck up proprietary IP company like Intel, MS, Sun, Apple and the like. If they want to be a proprietary IP company, that's fine. Just don't use Linus as some marketting ploy. That's like a slap in the face to anyone who supports the ideology of the GPL. I know I won't be buying a Crusoe until Transmeta releases some information about their code morphing technology.
But I must admit they created a system that will live up to its intentions. To be an efficient, fairly fast and usable webpad/linux PC. As long as they come with wireless net connections or at the very least a PCMCIA port, and cost me no more than $1000, they will help change the way I use computers to store my information. Palm Pilots just don't cut it with their 8MB of ram. Crusoe is just a step up, not the portable server you were expecting. And NO, they don't perform like a P3 500 or 700, they're different. Wish they'd post REAL benchmark numbers on their site and stop causing all this bad PR by trying to live up to the high end systems.
You want performance, go with a Pentium. You want something usable for several hours at home while you are watching the TV, that uses your home net to access all your personal information and the web, that is fast enough to get your work done, potentially upgradable to support other OSs, light weight and cool running, get a Crusoe... but only after Transmeta releases their IP on code morphing. We can at least hold out for that.
I remember doing similar benchmarks when I worked at Sun. We were introducing some PC emulation, and normal benchmarks didn't give the whole story. So, we created our own. One example was to do a search and replace on a 250 page Word document. Another was running a Black-Schols analysis in Excel. Do any readers have a new Sony Picture Book they can spare?
****
"I'd never want to join a club that would have me as a member" - G. Marx
Too bad Intel has low power faster chips already that dont emulate code. How about the xscale, or the low power P3 due to come out soon? I dont see why they are getting so much hype for a slower and more power demanding chip.
We can't forget that the goal of Transmeta wasn't to create amazingly high speed processors compared to today's standard, but to make comparable (and slightly better) processors for mobile Internet-connected computers (or web slates, laptops, etc.). It's ashame that people are "dissing" the company, so to speak, because the benchmarks (very old-school traditional ones, mind you) don't amaze them. Give Transmeta time, and I think you'll be incredibly pleased at what they produce. After all, working with code-morphing technology is just a bit challenging ;-D
______________________________
Eric Krout
If you celebrate Xmas, befriend me (538
First, I understand the argument that benchmarks aren't necessarily relevant. I can only type so fast in Microsoft Word, so provided it runs at a decent speed, I'm happy. For the user who just wants a cheap laptop, these might provide adequated performance coupled with a long battery life; that's just great.
But I'm a nerd, and I want to see the benchmarks. First, I remember reading *another* article about this from Slashdot, where they claimed that the Crusoe really shined at memory bandwidth, perhaps because it had so much integrated on the chip. That's really important nowadays, and people tend to ignore it.
Also, if anyone has one of these, could you please post results for the BYTEMarks? The Unix port includes numbers for memory bandwidth as well as integer and floating point, and the benchmark statistically repeats the tests until it gets a steady-state result for performance, which means that the Crusoe would get to optimize them after the first few passes through, and therefore this test should be more than fair to the Crusoe.
Also, the Crusoe should be compared against an equivalent (specced or priced) *laptop* configuration; remember that.
---
pb Reply or e-mail; don't vaguely moderate.
pb Reply or e-mail; don't vaguely moderate.
You may have a point, but the benchmarks simply are not taking into account battery life -- which is very important to the non-quake crowd. What they need is someway to benchmark the performance over the duration of battery life. Start the tests going with a full charge and see not only how fast, but how long it runs for. Perhaps someone could workout how many D.net blocks or S@h WUs can be completed on a single charge with some of these transmeta portables...
Usability, however, has never really featured as a big item on Slashdot. For example, in coverage of Gnome and KDE, most of the articles seem to be more about the politics, personalities involved, and so on than the actual usability of the environments.
The overall usefulness of computer products is difficult to quantify, and is one of the reasons why companies like Apple have had a tough time. Issues of user-friendliness are thought of as the whinings of computer-illiterates.
Interestingly enough, now we arrive at the Transmeta speed reviews, and it appears that many of the same people who decry anything "pretty" are offering up excuses as to why their favorite chip doesn't run as fast as the others. "But it's more *useful*" they complain.
Hmm... usefulness rather than pure numbers as a benchmark. What an interesting concept.
Read the EFF's Fair Use FAQ
Instead of having Transmeta go off and build a new style of (no doubt self-serving) benchmark practice, how about emulating the SPEC benchmarks with their "train" and "ref" benchmark runs.
For those of you who aren't SPECheads, "train" benchmark runs are used to profile and optimize your code for the final, measured "ref" benchmark set (although I think this only occurs for reporting of peak numbers, as there's a lot of opportunity for benchmark-specific compiler skullduggery here).
This is a well-known practice. It models well the fact that while there is a fair bit of common coverage between different runs of a program, it's not perfect. It affords opportunities for the "other camp" (IA-32) to do their own optimizations.
Granted, there's a difference between standard profile-driven optimization and on-the-fly optimization, but there's a lot less difference than you'd think in many applications.
A bit off my topic, here, but by the way: locality is typically much worse in real applications than it is in SPEC-style benchmarks, even the more realistic ones. Don't be too optimistic about all of that 90%/10% stuff you learned in undergrad CS; sometimes it applies, but a lot of bigger apps are not nearly as tidy as xlisp and m88ksim - or, god help us, the long-departed eqntott ("I think I'm an 8-line microbenchmark").
Most of my work is done on a server (Sun UltraSPARC running Solaris 7) colocated at an ISP, so all *I* really need at home is a fancy dumb terminal (SSH), a web browser (Netscape), and some entertainment (MP3s). Your needs might (and obviously are) different. What I'm saying is that not *EVERYBODY* needs the latest and greatest CPU and video card for the routine things most people use a computer for.
Of course, for multimedia applications, "faster is better"; same for engineering work, but I dont think Johnny writing his 10 page research paper for school needs a 1Ghz CPU and a 32mb video card.
As for "luddite xterm-bound world", I prefer to use *nix as my desktop OS of choice, but that does not mean its the ONLY operating system I use. I've got this *nix PC, an identical (hardware-wise) Windows 2000 machine (for ham radio applications), 2 GRiD laptops running DOS, and even a VAX 6000 in the garage. Certain commercial OSes *do* do some things better than others (Visio comes to mind, for one), so dont be so quick to judge.
After all, you don't expect your laptop to have a 19" screen? Four speaker Dolby (tm) surround sound? In a laptop, you want it to be light, easy to carry, fast *enough* and to have *enough* battery life for your needs.
As has been said in previous articles, many people would be THRILLED with a Pentium 200 based laptop, if only it would have a reasonable battery life. The Crusoe chips appear to give FAR better performance than that, with lower power usage ( gotta love intigrated memory controllers! ).
Who REALLY needs a P-III Coppermine 800 in their laptop? And if you do, don't bitch about battery life, you're getting other benefits.
The fact that no one understands you doesn't mean you're an artist.
Ok. I'm sorry, but I think it is almost humorous to watch /. react when positive things come from 'evil' places or negative results come from 'good' places. I'm really not trying to troll here, but I'm beginning to think of silicon valley business as /. sports. Root for the home team!
Transmeta has posted poor results, so now we want to change the way we test, etc. They have sacrificed speed for battery life. I think that is acceptable, I want to see longer lasting laptops (even at the expense of speed). I wonder if Linus didn't work for Transmeta if it would get such a defensive response.
-Moondog