Wine vs Windows Benchmarks

PeterBrett writes "Tom Wickline recently posted to the Wine development list announcing that he'd done some benchmarks comparing Windows XP to Wine. They should be taken with the requisite dose of salt, but Wine has certainly come a long way."

Comments

on a dev list

[mrcdeckard]

while i realise that postings to a dev list shouldn't be taken as gospel, why would a dev list posting of benchmarks be assumed to be doctored? of course i would expect this from a marketing dept, but a dev list?

chris

Re:on a dev list

[i kan reed]

pride. i'm a developer of software, and I don't allow myself to test my own code beyond a certain point because i'll be too proud of my accomplishments to accept mistakes or failures.

Re:on a dev list

[leuk_he]

Some of the tests look really weird.

That is the key phrase(but you have to look literally). The point is that you need to test the output of the benchmarks, not just look at the frame rates. You can create a REAL FAST benchmark by not implementing some api functions. The output might look reasonable, but if you zoom into some edges you might find additional oddities. That is the main point what is mising in this benchmark.

Rememeber driver writers made some unacceptable shortcuts in the past to increase performance.

Maybe a grain of salt, but it's what I'd predict

[strider44]

The results aren't exactly surprising - Wine is excelling in what Linux is generally better than Windows in doing - memory management, hard drive speed, and related matters (stressing generally there, because of course different apps give different results). This is Gentoo after all, it's built for speed. Then the heavier the load on the video drivers the more the superiority of the Windows drivers takes hold, so for the graphical stuff things don't work as fast.

Congrats to the Wine devs!

wine or driver test?

[shoelace_822695]

To me this seems to be more a test of the linux implementation of teh video card drivers.. and NOT the wine system itself.

i think a wider suite of tests would be required.. and not just the preformance/gaming orinted stuff.

Strange choice of benchmarks...

[Sathias]

Seems a bit strange to me to do a current comparison by using a version of 3d Mark that is 5 years old. If you were going to test out a 6800 on Windows alone you would use 2003 or 2005, the fact they didn't use that one in their Wine comparison suggests to me it couldn't run the later versions at all. The fact that 2000 ran better than under XP, but 2001 ran considerably worse suggests this as well.

If this is the case, the results in regard to game performance are out-dated at best.

WINE not a Windows replacement

[typical]

Okay, I *like* WINE. It's a great piece of software, and very useful. But it doesn't make Linux a drop-in Windows replacement. If you decide "Gee, I think I should use Linux as a desktop" (I do), then it's icing on the cake if WINE runs something. It's not reasonable to simply expect a given piece of software to run flawlessly under WINE, though.

no salt, but lies and damned stats

[SuperBanana]

while i realise that postings to a dev list shouldn't be taken as gospel, why would a dev list posting of benchmarks be assumed to be doctored?

Nobody said they were doctored; the slashdot editor said "take it with a grain of salt". I see a lot of reasons to do so:

• there is no sample size (ie, was each benchmark on each platform run 10 times, or just once?) or variance (if it WAS run 10 times- how much did the results vary?)
• The benchmarks all have wildly different results. Either the benchmarks are that way normally, or WINE (or Linux) is inconsistent. The data is presented such that, again, we have no clue as to the consistency of the results.
• In a number of the benchmark categories for PC Mark 2004, Linux is less than 1% faster. Usually that kind of difference is thrown in the "statistical anomaly" bucket, but the developer happily gave it the "green" mark, when it should have received a "grey" (ie, "not clear"). If the sub-1% wins had been thrown out, Windows would have won by at least an equal margin.
• Equal weight was given to the insignificant "wins", as was the massive failures.
• The developer breaks down the number of Wine failures into 4 categories, but groups Wine successes into one. As a result, it appears Wine is the overall winner, when in fact Wine was slower in 63 cases, and faster in 67.

Honestly? The results probably aren't manipulated, but the presentation is very clearly set up with a number of tricks (perhaps without him/her realizing it) to give the impression that Wine "kicked some serious ass", when for the most part, it did horribly.

Re:no salt, but lies and damned stats

[Anonymous Coward]

I think the main reason the 1% less is given a green is because this is targeted at developers of WINE, and seeing that wine is 1% as close as real windows means that that area is "done" being optimized. The areas where wine needs to focus are on the cases where WINE is significantly slower. WINE really only needs to be as fast as windows, not faster.

Re:no salt, but lies and damned stats

[Savantissimo]

You forgot the really important issue: in 18 of the tests, some pretty important, Wine didn't complete the test at all.

Re:no salt, but lies and damned stats

[mcvos]

That's probably because this is aimed at developers. It shows in which areas they need to improve. Once Wine is equal to XP in a test, they're done, and should focus on other areas. From that point of view, it makes sense to lump all successes in one category, and distinguish between levels of failures.

This benchmark isn't a Wine vs. XP contest, it's a test to see if Wine is at least as good as XP, and it failed in 81 categories, which means there's still some work to be done.

Re: Very Impressive!

[strider44]

As I understand it the essence of Wine is reverse engineering the Windows DLLs.

You might understand it that way, but you'd be wrong. All Wine does is implement the published API of Windows using Linux commands. Absolutely no reverse engineering is done.

Re:Compatibility more important than speed!

[misleb]

What about higher end MS applications like VisualStudio or the .Net framework?

At some point you have to ask yourself why you are running Linux at all.

-matthew

Re:amount of work done

[mmjb]

They are not altogether meaningless. For those who have never tried Wine, the article should give hope of success to those who want to give it a go - albeit more anecdotal than proof.

It's easy to be faster when you are doing less work.

To say that Wine developers have it easy is shamefully disrespectful to their efforts. (Unless you take the viewpoint that not having to work with MS code simplifies the work!) For Wine to work at all is commendable - to be (sometimes) faster is truly amazing, IMHO.

Don't be silly

[something_wicked_thi]

These results aren't presented to try to make Wine look better, nor is the author, consciously or unconsciously, trying to make it appear faster. These results simply were not meant to be used to say that Wine is better than Windows, and only Slashdot would try to make it appear that way. The real point of these comparisons, as is apparent if you read the Wine weekly summaries, is to give the Wine developers an idea of what areas need to be improved, and what areas are adequate. Green obviously means "at least as fast as Windows", which means that it's good. There is no point in grouping them any other way, since they don't care if they are 50% better or 1% better. Also, your criticisms of why this benchmark doesn't give a good idea of the relative speeds of Wine and Windows are quite wide of the mark (though they are valid complaints). The real reason why this benchmark cannot be used to gauge relative speeds is that it doesn't cover real world work loads. They measure a very small number of very specific things, mostly related to gaming and 3D performance. The benchmarks they ran that weren't related to that were designed to test the *hardware* speed, not the speed of the API. The Wine developers know this, and that's what the comment about taking it with a grain of salt means. It's probably adequate to give a rough idea of what parts of Wine need to be improved, but it is nowhere close to a comprehensive comparison of the speed of Windows and Wine, and was never meant as such.

Re:What do you think reverse engineering is ?

[BostonPilot]

With 30 years as a software engineer, it's only very recently that I've ever seen "reverse engineering" used in the context of reading source code. In the past we called that.... um... "reading the source code". I can understand this interpretation, and it fits in with the general idea of "deduce design by observing details" but I think this particular interpretation trivializes the term a bit. Can you imagine someone saying they "reverse engineered a book" because they read it and understood what the author intended?

Throughout most of my career, I've understood reverse engineering to be what you do when you DON'T have the source code. I think the wikipedia entry mentions that this is the most common interpretation. It can be extremely difficult and time consuming. I've done it on major projects a few times in my career. It is neither easy nor efficient, but sometimes the only choice you have.

It's also common to talk about reverse engineering hardware, where the innards of a chip may be deduced by observing its inputs and outputs. I worked on a project at a startup where we had to do that, because the original hardware developers were long gone, and no VHDL could be discovered, yet we had to write drivers for their (buggy) chips in order to make a deadline. At the same company we had to reverse engineer the workings of a (buggy) Fiber Channel PCI card, because the manufacturer would not give us the support we needed to make our deadlines. They were rather surprised when we talked to them about the details of their (proprietary, embedded in an ASIC) DMA engine that we deduced via logic analyzers and oscilloscopes.

Those projects were SO difficult compared to reading (even obscure) code, that I really think that using the one phrase for both activities is confusing and sometimes even deceptive.

Re:Maybe a grain of salt, but it's what I'd predic

[Spacejock]

I agree with you. When I was a kid we had 8 bit computers and you learned by doing. In those days you of fritzing your hardware by moving it too suddenly, causing a machine lockup with crappy code was the least of your problems.

Gentoo evokes that era for me, and you can't have a go at people for breaking something when they're busy learning. So what if they over-optimise compiler flags and break things? When they fix them up they're learning a valuable lesson, and that lesson isn't 'next time use a binary distro'

Anyway, today's school-age gentoo n00bs are tomorrow's crop of system admins.