Researchers Sour on Vista Service Pack 1 Performance

Stony Stevenson passed us a link indicating that a group of researchers has described Microsoft's upcoming Windows Vista Service Pack 1 as basically a performance dud. Researchers from the Devil Mountain Software group is claiming that a series of in-house benchmark tests showed that users hoping to receive a speed boost from the update will be disappointed. "Devil Mountain ran its DMS Clarity Studio framework on a laptop Barth described as a "barn burner" -- dual-core processor, dedicated graphics, and either 1GB or 2GB of memory -- to compare performance of the SP1 release candidate that Microsoft released last week with the RTM version that hit general distribution last January. The Vista RTM was not updated with any of the bug fixes, patches or performance packs that Microsoft has pushed through Windows Update since the operating system's debut. 'One gigabyte, 2GB [of memory], it didn't make a difference,' said [CTO Craig] Barth. 'SP1 was never more than 1% or 2% faster.'"

Comments

Are we shocked?

[faloi]

Microsoft has all but given up on Vista. A lot of corporate customers are going to sit it out and wait for the next iteration of the OS to come out. People who have it generally aren't that impressed, at least among the family and friends I've spoken to about it (not a large sample set, I'll grant you). Vista is the new ME, the sooner it dies and MS dumps it the better off we'll all be.

Re:Are we shocked?

[jeffasselin]

The RAM usage at startup for a newly-installed system is simply absurd. 600-700MB is not an exaggeration. The graphics card needs for the new environment (without which it's mostly XP right? It's not like there's a new object-oriented file system in there right?) are quite hgh for most business needs.

The slow file copy isn't a joke. We're talking 1hr+ to copy 2.5GB to a FW hard drive from internal SATA. That's about 25MB/min, 120 times slower than the peak speed of FW. I think you can get more out of a parallel port.

There are some nice additions. But it's not worth the trouble, as some of the flaws totally override those.

Re:Are we shocked?

[MightyYar]

I really don't see where all the Vista hate is coming from. Vista is fine if you buy all-new hardware and all new software, and you make sure that the new stuff is Vista compatible. My experience with Vista has been okay, but there have been some expensive warts:

• Vista didn't work with my wireless router. Solution? New router.
• Vista didn't work with the brand-new Vonage USB stick. Solution? None - but there is a workaround that is a bit of a PITA.
• Vista will not work well with Office XP or earlier. An expensive upgrade to 2003 or 2007 is necessary - and 2007 comes with a whole separate set of issues...
• Other software and hardware was similarly affected. It seems like a waste to ditch your old printers/scanners/etc when they still work fine - even if the replacement is cheap or even free.
• Vista nannies you a whole lot, and you can't tell it to STFU without disabling security in such a way that you might as well be running XP. Sometimes you are being insecure on purpose (like when MS's own installer tells you to disable your antivirus before continuing).
• Someone needs to tell third party vendors that running in Administrator mode is about 10 years out-of-fashion. Again, buying new software largely fixes this problem.

So after you are faced with buying new hardware AND new software, it makes you feel a lot less locked-in to MS and compelled to get a Windows machine. I think this is part of why Mac is seeing its fortunes reversed a bit - for the few applications that you can't find for MacOS (or Linux), you can just boot into your old copy of Windows.

Has it ever improved efficiency?

[Ckwop]

Without wishing to troll, when has a Window service pack ever improved the speed of a Windows OS?

In fact, and I'm sure someone on Slashdot has raw data on this (that perhaps even shows I'm wrong), Apple are the only company who has ever achieved this on a regular basis.

I've found in my rather short development career is something scarily similar to the first law of thermodynamics: "Bad code once created can never be destroyed." In most commercial situations, the risk of breaking a routine far outweighs the benefit the change brings.

We've built an entire area of study, refactoring, on trying to sell the importance of keeping code clean. I'm still not 100% convinced that the case for refactoring has been made. If you spend three months refactoring, is the simpler overall structure really going to speed up development sufficiently to justify the capital outlay? In all but the very worst code-bases, the answer is unclear.Bear in mind, refactoring my cause you to notice bugs that you can't fix because it would break an interface. Now your code has to be badly structured to support this bad business logic. This can be enough to render the effort useless.

This is why service packs rarely improve functionality or performance. Windows XP SP2 is a notable exception. The risk is simply too great.

Simon

Why I even care one bit

[Mascot]

Vista has one great selling point as far as I'm concerned: DX10. It's inevitable that games will eventually require it, though so far it's not exactly a big deal.

So I notice Crysis has a "Very High" setting that's disabled for me in XP. Ok, I think, the first half or so of the game runs ok with High settings, so maybe it might just barely be playable on Very High. Just to be able to see what it looks like.

I boot into Vista and install the game there. Lo and behold, it runs at almost exactly half the FPS on High compared to in XP. Had to drop it to Medium to be even remotely playable. Needless to say, Very High is what I'd need to be to enjoy it with everything at max.

Is the culprit crap drivers for my hardware, general performance drain by Vista, or DRM using everything it can to make sure I'm actually allowed to use the computer today? I don't know, but I do know Vista has made me seriously try a Linux on a desktop for the first time (only used it for servers until now). If only more games supported it, or ran under Wine, I'd be happy as can be.

Re:Why I even care one bit

[AbRASiON]

I can give you 2 pieces of information.

Firstly, I can confirm for you, yep Vista sucked for me too, same driver versions, fully patched machines and the Vista install has several bullshit disk thrash services disabled, it still ran at 34 FPS avg in the benchmark at X settings.

XP ran at 45 FPS avg, same system, same benchmark and settings.

Also the "DX10" features in Vista ARE available in XP with some ini hacking, do a google on it, I think DIGG covered it.
Vista, more like shitsta.

Didn't we have a similiar story five days ago?

[denzacar]

http://it.slashdot.org/article.pl?sid=07/11/18/188235

Only, that one was from PC World Canada.
AND... they at least listed the RC's version (0.275) and explained the tests (well, kinda...),the difference in performance AND the hardware used. http://www.pcworld.ca/news/column/3eef651f0a010408008b33e8065121c5/pg1.htm
WTF is a "barn burner"?

Also, saying "Office-based test script was "statistically insignificant,"...while a multitasking test panel produced results for SP1 less than 1% faster than RTM." doesn't really say much.

Adding to that the first (T)FA actually bothered to mention WHAT was the RC about...

Instead, Microsoft says, the service pack beta improves stability, performance, and reliability when reactivating a machine from Hibernate or Suspend mode; enhances device-driver support; increases security; and adds support for new standards such as Extended File Allocation Table (intended to enhance flash storage on notebooks, not desktops). ... kinda makes this (T)FA even more non-informative in comparison.

In fact... first thing that comes to my mind after reading TFA (the "Barth said"-part) is Samuel L. Jackson in Pulp Fiction:
"Check out the big brain on Barth!"

Re:Optimization

[TheRaven64]

Most worthwhile optimisation is done by rethinking the design, and to a lesser degree hand-coding parts where you know the realities better than the compiler can guess, and just how to exploit that. Micro-optimisations in the right place (not even at the assembly level, just tweaking a few algorithms or data structures, or even the code layout) can give huge benefits. I got a 25% speed gain from some code I was working on a few years ago just be moving a couple of functions into a header and marking them as static inline so the compiler could inline them. Memoisation of frequently-called functions can also give some benefits.

The hard part is usually not the optimisation, it's working out where the optimisations need to go. This typically involves wading through huge amounts of data from profiling runs.

Vista's not slow

[holophrastic]

I can't figure out if I'm just brilliant when it comes to selecting computer components, or I'm the only one who doesn't take a bat to my machine and then wonder why the machine no longer works. You guys keep saying that Vista is slow, and I'm forced to believe you when you say that yours is slow. But mine, mine is not slow. Hell, forget mine. I just bought my grandparents a machine. $1'800.00 got them a 2GHz, 2GB, 24", office, ultimate. No dedicated graphics card. No dedicated sound card. No dedicated NIC. No dedicated anything. It's responsive, it's reliable, it's stable, and it works as fast as anyone would want it to work in full aeroglass beauty. Now, they aren't running photoshop, nor any CAD app, but they are running office, a bunch of games, digital camera stuff, and the typing of the dead.

So why is your machine so slow?

Re:Vista's not slow

[holophrastic]

Heh, what a benchmark. I'm with you on the point of typical versus power users. Certainly I've taught my grandparents and set things up so that the system stays clean. However, I hardly think that we power users are a good standard.

Not only are we happy to push our machines well beyond reasonable, we also tend to take hardware that was never meant to approach even their own specs -- consumer hardware specs are maximums, not typicals.

For my own Vista machines, I didn't go with the minimum ram -- and 2GB is minimum for Vista. I didn't go for half of Vista, I meant Business or Ultimate. I didn't go for the cheap hard drive, the centrino processor, or the slow ram.

The other issue with using power users as any sort of benchmark is that our needs and desires and playfulness grow much faster than the industry -- in part because we are driving the industry. A new super car is invented, and some car guy sees how fast it can go before it breaks. A new plane, and a test pilot takes it out. We get a new OS, and we see how many things it can handle concurrently.

Game developers do the same thing. They build games with adjustable detail. We see that as a way to dial down the graphics for lower machines. We forget, or ignore, the fact that the developers build the very same detail adjustment to dial it up for future machines. But we want to play every new game at maximum detail, which effectively means that the game is old the moment we play it, and that it wasn't developed to take advantage of future hardware. Obviously game developers build for future hardware. We power users push to play that higher level now, and complain when we can't see the best possible version the instant the game is released. Maybe those game developers should hide the higher detail levels from us until six months after release.

Ultimately (heh) I don't see Vista as anything more than a much better OS than XP (I mean in the network, business, and other internals, not aeroglass) that, as with every other software development, will take advantage of hardware improvements over the next two years. We'd be upset if it didn't.

The question I pose to you is not whether Vista now is faster than XP now. I pose to you is Vista faster now than when XP was new? And even more appropriately: will Vista be faster at the end of its life-time than XP is now, at the end of its own.

Re:Straw Man?

[ozmanjusri]

Either way, Vista performance is poor and not getting better.

It's not just Vista though. Microsoft Office 2007 on Windows Vista consumes over 12x as much memory and nearly 3x as much processing power as Office 2000.

The missed point...

[TheNetAvenger]

The things that reviewers seem to be missing...

1) Some of the performance updates scheduled for SP1 were already released as Updates.

2) Performance on a System of 1GB (the sweet spot) will see virtually no improvement, and they are reviewing systems with 1GB and 2GB or more. If you baseline the performance difference on a 512mb system the performance difference is more dramatic.

3) There are also a few optimization that don't affect most users. Readyboost got a significant jump in how it improves performance, and there has been refining of Superfetch as well. This includes not only USB flash, but Solid State and hybrid Drives will see significant boosts.

4) File copying in RTM did have some performance problems but the majority of the problem was the screen not accurately reporting it was already copying files when it said 'calculating time', so SP1 gets about a 10% boost, but the dialog reports the process more accurately as well.

If Windows Update wasn't doing its job and the updates hadn't already been being released, SP1 would be more of a one time dramatic increase. Also they need to be looking at lower end system when testing if they want to see more SP1 improvements.

Finally, older and pre-Vista designed system configurations see more of a bump as well. If you test SP1 on a system that has the specific chipsets and HD Audio, etc that is designed for Vista, SP1 won't add a lot, as the system components were already designed and optimized for Vista.