Researchers Sour on Vista Service Pack 1 Performance

Stony Stevenson passed us a link indicating that a group of researchers has described Microsoft's upcoming Windows Vista Service Pack 1 as basically a performance dud. Researchers from the Devil Mountain Software group is claiming that a series of in-house benchmark tests showed that users hoping to receive a speed boost from the update will be disappointed. "Devil Mountain ran its DMS Clarity Studio framework on a laptop Barth described as a "barn burner" -- dual-core processor, dedicated graphics, and either 1GB or 2GB of memory -- to compare performance of the SP1 release candidate that Microsoft released last week with the RTM version that hit general distribution last January. The Vista RTM was not updated with any of the bug fixes, patches or performance packs that Microsoft has pushed through Windows Update since the operating system's debut. 'One gigabyte, 2GB [of memory], it didn't make a difference,' said [CTO Craig] Barth. 'SP1 was never more than 1% or 2% faster.'"

Comments

Straw Man?

[lseltzer]

Did Microsoft say it would improve overall system performance?

Re:Straw Man?

[faloi]

Yes, they did. In the SP 1 white paper. They talk a lot about some of the specific improvements, and are sort of vague on exactly why there'd be an overall performance increase. They certainly give the impression it would improve overall performance.

Re:Straw Man?

[trianglman]

No. It's fair to call a straw man when someone puts words in someone else's mouth and then defeats that argument. In this example, (I did not RTFA, nor anything else related to this btw)if Microsoft did not say anything about performance, but this group tore MS apart because of a lack of performance improvement, it would be a straw man because this group is attacking a claim MS never made. On the other hand, if MS did say performance would be improved, it wouldn't be. From what others have said, and my own personal expectations of this SP, this is probably a straw man. I wouldn't expect a service pack designed to fix security holes and other issues would by default improve performance significantly. Service packs are, generally, a roll up of all the previous security updates, plus any additional security or features they want to add.

An example from the wikipedia article:

An example of a straw man fallacy:
Person A: I don't think children should play on busy streets.
Person B: I think that it would be foolish to lock children up all day.

Re:Straw Man?

[QRDeNameland]

Perhaps you should google on logical fallacies. All that saying "straw man" means is that someone is making an argument against a claim that was never made. If Microsoft never claimed SP1 would improve performance, than it would truly be a "straw man" criticism to berate them because SP1 does not improve performance, and thus the "straw man" defense is valid. However, if MS *did* tout SP1 as improving performance, then the "straw man" accusation is invalid as the article would have a valid point in pointing out that performance gains appear to be dismal.

The guy who posted that MS *did* claim performance improvement makes an actual argument that the OP's "straw man" claim *is* invalid, which is perfectly fine. However, you are simply implying that *any* claim of "straw man" is a "diversion tactic", which is not.

Re:Straw Man?

[QRDeNameland]

No, you see, *that* was good as far as an argument against the OP's claim of "straw man". You actually made an argument as to why the article is not making a straw man argument, with evidence to back it up, though it is extacly the same one the the first response from 'faloi'. Great, so far I agree with that, and I said as much.

But that was not *my* argument. My argument was that you can't simply deny any claim of "straw man" based solely upon your perception that it is often misused, which is where you started. And appropriately enough, that makes your last response to me......a "straw man" argument! To which I can only respond...refer to my previous post.

Re:Straw Man?

[deathy_epl ccs]

Now this is a picture of Chewbacca.

...

Lookit the silly monkey.

Re:Straw Man?

[Daniel Phillips]

It's fair to call a straw man when someone puts words in someone else's mouth and then defeats that argument. Is it fair to try to divert attention away from an actual issue (Vista performance is terrible and is not improved by the latest service pack) to a stupid wankfest about whether Microsoft actually claimed they would improve the poor Vista performance? Either way, Vista performance is poor and not getting better.

Meanwhile, I hear the Walmart Green PC at $199 is selling like hotcakes, because it performs very well running Linux + Enlightenment. Perhaps this shows that people really do care about poor Vista performance. And not what Microsoft claimed they would try to do about it.

Re:Straw Man?

[Daniel Phillips]

Here is what Microsoft specifically claimed about SP1 performance (thanks to faoli for the link):

Performance
The following list describes some of the performance improvements that Windows Vista SP1 will include
      Improves the speed of copying and extracting files.
      Improves the time to become active from Hibernate and Resume modes.
      Improves the performance of domain-joined PCs when operating off the domain; in the current release
        version of Windows Vista, users would experience long delays when opening the File dialog box.
      Improves performance of Windows® Internet Explorer® 7 in Windows Vista, reducing CPU utilization and
        speeding JavaScript parsing.
      Improves battery life by reducing CPU utilization by not redrawing the screen as frequently, on certain
        computers.
      Improves the logon experience by removing the occasional 10-second delay between pressing CTL-
        ALT-DEL and the password prompt displaying.
      Addresses an issue in the current version of Windows Vista that makes browsing network file shares
        consume significant bandwidth and not perform as fast as expected.

Hmm, file shares are slow? Perhaps Microsoft should switch to Samba, which is fast.

Re:Straw Man?

[ozmanjusri]

Either way, Vista performance is poor and not getting better.

It's not just Vista though. Microsoft Office 2007 on Windows Vista consumes over 12x as much memory and nearly 3x as much processing power as Office 2000.

Re:Straw Man?

[QuietObserver]

If you take a look at the link included with the post issued by your GP (http://www.microsoft.com/downloads/details.aspx?FamilyID=090DEAF6-2EAA-4AAA-8B3B-2E199DB4A97D&displaylang=en) you'll see that Microsoft did, indeed, promise that Vista SP1 would improve performance, as stated in the following (taken directly from the overview paragraph on that page; emphasis added):

In addition to previously released updates, SP1 will contain changes focused on addressing specific reliability and performance issues, supporting new types of hardware, and adding support for several emerging standards.

Therefore, this article cannot be, by any stretch of the imagination, considered a straw man argument.

Are we shocked?

[faloi]

Microsoft has all but given up on Vista. A lot of corporate customers are going to sit it out and wait for the next iteration of the OS to come out. People who have it generally aren't that impressed, at least among the family and friends I've spoken to about it (not a large sample set, I'll grant you). Vista is the new ME, the sooner it dies and MS dumps it the better off we'll all be.

Re:Are we shocked?

[king-manic]

Vista is the new ME, the sooner it dies and MS dumps it the better off we'll all be. Vista would have to re-animate the dead into blood thirsty zombies before it could rival the utter horror of ME.

Re:Are we shocked?

[cronot]

Vista would have to re-animate the dead into blood thirsty zombies before it could rival the utter horror of ME.

Gosh, I sure wouldn't like to meet you.

Re:Are we shocked?

[WombatDeath]

Won't happen - they wouldn't give users the ability to reanimate the dead without the permission of the copyright holder (presumably FSM or Odin or someone). I suppose they could get official backing by releasing something (Holy Windows?) which makes you pray for half an hour before booting but, now that I think about it, that's pretty much the current position... ...oh, shit.

Can someone lend me a cricket bat, please?

Re:Are we shocked?

[Lachlan Hunt]

Although it certainly had it's problems, at least Win ME was usable. Vista gets in the way of absolutely everything! I have never been so irritated with an OS in under 5 minutes of use, until vista came along.

Using a friend's laptop running vista, logged in as an administrator, trying to copy harmless files from a public folder on my mac to the my documents folder on vista was forbidden. I had to copy to my Win XP machine first and then from there to Vista. Once tried to use ipconfig /release and /renew to fix a conflicting IP address, but it gave permission denied error! I had to explicitly select Run As Administrator from the context menu to get elevated permissions just to run ipconfig. Bloody oath! Also, the windows explorer UI is so bloody awful and unusable, it's not even funny.

Seriously, if my only choice was ME or Vista, I'd go back to ME any day. But luckily I can just stick with OS X Leopard and Win XP.

Re:Are we shocked?

[syousef]

So what you're saying is that you don't care because it's not your primary OS. Those that do care may be thinking of it running as their primary OS. Heck they may be forced to do so at work in a couple of years. Their LIVING may depend on it.

I do use XP as my primary OS at home and at work and you bet I care. It ain't my spare car. It's my primary ride.

How is the parent modded as insightful? He's saying he doesn't give a shit because he hardly uses it.

Re:Are we shocked?

[jeffasselin]

The RAM usage at startup for a newly-installed system is simply absurd. 600-700MB is not an exaggeration. The graphics card needs for the new environment (without which it's mostly XP right? It's not like there's a new object-oriented file system in there right?) are quite hgh for most business needs.

The slow file copy isn't a joke. We're talking 1hr+ to copy 2.5GB to a FW hard drive from internal SATA. That's about 25MB/min, 120 times slower than the peak speed of FW. I think you can get more out of a parallel port.

There are some nice additions. But it's not worth the trouble, as some of the flaws totally override those.

Re:Are we shocked?

[Slack3r78]

The RAM usage at startup for a newly-installed system is simply absurd. 600-700MB is not an exaggeration. The graphics card needs for the new environment (without which it's mostly XP right? It's not like there's a new object-oriented file system in there right?) are quite hgh for most business needs.

And yet another person who doesn't understand the new memory manager. High levels of allocated memory are a good thing for performance. Coding Horror has a decent primer on all of this, but the short version of the story is that people who are used to how Windows has traditionally handled memory management rather than how an ideal memory manager should work love to complain about Vista being a memory hog when, in fact, I'd suggest that the Vista memory manager may arguably be one of the best out there right now.

Re:Are we shocked?

[roystgnr]

Bad permissions cause Vista to copy files VERY slowly because it has to reset them on all files.

On the Lame Excuses List, this falls somewhere above "You can't take bottled water on an airplane or the terrorists might win" but still doesn't beat out "He only hits me because he loves me."

If the equivalents of "cp -r" and "cp -pr" take noticeably different amounts of time to complete on your operating system, something is broken, because a multi-gigahertz processor can finish fiddling with even complicated permission bits long before a 50MB/s disk needs to have them ready to write.

Optimization

[ktappe]

50 million lines of code and they couldn't find anything that needed optimization?? Or were their priorities elsewhere? These days, optimization always seems to be relegated to "low man on the totem pole."

Re:Optimization

[Anonymous Coward]

Well I did go with a compsci degree but I'm sure I could find something. Here's my proposed patch:

+/*

40 million lines of DRM, WGA, Windows Media Ultra Control Restricted Mode Crap

+*/

Done!

Re:Optimization

[arth1]

Most worthwhile optimisation is done by rethinking the design, and to a lesser degree hand-coding parts where you know the realities better than the compiler can guess, and just how to exploit that.
Neither is something Microsoft is likely to do -- the first costs too much (including accepting incompatibilities and devising workarounds for them), and the second requires ace programmers, not run-off-the-mill visual-anything. Changing a few compiler flags here and there, or re-compiling with a new compiler version is cheap, but usually won't have much noticeable effect. However, it's what you're most likely to see from huge corporations.

Re:Optimization

[UncleTogie]

Does average joe care about optimizations? Probably not. Are they important? To people like you and I, sure, but not to average joe.

Yes, it DOES matter to Joe. Joe, however, won't call it "code optimization". Joe will simply say that "Vista runs slower than my XP did!" He doesn't care WHY it's so, but even Joe can tell the difference in speed.

We have a lot of Joes come through our shop. They notice.

Re:Optimization

[sumdumass]

Well, seeing how this machine was so "hot" in the hardware section, it could be that the bottleneck wasn't in the OS at all. IT could be that it has cycles to spare but is waiting on the memory bus to see any increase in performance. They could have been maxing out everything that would have restricted the OS from performing and never saw the "issue" in the first place.

Of course there was/is an issue, Vista just seems slow. In the former example, they wouldn't have seen the issue because something else would be slowing it down. But on a lesser machine, I'm wondering if the optimizations would have a more dramatic effect. I mean a machine where the memory or processor is limited and the actual execution of the code was keeping it slow. Will it allow the code to be executed faster on a processor that is maxed out all the time?

Re:Optimization

[TheRaven64]

Most worthwhile optimisation is done by rethinking the design, and to a lesser degree hand-coding parts where you know the realities better than the compiler can guess, and just how to exploit that. Micro-optimisations in the right place (not even at the assembly level, just tweaking a few algorithms or data structures, or even the code layout) can give huge benefits. I got a 25% speed gain from some code I was working on a few years ago just be moving a couple of functions into a header and marking them as static inline so the compiler could inline them. Memoisation of frequently-called functions can also give some benefits.

The hard part is usually not the optimisation, it's working out where the optimisations need to go. This typically involves wading through huge amounts of data from profiling runs.

Standard reply from most vendors is...

[Hymer]

"This is a BETA, it is not finished yet. Everything will be alright when it is released."

Game over man!!

[djupedal]

Last one out of Redmond, please turn off that god damn useless big ass table...

Has it ever improved efficiency?

[Ckwop]

Without wishing to troll, when has a Window service pack ever improved the speed of a Windows OS?

In fact, and I'm sure someone on Slashdot has raw data on this (that perhaps even shows I'm wrong), Apple are the only company who has ever achieved this on a regular basis.

I've found in my rather short development career is something scarily similar to the first law of thermodynamics: "Bad code once created can never be destroyed." In most commercial situations, the risk of breaking a routine far outweighs the benefit the change brings.

We've built an entire area of study, refactoring, on trying to sell the importance of keeping code clean. I'm still not 100% convinced that the case for refactoring has been made. If you spend three months refactoring, is the simpler overall structure really going to speed up development sufficiently to justify the capital outlay? In all but the very worst code-bases, the answer is unclear.Bear in mind, refactoring my cause you to notice bugs that you can't fix because it would break an interface. Now your code has to be badly structured to support this bad business logic. This can be enough to render the effort useless.

This is why service packs rarely improve functionality or performance. Windows XP SP2 is a notable exception. The risk is simply too great.

Simon

Why I even care one bit

[Mascot]

Vista has one great selling point as far as I'm concerned: DX10. It's inevitable that games will eventually require it, though so far it's not exactly a big deal.

So I notice Crysis has a "Very High" setting that's disabled for me in XP. Ok, I think, the first half or so of the game runs ok with High settings, so maybe it might just barely be playable on Very High. Just to be able to see what it looks like.

I boot into Vista and install the game there. Lo and behold, it runs at almost exactly half the FPS on High compared to in XP. Had to drop it to Medium to be even remotely playable. Needless to say, Very High is what I'd need to be to enjoy it with everything at max.

Is the culprit crap drivers for my hardware, general performance drain by Vista, or DRM using everything it can to make sure I'm actually allowed to use the computer today? I don't know, but I do know Vista has made me seriously try a Linux on a desktop for the first time (only used it for servers until now). If only more games supported it, or ran under Wine, I'd be happy as can be.

How to "speed up" Vista

[Toreo asesino]

Do some research and you'll find you don't need a service pack to tune Vista:

Turn off: Volume Shadow Copy (files won't be versioned automatically any more), indexing service (rapid searching won't work any more), and SuperFetch (apps wont be pre-loaded and so will start slower, but you'll have more "free memory" on average - a debatable benefit anyway).

You'll notice XP levels of disc activity (barely any) and lot's more free memory. That's because Vista's not doing anything. Personally, I like to be able to search instantly, have apps load instantly, and have my critical files backed up transparently; so I don't mind the "bloat".

Anyway, if you actually know how Windows works, you'll know what you don't want running and what you do. Turn off the stuff you don't want, but most people are fine with the defaults even if it means using more resources.

Re:How to "speed up" Vista

[j.sanchez1]

Do some research and you'll find you don't need a service pack to tune Vista:

I agree...Black Viper to the rescue. I printed out his list of services for XP and still use it to this day when tweaking systems for friends/family.

Re:How to "speed up" Vista

[Drencrom]

It seems you have to do a lot of research to get vista working decently. I guess this proves that it is not yet ready for the desktop :)

Re:How to "speed up" Vista

[stewbacca]

Anyway, if you actually know how Windows works, you'll know what you don't want running and what you do.

I think that sentence basically makes the point for all Mac users on the planet.

Fixed the headline for you

[trifish]

Researchers Sour on Vista SP1 RC1 Performance

Re:That's a release candidate

[mkraft]

A release candidate should be identical to the actual release; that's why it's called a "release candidate" and not a beta version. The only things that would be changed between the RC and the release are any major bugs such as crashes, exploits, etc. Any performance tweaks would have already been done by the time it hit release candidate status. Similarly any debugging code that would slow things down would have also been removed.

Re:That's a release candidate

[GDubs]

Tell that to the KDE team.

Bias

[Nanite]

I'm sure the only thing tying you to Windows these days is your own aging skill-set. Let's face it, Windows has always been your bread-and-butter as a programmer right? Well one could see why you would feel slighted when others bash what you've spent a large amount of your life learning and suffering with. The cold truth is: The Windows skill-set is in danger if MS keeps dropping the ball. Every time MS drops a steaming pile of OS on the market, more people make the switch to Apple, or Linux, and your skill-set degrades just a notch. The thought of mass defections from Windows probably makes you wake up in a cold sweat at night. Well, I'm not going to sugar-coat it: Vista is turning many people elsewhere, and Apple is making all the right moves in the market right now to swiftly pick those disenfranchised folks up. It's only a matter of time before the market tips and non-windows machines are the minority in many areas. It may not be tomorrow, or even ten years from now, but I've lost all hope in MS pulling up from the tailspin they are in.

In closing, I think that there is no better time then RIGHT NOW to expand your skill-set to include Windows agnostic developing. Because I'm of the opinion that there is a huge shift happening in the market right now, just very slowly...

Give up...

[__aamisb9940]

...Vista is NOT about performance. It's about security. The market demanded a 'more secure' Windows, and Microsoft delivered. The market once demanded speed, and MS delivered Windows 98.

Windows is like "Star Trek" movies

[BearRanger]

Let's see. . . 95, 98, ME, XP, Vista. . . Yup, only the even numbered releases are any good. Just to be safe they'd best rebrand Windows 7 as Windows 8. ;-)

Woo hoo

[JRHelgeson]

So my DRM is being upgraded? Should I be excited?
The worst thing Microsoft has ever done was put Mickey Mouse in charge of kernel development. Letting Hollywood dictate the kernel design will prove to be the undoing of the Windows platform.