Slashdot Mirror
Mark Russinovich On Vista Network Slowdown
koro666 writes "In his latest blog post, Mark Russinovich analyzes the network slowdown experienced by some users when playing multimedia content. 'Tests of MMCSS during Vista development showed that... heavy network traffic can cause enough long-running DPCs to prevent playback threads from keeping up with their media streaming requirements, resulting in glitching. MMCSS' glitch-resistant mechanisms were therefore extended to include throttling of network activity. It does so by issuing a command to the NDIS device driver... [to] pass along, at most 10 packets per millisecond (10,000 packets per second)... [T]he networking team is actively working with the MMCSS team on a fix that allows for not so dramatically penalizing network traffic, while still delivering a glitch-resistant experience.'"
Aaah, it's those pesky DPCs in the MMCSS. It's so obvious really.
Get your own free personal location tracker
Once again, over-complication and stupid engineering lead to a humiliatingly bad operating system. It's obvious it didn't receive a modicum of real testing.
Sam ty sig.
So why can my Windows 98/95/2000/ME/XP computers play mp3s without this happening?
Not a Twitter sockpuppet... but I wish I was.
10,000 packets/second ought to be enough for anyone.
"No one likes working in a hamster wheel, and your shop smells of cedar shavings from here." - TaleSpinner
I have [Cancel][Allow] no idea [Cancel][Allow] what slowdowns they [Cancel][Allow] could possibly be [Cancel][Allow] talking about!
But don't let logic or common sense get in your way.
Can't wait until the "glitch-resistant mechanisms" migrate throughout the rest of the Vista. It's gonna be awesome!
Because the standard Ethernet frame size is about 1500 bytes, a limit of 10,000 packets per second equals a maximum throughput of roughly 15MB/s. 100Mb networks can handle at most 12MB/s, so if your system is on a 100Mb network, you typically won't see any slowdown. However, if you have a 1Gb network infrastructure and both the sending system and your Vista receiving system have 1Gb network adapters, you'll see throughput drop to roughly 15%."
That is one of the dumbest things I have heard in a while. Let's see:
What an over-engineered non-solution to what should have been a non-problem in the first place. Microsoft is supposed to employ some of the smartest engineers in the world: can none of them optimise their code?
I find this totally interesting. It goes to the heart of what is wrong with Microsoft these days... All seperate groups of folks, not talking to each other, to try and do "what is best" for the user, and then totally stomping on each other. Instead of really looking at thread management and optimising the kernel, they cludge together something to make multi media work by simplying saying "in certain situations, I can't guarentee the thread because of a crappy kernel, so I am going to tell everyone else to slow down".
It is these sorts of things and things like the teams and teams debating the "Shutdown Menu" in Vista that are really showing Microsoft needs to really change if they are going to survive. It amazes me how a bunch of open source developers with all their own agendas do a better job then a bunch of folks all paid by the same company. Of course then there is Apple of an example of a group that shows you can pull it off and still all look like the same organisation.
D.O.U.O.S.V.A.V.V.M.
I haven't been really on the lookout for it, but I haven't seen any posts explaining that as the cause. I'd expect if that really were the cause, there'd be a much bigger outrage from people and it would have blown up and I'd see articles on it whether I wanted to or not. I don't really see any useful DRM techniques for unprotected MP3s anyway. There'd really be nothing that MS could do with that sort of information.
However, this actually does make sense. In all honesty, they probably would have worked on a better answer than cutting back on networking, but with the time crunch on releasing it, they probably cut corners here and there (and by probably, i mean definitely and by here and there, i mean everywhere). They probably viewed this as an acceptable cut for the time being because for a majority of users, they use very little of their networking bandwidth. If its just a PC connected to the internet, they'd most likely never notice. The only time this would be an issue is for heavy network usage, which would normally only occur on work-related machines because let's face it, aside from geeks and techies, not many people have systems set up that max out their network bandwidth, so, if they were work-related machines, well, they probably wouldn't be playing that much music to begin with.
I'm not a MS shill, though I don't assume everything they do has evil intentions. We have to admit that they are great code writers, just not the best. Just because they do shady things here and there (mostly in business practices however) doesn't mean everything they do is evil. This was a problem they ran into and they made a workaround that would only affect a relatively small amount of their users. They were probably hoping no one would notice it at all until they either A) had a fix or B) just let it go because maybe no one would notice it.
Remember, this wouldn't really slow down your internet unless you have an *extremely* high bandwidth and even then, bottlenecks on the information before reaching you would probably still mask the problem. This is only an issue on system that have heavy network usage on some sort of intranet or other type of local area network, because these would account for the majority of networks that could even use a decent amount of your possible networking bandwidth.
Why should Vista have any problems playing audio and videos?! I have an ancient 550Mhz PIII with only 256 megs of ram running W2K and it plays MP3s and video (divx and xvid) much more smoothly than my wife's Vista system (2600+ AMD, GeForce 6800XT 512MB 256-bit GDDR2, 1.5 gigs Ram). My wife's system used to run XP Pro and it rocked for everything, including games. Now even old games such as Sonic Heroes will barely run on Vista.
I gave it an honest chance, but Vista is a turd. If it can't play decades old MP3 technology MS should really give it up.
If someone says he and his monkey have nothing to hide, they almost certainly do.
No.
Have you ever seen a talk by Mark?
While he might be on the Microsoft payroll, he is definitely NOT one to sell-out.
www.christopherlewis.com
Microsoft should hire Con Kolivas to fix their Completely Unfair Scheduler :)
As goes without saying, arbitrarily throttling one particular task, at some arbitrary level, is the wrong thing.
Perhaps this could go in Wikipedia under "Kludge"?
Well, I use and love some MS software like the next guy.
You mean, not at all?
"I think the real fear is the DRM layers that are built into Vista are at the root of this problem"
If you RTFA it explains that it isn't DRM causing the problem.
The Good Life
I think this is a great example of why the open source development model will lead to better outcomes. Microsoft apparently tweaked Windows for profit instead of to improve efficiency or user experience. This design flaw would have been identified immediately in the open source world and would have been rightly discarded.
http://it.slashdot.org/comments.pl?sid=280101&cid= 20366549 2 8200 = 20377327
http://it.slashdot.org/article.pl?sid=07/08/26/16
http://it.slashdot.org/comments.pl?sid=280101&cid
If they are meant to be humourous then my sense of humour must be completely broken. I'm sure there are more comments of equal paranoia to be found in previous installments of this saga.
it was an IMPOSED, HARDCODED limit WITHOUT ASKING the user. They could just add a registry entry of "maximum network packets per millisecond when playing multimedia files" or something.
Microsoft has a long history of hardcoding stuff without thinking of power users. Remember the 10-limit for open TCP connections per program? They did this because viruses and malware open many TCP connections. "Hey, what about P2P?" "What's P2P?".
Wow, your incredibly well-thought argument has swayed me! Time to burn my Vista DVD, reformat my computer and install Ubuntu! ... No, wait, that was a lie. Sorry about that.
"It does not do to leave a live dragon out of your calculations, if you live near him." - Tolkien
My pocket watch plays MP3s while keeping time, as well as keeping track of my schedule, so why is it my top of the line dual-core machine with 4gb of ram, the fastest disks on the planet, and the newest, most innovative, most secure, and best operating system there ever was can't do the same? OMG... It's a virus... "format c:"... all better... *wry sarcasm*
I'd had two CPU's and Gigabit Ethernet for three years by the time that Vista was on sale to the public. That's not simply "short-sighted with respect to today's systems", that's a total let down to businesses who have high-performance workstations.
While I was a consultant years ago during college, a co-worker had an old 486 machine and had Winamp playing a file off of a 3.5" floppy disc and it worked perfectly.
Sure, video (especially HD content) has much higher bandwidth demands, but local video playback has NEVER been a problem on any machine I've owned in the last 10 years. I remember IBM thinkpads with PII 266 processors that could easily play DVDs.
The only explanation for Vista's media playback design decision must be to compensate for the huge processing overhead that Vista creates. Poor fundamental design decisions necessitated hacks like this prioritization scheme.
-ted
Although I think some kind of multimedia-aware scheduler is absolutely necessary, it simply looks insane to hard code the network device to drop the rate to 10k packets/sec. This makes me sort of feel like they added MMCSS on the last moment of vista design, or they simply didn't think hard enough.
There can be two possible solutions
1) dynamically change the packet rate depending on the multimedia workload/processor/whatsoever
2) dropping network packets not on the NDIS, but on the TCP/IP driver, which starts dropping packets when multimedia processes runs out of processing power
Yes, I understand that your PC plays mp3s without any problem even on a GB ethernet connection. However, I don't think the MMCSS is a flawed concept/unneeded feature/whatsoever. The MMCSS is for improving multimedia performance on EXTREMELY heavily-loaded processors. I use XP, and my PC is occasionally heavily loaded with a dozen threads, and in those cases I occasionally experience glitches. Thus, I have to manually adjust thread priorities, but it's annoying anyway.
The problem is a flawed implementation - nothing else.
ps : I wish there was some method to reserve some processing power for launching taskmgr, so that I can kill problematic processes with ease. Any suggestions?
Oh fuss and bother. Another apparent glitch in my plan to pass as young. It's back to the drawing board to get out the slide rule and adding machine.
Just what is a "glitch-resistant experience" ? Before he was in M$ payroll, Mark would have called this something else. Sorry, but his "technical authority" value, that Microsoft are hoping to use to explain away their bugs, has lost all value now he's making their excuses for them.
When my old XP HD crashed I was forced to use Vista exclusively for several weeks. It was like my computer was sick and in the hospital. No TV from my ATI x800 All-in-Wonder (though I did get the FM radio working after a week or two), sucky video game frame rates, unstable network card and sound card drivers and crap multimedia playback. P2P kept crashing the network stack.
Some people say that this isn't Microsoft's fault, it's those third party driver writers to blame. I say fuck that, these folks can write good drivers for the exact same computer in several other operating systems. It's Vista's fault.
MS fanboys will all come out and say their systems all work perfectly. Horseshit. I've now had hands on with more than two dozen Vista machines ranging from laptops to upgrades and in every single case, that's 100% MS fanboys, not 99%, not 80%, all of them had stuttering media playback.
There is no excuse for this sort of crap. My goodness it was such a relief to get an XP install back. My computer was perkier and all of a sudden everything worked again.
If Microsoft does not fix this with the mother of all service pack releases rewriting Vista from the core out then my next post-XP os will not be Windows. My best guess is Vista SP1 will be lipstick on a pig rather than the thorough cleaning out that poor excuse for a beta release really needs though.
There's good reasons to attack Slashdot. Its business practices, internal workings, and products have a history of abuse, contradiction, and stupidity behind them. I wave no flag for Slashdot and Subscriptions are a pain in the ass that's priced itself out of the market but the look and feel of Digg is heading for the basement. I'd be careful of this because when Slashdot sort themselves out the only thing you'll be left with is hate. As that has nothing to grip on people will see it for what it is and walk away.
Slashdot is focused on improving its products and reaching out to new markets. Meanwile, the comments on Digg continue to get less informative and relevent to people outside its core audience. From being some great visionary power that could tear down someone's server with the mere waving of a hand it's become the problem. It has no clear forward vision and most servers just shrug off the famed Digging. Slashdot has changed. The world has changed. Meanwhile, Digg just tears itself up in frustration.
Wake up.
As written in the article...
Besides activity by other threads, media playback can also be affected by network activity. When a network packet arrives at system, it triggers a CPU interrupt, which causes the device driver for the device at which the packet arrived to execute an Interrupt Service Routine (ISR). Other device interrupts are blocked while ISRs run, so ISRs typically do some device book-keeping and then perform the more lengthy transfer of data to or from their device in a Deferred Procedure Call (DPC) that runs with device interrupts enabled. While DPCs execute with interrupts enabled, they take precedence over all thread execution, regardless of priority, on the processor on which they run, and can therefore impede media playback threads. They're saying that every packet received causes an interrupt request, which causes the CPU to get loaded at high transfer speeds.
Apparently they haven't heard of interrupt moderation or polling, technologies that are used by network cards to offload the CPU.
Even my Marvell semi-hardware (I think) Gigabit on-board network card used about 14% CPU (Barton 1833Mhz) when transferring files at about 45Mbps.
I don't know, everything seems really stupid, and I'm not sure it's just a "bug", or their description is just a part of what really happens behind the scene.
I think it's right to at least pose the question, though. Before, Mark was a Windows expert working independently, and was able to voice opinions as he saw fit. Now he's a Windows expert being paid by the company that makes Windows - the very success of Windows Vista will dictate how long his job lasts. He now has an interest in assuring customers and investors that things aren't as bad as they might be. Now, it's all about the bottom line. Of course, he's built up a lot of trust amongst the community, trust which MS themselves are now paying money for. Whilst we can continue to trust him, until proved otherwise, it's not wrong to least ask the question.
Back when sysinternals was still independent, Mark provided real information with real criticism when things were wrong. Apparently, things have changed.
His "analysis" here is not much more than a series of rationalizations and excuses:
"Network DPC receive processing is among the most expensive, because it includes handing packets to the TCP/IP driver, which can result in lengthy computation. The TCP/IP driver verifies each packet, determines the packet's protocol, updates the connection state, finds the receiving application, and copies the received data into the application's buffers." (emphasis mine)
The issue at hand is related to gigE NICs. Please find me a single gigE NIC that does not support TCP/IP checksum offload (even the lowly Realtek does).
His graph showing 40% CPU utilization during a file copy must be a joke or an admission of a dismally performing network stack. There are only 2 possible explanations for that number:
1. His file copy was saturating a 1gigE link - if you've saturated the link, 40% is not great but is decent. However, the test is not applicable to most people who've seen the issue. It also means there is another 60% of the CPU for processing audio - that should be plenty.
2. His file copy was nowhere near saturating the link and Vista's network stack is horribly inefficient. My experience with pervious incarnations of Windows (2K, 2K3 and XP) has shown that under ideal conditions a single file copy will max out (because of inefficiencies in CIFS but that's another story) at ~35MB/s (roughly 1/3 of a gigE link in one direction). If Vista performs at roughly the same rate, then 40% CPU for 35MB/s is terrible. No wonder there is a degradation problem that required network throttling.
Looking down further to the NDIS packet graph, it appears that it is indeed explanation 2 that is correct. Peak throughput through the system was 24.6MB/s (17215*1500). If this test was similar to the CPU test for the previous screenshot, we are seeing 40% for 24.6MB/s. It appears the system will saturate its CPU at 50MB/s half-duplex?!? That's horrible. Or Mark is showing different numbers from different tests. I'm not sure which I want to believe.
Something appears to be very wrong with the network stack in these experiments. I don't have Vista. Can anyone test this?
P2P is also widely used to dowload free software! Mainly Linux distros! Oh... Nevermind.
Rethinking email
http://courtneymalone.com/2007/08/28/a-note-on-vis ta-network-speed/
So, basically the GP poster was right: 1% goes to WGA.
GPG 0x1B479C78