Slashdot Mirror
Vista Runs Out of Memory While Copying Files
ta bu shi da yu writes "It appears that, incredibly, Vista can run out of memory while copying files. ZDNet is reporting that not only does it run out of memory after copying 16,400+ files, but that 'often there is little indication that file copy operations haven't completed correctly.' Apparently a fix was scheduled for SP1 but didn't make it; there is a hotfix that you must request."
the box I "make use of" has just 15,000 mp3s...
They who would give up an essential liberty for temporary security, deserve neither liberty or security - Ben Franklin
16k files should be enough for everybody.
At the end, there will be free therapy. And Cake!
Physics is nothing like religion. If it was, we'd have an easier time trying to raise money!
And that is one of many reasons we are all still running XP
Copyright 2010. All rights reserved. This comment may not be copied in any way including, but not limited to caching.
Isn't it a little odd that if you strip off the first and last digits of the number "16,400", it's 640, as in 'no one needs more than 640k"?
Start a happiness pandemic
M$ is scared that people will try to copy their documents to another computer before reverting back to XP. Smart, very smart Micro$oft! On a tech note, what kind of number is 14,600? I would have thought 16,384 would be better.
void r() { printf("recursion is "); r(); }
I don't want to start a holy war here, but what is the deal with you Vista fanatics? I've been sitting here at my freelance gig in front of a Vista PC (an Intel Core 2 Duo w/4 gigs of RAM) for about 20 minutes now while it attempts to copy a 17 Meg file from one folder on the hard drive to another folder. 20 minutes. At home, on my ancient Mac running OS 9, which by all standards should be a lot slower than this Vista PC, the same operation would take about 2 minutes. If that.
In addition, during this file transfer, Firefox will not work. And everything else has ground to a halt. Even Notepad is straining to keep up as I type this.
I won't bore you with the laundry list of other problems that I've encountered while working on various Vista PCs, but suffice it to say there have been many, not the least of which is I've never seen a Vista PC that has run faster than its Mac OSX counterpart, despite the Vista PC's same chip architecture. My 286/12 with 2 megs of ram runs faster than this 2.4ghz mhz machine at times. From a productivity standpoint, I don't get how people can claim that Vista is a superior operating system.
Vista lovers, flame me if you'd like, but I'd rather hear some intelligent reasons why anyone would choose to use Vista over other faster, cheaper, more stable systems.
they can only send 16,000 files to the RIAA and MPAA to check, at once.
If you are using Vista as a server, you pretty much deserve what ever happens to you.
Well.. maybe. Or Maybe not. But Definitely not sort of.
Oh, the fools! If only they'd built it to let you copy 16,401 files!
I have 13K+ music tracks on a backup disk. If I try to copy them with the Explorer UI, it does nothing - No error message or anything. I reverted to Robocopy, which works fine. You must be doing the same thing. Doesn't anyone at Microsoft have a big music collection to copy, or do they just use their Macs and iPods for that?
Imagine how much harder physics would be if electrons had feelings! -Feynman, maybe
Maybe you're backing up to an external hard drive?
The underlying problem is a Windows OLE component memory leak. Microsoft has a hotfix for the issue at http://support.microsoft.com/kb/942435/en-us
I want an OS that lets me re-organize my pr0n anytime I want. I *need* to be able to select 50K-100K files at a time and move them from place to place without slowdowns. Ever try, in Windows, to search your network for all the *.jpg files, select a few hundred thousand of them in the search window, and drag them to the new firewire disk you just plugged in? It's painful, lemme tell ya.
Anybody want to suggest an OS that would work for me? I'm serious.
-- "I never gave these stories much credence." - HAL 9000
.. when your GUI is using 2 gigs of RAM.
The reason being is I've setup a Vista system and copied about 100,000 files (totaling about 60GB) drive to drive in a single operation, without error. So while I'm not saying this isn't a Vista error, I'm wondering what else has to be done to trigger it. The persisting across reboots, even if you break it down smaller really makes it sound like another program is somehow interfering with the copy. I'll have to mess around with it at work, we have Vista test machines and Cadence installs north of 250,000 files when you install its libraries. I know it installs fine, though that isn't a copy strictly speaking as it is files being extracted from archives.
I'm just wondering if perhaps there isn't more to this than just "OMG Vista runs out of memory!" If it is a memory issue, why then haven't I encountered it, doing far larger amounts of files?
How the F%$^ can this be a problem? A file copy is a simple operation. There's simply no excuse for this... This should have never been a problem in the first place. What pisses me off is that I need to buy a new laptop, Vista is now forced down my throat, and I have no option to get XP pre-installed.
Actually, the bug is in the shell, not the kernel and only files with altnerate data streams trigger the leak. The KB article that Adrian links to states that very clearly, but he's been on an anti-Windows rampage lately that's blinded him to the facts.
Very few files have data streams, so the vast majority of users won't ever see a problem. Kaspersky choses to pollute every single file with a stream, however, which is why systems with it installed exhibit the problem.
If it is *just* kapersky that causes the problem, then why is MS issuing a hotfix instead of kapersky?
When information is power, privacy is freedom.
Very probably it's 16,384, as in 2^14. I'm sure it was a hardcoded limit. So typical, Microsoft... so typical.
FTA: The "Out of Memory" error (which is affectionately known at the PC Doc HQ as the "Out of Cheese" error
I don't think there has ever been a version of Windows that could deal with large numbers of files. Particularly if you are using the GUI interface. The whole thing is a toy operating system, really.
A few years ago, while investigating a similar problem with a production server (a SERVER not a client machine) the machine would gradually grind to a halt doing the copy, while still responding (but slowly) to other operations.
I found that the "copy" command did much better than a drag and drop operation, but still would have a problem eventually. Finally, I found that this was a known problem, and that to solve it, a dedicated MS employee had written a utility called "robocopy" the "robo" not being for "robot", but for "robust" (really, it said that!).
Using that usually got the job done, much more slowly than it should have, but at least I didn't have to re-boot the machine daily to clear things up.
Now that Gates is too busy with other things to take tours of the data center, really, Microsoft should do itself a favor and ditch the VMS underpinnings of Windows (some of which they have probably forgotten how to maintain) and build your nice GUI on top of BSD or something similar. That way you won't break your budget (in manpower and electricity) trying to match the Google server farms.
Once that's done you will have the experience needed to do the same on the desktop. You will be doing the world, and yourselves a favor. Thanks in advance!
Where are you shopping? Most computer manufacturers still offer Windows XP instead of Vista as an option on their computers.
You know you can opt for a Macintosh. With Leopard coming out and all. Plus, if you *need* Windows, it is just a reboot away or something.
and then you take the 640, divide by 2, get 320, remove the last 0, and get 32, the reverse of 23.
Then you multiply 23 by 30, get 690, take 42, reverse it to 24 and subtruct this from 690. You get 666.
What does that tell you? Ha? Ha?
You can't handle the truth.
Very few files have data streams, so the vast majority of users won't ever see a problem. Kaspersky choses to pollute every single file with a stream, however, which is why systems with it installed exhibit the problem.
So it's Kaspersky's fault that alternate data streams are apparently no longer supported by Vista, despite being a basic part of NTFS?
The original rant may be found here.
"How the F%$^ can this be a problem?"
To ensure backward comparabillity. I'm a techno luddite. I got my first DVD drive this year and was slow to get CD drives. All my systems have floppy drives.
98 can be made to puke copying a big file from a floppy. If the floppy is bad you may as well reboot. Delete a few gigs from a hard drive and it goes awy for ages and will more often then not kill the gui task. This is very repeatable. Again, if the CD is bad, reboot.
I can make XP croak as well copying huge files from a CD or floppy which is handles very very badly (see 98). And my biggest drive is 20G (albeit a damn fast one). It seems to do ok copying big files from hard disk to hard disk but even with SCSI RAID with huge caches and the correct drivers you can't expect much left of your CPU when its doing this. Do two at once and you may as well go rebuild your transmission while you're waiting. Apparantly DMA and interrupts are unknown concepts at Redmond; PDP-11's did this just fine (unless you turned off DMA and interrupts in which case it was no faster than a 4Mhz Z-80 CP/M system)
There's really no excuse for this. In the days of 8 bit microprocessor systems we still went out and got the biggest pre-production drives we could to see if they'd copy ok. They may have filled a room but the Navy did indeed have 100 megs online pumping its data through an 8085. Eventually. We knew it'd work cause we tried it. This was 1981.
This is why they use real (IBM, SUN) computers to serve up say, the root or com zone. The root zone isn't big but the com zone is. Copying it isn't a problem on any unix system I've tried, just don't try to load it into BIND on anything but a massive computer or it'll just hang. And not gracefully either.
Windows is for games and sometimes works well enough to run some office tools. As long as you don't need accuracy.
Big files or LOTS of small files are a problem for computers. This isn't news folks. It's just sloppy carelessness.
Need Mercedes parts ?
What does that tell you? Ha? Ha?
You've got too much time on your hands?
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
...why would anyone want to copy 16400+ files? What if a person were to upgrade their hard drive.Linux Zealots: Smarter than Mac Zealots, but still zealots.
mod parent as stupid.
Likely, they're allocating memory to store file attributes or some such that are not being free'd when done with. Hence running out of memory. If you had coded a day in your life you'd see that.
Tom
Someday, I'll have a real sig.
only files with altnerate data streams trigger the leak
Well that's what you get for crossing the streams. Egon warned us. Kaspersky's risking total protonic reversal. I guess they were fuzzy on the whole good/bad thing.
Developers: We can use your help.
This is a minor problem, absolutely rare event, occurs with next to no regu...
**OUT OF MEMORY ERROR, SYSTEM HALT**
Karma Whoring for Fun and Profit.
Actually, a lot of the problems I've noticed with XP is related to the stupid fucking way that Windows handles it's file cache. It will literally swap out PROGRAMS YOU ARE ACTIVELY USING to expand the file cache during a large copy/read operation.
Anybody that has ever tried to alt-tab while copying huge files knows about this.... then you sit and wait for the pages to be swapped back into memory. And you might as well get some coffee, cuz with the hard drive already being pegged for the copy operation, it's gonna take awhile. Oh, and once it's finally done and you need to alt-tab back to the original program.... well, hope you need more coffee.
Lately I've been playing with a program called CachemanXP. Google it. It seems to give you more control over the memory and process management functions of Windows. It also lets you do a 'kill -9' equiv, which (as far as I'm aware) even Task Manager won't do, as it insists on trying to do a graceful shutdown first.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
We have found the first slashdot user that has no porn.
There is a similar bug in all unix, probably linux too. :)
If there are 16k files in a simple directory, type rm *.
If it works, try the same with 32k files.
Then 64k files. Eventually it will fail
Patents Drive Free Software as Hurricanes Drive Construction Industry
Win95, Win98, and WinME all can't handle more than 512MB of RAM anyways
SYMPTOMS
If a computer that is running any of the versions of Windows that are listed above contains more than 512 megabytes (for example, 768 megabytes) of physical memory (RAM), you may experience one or more of the following symptoms:
You may be unable to open an MS-DOS session (or command prompt) while Windows is running.
The computer may stop responding (hang) while Windows is starting, or halt and display the following error message:
Here is one of their suggested workarounds:
Reduce the amount of memory that is installed in your computer to 512 MB or less.
Here is their support article on it http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q253912&ID=KB;EN-US;Q253912
ZERO ZERO ONE ZERO ONE ZERO ONE ONE! Just brushing up for my next big invention: Ethernet over Voice (EoV)
That sort of depends if you are talking average home user or average work user. The average home user may actually have this kind of problem - since downloads to the tmp directory are then copied to the correct folder once downloads are complete. Update EQII, WOW & FFXI & you've gone a long ways towards 16K files. Add in patch Tuesday, and your average user is probably going to hit real close to 16K files if they try to keep the PC up for a month.
I probably come reasonably close to 16K files copied in a week on my work PC, so a crash like that would hit me every other week or so - not something I would consider 'Enterprise Ready'.
MS has a habit of programming for the home environment & pushing it into the Business environment.
their bug is legitimate. Unix does shell globbing (yes, it IS fucking retarded), which means that wildcard expansion is done by the shell, and there's a definite file line limit. This can easily be solved using an xargs (or find or something) command, and most shell geeks know how to/that they have to do this, and I don't think any of the GUIs have a problem similar to this.
Actually, it's more like 90 / 90000, but you're only a few orders of magnitude off.
Not necessarily his fault, maybe he used Excel to calculate it...
It's forced down your throat because you couldn't be bothered to source a laptop from anywhere but CC?
Give me a break.
"It does not do to leave a live dragon out of your calculations, if you live near him." - Tolkien
It's not a simple operation. Think about how a file copy works within Vista:
1. The file is opened.
2. The file is scanned for viruses.
3. The file is scanned for adware.
4. The file is scanned for DRM violations.
5. The user is asked if they're really sure they want to copy the file.
6. The user is asked again if they're sure they want to copy it.
7. The OS makes a judgement on how long it will take to copy so it can update the pretty stats in the gui.
8. Lots of flashy graphics and widgets are loaded to show you a pretty animation while you wait.
9. The file is copied.
10. The destination file is verified that it is intact.
11. The destination file is scanned for viruses.
12. The destination file is scanned for adware.
13. The destination file is scanned for DRM violations.
14. The file is successfully copied.
Hell - I'm surprised their OS can even handle copying 1,600 files, let alone 16,000.
Just disrupt the deflector shield with a tachyon burst.
"Very few files have data streams, so the vast majority of users won't ever see a problem. Kaspersky choses to pollute every single file with a stream, however, which is why systems with it installed exhibit the problem."
Yeah, that's the typical Windows world attitude.
The operating system is specified to do certain things. It doesn't do them. Well, if not many people use this feature, so what? One of the way we make the feature list long is by including lots of features that don't work, but we figure nobody will use them and nobody will find out...
"Waiter, there's a fly in my soup."
"What kind of soup?"
"The orange scented celery puree.
"Oh, hardly anyone orders that. You should expect flies in it. It's your own fault for being foolish enough to order it."
"How to Do Nothing," kids activities, back in print!
"Vista is now forced down my throat"
I got news for you, that is Vista, but it isn't your throat it being forced into.
There are no loopholes. It's either legal or it's not.
you just keep on trying till you run out of cake.
How can you expect consistent moderation? Because of the way the system works, the first mod can effectively hide a post from other modders (yes they advice you to mod at -1 but who does that?) so wether a post gets seen or not depends on who gets to see it first. Mod it down, and it is gone. Mod it up, and more people see it, this includes downward modders but also upward modders, so it totally depends on the first mod.
Then there is the fact that moderation is totally random. So random that sometimes I get them constantly while othertimes I go for weeks without. Meta moderation is even worse.
Then there is mods themselves, not the moderators, but the way you can mod. There is no simple "FALSE" mod. or "you are an idiot" mod. Or a "True" mod. Flamebait and troll are way to often used to mean "I don't agree with you." Insightfull and intresting far too often for someone just stating the facts.
So don't expect sense from the moderation system, just accept that some of the best posts will disappear forever because a mod didn't like it, and that some crap will be modded up.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
Microsoft employees buy another copy of their 13K+ music collections if they want another copy.
Insert self-referential sig here.
This has to do with shell expansion and the argument list that gets passed to exec(). It's fixed in a recent Linux kernel version, it now allows argument lists limited only by available memory.
Of course there were always ways around it.
-- Alastair
Shouldn't this be "divide by 8.3" in the true DOS spirit of things? ;-)
Ever wondered whats wrong with the world? http://www.ishmael.org/
I expected this very problem to happen and have taken corrective measures. Simply, I use one large file for everything.
Think about the innovation that is being created here. I can access all the spreadsheets I have ever created whilst updating my current webpage project, search for an e-mail archive and read the latest TPS Report Coversheet without changing files. It means I don't have to partition my hard drive. And I only have *one* file ever to backup.
This has simplified my daily work to the point I fired all my IT staff. Thanks Microsoft!
First of all, the issue is how Explorer handles extended attributes (EAs), which are distinct from alternate data streams (ADSes). The kernel and NTFS have always provided full support for EAs and ADSes (since NT 3.1). Explorer (and for that matter Win32) has never had very good support for ADSes, and almost nonexistent support for EAs. EAs were implemented in support of the OS/2 subsystem. ADSes are the 'official' way to attach metadata to a file, and scale better than EAs. The only Win32 functions that have ever provided access to EAs are the BackupRead and BackupWrite functions which are designed to handle all metadata on a file transparently. Looking at the imports from shell32.dll to ntdll.dll on Vista, it looks like the shell bypasses Win32 when dealing with EAs, invoking the syscalls NtQueryEaFile and NtSetEaFile directly (bypassing API layers like this is something Microsoft tells ISVs is a big no-no).
This is just Yet Another Windows 95 shell bug (yes Vista uses the same shell architecture ported through each version from Win95). It is not the end of support for EAs or ADSes. If anything, it's a belated attempt at better support, done poorly. The shell has always been, IMO, one of the lower quality windows components, especially when it comes to properly interfacing with lower layers. This bug does not surprise me. I've been using robocopy for nontrivial file transfer for a while now.
For those situations...
/s
Run -> "cmd" -> del %dir\*.*
It will clear most stuff and you will see error messages fly by... redirect output to a file for later examination if desired.
I use the good old 'del' whenever I know I will be deleting something like 20k files and do not wish to waste time waiting for windows to prepare for that operation... why the heck does Windows need to scan directories to be deleted before deleting them is beyond me, just delete them and be done with it. Same thing for copying, Windows wastes time scanning the source directory for no apparent reason since it won't tell you you have insufficient disk space to complete the operation until the target drive runs out of disk space... or any other errors for that matter, until it runs into them while carrying out the actual operation.
Linux has quirks, so does Windows. Linux has the excuse of being an relatively immature desktop OS but on the Windows side, it can only be written off as the result of half-ass design decisions.
Yes, but everyone knows one-in-a-million chances come up nine-times out of ten.
Not a sentence!
Yup. I've just about given up trying to copy gigabytes of files from large drives to backup drives because of this. I mean, once you've done this ONE TIME, it should be OBVIOUS to ANY designer that it needs to be fixed to allow the copy to continue - or resume - or SOMETHING other than just DYING.
Also, if Windows sees a zero-byte file, it can't handle it. I have to boot Linux and use it to delete the file.
Daily while working with clients I ask myself how anybody could use this garbage on a daily basis. When I reboot into Windows on my dual-boot openSuse/XP machine, I always dread it because I KNOW something is not going to work properly, or something extraneous will have to be done BEFORE I can do what I need to do.
Pathetic OS. Just pathetic.
Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
Pop quiz: How do you calculate the remaining time?
Don't. It's going to be wrong, anyway.
How do you handle infinitely recursing soft links?
Does Windows even support soft links?
That does not make a bit of sense, either one of you!
Added Pressly: "Oh, and by the way, milk is nothing but liquid meat."
Yes but it can be fixed, or rather, worked around. It's been this way for literally decades.
Need Mercedes parts ?
find -name "*base*" -exec chown us {} \; ; ln -s
Someone else in this thread mentioned CachemanXP. This non-crippled shareware allows you to limit the amount of free RAM XP uses for file caching (and has a nifty "kill" option, for when you absolutely positively wanna toast the sucka -- the reason I checked it out).
I come here for the love
It's a shame that Open Source chooses names that turn most people off.
If you want news from today, you have to come back tomorrow.
Also, if Windows sees a zero-byte file, it can't handle it. I have to boot Linux and use it to delete the file. The strange thing is, both of these are idiosyncracies of the explorer.exe shell. Both of these things work fine from the command line from within windows, and if you want lots of move/copy options, you use xcopy.
Which is sad, because clearly its not required for the OS to have these problems, but they exhibit these problems in the GUI shell, where the vast vast majority of people will encounter them.
Honestly, how can [Cancel or Allow] anyone co[Cancel or Allow]py that m[Cancel or Allow]any files anyway?
From the KB article describing the problem, ie TRFA (where R = Real):
http://support.microsoft.com/kb/942435/en-us This problem occurs because of a memory leak in the Windows OLE component. This memory leak is triggered by the way that Windows Explorer deals with the extended attributes of the files. This is 100% in the shell and UI layers, not in the kernel.
with XP while the activation pissed some people off only home users and small buisnesses who bought legit had to put up with it. Corps, educational establishments and pirates got a slightly different version which did not have that misfeature. Most of the rest of the bad things about XP were bugs and bloat. Bugs were fixed and bloat became less significant with time. Vista pushes it's activation on everyone except those prepared to buy big brand OEM and prepare seperate images for every brand of machine.
but sadly I suspect you are right, apps are king and once vista is the only version of windows that is reasonablly availible people will use it whether they like it or not.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
No. When removing a file, Windows will always tell you beforehand if you lack disk space for the operation. It is a feature, stop complaining.
Rethinking email