Speeding up Firewire File Transfers?

Milo_Mindbender asks: "I've got a pretty common problem: copying a ton of files from an old Windows XP computer to a new one. After noticing how long transfers were taking over my 100mbps Ethernet, I hooked up a IEEE1394/Firewire cable and things were much faster. Strangely though, Windows is still only using about 10% of the cable's 400mbps bandwidth. Does anyone know any tips/tricks for speeding this up or any Shareware mass-file-copy tools that would be faster than Explorer/file sharing? Right now, the older machine is setup with Windows file sharing and the new machine is copying from it, neither machine is using much CPU and the disks are nowhere near their max speed. The number and size of the files might be what's slowing it down, since it's gigabytes of files in the 100-200k size range."

Comments

archive then move?

[vjl]

Have you tried to archive/compress them first [gzip/zip/etc], then move the big file over? Lots of small files take longer to move than fewer larger files. /vjl/

Re:archive then move?

[biglig2]

Nothing to do with compression (although that may help); it's about one big file being faster to copy than lot's of small files that add up to the same size. Even if you zip them up without compressing (it'll be an option somewhere) then this will help.

Another thing is that even without looking at third party tools, you should be using XCOPY in preference to windows explorer.

There is an Exchange server utility that is optimised for moving gigantic files very fast; doubtless you can find similar programs about.

Re:archive then move?

[blincoln]

Robocopy is approximately a hundred trillion trillion trillion times better than xcopy.

To put that in perspective, you would need to weld fourteen quadrillion VW Beetles end to end, then use the resulting Beetle Bar as a lever and an object with the displacement of eleven million Libraries of Congress as the fulcrum in order to give xcopy the same Windows command-line file copying power as Robocopy.

Re:archive then move?

[mgv]

Have you tried to archive/compress them first [gzip/zip/etc], then move the big file over? Lots of small files take longer to move than fewer larger files.

Is it just possible that you are confusing bits with bytes per second? 400 Mb/s is about 40 MB/s (or pretty close, especially as you rarely the full theoretical 50 MB/s that you would think this would equate to).

Michael

Move the old hard drive, then copy

[TheArtfulTodger]

Why not just plug the old hard drive on the secondary channel on the new PC, reboot and then just file copy? Or do I need to reread the question?

Re:Move the old hard drive, then copy

[BrynM]

The problem with doing that is Windows can detect the "magic" folders... Mods: If you don't know what "magic" is, please mod some other comment. Thank you.

Do you mean the software named "Magic Folders"? Or perhaps you have some kind of Folder of Holding with compression created by a high-level Magic User. Most likely you mean the Special Folders that are used by Windows, but then again you may just be spouting about something you actually know little about.

Or maybe you should just explain yourself and not flame the mods... You might even get modded +Insightful or +Informative then and you would have the advantage of explaining your term to the person you were answering - thus being genuinely helpful.

Re:Move the old hard drive, then copy

[BrynM]

BTW, I learned the term "magic" from the Microsoft tool "tweakgui". I didn't invent it.

I looked in TweakUI and it calls them "Special Folders" as well. My point is: Be careful about how you present things that you may only have cursory knowledge of here on Slashdot. I was being funny about it, but there are planty of users out there who will ream you for bad information. Instead of being authorative and telling the mods what to do, you could have replied in the form of a question such as "You could try, but won't windows have problems with the magic folders?" or something like that. Further, you could disclaim being an authority and just post something along the lines of "I think that...".

Ok, I'm done being a slashdot post nazi now :D

From my experience, XP stores the hard locations (ie: c:\Documents and Settings\BrynM\My Documents") for the special folders it needs in the registry. I've never had a problem slapping a previously used drive into a machine to copy files (I did this exact thing to recover files after and IDE failure that was corrupting NTFS just a couple of weeks ago). If you've had problems or know of them, then please post that.

Re:Move the old hard drive, then copy

[cgenman]

Special folders don't exist on the second drive. Only the real folders do. So your desktop folder is still in a folder on the drive, you just have to know where to look. In this case, that's Drive:\Documents and Settings\User\Desktop. The virtual folder itself ceases to be where you expect it to be, as Windows is not searching your secondary drive for them, but it continues to be where it is.

If you want to know where a folder really is on the drive, just do a search for a file you think is in the drive.

In terms of Folders, Windows adds the contents of the folders together. If instead of being meticulous about your upgrade, you simply copy over your entire "Documents and Settings" folder, all of the contents of the folders that were virtual before will be added to the new folders that are virtual now. This works better if you keep usernames the same between machines. Nothing should break. I'd recommend against directly copying your program folders or windows folders, but nothing from there would work anyway.

I've done upgrades like this before, direct drive-to-drive through the intermediary OS. It really is the fastest way to do it.

Here

[abscissa]

Here you go.

Firewire is crippled in Windows by default. You need the patch here to restore functionality.

Re:Here

[Anonymous Coward]

OK, discussion is over. Everyone go home.

Re:Here

[PayPaI]

Non login requiring link here
Article seems to imply that only 1394b (FW800) controllers are affected. Still, limiting to S100 is a pretty dumb move on MSFT's part (insert tinfoil rant about MS being in league with Intel over FW vs USB2 here).
As a side note, I use an external FW drive on my HP laptop running XPSP2 Pro, and I can pull >35MB/s from it. I do not have any machines with 1394b so I can't confirm that there is an issue with that, but my 1394 port is definitely not running at S100.

Re:Here

[Anonymous Coward]

Thanks for the link but in order to download the fix MS requires you to have that Windows Genuine Advantage crap installed.

Until Microsoft gets a truthful disclosure I refuse to install it.

Re:Here

[kyouteki]

Errors corrected below: Windows is crippled by default. You need the patch here to restore functionality.

Sleep on it.

[Bob Cat - NYMPHS]

I bet when you wake up in the morning, things will look much brighter. They always do.

IF you wake up. Muahahahahahaaa....

Find the bottleneck

[philocipher]

If your not maxing out your connection, any number of things could be limiting the speed of the transfer; cpu, bus speed, harddrive performance etc. Use a system monitoring utility to see what's at 100% utilization, and then upgrade the part. Transfering larger files that are sequential on the disk will also help.

What are you moving now?

[digerata]

"...since it's gigabytes of files in the 100-200k size range."

That's quite a collection of pr0n!

Re:It's the hard drive, not the fire-wire

[Anonymous Coward]

your hard drive is most likely not fast enough to receive the full 400 Mb/s stream from the firewire. The fastest SCSi drives are 320Mb/s and that's not sustained.

You are confusing MByte/s and MBit/s. Firewire is 400 MBit/s, while SCSI is 320 MByte per second.

File size is the problem

[inio]

Your file size, and disk seek time, are the problem. Lets say your drive has a 5ms seek time (that's pretty damn fast). writing each file actually requires three writes: to the file allocation tabe, to the directory, and the contents of the file itself. Assuming the writes take another 5ms, that's 20ms per file. that limits you to 50 files per second. At 200kiB per file that's about 10 megs per second.

Re:Linux with NFS or maybe ghost?

[BandwidthHog]

All in all Explorer sucks ass at file operations. You would probably get beter transfer rates with ftp.

Even IP Over Carrier Pigeons would yield some sort of improvement.

If you’re gonna use an operating environment that shits all over your desktop anyway, ya might as well go all out.

holy crap

[fishdan]

how is THIS not well known? MSFT blows. Thanks for the great tip

Re:Use FTP

[andy753421]

If you're using tcp/udp and you're really worried about protocols slowing you down here's a nice way to bypass them almost completely.
user@destination $ netcat -l -p 12345 | tar -x
user@source $ tar -c /usr/src/linux-2.6.17.1 | netcat desktop 12345

Re:It's the hard drive, not the fire-wire

[ArbitraryConstant]

Firewire 400 is 400 megabits per second.

A modern SATA drive can do just shy of 70 megabytes per second, which is 560 megabits.

Try searching for an rsync clone for win32

[rwa2]

Well, I could think of a lot of ways to speed it up under Linux using various combinations of rsync, and... well, really just rsync. See if there's a good rsync clone for Win32 that will preserve your precious file attributes. Even running it under cygwin may be better in the long run, especially because inevitably (speaking from experience) your large copy will be interrupted halfway through by an "unreadable file" or some such rubbish, and you'll find yourself having to try to fix it and start the copy all over again from the beginning, or else trying to just transfer the remaining directories you think you're missing.

Using cygwin's rsync via ssh: (after running "ssh-host-config" on your new box and setting a "passwd" as Administrator )

rsync -azve ssh --progress /cygdrive/c/pr0n/ Administrator@newxpbox:/cygdrive/c/pr0n/

will do the trick, and you can just keep running it over and over again until all the files are mirrored. It will take a long time to buld a list of all the files you need to transfer, but it will only tranfer the files you're missing, and will attempt to do some compression (which should help because you're more IO bound than CPU bound, but just remove the -z if your CPU is pegged). Plus, you'll find rsync & scp damn useful for many other common tasks you take on.

The bottleneck is probably your windows filesystem, and cygwin's extra abstraction layer will only make that worse. But using rsync under cygwin means you only have to transfer the files once - which will be a much bigger time saver than trying and failing to do the entire transfer several times.

If you were doing this often, I'm guessing you might see an improvement if you defragment your old drive first, but you obviously don't really want to waste time on that for a once and final transfer.

Also, the Windows TCP/IP stack is typically tuned for 2 - 10Mbps links. Here's some information on how to fix that: http://rdweb.cns.vt.edu/public/notes/win2k-tcpip.h tm It's mainly geared towards improving throughput on high-capacity WAN links, but parts are also relevant to achieving decent performance on 100Mbps+ networks as well. Also remember that a lot of network drivers suck too and are incapable of pushing the throughput even to a fraction of its rating... that's been a factor too, especially on cheap windows crap. An updated NIC driver /might/ get your net transfer to catch up with your firewire transfer somewhat.

Since you're getting 40Mbps / 400Mbps firewire, you're really not doing too bad. Converting to bytes, 5MB/s is a decent fraction of the 20MB/s to 50MB/s raw speed of your older hard drives, and actually seems reasonable given that you're sending lots of small files and not a few big ones where you can actually make good use of your drive's readahead cache.

Pay attention to units

[Bogtha]

Strangely though, Windows is still only using about 10% of the cable's 400mbps bandwidth.

Are you sure you aren't confusing mbps with MBps? 400mbps is equal to 50 megabytes per second, and "12.5% of the cable's bandwidth" sounds suspiciously like your description of the problem, "about 10% of the cable's bandwidth".

Re:Pay attention to units

[mattmacf]

Are you sure you aren't confusing mbps with MBps? 400mbps is equal to 50 megabytes per second...

Are you sure you aren't confusing mbps with Mbps?

Last I checked, 400mbps is equal to 0.4 bytes per second. I remember getting speeds like that back in the days of dial-up. Like back in the day when you picked up the phone and read off ones and zeros to your friend on the other line as he copied your fortran program by punching holes in his punch card. Oh yeah, and uphill both ways. In the snow.

; )

[/toungeincheek]

Re:Pay attention to units

[BillyBlaze]

Nope, 400 mbps would be .4 bits per second, which is .05 bytes per second, or about .000071 Libraries of Congress per millenium.

Similar Problem on Macs

[R3d M3rcury]

I did a freelance gig back in '98 where I had to use a Mac (an 8600/300 w/64 megs of RAM). It took well over 20 minutes to copy a 17 meg file from one folder on the hard drive to another. 20 minutes! At home, on my Pentium Pro 200 running NT4, the same operation would take about 2 minutes.

(Admit it. You knew this was coming.)

Re:xcopy?

[1u3hr]

Is there now a version that supports long filenames?

See my earlier post about XXcopy, http://www.xxcopy.com./

Have you tried this?

[Anonymous Coward]

There is a problem in Windows XP SP2 with firewire transfer. Albeit that it could be numberous small files creating problems but it should be faster than 100mbps ethernet. Try this blog regarding Windows XP SP2 Firewire Slowness for a link to the KB and a links to few other work arounds or just go direct to the KB article.

Re:It's the hard drive, not the fire-wire

[ArbitraryConstant]

"Modern? This is from an old computer to a new computer - and those speeds would be for consecutive data. Which I suppose is fair, given that the user defragmented completely. But an IDE drive's speed is far less 400mbit/s."

They're just about as fast as SATA drives, since ATA-100 is still faster than the sustained speeds of the drives (100 is megabytes in this case). This is why ATA-133 never caught on -- it's faster than any of the drives you'd connect it to. It wasn't SATA's speed that made it popular, it's the numerous other advantages (thinner cable, cheaper, hotplug, etc).

Re:It's the hard drive, not the fire-wire

[PB_TPU_40]

Acuatlly hardrives both IDE/EIDE, and normal scsi are measured in MB/s note the capital B meaning BYTEs not BITs. External communication channels are measured in bits/s. Only the new SATA, and SerialSCSI drives are bits/sec. If you calculate it out, a 3.0Gb/s SATA drive pulls about 375MB/s burst rate. Its marketing manipulation.

I'm not sure why you're transfers aren't that fast, for me firewire from my external harddrive is just as fast and getting stuff off my fileserver *6 disk raid 0*. I have yet to max out my gigabit system, and that was copying 3 dvd images to my desktop (U160 SCSI) *I was doing this to determine upper bandwidth from my server* and even then I was sustaining only about 60 MB/s, note thats sustained.

My firewire disk drive copies at a slightly slower rate, but at about 30MB/sec, however considering the theoretical MAX with just transfer, no protocol or system overhead is 50MB/sec. Another question is how much ram do you have, cause most certainly if your sending drive is slow, or your recieving is slow, everything is limited to that speed.

Optimize for performance

[Svenne]

No one's mentioned this?

Bring up the properties of the firewire disk in "Device Manager". Go to the Policies tab and make sure it's set to "Optimize for performance".

Re:It's a good thing you don't write Windows...

[$RANDOMLUSER]

I don't know which is worse, how breathtakingly wrong this post is (Windows: now it sees into the future) or the fact that this AC post got modded "insightful".

NSCopy

[megabyte405]

I use NSCopy for any decently-sized Windows File Sharing file transfer - it can copy a whole directory tree and throttle the speed down or up (to maximum "plaid") Just google for it, it's free.

If you want more speed, I'd say get FireZilla (an FTP client) and FireZilla Server (an easy to use FTP server), both open source and free. Set up the server on the "source" computer, and download as fast as you can! It will use the bandwidth much better.

One of the other suggestions about moving the hard drive would work too, but these are within your original constraint of improving network transfer performance. If you move the hard drive, use a text-mode (like xcopy) utility to copy from the command prompt - far faster than using Explorer.

Good luck!

Synctoy

[joeaic]

If you don't want to use the command line xcopy, then I suggest you download a copy of Synctoy from Microsoft.

Whitepaper: http://www.microsoft.com/downloads/details.aspx?fa milyid=49818CF1-2287-40EA-8A6F-57BD8695F23D&displa ylang=en

Download:
http://www.microsoft.com/downloads/details.aspx?fa milyid=E0FC1154-C975-4814-9649-CCE41AF06EB7&displa ylang=en