Slashdot Mirror
Speeding up Firewire File Transfers?
Milo_Mindbender asks: "I've got a pretty common problem: copying a ton of files from an old Windows XP computer to a new one. After noticing how long transfers were taking over my 100mbps Ethernet, I hooked up a IEEE1394/Firewire cable and things were much faster. Strangely though, Windows is still only using about 10% of the cable's 400mbps bandwidth. Does anyone know any tips/tricks for speeding this up or any Shareware mass-file-copy tools that would be faster than Explorer/file sharing? Right now, the older machine is setup with Windows file sharing and the new machine is copying from it, neither machine is using much CPU and the disks are nowhere near their max speed. The number and size of the files might be what's slowing it down, since it's gigabytes of files in the 100-200k size range."
Have you tried to archive/compress them first [gzip/zip/etc], then move the big file over? Lots of small files take longer to move than fewer larger files. /vjl/
My Daily photo website.
Why not just plug the old hard drive on the secondary channel on the new PC, reboot and then just file copy? Or do I need to reread the question?
Here you go.
Firewire is crippled in Windows by default. You need the patch here to restore functionality.
I bet when you wake up in the morning, things will look much brighter. They always do.
IF you wake up. Muahahahahahaaa....
The latest Slashdot meme.
If your not maxing out your connection, any number of things could be limiting the speed of the transfer; cpu, bus speed, harddrive performance etc. Use a system monitoring utility to see what's at 100% utilization, and then upgrade the part. Transfering larger files that are sequential on the disk will also help.
That's quite a collection of pr0n!
1;
Agreed 100%. Any time I need to transfer massive amounts of files (and physically moving the hard drive from one machine to another is not an option), Filezilla never lets me down. Server on one side, client on the other... they're a quick setup and just as easy to install as remove when they're no longer needed. Its amazing how much data can fly through a router that's actually working it full capacity.
your hard drive is most likely not fast enough to receive the full 400 Mb/s stream from the firewire. The fastest SCSi drives are 320Mb/s and that's not sustained.
You are confusing MByte/s and MBit/s. Firewire is 400 MBit/s, while SCSI is 320 MByte per second.
Get 2 knoppix discs and use nfs. Firewire and ethernet arn't whats slowing you down. I always get 92mbit transfer rate from my desktop (ext3) to laptop (ext3 and fat32). You do run the risk of losing file attributes though.
Is there any enterprise software you can use like ghost?
All in all Explorer sucks ass at file operations. You would probably get beter transfer rates with ftp.
What is the manufacturer of Firewire controllers in the computers you are using? VIA controllers are usually not the best Firewire controllers. Texas Instruments controllers are usually better. For that matter, depending on the situation try compressing the files. Also, do not depend on time remaining in Windows I have found it wildly inaccurate at times. Windows seems to estimate the time remaining to be way too high, so YMMV, literally.
Impersonating Tycho from Penny Arcade since before there was a PA.
Your file size, and disk seek time, are the problem. Lets say your drive has a 5ms seek time (that's pretty damn fast). writing each file actually requires three writes: to the file allocation tabe, to the directory, and the contents of the file itself. Assuming the writes take another 5ms, that's 20ms per file. that limits you to 50 files per second. At 200kiB per file that's about 10 megs per second.
If you want to keep the files separate.. use Robocopy.. (free from MSFT)..
If you don't mind the files being in one glump.. use a Win32 port of Tar..
Both options above seem to speed up Firewire (or any) transfer.
how is THIS not well known? MSFT blows. Thanks for the great tip
Nothing great was ever achieved without enthusiasm
If you're using tcp/udp and you're really worried about protocols slowing you down here's a nice way to bypass them almost completely. /usr/src/linux-2.6.17.1 | netcat desktop 12345
user@destination $ netcat -l -p 12345 | tar -x
user@source $ tar -c
I haven't used xcopy in a while. Certainly not on an XP machine. The last time I used it was on a w98 box and it truncated all of the filenames to 8.3 format (7~.3 really). Is there now a version that supports long filenames?
Some mornings it's hardly worth chewing through the restraints to get out of bed.
Assuming you can't change any of your hardware, and don't want to go for any massively difficult software, just use Windows' built-in xcopy32 command. Yes, it's command-line, but it's the fastest free-and-easy utility out there.
And, as others have pointed out, you are NEVER going to hit 400 mbps, no matter what software you're using. 400 mbps is 50 MB/s, do you honestly think that old PC has a drive capable of sustained 50 MB/s transfers? Do you have a drive in the FW enclosure that is capable of that? Only the absolute latest PATA drives are capable of sustaining that, and even then, only over sequential transfers. Any random access, and you're dropping well under that.
If you want a perfect copy of the drive, use a sector copier, like Norton Ghost or Partition Magic.
Another non-functioning site was "uncertainty.microsoft.com."
The purpose of that site was not known.
Firewire 400 is 400 megabits per second.
A modern SATA drive can do just shy of 70 megabytes per second, which is 560 megabits.
I rarely criticize things I don't care about.
Well, I could think of a lot of ways to speed it up under Linux using various combinations of rsync, and... well, really just rsync. See if there's a good rsync clone for Win32 that will preserve your precious file attributes. Even running it under cygwin may be better in the long run, especially because inevitably (speaking from experience) your large copy will be interrupted halfway through by an "unreadable file" or some such rubbish, and you'll find yourself having to try to fix it and start the copy all over again from the beginning, or else trying to just transfer the remaining directories you think you're missing.
/cygdrive/c/pr0n/ Administrator@newxpbox:/cygdrive/c/pr0n/
h tm It's mainly geared towards improving throughput on high-capacity WAN links, but parts are also relevant to achieving decent performance on 100Mbps+ networks as well. Also remember that a lot of network drivers suck too and are incapable of pushing the throughput even to a fraction of its rating... that's been a factor too, especially on cheap windows crap. An updated NIC driver /might/ get your net transfer to catch up with your firewire transfer somewhat.
Using cygwin's rsync via ssh: (after running "ssh-host-config" on your new box and setting a "passwd" as Administrator )
rsync -azve ssh --progress
will do the trick, and you can just keep running it over and over again until all the files are mirrored. It will take a long time to buld a list of all the files you need to transfer, but it will only tranfer the files you're missing, and will attempt to do some compression (which should help because you're more IO bound than CPU bound, but just remove the -z if your CPU is pegged). Plus, you'll find rsync & scp damn useful for many other common tasks you take on.
The bottleneck is probably your windows filesystem, and cygwin's extra abstraction layer will only make that worse. But using rsync under cygwin means you only have to transfer the files once - which will be a much bigger time saver than trying and failing to do the entire transfer several times.
If you were doing this often, I'm guessing you might see an improvement if you defragment your old drive first, but you obviously don't really want to waste time on that for a once and final transfer.
Also, the Windows TCP/IP stack is typically tuned for 2 - 10Mbps links. Here's some information on how to fix that: http://rdweb.cns.vt.edu/public/notes/win2k-tcpip.
Since you're getting 40Mbps / 400Mbps firewire, you're really not doing too bad. Converting to bytes, 5MB/s is a decent fraction of the 20MB/s to 50MB/s raw speed of your older hard drives, and actually seems reasonable given that you're sending lots of small files and not a few big ones where you can actually make good use of your drive's readahead cache.
Are you sure you aren't confusing mbps with MBps? 400mbps is equal to 50 megabytes per second, and "12.5% of the cable's bandwidth" sounds suspiciously like your description of the problem, "about 10% of the cable's bandwidth".
Bogtha Bogtha Bogtha
"Your drive doesn't keep up with Firewire. Sorry, but it's true. Disks aren't as fast as you think -- particularly that cheap ATA thing you are using."
Firewire 400 is 400 mbit/s. A modern 7200 rpm SATA desktop drive can sustain just shy of 70 mbyte/s, which is 560 mbit/s.
I rarely criticize things I don't care about.
First of all, I'm confused by what you mean when you say it's only using 10% of bandwidth. 400Mbps, means you're gonna get something like a max of about 40 Megabytes per sec transfer. (remember, 8 bits per byte, plus some overhead) Are you seriously only getting 4 MB/s?
As far as copying faster. You might want to try robocopy from the Windows 2003 resource kit or xxcopy. I've tried xxcopy and it seems to buffer things well, such that I can do a sustained 25 MB/s or so when backing up files to my 500 Gig USB 2.0 external drive.
The explorer method of copying seems to have a lot of overhead, and doesn't buffer the transfers well. At least not with lot's of small files.
While I'm sure it's great to get the firewire working at full speed, why don't you just put the drives on separate IDE channels in the same machine. You'll get a much higher throughput.
You'd still need to use something like xcopy32 (or boot in linux and use tar - if both drives are fat32)... or find a windows version of tar (url:http://unxutils.sourceforge.net/)
Are you sure that you're not missing the bit -vs- bytes distinction? A difference of about 1/10 would appear if you are.
Those who know, do not speak. Those who speak, do not know. ~Lao Tzu
It's a good thing you don't write Windows for a living. I'll bet you anything that those write operations are coalesced over several files at a time. Nice try, though.
I did a freelance gig back in '98 where I had to use a Mac (an 8600/300 w/64 megs of RAM). It took well over 20 minutes to copy a 17 meg file from one folder on the hard drive to another. 20 minutes! At home, on my Pentium Pro 200 running NT4, the same operation would take about 2 minutes.
(Admit it. You knew this was coming.)
A modern SATA drive [...]
Modern? This is from an old computer to a new computer - and those speeds would be for consecutive data. Which I suppose is fair, given that the user defragmented completely. But an IDE drive's speed is far less 400mbit/s.
Death by snoo-snoo!
There is a problem in Windows XP SP2 with firewire transfer. Albeit that it could be numberous small files creating problems but it should be faster than 100mbps ethernet. Try this blog regarding Windows XP SP2 Firewire Slowness for a link to the KB and a links to few other work arounds or just go direct to the KB article.
"Modern? This is from an old computer to a new computer - and those speeds would be for consecutive data. Which I suppose is fair, given that the user defragmented completely. But an IDE drive's speed is far less 400mbit/s."
They're just about as fast as SATA drives, since ATA-100 is still faster than the sustained speeds of the drives (100 is megabytes in this case). This is why ATA-133 never caught on -- it's faster than any of the drives you'd connect it to. It wasn't SATA's speed that made it popular, it's the numerous other advantages (thinner cable, cheaper, hotplug, etc).
I rarely criticize things I don't care about.
Acuatlly hardrives both IDE/EIDE, and normal scsi are measured in MB/s note the capital B meaning BYTEs not BITs. External communication channels are measured in bits/s. Only the new SATA, and SerialSCSI drives are bits/sec. If you calculate it out, a 3.0Gb/s SATA drive pulls about 375MB/s burst rate. Its marketing manipulation.
I'm not sure why you're transfers aren't that fast, for me firewire from my external harddrive is just as fast and getting stuff off my fileserver *6 disk raid 0*. I have yet to max out my gigabit system, and that was copying 3 dvd images to my desktop (U160 SCSI) *I was doing this to determine upper bandwidth from my server* and even then I was sustaining only about 60 MB/s, note thats sustained.
My firewire disk drive copies at a slightly slower rate, but at about 30MB/sec, however considering the theoretical MAX with just transfer, no protocol or system overhead is 50MB/sec. Another question is how much ram do you have, cause most certainly if your sending drive is slow, or your recieving is slow, everything is limited to that speed.
-PB_TPU_40 The trick to flying is to throw yourself at the ground and miss.
Use gigabit network cards? Faster than Firewire.
The fastest way to do this is to put the old drive in the new machine (or perhaps an external drive enclosure if we are talking about a laptop) and copy that way.
If you are worried about special file or folder attributes then use MSBackup to copy the drive to a backup file as it will preserve everything.
I was about to post in a similar vein, wondering how 10% of 400 Mbps can be faster than 100 Mbps, but after reading your post it makes sense; the question submitter is saturating the LAN connection and getting ~10 MBps, and is also saturating the firewire connection thinking it can do faster because he is expecting it to go 400 MBps.
Why not just plug in the old machine's hard disk to the new machine? Leave the lid off, have an IDE ribbon cable dangling over the side of the case, prop the disk upright with a chipped mug and a spare copy of Tanenbaum's Minix book... this is the correct old-school approach to moving data and many times faster than anything involving slinging a cable between the two boxes.
-- Ed Avis ed@membled.com
No one's mentioned this?
Bring up the properties of the firewire disk in "Device Manager". Go to the Policies tab and make sure it's set to "Optimize for performance".
Slagborr
While I've done this very thing with success, I would advise md5ing the two files afterwards, as there is a lack of error correction here.
What I actually usually do is run a pass of rsync over the directories. If the files are the same, rsync will do very little work, if there is anything needs fixing, it will get done.
Combination - fun iPhone puzzling
Oxford 911 or 911+ seem to work the best in my experience (and even more so if you ever have to use OS X).
TCP has error detection (an correction by retransmission). But nevertheless I would recomend using tar -v and checking the last transmitted file for correct length, because netcat doesn't detect end-of-file on input (and manual says it's a feature).
Pretty simple, it's just a regular HTML link. What you're doing is creating an "auto-link". The auto-link will just display an URL as a link (like http://slashdot.org/). What you want to do is look at the "Allowed HTML" section when you post. If you don't know how to use one of the HTML tags, just google it like this. Slashcode will put the domain name of the address in brackets next to the link automagically.
I write most of my comments in HTML and have done so for years now. Come to think of it, I can't remember a time when I posted without any HTML. For Example, here's the source code for the first paragraph of this comment (wrapped in <ecode> tags so you can see it):
You'll find that the <em> (emphasis) tag surrounded by the <blockquote> tag is quite common when quoting what you are replying to. So much so that the CSS now seems to recognize it and do it's fancy(pants) formatting. I suspect blockquote is doing this doing this, but have been using the two tags together for so long now it's habit ;)
Like they say: "News for Nerds".US Democracy:The best person for the job (among These pre-selected choices...)
No... for a raptor, that is pretty much the sustained rate. The burst transfer rates are often 110+ MB/s.
A modern 7200 rpm SATA desktop drive can sustain just shy of 70 mbyte/s
Just for the first 50 gb or so.
Although I normally think the constant `Linux pwns M$' thing often goes over the top around here, I would actually say there's probably a good chance the parent is just stating the truth as he's experienced it here. The protocols he mentions, USB and Firewire, have at different times had issues running at full speed on Windows XP. Don't quote me on this, as I'm not 100% sure which SP it was, but I'm pretty sure before SP1, Windows XP didn't have proper support for USB 2.0. USB 1 speeds worked fine but I definitely recall having issues personally running at full speed. I imagine there was a patch for it fairly quickly but in the usual tradition of patching Windows most people didn't bother. There are also quite a few comments above regarding the recent issue with SP2 and breaking Firewire 400 speed -- it throttles it back to just 100 speed. There is, again, a simple patch to fix it. But my point is, is that it's quite possible to have problems with USB/Firewire under Windows and not have an issue (at least after it was implemented ^_^) with those two protocols under Linux.
The Shoes of the Fisherman's Wife Are Some Jive Ass Slippers
One more X in xxcopy and you have a program the man actually needs.
I use NSCopy for any decently-sized Windows File Sharing file transfer - it can copy a whole directory tree and throttle the speed down or up (to maximum "plaid") Just google for it, it's free.
If you want more speed, I'd say get FireZilla (an FTP client) and FireZilla Server (an easy to use FTP server), both open source and free. Set up the server on the "source" computer, and download as fast as you can! It will use the bandwidth much better.
One of the other suggestions about moving the hard drive would work too, but these are within your original constraint of improving network transfer performance. If you move the hard drive, use a text-mode (like xcopy) utility to copy from the command prompt - far faster than using Explorer.
Good luck!
I recognize people by their sigs. Is that a bad thing?
It would be faster to 'tar' directly to the target machine. Tar will stream the output and is probably much more efficient than Windows Explorer.
/path/to/local/drive ./* /path/to/remote/drive/backup.tar
Also, booting the machine from a Linux boot CD and mounting the drive read-only and using tar to move the files is probably the fastest method. I do this all the time to recover/backup machines at work.
boot gentoo install cd...
mount network share/external drive
mount local drive '-o ro'
cd
tar -vcf
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
Well, for the purposes of figuring out whether or not Firewire is a bottleneck, it's enough.
I rarely criticize things I don't care about.
http://winhlp.com/WxDMA.htm
I've seen it happen multiple times on my 2000 box. A 35 MB/sec drive drops down to about 4 MB/sec. It's worst when your DVD/CD drive is on the same controller. You throw in a bad disk and windows decides that flakey things are going on and reverts you back to PIO.
A real PITA - and a good reason for always putting optical drives on their own controller.
"This is, however, a burst transfer rate. It is not sustained. So even a modern SATA -- yes, even the expensive 10,000 RPM one -- won't saturate the bandwidth of FireWire 400"
hmm...
$ dd if=/dev/zero of=zero bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 161.013 seconds, 66.7 MB/s
Looks pretty sustained to me.
I rarely criticize things I don't care about.
It might be true if you are doing a sector by sector read, and then only if you have platters with high areal density (ie extremely recent drives). Windows file copy; however, does not optimize its disk access in any way and so you have lots of delays while moving from file to file. So, yeah, Firewire (400) might be a bottleneck if you are doing drive imaging or copying lots of very large files, but sustaining ~50MB/s during random access is still a pipe-dream on normal consumer drives at this point in time.
It is remarkable that a /. user, who should have some measure of tech-savvy, mistakes megabits per second with megabytes per second. This is pretty basic stuff, kiddies. Firewire (the IEEE1394A type) tops out at 400 Mb/s, which is 50 MB/s. Eight bits in a byte, remember? SATA is 150 MB/s or 1.2 Gb/s. SATA-2 doubles that. Ultra320 SCSI is 320 MB/s or 2.56 Gb/s. The newer IEEE1394B is 800 Mb/s, but you will only typically find that on a Mac.
Just because you are gay, Anonymous homophobe Coward, and you recommend Linux, that doesn't make it a "fag's OS". It does make you a repressed "fag" cruising Slashdot for humiliation, regardless of your OS choices.
--
make install -not war
Moderation -2
50% Troll
50% Offtopic
Topic is "speeding up FireWire transfers". I have done that by using Linux instead of Windows. I told my experience in detail, including how to switch.
TrollMods love Windows, or maybe just going slow.
--
make install -not war
Well, then don't put anything you want to transfer between drives in the recycle bin.
I've done this type of copy--physically move the source drive into the computer with the target drive--many times with Windows 98, 2000, and XP. I've copied files and folders to and from 'My Documents', 'Desktop', 'Application Data', and other special folders. I've never had an issue with this approach.
Whatever.
10% of firewire 400 is 40mbps. fast ethernet is more than that even with the collisions. So I dont know how you got faster transfers with firewire.
One option is to just pull the drive from the old machine and use it as the slave drive. I use this when moving large files. Another option is to have a gigabit card, now around $14 everywhere. Newer PCs already have gigabit cards. Just use a crossover cable if you wont buy the (also cheap) gigabit switch.
As far as firewire is concerned, I've never used it to transfer files since I've always had faster alternatives.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
Why not just boot Knoppix or some other CD with a driver that does not suck?
Friends don't help friends install M$ junk.
Okay, as much question as it is suggestion... Would turning off System Restore on the source and destination drives help out?
initiated from files; location: /someplace ; tar xvf - )'
tar cvf - file1 file2..filen | ssh user@host '( cd
initiated from files' destination: /someplace ; tar cvf - file1 file2..filen)' | tar xvf -
ssh user@host '( cd
Not exactly a new trick but one that bears repeating. You get prompted for a password and then the transfer commences. Using something like this, you don't pause for confirmation in between files like you would using most other methods.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
That seems like a pretty precise number, I wonder if there just might be 1395 or even more?
seriously, though, thanks for the pointer; I'll be (re-re-re-re-re-re-re...-re)patching my XP box tonite!
This space intentionally left (almost) blank.
Boot both boxes under Knoppix (or other Live CD), and SCP the files.
You are being MICROattacked, from various angles, in a SOFT manner.
First off, I'm not sure NTFS has a file allocation table. But more relevantly, there are two filesystems I know of that do lots of small files really well: Reiser4 and XFS. At least in the case of Reiser4, Microsoft could license it.
The key feature here is lazy allocation. It not only keeps your drive from getting as fragmented, it also means that when it does decide it has to write, it's writing all the files at once, and can make intelligent decisions like, write all the metadata out, then write all the files, then go back and trip the flag that says the transaction worked. That's something like 3 seeks for an arbitrary number of files (assuming all those files fit in RAM).
The disadvantage is, it takes a bit more CPU, and it takes quite a bit of intelligent guessing to figure out when it should start flushing, to keep from the all-too-frequenty problem of having a gig of ram full and having to flush it all at once, locking up much of the rest of the system until it's done. But, that seems to be getting better all the time -- my Reiser4 box, under heavy load, usually figures out how to flush often enough that it still has enough RAM free to keep buffering till the flush is done.
Don't thank God, thank a doctor!
If it's only using 10% of a 400mbps link, that would be 40mbps. How is that faster than a 100mbit ethernet link?
Arguing about vi versus Emacs is like arguing whether it's better to make fire by rubbing sticks or banging rocks.
You cannot expect many tens of MiB/s when reading files, the sustained transfer rate is never reached in practice. Writing is slower, still. You're getting 5 MiB/s for essentially random reads and writes, and that's about all you can expect. Yes, that means hard disks are actually slow. They basically haven't gotten any faster over the last couple of years, only bigger.
Can it be sped up? No. dd or dump could read faster, but the former would shred the target filesystem and the latter couldn't speed up the writing. So let it copy overnight and be done with it.
He's finding out that hard drives move data at a fraction of their stated max speed when moving large amounts of data. The only way you're going to fill up a FW channel to its capacity is to have very fast devices at both ends. This generally means RAID 0, 5, 6 or some combination thereof like 50 or 60. Raid 10 is more for redundancy in case a disk dies than performance, I mention it for completeness and to support its use over straight RAID0 for the uninitiated.
The cesspool just got a check and balance.
If you don't want to use the command line xcopy, then I suggest you download a copy of Synctoy from Microsoft.
a milyid=49818CF1-2287-40EA-8A6F-57BD8695F23D&displa ylang=en
a milyid=E0FC1154-C975-4814-9649-CCE41AF06EB7&displa ylang=en
Whitepaper: http://www.microsoft.com/downloads/details.aspx?f
Download:
http://www.microsoft.com/downloads/details.aspx?f
From here to there and there to here funny things are everywhere.
Umm Yea instead of doing that you can use the Windows XP Files & Settings Transfer ON the CD just Insert your XP CD or just borrow someone else's and then use that tool to move all of your crap to a new computer using USB 2.0 Data Transfer Cord. (480 Mbps)
"FIREHOSE gives you a basic data transfer over multiple network devices supporting TCP/IP layers. Stripe multiple 100Mbit, Gigabit, 10 Gigabit, or firewire to give one humungous pipe for firehosing your gigabytes and gigabytes of data.
"Unlike RAID striping, FIREHOSE striping load balances the network devices so every ounce of bandwidth is utilized. Combine a 400Mbit firewire eth device with a 100Mbit eth device to get 500Mbits of power. Combine 10 100Mbit ethernet ports for a gigabit pipe. The number of devices which can be striped is limited only by imagination and budget."
http://heroinewarrior.com/firehose.php3
n/t
All the techniques ever used to make men moral have been themselves thoroughly immoral... (Nietzsche)
/. was not accepting my post last night. I have a response ready @ home. Will post tonight.
The cesspool just got a check and balance.
Any modern IDE drive (40G+) should be able to sustain 40 - 50M/s for sequential reads and writes.
Ah, thank you sir, I stand corrected.
On the other hand, it looks like I need to upgrade my own hardware..
Death by snoo-snoo!