Speeding up Firewire File Transfers?

Milo_Mindbender asks: "I've got a pretty common problem: copying a ton of files from an old Windows XP computer to a new one. After noticing how long transfers were taking over my 100mbps Ethernet, I hooked up a IEEE1394/Firewire cable and things were much faster. Strangely though, Windows is still only using about 10% of the cable's 400mbps bandwidth. Does anyone know any tips/tricks for speeding this up or any Shareware mass-file-copy tools that would be faster than Explorer/file sharing? Right now, the older machine is setup with Windows file sharing and the new machine is copying from it, neither machine is using much CPU and the disks are nowhere near their max speed. The number and size of the files might be what's slowing it down, since it's gigabytes of files in the 100-200k size range."

Comments

archive then move?

[vjl]

Have you tried to archive/compress them first [gzip/zip/etc], then move the big file over? Lots of small files take longer to move than fewer larger files. /vjl/

Re:archive then move?

[quokkapox]

archive/compress them first [gzip/zip/etc], then move the big file over?

Pr0n jpgs do not compress very well. WTF not just let it run overnight?

Re:archive then move?

[biglig2]

Nothing to do with compression (although that may help); it's about one big file being faster to copy than lot's of small files that add up to the same size. Even if you zip them up without compressing (it'll be an option somewhere) then this will help.

Another thing is that even without looking at third party tools, you should be using XCOPY in preference to windows explorer.

There is an Exchange server utility that is optimised for moving gigantic files very fast; doubtless you can find similar programs about.

Re:archive then move?

[dwater]

The point is not to make them smaller - it is to make them all one file. Of couse, smaller is better too.

I wonder how the microsoft backup too would work in this situation. Backup to a file, copy the file, then restore from it. I've done it in the past, though not for the same reason, and it's worked...not sure about speed.

Clearly, moving the disk would be the best option. I often use a firewire Wiebetech Drive (useful to have around if you often find the need to do such things) dock to perform a similar task - it'd mean you don't have to open one of the cases, at least.

On a side note, why is http://www.wiebetech.com/ inaccessible from China - it's been inaccessible (at least) ever since I arrived here 3 years ago? Groklaw was blocked for a while, but is now ok; Wikipedia seems to come and go ('go' at the moment); even slashdot occasionally does not work; but wiebetech's web site has consistently never worked. Yes, I try from multiple access points...

Curious.

Re:archive then move?

[sirsnork]

Put a freeware FTP server on one of them and FTP everything. Windows file copy is TERRIBLE if you want speed

Re:archive then move?

[blincoln]

Robocopy is approximately a hundred trillion trillion trillion times better than xcopy.

To put that in perspective, you would need to weld fourteen quadrillion VW Beetles end to end, then use the resulting Beetle Bar as a lever and an object with the displacement of eleven million Libraries of Congress as the fulcrum in order to give xcopy the same Windows command-line file copying power as Robocopy.

Re:archive then move?

[1u3hr]

you should be using XCOPY in preference to windows explorer.

Better, use xxcopy. Similar CLI, free; avoids the common problem of long/short file names getting scrambled. The "pro" version apparently has network features, but I've never used that.

Re:archive then move?

[personman21]

You could always put them in an encrypted (http://www.truecrypt.org/) volume. Then it's one big file, and you don't have to worry about who stumbles upon your pr0n collection. Two birds with one stone?

Re:archive then move?

[Anonymous Coward]

and you don't have to worry about who stumbles upon your pr0n collection. Two birds with one stone?

if thats what floats your boat.....

Alex

Re:archive then move?

[mgv]

Have you tried to archive/compress them first [gzip/zip/etc], then move the big file over? Lots of small files take longer to move than fewer larger files.

Is it just possible that you are confusing bits with bytes per second? 400 Mb/s is about 40 MB/s (or pretty close, especially as you rarely the full theoretical 50 MB/s that you would think this would equate to).

Michael

Re:archive then move?

[mathew7]

FTP is great for big files, but for 4K files SMB (aka windows file sharing) is better. FTP has a big overhead: it creates a new TCP connection for every file and closes it when the file is transfered.
I'm looking right now for a TAR/netcat solution (I used it once, but don't remember netcat's options).

Re:archive then move?

[mathew7]

I once used netcat & tar to transfer lots of small files, but it's a complicated task (archive everything in a big file which is instantly transfered to the other computer).
On source computer you have to create a tar archive redirected to netcat, which is instructed to connect to another netcat session (destination computer) that provides its output to tar extraction.
Problem is if you give something wrong and need to abort, you will have to restart the server part (destination computer).
Server(destination): nc -L -p 22|tar -xvf -
Client(source): tar -cf - *|nc sss.sss.sss.sss 22
The only problem I see with Windows is I don't know if netcat's STD_ERR will be transferred to tar (only STD_OUT should be tranfered).
This is the fastest way I can see of tranfering lots of small files between computers. You can even use compression with TAR (add z or j to tar parameters;e.g.: tar -cjf - *) if you really are bandwidth limited. But for firewire I saw that between 2 3G P4s you cannot tranfer more than 30MB/s (100% CPU load) because the TCP checksumming/compression is done on the CPU, not accelerated by LAN chips as on ethernet. I don't think you will see inprovement in TAR compression, since TCP will always try to compress.

Re:archive then move?

[drachenstern]

Ahha, but in relation to the dupe that is sure to be posted to the front page soon, how many highways* would that provide the ability to build? corollary question, what's the best transport mechanism to move that many LoCs?

*that would be an information highway

Re:archive then move?

[Karl Cocknozzle]

There is an Exchange server utility that is optimised for moving gigantic files very fast; doubtless you can find similar programs about.

Perhaps you mean ROBOCOPY.EXE (RobustCopy)? It is a tool in the Wind2k/2k3 Admin toolkit designed to do just what you describe: Faithfully, accurately, (and efficiently) copy large files from one place to another.

Re:archive then move?

[j00r0m4nc3r]

So what you're saying is that you like Robocopy?

Re:archive then move?

[chenski]

I've done like this so I wouldn't have to use temporary disk space to store the archive file and takes care of the small files problem. Basically pipe the output of tar to the other machine through rsh...

tar -cz /path/to/files | rsh othermachine tar -xz

(hope the syntax is correct. been a while since i've tried it)

Re:archive then move?

[mad.frog]

Bah. Everyone knows that xxcopy is old tech; it has only two x's.

I much prefer xxxxxxxxxcopy.

Re:archive then move?

[Ratbert42]

you should be using XCOPY in preference to windows explorer

We used to run a Windows to Windows performance lab to collect file transfer statistics. FTP beat the pants off of any Windows networking thing, even XCOPY.

Re:archive then move?

[LurkerXXX]

Robocopy has nice features, but it's slow. I've started using HAS instead. It costs, but It's got more features, a nice GUI if you happen to like that, and most importantly, it's about 3x faster on my servers than Robocopy.

Re:archive then move?

[Anonymous Coward]

Great Firewall of China. Look it up.

Re:archive then move?

[OpenSourceOfAllEvil]

I've done similar things but over gigabit ethernet. And the Windows overhead of multiple files is still the killer. As a side note, the brand and model of gigabit card makes a difference too.

However I am dubious of the time in compressing to one file and transferring to the other computer and decompressing, even with minimal or no compression is actually a time saver. Has anyone actually verified this? This seems particurly wasteful when compressing and decompressing to the same drive. It's the equivalent of copying the file 3 times. Twice locally and once over the wire.

Instead of XCOPY, I would recommend ROBOCOPY also from MS. This command line util was also written by Microsoft and does several things including retries from an MS sharing bug as well as ETAs and exclusion lists. I believe it is or was included in their free PowerToys but I recall having a hard time locating an XP version of it on the MS site the last time I needed it.

Re:archive then move?

[nolife]

I've noticed this as well. At work we move hundreds of thousands of files from firwire/usb drives and other computers on a frequent basis. Windows takes a LOOONG time to handle large directories and large amounts of files. Even selecting a directory on a dual CPU/8GB ram server class hardware with RAID 0+1 10-15K ultra SCSI drives to get the properites of that directory (total size and file count)can take over 30 minutes if it contains over 50k files in the below sub directories. Try moving those directories and files to or from a firewire drive can literally take 24-48 hours even though the total bytes may only be 500-750GB. Doing the same transfer over to a network computer takes even longer. Hell, even moving them from one SAN to another SAN from the same physical server with 2 HBA's (one to each SAN) is terrible.
We completely skip the gui and go command line with robocopy, still slow but prevents strange mid transfer lockups and eliminates the confusion from using the GUI because robocopy reports files as they are transfered and reports the final results when complete. Not just that inaccurate copy dialog box that the Windows gui provides.

Move the old hard drive, then copy

[TheArtfulTodger]

Why not just plug the old hard drive on the secondary channel on the new PC, reboot and then just file copy? Or do I need to reread the question?

Re:Move the old hard drive, then copy

[TheArtfulTodger]

So what is this "magic" you speak of? Could I get around it by ftp'ing to localhost to get the files (even though this is perverted)? Thanks.

Re:Move the old hard drive, then copy

[BrynM]

The problem with doing that is Windows can detect the "magic" folders... Mods: If you don't know what "magic" is, please mod some other comment. Thank you.

Do you mean the software named "Magic Folders"? Or perhaps you have some kind of Folder of Holding with compression created by a high-level Magic User. Most likely you mean the Special Folders that are used by Windows, but then again you may just be spouting about something you actually know little about.

Or maybe you should just explain yourself and not flame the mods... You might even get modded +Insightful or +Informative then and you would have the advantage of explaining your term to the person you were answering - thus being genuinely helpful.

Re:Move the old hard drive, then copy

[Elemenope]

Mods: If you don't know what "magic" is, please mod some other comment. Thank you.

Normally I avoid posting on the techie parts of slashdot, as I know approximately fuckall in the power/knowledge sense when it comes to the esoteric hardware and coding problems that are usually discussed. I just read, hopefully learn something, and avoid sticking my foot in my mouth. But, there was something about this that really rubbed me sideways in an uncomfortable "AAAHHAGH, you're rubbing be with stingling nettles!!!!111!!!11!" sort of way.

I'll admit that I haven't the foggiest what 'magic' is. Problem is, after the comment, I ain't any closer to knowing 'magic'. If I was the poor guy who asked the original question, I'd still be clueless. But that's not my real problem.

My real problem is this: the poster actually anticipated that a majority of mods wouldn't know what he was talking about; in fact, said as much. Doesn't that strike one as a tad worthless on the communication side? I mean, when you are communicating, isn't important that the receiver kno what the hell you are talking about? If you assume right from go that nobody is gonna have a clue, haven't you already failed at the whole communication thing? I surely think so.

This, incidentally, is why normies hate the geek culture. It's the smug superiority of "you don't understand my ridiculously technical argot? I spit on thee, and piss on thy great-grandmother's grave! You are UNWORTHY!!!" That attitude is why so many of us got beat up in high school. But it's even more insulting, and unnecessary, if you are talking to other geeks, who, given the barest of clues or rudimentary explanation, could probably figure out just what the hell you are talking about.

Yeah, I know some folks get cranky because some mods don't understand what it is they are modding. And I sympathize...but isn't it the safer part of valor to take the extra time to make sure one is understood, rather than either be unclear and get modded down, or be a jerk and scare away anyone with whom you might try to converse with?

BTW, feel free to burninate this rant if you feel it justified to do so. ;)

Re:Move the old hard drive, then copy

[Bob Cat - NYMPHS]

Yes, I meant the special folders. Thanks for the link. However, it just seemed simpler to give a warning. Doing the file operations over the network obviates all concern about stray magic.

If you've got a system you don't care about, and another system disk you were planning on wiping anyway, install the second disk and try various copy/delete/move operations between the original special folders and the newly installed ones on the second disk. Use Explorer, magic is not quite as strong using xcopy from the command line.

BTW, I learned the term "magic" from the Microsoft tool "tweakgui". I didn't invent it.

Re:Move the old hard drive, then copy

[Bob Cat - NYMPHS]

Sorry, I was just trying to avoid karma burn from mods who might think I was being a smartass about Microsoft products. How's that working for me? :)

I replied further upthread to the guy who knew what I was talking about. What I *really* hoped would happen was that a few hackers would try to find out what I (and Microsoft) meant by "magic", but on Googling right now, it is not as obvious as it used to be in 1995, when the magic was first conjured. Here be dragons:

http://en.wikipedia.org/wiki/Special_Folders

A Special Folder can either be a reference to a physical file system directory, or a reference to a "virtual" folder. In the former case, they are analogous to environment variables -- in fact, many of the environment variables that are set in a user's session are defined by where the special folders are set to point to.
Virtual folders, however, do not actually exist on the file system; they are instead presented through Windows Explorer as a tree of folders that the user can navigate. This is known as the Shell namespace.

What happens when you add a disk full of special folders to another disk full of special folders - that have the same names and attributes? Can the OS keep the namespace straight? I will leave this as an exercise to those readers with disks they don't mind trashing.

Re:Move the old hard drive, then copy

[Elemenope]

Sorry I was cranky. I can understand not wanting to be burned by roving retardo-mods. Thanks for the info, it was helpful!

Re:Move the old hard drive, then copy

[Bob Cat - NYMPHS]

The answer to the first question is elsewhere in this thread.

The answer to the second question is "I don't know, but I think maybe, and since it's a brilliantly devious idea, I hope so."

As for me, I'm never putting an old system disk on a new system as a second disk. After what happened last time.

Re:Move the old hard drive, then copy

[BrynM]

BTW, I learned the term "magic" from the Microsoft tool "tweakgui". I didn't invent it.

I looked in TweakUI and it calls them "Special Folders" as well. My point is: Be careful about how you present things that you may only have cursory knowledge of here on Slashdot. I was being funny about it, but there are planty of users out there who will ream you for bad information. Instead of being authorative and telling the mods what to do, you could have replied in the form of a question such as "You could try, but won't windows have problems with the magic folders?" or something like that. Further, you could disclaim being an authority and just post something along the lines of "I think that...".

Ok, I'm done being a slashdot post nazi now :D

From my experience, XP stores the hard locations (ie: c:\Documents and Settings\BrynM\My Documents") for the special folders it needs in the registry. I've never had a problem slapping a previously used drive into a machine to copy files (I did this exact thing to recover files after and IDE failure that was corrupting NTFS just a couple of weeks ago). If you've had problems or know of them, then please post that.

Re:Move the old hard drive, then copy

[cgenman]

Special folders don't exist on the second drive. Only the real folders do. So your desktop folder is still in a folder on the drive, you just have to know where to look. In this case, that's Drive:\Documents and Settings\User\Desktop. The virtual folder itself ceases to be where you expect it to be, as Windows is not searching your secondary drive for them, but it continues to be where it is.

If you want to know where a folder really is on the drive, just do a search for a file you think is in the drive.

In terms of Folders, Windows adds the contents of the folders together. If instead of being meticulous about your upgrade, you simply copy over your entire "Documents and Settings" folder, all of the contents of the folders that were virtual before will be added to the new folders that are virtual now. This works better if you keep usernames the same between machines. Nothing should break. I'd recommend against directly copying your program folders or windows folders, but nothing from there would work anyway.

I've done upgrades like this before, direct drive-to-drive through the intermediary OS. It really is the fastest way to do it.

Re:Move the old hard drive, then copy

[BrynM]

As for me, I'm never putting an old system disk on a new system as a second disk. After what happened last time.

Aha! You did have problems. Please post what happened so the rest of us may avoid it. Even if you weren't sure what was going on when it happened, you might get replies that help clear things up. I hope you don't think I'm nitpicking you, I'm just trying to help you and I have the Karma to burn on helping other posters (modded to oblivion for being offtopic). Believe it or not, I've learned quite a bit by posting "A happened, then B happened so I don't do that anymore" here on Slashdot.

Re:Move the old hard drive, then copy

[Vengie]

Be careful about how you present things that you may only have cursory knowledge of here on Slashdot. This is what i want to be included in every thread that discusses courts and the law. IAAL. :(

Re:Move the old hard drive, then copy

[east coast]

That attitude is why so many of us got beat up in high school.

You got beat down in high school? I'm sorry...

I didn't have that problem because all those bitches knew I was speaking the truth when I told them I was "down with the AK" (ala Ice Cube)

Re:Move the old hard drive, then copy

[Anonymous Coward]

FWIW, I haven't used Windows full-time since 1998, and I knew exactly what you meant by "magic folders". I suspect that was the original terminology for them.

Re:Move the old hard drive, then copy

[Bob Cat - NYMPHS]

I have karma to burn, too, so I'll try to remember. Maybe I'll trash a system later to be sure. :)

I think the most obvious thing is the recycle bin, since anything on either drive ends up in the same magic recycle bin. This shouldn't normally be a problem, unless you had a bigger %-sized bin on the second drive with something you wanted to restore, and the settings were at default (use one setting for all drives). In any case, this is evidence Windows DOES spread the magic (I swear it was called that in the original tweakui) in certain cases. I believe I tried to move D: "My Documents" to c: "My Documents" and got infinitely nested folders. Yes, it's coming back to me now (shudder). This was in WIN98 I think, I had to boot to a command shell (off a floppy? maybe, shouldn't have had to.) and get to the top mydocu~1 folder, attrib the contents -s -h -a -r and delete the phony mydocu~1 folder (I think I looked for a way to kill the magic unsuccesfully.

Maybe this kind of thing doesn't happen under XP, but it NEVER happens if you just share C: and copy across the network. Better yet, share a folder called /incoming/ on the target, and if you have multiple accounts on the old pc log onto each one and copy it's data into /incoming/user1, etc.

I had some similar things happen with OS/2 2.0, but that was more than 10 years ago, and no one cares anymore.

Someone mod me up for excessive typing, my RSI is killing me.

Re:Move the old hard drive, then copy

[Lehk228]

the "magic" is things like my documents and program files

i had a problem once copying an entire user directory, which totally screwed up a machine because it suddenly had magic folders appearing and disappearing

Re:Move the old hard drive, then copy

[SuperMog2002]

Just FYI, I have run in to a problem doing exactly that. There is an option in the Users control panel to make a user's My Documents folder private. If that option was checked, Windows will flat-out refuse to touch those files. The work-around I've always used is to log off, then log back on as Administrator. (If it's Win XP Home Edition, to log on as Administrator, you have to reboot the computer in to Safe Mode and press ctrl-alt-del twice at the Welcome screen to force you to the standard login screen (if necessary). The administrator password is blank by default). As Administrator, a new option is added to the file context menu in exporer: Take Ownership. This will seize ownership of the files, clearing the privacy flag in the process. You can then manipulate them as you please.

Re:Move the old hard drive, then copy

[Anonymous Coward]

BTW, this is way, way, WAY off topic, but does anyone else think that Elemenope is protesting far, far too much considering he reads D&D Wikipedia articles? ;)

Re:Move the old hard drive, then copy

[Anonymous Coward]

But, if you call the folder /incoming/, and you move the files from /, you add 9 chars to the path, which will NOT, in my experience, let a lot of files copy. Even worse, unless you use a copy manager such as foldermatch or viceversa plus (anyone knows of a free good one?), you may not find out until you need those files!
Examples of stuff that gets lost are download folders and iTunes libraries (or any well organized library with classical music in it).

Re:Move the old hard drive, then copy

[Bob Cat - NYMPHS]

What if they are encrypted?

Re:Move the old hard drive, then copy

[Elemenope]

Hey, anaonymous moron, the OP linked to the article I was referring to.

Re:Move the old hard drive, then copy

[Elemenope]

And if I could only spell 'anonymous'...

Re:Move the old hard drive, then copy

[Elemenope]

LOL, no. I was too big. ;) But many geeks complain of this problem, or so I hear. In any case, I sure wasn't 'popular' until I found my people: speech and debate team, mega mecha geeks 'r us. I eventually became the team captain.

Re:Move the old hard drive, then copy

[Anonymous Coward]

Wow, you're really defensive, aren't you? A little too defensive, if you catch my drift. It's okay, you won't be judged for being a gaming nerd here on Slashdot. You can come out of the closet.

Re:Move the old hard drive, then copy

[Anonymous Coward]

I'm just trying to help you and I have the Karma to burn on helping other posters ...

Can't post and mod in same discussion

(modded to oblivion for being offtopic)

Concentrate more on promoting than on demoting. The real goal here is to find the juicy good stuff and let others read it. Do not promote personal agendas. Do not let your opinions factor in.

- The moderation guidelines.

I'm only posting this due to the unbelieveable number of off-topic and troll mods that I have metamoderated recently. It's not what those mods are for. Learn the system.

Re:Move the old hard drive, then copy

[Anonymous Coward]

You don't have a clue what you're talking about. Those "special folders" are only special when windows boots from that drive. They are "special" because of a bunch of registry settings. If you take that drive out and place it into some other machine as a slave drive just to copy the files over, there's nothing special about them. Windows will not magically recognize them as special.

Maybe you should leave your basement, head out into the real world and get some real experience other than just spewing some theory you came up with or read some where on the internet.

Re:Move the old hard drive, then copy

[SuperMog2002]

I don't really know for sure if Windows actaully encrypts the files or not when you set them to private. I'm guessing it just sets the permissions to the Windows equivalent of 700 on the individual files (you can still read the folder) because I know for certain the method I just described works, and works quickly, as I've used it several times. Now, if they've been encrypted by another method, then I'd guess you're up a creek if you've lost your decrpytion key.

Re:Move the old hard drive, then copy

[Anonymous Coward]

> And if I could only spell 'anonymous'...

Or "LMNOP". :-)

Re:Move the old hard drive, then copy

[Bob Cat - NYMPHS]

They're encrypted if you set them to be under properties > general > advanced. The only (official) way to copy them out is to log onto that drive as the user who owns them.

Re:Move the old hard drive, then copy

[Bob Cat - NYMPHS]

THANK YOU!

This must mean I'm not stupid OR crazy. Now, why are there so many posts telling me to get out of the basement or return my computer to the store? I'm guessing those experts did not take the particular steps we did, so we must be wrong. But we're not.

Here

[abscissa]

Here you go.

Firewire is crippled in Windows by default. You need the patch here to restore functionality.

Re:Here

[Anonymous Coward]

OK, discussion is over. Everyone go home.

Re:Here

[PayPaI]

Non login requiring link here
Article seems to imply that only 1394b (FW800) controllers are affected. Still, limiting to S100 is a pretty dumb move on MSFT's part (insert tinfoil rant about MS being in league with Intel over FW vs USB2 here).
As a side note, I use an external FW drive on my HP laptop running XPSP2 Pro, and I can pull >35MB/s from it. I do not have any machines with 1394b so I can't confirm that there is an issue with that, but my 1394 port is definitely not running at S100.

Re:Here

[Anonymous Coward]

Thanks for the link but in order to download the fix MS requires you to have that Windows Genuine Advantage crap installed.

Until Microsoft gets a truthful disclosure I refuse to install it.

Re:Here

[Anonymous Coward]

Boot up with a Linux Live CD that includes WINE (or boot the live CD and install it), then download the patch. It will ask you to download and install a Firefox plugin (if you use Firefox to view the site) from MS in exe format, but if you scroll to the bottom of the page there is a link to an "alternative method". Use that, download the exe it gives you and run it under WINE, copy and paste the code it gives you into the text box provided then it will allow you to download the patch as it sees WINE as a valid version of Windows.

Re:Here

[kyouteki]

Errors corrected below: Windows is crippled by default. You need the patch here to restore functionality.

It's the hard drive, not the fire-wire

[johnnywheeze]

This being slashdot, I'm sure someone will correct me if I'm wrong, but your hard drive is most likely not fast enough to receive the full 400 Mb/s stream from the firewire. The fastest SCSi drives are 320Mb/s and that's not sustained.

To get full firewire transfer goodness, you need a raid of fast drives, on both systems.

Re:It's the hard drive, not the fire-wire

[Anonymous Coward]

your hard drive is most likely not fast enough to receive the full 400 Mb/s stream from the firewire. The fastest SCSi drives are 320Mb/s and that's not sustained.

You are confusing MByte/s and MBit/s. Firewire is 400 MBit/s, while SCSI is 320 MByte per second.

Re:It's the hard drive, not the fire-wire

[ArbitraryConstant]

Firewire 400 is 400 megabits per second.

A modern SATA drive can do just shy of 70 megabytes per second, which is 560 megabits.

Re:It's the hard drive, not the fire-wire

[Daxster]

A modern SATA drive [...]
Modern? This is from an old computer to a new computer - and those speeds would be for consecutive data. Which I suppose is fair, given that the user defragmented completely. But an IDE drive's speed is far less 400mbit/s.

Re:It's the hard drive, not the fire-wire

[ArbitraryConstant]

"Modern? This is from an old computer to a new computer - and those speeds would be for consecutive data. Which I suppose is fair, given that the user defragmented completely. But an IDE drive's speed is far less 400mbit/s."

They're just about as fast as SATA drives, since ATA-100 is still faster than the sustained speeds of the drives (100 is megabytes in this case). This is why ATA-133 never caught on -- it's faster than any of the drives you'd connect it to. It wasn't SATA's speed that made it popular, it's the numerous other advantages (thinner cable, cheaper, hotplug, etc).

Re:It's the hard drive, not the fire-wire

[PB_TPU_40]

Acuatlly hardrives both IDE/EIDE, and normal scsi are measured in MB/s note the capital B meaning BYTEs not BITs. External communication channels are measured in bits/s. Only the new SATA, and SerialSCSI drives are bits/sec. If you calculate it out, a 3.0Gb/s SATA drive pulls about 375MB/s burst rate. Its marketing manipulation.

I'm not sure why you're transfers aren't that fast, for me firewire from my external harddrive is just as fast and getting stuff off my fileserver *6 disk raid 0*. I have yet to max out my gigabit system, and that was copying 3 dvd images to my desktop (U160 SCSI) *I was doing this to determine upper bandwidth from my server* and even then I was sustaining only about 60 MB/s, note thats sustained.

My firewire disk drive copies at a slightly slower rate, but at about 30MB/sec, however considering the theoretical MAX with just transfer, no protocol or system overhead is 50MB/sec. Another question is how much ram do you have, cause most certainly if your sending drive is slow, or your recieving is slow, everything is limited to that speed.

Re:It's the hard drive, not the fire-wire

[Anonymous Coward]

A modern SATA drive can do just shy of 70 megabytes per second, which is 560 megabits.

This is, however, a burst transfer rate. It is not sustained. So even a modern SATA -- yes, even the expensive 10,000 RPM one -- won't saturate the bandwidth of FireWire 400, much less FireWire 800. It might give USB 2.0 a run for its money, as USB has much greater overhead due to its offloading processing to the computer's CPU and other design constraints (e.g., FireWire was designed for 400 Mbits/sec from the start, USB was designed for only 12 Mbits/sec from the start).

Re:It's the hard drive, not the fire-wire

[bwoodring]

No... for a raptor, that is pretty much the sustained rate. The burst transfer rates are often 110+ MB/s.

Re:It's the hard drive, not the fire-wire

[ArbitraryConstant]

"This is, however, a burst transfer rate. It is not sustained. So even a modern SATA -- yes, even the expensive 10,000 RPM one -- won't saturate the bandwidth of FireWire 400"

hmm...

$ dd if=/dev/zero of=zero bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 161.013 seconds, 66.7 MB/s

Looks pretty sustained to me.

Re:It's the hard drive, not the fire-wire

[MacBoy]

It is remarkable that a /. user, who should have some measure of tech-savvy, mistakes megabits per second with megabytes per second. This is pretty basic stuff, kiddies. Firewire (the IEEE1394A type) tops out at 400 Mb/s, which is 50 MB/s. Eight bits in a byte, remember? SATA is 150 MB/s or 1.2 Gb/s. SATA-2 doubles that. Ultra320 SCSI is 320 MB/s or 2.56 Gb/s. The newer IEEE1394B is 800 Mb/s, but you will only typically find that on a Mac.

Re:It's the hard drive, not the fire-wire

[Anonymous Coward]

You are thinking of SATA drives, which use burst data transfer; SCSI drives can do SUSTAINED data transfer at their maximum rate, whether it be 80GB/s, 160GB/s or 320GB/s; they are also thoroughly tested and most are warranteed for almost a decade (even with continuous, vigorous and full speed use)

Aside from that, has anybody thought to have the gent check his drivers? The default windows drivers deliberately hobble the Firewire data transfer to limit the amount of processor resources that Firewire can use at any given moment; one can usually find the proper 3rd party drivers for the Firewire ports from the manufacturer's website, and they make a world of difference in terms of capabilities (especially if they are the newest revision)

Re:It's the hard drive, not the fire-wire

[drsmithy]

But an IDE drive's speed is far less 400mbit/s.

Any modern IDE drive (40G+) should be able to sustain 40 - 50M/s for sequential reads and writes.

Re:It's the hard drive, not the fire-wire

[Daxster]

Ah, thank you sir, I stand corrected.
On the other hand, it looks like I need to upgrade my own hardware..

Use FTP

[Anonymous Coward]

I'd use FTP. It has less overhead than Windows shares, and I often get up to wirespeed when I copy files (at least on 100MBit, on 1GBit Lan the Disk will slow you down).

Re:Use FTP

[Goose42]

Agreed 100%. Any time I need to transfer massive amounts of files (and physically moving the hard drive from one machine to another is not an option), Filezilla never lets me down. Server on one side, client on the other... they're a quick setup and just as easy to install as remove when they're no longer needed. Its amazing how much data can fly through a router that's actually working it full capacity.

Re:Use FTP

[andy753421]

If you're using tcp/udp and you're really worried about protocols slowing you down here's a nice way to bypass them almost completely.
user@destination $ netcat -l -p 12345 | tar -x
user@source $ tar -c /usr/src/linux-2.6.17.1 | netcat desktop 12345

Re:Use FTP

[Chris_Jefferson]

While I've done this very thing with success, I would advise md5ing the two files afterwards, as there is a lack of error correction here.

What I actually usually do is run a pass of rsync over the directories. If the files are the same, rsync will do very little work, if there is anything needs fixing, it will get done.

Re:Use FTP

[deefox]

TCP has error detection (an correction by retransmission). But nevertheless I would recomend using tar -v and checking the last transmitted file for correct length, because netcat doesn't detect end-of-file on input (and manual says it's a feature).

Re:Use FTP

[Anonymous Coward]

Cool! Now how much of 400mbps FireWire can I get out of this method?

Sleep on it.

[Bob Cat - NYMPHS]

I bet when you wake up in the morning, things will look much brighter. They always do.

IF you wake up. Muahahahahahaaa....

Find the bottleneck

[philocipher]

If your not maxing out your connection, any number of things could be limiting the speed of the transfer; cpu, bus speed, harddrive performance etc. Use a system monitoring utility to see what's at 100% utilization, and then upgrade the part. Transfering larger files that are sequential on the disk will also help.

Re:Find the bottleneck

[99BottlesOfBeerInMyF]

Since the poster already mentioned CPU, I suspect they know enough to look at the basic utilization stats. Most likely, however, the limiting factor is either the hard drive speed or the fact that Windows explorer is a piece of crap.

btw

[Anonymous Coward]

btw i am fairly sure that with firewire you won't see much cpu usage. I think that's only USB; firewire devices can talk to each other without a cpu.

What are you moving now?

[digerata]

"...since it's gigabytes of files in the 100-200k size range."

That's quite a collection of pr0n!

Re:What are you moving now?

[CelticWhisper]

"...since it's gigabytes of files in the 100-200k size range."
That's quite a collection of pr0n!

You've obviously never seen my pr0n archives. Which reminds me, anyone know of good prices on RAID cards? Newegg maybe?

Re:What are you moving now?

[Anonymous Coward]

You've obviously never seen my pr0n archives.

If we're going to be measuring disk sizes...

[user@loonix user]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 42G 38G 1.7G 96% /
/dev/hda1 99M 17M 78M 18% /boot
none 126M 0 126M 0% /dev/shm
/dev/md0 1.2T 692G 447G 61% /good_stuff

Linux with NFS or maybe ghost?

[Anonymous Coward]

Get 2 knoppix discs and use nfs. Firewire and ethernet arn't whats slowing you down. I always get 92mbit transfer rate from my desktop (ext3) to laptop (ext3 and fat32). You do run the risk of losing file attributes though.

Is there any enterprise software you can use like ghost?

All in all Explorer sucks ass at file operations. You would probably get beter transfer rates with ftp.

Re:Linux with NFS or maybe ghost?

[BandwidthHog]

All in all Explorer sucks ass at file operations. You would probably get beter transfer rates with ftp.

Even IP Over Carrier Pigeons would yield some sort of improvement.

If you’re gonna use an operating environment that shits all over your desktop anyway, ya might as well go all out.

Some things to try

[Tycho]

What is the manufacturer of Firewire controllers in the computers you are using? VIA controllers are usually not the best Firewire controllers. Texas Instruments controllers are usually better. For that matter, depending on the situation try compressing the files. Also, do not depend on time remaining in Windows I have found it wildly inaccurate at times. Windows seems to estimate the time remaining to be way too high, so YMMV, literally.

Re:Some things to try

[bazorg]

why should I use mileage literally when we're talking about bits or bytes per second?

File size is the problem

[inio]

Your file size, and disk seek time, are the problem. Lets say your drive has a 5ms seek time (that's pretty damn fast). writing each file actually requires three writes: to the file allocation tabe, to the directory, and the contents of the file itself. Assuming the writes take another 5ms, that's 20ms per file. that limits you to 50 files per second. At 200kiB per file that's about 10 megs per second.

Re:File size is the problem

[Animats]

You're asking Windows to create, write, and close maybe 500 files per second. Windows file creation isn't that fast. What's the file system format on the destination side?

Try transferring a 1GB file and report how long that takes.

Tar and Robocopy..

[cowmix]

If you want to keep the files separate.. use Robocopy.. (free from MSFT)..

If you don't mind the files being in one glump.. use a Win32 port of Tar..

Both options above seem to speed up Firewire (or any) transfer.

holy crap

[fishdan]

how is THIS not well known? MSFT blows. Thanks for the great tip

Re:holy crap

[DrSkwid]

for once, wikipedia to the rescue

http://en.wikipedia.org/wiki/Firewire#Operating_sy stem_support

xcopy?

[Kozar_The_Malignant]

I haven't used xcopy in a while. Certainly not on an XP machine. The last time I used it was on a w98 box and it truncated all of the filenames to 8.3 format (7~.3 really). Is there now a version that supports long filenames?

Re:xcopy?

[Anonymous Coward]

Yes, xcopy for NT is very good. I use it all the time. You need a string of options as long as your arm to do proper backups, though, and it chokes if the number of files is too large. In that case, you could try GNU cp.

Re:xcopy?

[1u3hr]

Is there now a version that supports long filenames?

See my earlier post about XXcopy, http://www.xxcopy.com./

Re:xcopy?

[Jaruzel]

I second XXCopy. I have it scheduled on my file server, as a free once-a-day mirroring utility from drive D: to drive E:. It works great, and outputs what it did in a very nice manner.

-Jar.

Re:xcopy?

[vjl]

And I third that! We use xxcopy at work for a client to server copy procedure and it is very fast. I actually had forgotten about that solution, as we use it to move a lot of small files over the network from one client PC to the server, and it does copy fast.

xxcopy does cost money though [but i believe you can try it out for free, as it simply has nag screens if i recall]. If the original poster doesn't mind opening up a command shell [aka DOS window], xxcopy would be the way to go, without requiring zippin' the files up first.

/vjl/

Re:xcopy?

[prescor]

He's copying pr0n. Is there an xxxcopy?

Re:xcopy?

[itwerx]

You need a string of options as long as your arm...

My mom took Thalidomide, you insensitive clod!

Lucky you - no options to type! :)

(Er, hehheh, I guess that's true in more than one way, isn't it?)

Re:xcopy?

[BigCheese]

I've had problems with cp too. Try the cygwin rsync.

xcopy32

[Anonymous Freak]

Assuming you can't change any of your hardware, and don't want to go for any massively difficult software, just use Windows' built-in xcopy32 command. Yes, it's command-line, but it's the fastest free-and-easy utility out there.

And, as others have pointed out, you are NEVER going to hit 400 mbps, no matter what software you're using. 400 mbps is 50 MB/s, do you honestly think that old PC has a drive capable of sustained 50 MB/s transfers? Do you have a drive in the FW enclosure that is capable of that? Only the absolute latest PATA drives are capable of sustaining that, and even then, only over sequential transfers. Any random access, and you're dropping well under that.

If you want a perfect copy of the drive, use a sector copier, like Norton Ghost or Partition Magic.

Try searching for an rsync clone for win32

[rwa2]

Well, I could think of a lot of ways to speed it up under Linux using various combinations of rsync, and... well, really just rsync. See if there's a good rsync clone for Win32 that will preserve your precious file attributes. Even running it under cygwin may be better in the long run, especially because inevitably (speaking from experience) your large copy will be interrupted halfway through by an "unreadable file" or some such rubbish, and you'll find yourself having to try to fix it and start the copy all over again from the beginning, or else trying to just transfer the remaining directories you think you're missing.

Using cygwin's rsync via ssh: (after running "ssh-host-config" on your new box and setting a "passwd" as Administrator )

rsync -azve ssh --progress /cygdrive/c/pr0n/ Administrator@newxpbox:/cygdrive/c/pr0n/

will do the trick, and you can just keep running it over and over again until all the files are mirrored. It will take a long time to buld a list of all the files you need to transfer, but it will only tranfer the files you're missing, and will attempt to do some compression (which should help because you're more IO bound than CPU bound, but just remove the -z if your CPU is pegged). Plus, you'll find rsync & scp damn useful for many other common tasks you take on.

The bottleneck is probably your windows filesystem, and cygwin's extra abstraction layer will only make that worse. But using rsync under cygwin means you only have to transfer the files once - which will be a much bigger time saver than trying and failing to do the entire transfer several times.

If you were doing this often, I'm guessing you might see an improvement if you defragment your old drive first, but you obviously don't really want to waste time on that for a once and final transfer.

Also, the Windows TCP/IP stack is typically tuned for 2 - 10Mbps links. Here's some information on how to fix that: http://rdweb.cns.vt.edu/public/notes/win2k-tcpip.h tm It's mainly geared towards improving throughput on high-capacity WAN links, but parts are also relevant to achieving decent performance on 100Mbps+ networks as well. Also remember that a lot of network drivers suck too and are incapable of pushing the throughput even to a fraction of its rating... that's been a factor too, especially on cheap windows crap. An updated NIC driver /might/ get your net transfer to catch up with your firewire transfer somewhat.

Since you're getting 40Mbps / 400Mbps firewire, you're really not doing too bad. Converting to bytes, 5MB/s is a decent fraction of the 20MB/s to 50MB/s raw speed of your older hard drives, and actually seems reasonable given that you're sending lots of small files and not a few big ones where you can actually make good use of your drive's readahead cache.

Re:Try searching for an rsync clone for win32

[Linux_Bastard]

Make that -e "ssh -1" and you have a winner. Protocol 1 has lower overhead. :wq

Pay attention to units

[Bogtha]

Strangely though, Windows is still only using about 10% of the cable's 400mbps bandwidth.

Are you sure you aren't confusing mbps with MBps? 400mbps is equal to 50 megabytes per second, and "12.5% of the cable's bandwidth" sounds suspiciously like your description of the problem, "about 10% of the cable's bandwidth".

Re:Pay attention to units

[mattmacf]

Are you sure you aren't confusing mbps with MBps? 400mbps is equal to 50 megabytes per second...

Are you sure you aren't confusing mbps with Mbps?

Last I checked, 400mbps is equal to 0.4 bytes per second. I remember getting speeds like that back in the days of dial-up. Like back in the day when you picked up the phone and read off ones and zeros to your friend on the other line as he copied your fortran program by punching holes in his punch card. Oh yeah, and uphill both ways. In the snow.

; )

[/toungeincheek]

Re:Pay attention to units

[BillyBlaze]

Nope, 400 mbps would be .4 bits per second, which is .05 bytes per second, or about .000071 Libraries of Congress per millenium.

Re:Pay attention to units

[Ben Jackson]

about .000071 Libraries of Congress per millenium.

Oh, a mere 71 uLOC/MY.[*]

[*] Assuming you meant an analog millenium, not a digital millenium, which would be MiY.

Re:Many Major Problems:

[ArbitraryConstant]

"Your drive doesn't keep up with Firewire. Sorry, but it's true. Disks aren't as fast as you think -- particularly that cheap ATA thing you are using."

Firewire 400 is 400 mbit/s. A modern 7200 rpm SATA desktop drive can sustain just shy of 70 mbyte/s, which is 560 mbit/s.

Some ideas

[sheldon]

First of all, I'm confused by what you mean when you say it's only using 10% of bandwidth. 400Mbps, means you're gonna get something like a max of about 40 Megabytes per sec transfer. (remember, 8 bits per byte, plus some overhead) Are you seriously only getting 4 MB/s?

As far as copying faster. You might want to try robocopy from the Windows 2003 resource kit or xxcopy. I've tried xxcopy and it seems to buffer things well, such that I can do a sustained 25 MB/s or so when backing up files to my 500 Gig USB 2.0 external drive.

The explorer method of copying seems to have a lot of overhead, and doesn't buffer the transfers well. At least not with lot's of small files.

Re:Some ideas

[Firehawke]

If he were using it to copy porn, shouldn't it be XXXCopy?

why use firewire?

[hazem]

While I'm sure it's great to get the firewire working at full speed, why don't you just put the drives on separate IDE channels in the same machine. You'll get a much higher throughput.

You'd still need to use something like xcopy32 (or boot in linux and use tar - if both drives are fat32)... or find a windows version of tar (url:http://unxutils.sourceforge.net/)

Re:Why use Firewire?

[CokoBWare]

Dude, although this makes sense, it's also mighty inconvenient, even for a hardware geek (which I am). Firewire would do the trick much better I think. Your method involves opening up both PCs, connecting the ribbons to the other computer. Firewire method involves taking a cable, plug it in to the ports on both machines, and it's ready to go. I think simplicity wins personally, IMHO.

Bits and Bytes

[jessecurry]

Are you sure that you're not missing the bit -vs- bytes distinction? A difference of about 1/10 would appear if you are.

It's a good thing you don't write Windows...

[Anonymous Coward]

It's a good thing you don't write Windows for a living. I'll bet you anything that those write operations are coalesced over several files at a time. Nice try, though.

Re:It's a good thing you don't write Windows...

[$RANDOMLUSER]

I don't know which is worse, how breathtakingly wrong this post is (Windows: now it sees into the future) or the fact that this AC post got modded "insightful".

Re:It's a good thing you don't write Windows...

[psmears]

I don't know which is worse, how breathtakingly wrong this post is (Windows: now it sees into the future) or the fact that this AC post got modded "insightful".

It doesn’t need to see into the future: just delay writing updates to the disk for a while. Most modern operating systems do this—unsurprisingly, as it usually makes things go faster :-)

Re:It's a good thing you don't write Windows...

[dgatwood]

Windows can't delay metadata writes on FAT32. See my comments earlier about synchronous metadata writes in FAT32 on Windows due to the whole floppy disk legacy.

Re:It's a good thing you don't write Windows...

[psmears]

Windows can't delay metadata writes on FAT32. See my comments earlier about synchronous metadata writes in FAT32 on Windows due to the whole floppy disk legacy.

I don’t have access to the Windows source code, but I see no reason in principle why the FAT32 driver couldn’t behave differently for different media, eg by checking the FILE_REMOVABLE_MEDIA and FILE_FLOPPY_DISK attributes in the Characteristics field of the device object, and delaying the metadata writes (or not) accordingly. Or is there something I’m missing?

Re:It's a good thing you don't write Windows...

[RedDirt]

Windows (XP at least) doesn't buffer writes to removable media by default. You can enable that feature by going into Disk Administrator, finding the firewire drive in the bottom-right quadrant, right-clicking on the grey label and picking properties. On the policy tab you can opt to have Windows buffer writes at the price of having to "eject" the disk before unplugging it.

Similar Problem on Macs

[R3d M3rcury]

I did a freelance gig back in '98 where I had to use a Mac (an 8600/300 w/64 megs of RAM). It took well over 20 minutes to copy a 17 meg file from one folder on the hard drive to another. 20 minutes! At home, on my Pentium Pro 200 running NT4, the same operation would take about 2 minutes.

(Admit it. You knew this was coming.)

Re:Similar Problem on Macs

[Ohreally_factor]

Why not provide a link to the original. Oh, wait, I know. . . BBEdit was struggling to keep up as you were typing the post. =)

Re:Similar Problem on Macs

[rf600r]

Most.

Irrelevant.

Post.

Ever.

Re:Similar Problem on Macs

[ElGanzoLoco]

Haha, the second I read the story headline I just knew someone would post that. :-D

I say, EXCELLENT!!!

Cheers,
ElGanzoLoco

Re:Similar Problem on Macs

[R3d M3rcury]

Well, I figured I'd better make sure I referenced something that explicitly says it's a joke. You know Mac people...we'd end up with 700 posts about how "Well, yeah, but that was before Mac OS X" or how they don't have the problem and I'm making it up, etc., etc.

Have you tried this?

[Anonymous Coward]

There is a problem in Windows XP SP2 with firewire transfer. Albeit that it could be numberous small files creating problems but it should be faster than 100mbps ethernet. Try this blog regarding Windows XP SP2 Firewire Slowness for a link to the KB and a links to few other work arounds or just go direct to the KB article.

Re:Have you tried this?

[Zork the Almighty]

How is it that the most useful comment in the whole thread is not modded informative ? This could easily be the problem! Mod up!

Gigabit?

[skinfitz]

Use gigabit network cards? Faster than Firewire.

The fastest way to do this is to put the old drive in the new machine (or perhaps an external drive enclosure if we are talking about a laptop) and copy that way.

If you are worried about special file or folder attributes then use MSBackup to copy the drive to a backup file as it will preserve everything.

Makes sense

[ichigo 2.0]

I was about to post in a similar vein, wondering how 10% of 400 Mbps can be faster than 100 Mbps, but after reading your post it makes sense; the question submitter is saturating the LAN connection and getting ~10 MBps, and is also saturating the firewire connection thinking it can do faster because he is expecting it to go 400 MBps.

Easy solution

[Anonymous Coward]

"The number and size of the files might be what's slowing it down, since it's gigabytes of files in the 100-200k size range."

Easy solution: download your porn in video files rather than jpegs in future

Why use Firewire?

[Ed Avis]

Why not just plug in the old machine's hard disk to the new machine? Leave the lid off, have an IDE ribbon cable dangling over the side of the case, prop the disk upright with a chipped mug and a spare copy of Tanenbaum's Minix book... this is the correct old-school approach to moving data and many times faster than anything involving slinging a cable between the two boxes.

Ooops, my bad.

[ronanm]

Sorry, I modded parent as informative before reading through his linked page fully. And now I'm posting in a discussion I've moderated. Ooops.

Anyway, I should have modded the article down. a) MS doesn't require a login it requires you to have Windows Genuine Advantage and b) the parent's link links back to MS and requires you to install the WGA crap.

Sorry again.

Optimize for performance

[Svenne]

No one's mentioned this?

Bring up the properties of the firewire disk in "Device Manager". Go to the Policies tab and make sure it's set to "Optimize for performance".

Re:Optimize for performance

[DigiShaman]

Quote from Microsoft technet article KB904569

In Windows XP or in Windows Server 2003, you can change the write cache policy to optimize write performance for a disk device and for data safety. You can select either the Optimize for quick removal option or the Optimize for performance option in Device Manager.
If the Optimize for performance option is selected, data that is written by an application is cached by the operation system, and lazy write is used for the cached data.

Basically, when its set to quick removal you have both read and writes in sync. When setup for performance, reads and writes are asyncronous. The later isn't problem unless you yank the data cable even if the drive is seams idle. In any event, just be sure to right-click over the removable device icon (located next to the clock) and select Safely Remove Hardware. You can also just double-click on it, and then press the Stop button. This ensures all pending "writes" are flushed back to disk as needed.

Re:Optimize for performance

[nolife]

I have another post in this article describing my speed issues as well. I've also tried the setting you suggested in the past and that had no noticable change at all. Maybe that is useful for short bursts with a few files. The bottleneck always seems to be how Windows treats or handles large amounts of files. It might not even be the amount of files but the latency or overhead involved with opening and closing a file multiplied by the number you have adds up to significant delays. Anything above 10k or more files becomes a problem regardless of the location, type of media, or connection and bandwidth that your medium has. It could by local between two hard drives, a portable drive and an internal, or over a network with TCP/IP or IPX.
Your tip combined with other general tips and practices from others will help but I have not found a combination that even comes close to the typical everyday throughput you should expect of the hardware being used.

OT

[Anonymous Coward]

How do you write links like you did in "Magic Folders" and "Holding"? When I follow the URL: instructions it shows the whole URL. The /. help and FAQ does not cover this.

Re:OT

[BrynM]

How do you write links like you did in "Magic Folders" and "Holding"? When I follow the URL: instructions it shows the whole URL. The /. help and FAQ does not cover this.

Pretty simple, it's just a regular HTML link. What you're doing is creating an "auto-link". The auto-link will just display an URL as a link (like http://slashdot.org/). What you want to do is look at the "Allowed HTML" section when you post. If you don't know how to use one of the HTML tags, just google it like this. Slashcode will put the domain name of the address in brackets next to the link automagically.

I write most of my comments in HTML and have done so for years now. Come to think of it, I can't remember a time when I posted without any HTML. For Example, here's the source code for the first paragraph of this comment (wrapped in <ecode> tags so you can see it):

<blockquote><em>How do you write links like you did in "Magic Folders" and "Holding"? When I follow the URL: instructions it shows the whole URL. The /. help and FAQ does not cover this.</em></blockquote>

<p>Pretty simple, it's just a regular HTML link. What you're doing is creating an "auto-link". The auto-link will just display an URL as a link (like <URL:http://slashdot.org>). What you want to do is look at the "Allowed HTML" section when you post. If you don't know how to use one, just google it like <a href="http://www.google.com/search?q=html+tag+em"> this</a>. <a href="http://en.wikipedia.org/wiki/Slashcode">Slas hcode</a> will put the domain name of the address in brackets next to the link automagically.</p>

You'll find that the <em> (emphasis) tag surrounded by the <blockquote> tag is quite common when quoting what you are replying to. So much so that the CSS now seems to recognize it and do it's fancy(pants) formatting. I suspect blockquote is doing this doing this, but have been using the two tags together for so long now it's habit ;)

Like they say: "News for Nerds". ...And that's our /. HTML formatting lesson for today. Hope it helps.

Re:OT

[Anonymous Coward]

Use HTML Formatted.

<a href="http://en.wikipedia.org/wiki/Bag_of_holding" >Holding</a>

/. has put and extra space in that code. When you write a post using HTML Formatted, you can use the HTML tags /. says you can (<b> <i> <p> <br> <a> <ol> <ul> <li> <dl> <dt> <dd> <em> <strong> <tt> <blockquote> <div> <ecode>)

Re:OT

[Anonymous Coward]

...And that's our /. HTML formatting lesson for today. Hope it helps.

Great, thanks

Oxford even better

[denjin]

Oxford 911 or 911+ seem to work the best in my experience (and even more so if you ever have to use OS X).

Re:Oxford even better

[dgatwood]

That's not a FireWire controller. Oxford makes bridge chipsets from FireWire to ATA. A FireWire controller generally refers to the bridge between an peripheral bus and FireWire, e.g. PCI to FireWire. My advice: TI chipset. Accept no substitutes. In particular, though, avoid NEC from what I've read.

As for the performance problems, if you have SP2 and FW800 ports (or a controller that's FW800-capable under the hood but uses FW400 PHYs), make sure you install the FireWire hot patch from Microsoft or FireWire performance will seriously suck.

Beyond that, my guess is that you're running into fundamental limitations in the performance of a FAT-based filesystem when reading large numbers of files coupled with the fact that Windows treats FAT32 in a mostly synchronous fashion as far as I can tell, with minimal caching. Why? Because if you eject a floppy after the light goes off, folks expect it to not corrupt anything. They also expect it to not show contents of one disk when you switch them and insert a different one. Unfortunately, AFAIK, Windows is still stuck with that broken model as a result of broken hardware that few people even use these days. Maybe it will be better in Vista.... (And folks made fun of the Mac for its electric eject....)

Linux (or any other non-Windows OS for that matter) should smoke Windows even copying files between the same FAT32 drives because it does a lot better buffer caching and doesn't flush filesystem state to disk constantly. FAT32 is slower (by design) than other filesystems even on other OSes, though. If you want to keep running Windows, move to NTFS, which AFAIK doesn't suffer from this problem.

Re:Many Major Problems:

[Lord Crc]

A modern 7200 rpm SATA desktop drive can sustain just shy of 70 mbyte/s

Just for the first 50 gb or so.

Parent perhaps not Troll

[xerxesdaphat]

Although I normally think the constant `Linux pwns M$' thing often goes over the top around here, I would actually say there's probably a good chance the parent is just stating the truth as he's experienced it here. The protocols he mentions, USB and Firewire, have at different times had issues running at full speed on Windows XP. Don't quote me on this, as I'm not 100% sure which SP it was, but I'm pretty sure before SP1, Windows XP didn't have proper support for USB 2.0. USB 1 speeds worked fine but I definitely recall having issues personally running at full speed. I imagine there was a patch for it fairly quickly but in the usual tradition of patching Windows most people didn't bother. There are also quite a few comments above regarding the recent issue with SP2 and breaking Firewire 400 speed -- it throttles it back to just 100 speed. There is, again, a simple patch to fix it. But my point is, is that it's quite possible to have problems with USB/Firewire under Windows and not have an issue (at least after it was implemented ^_^) with those two protocols under Linux.

Re:Parent perhaps not Troll

[Doc Ruby]

Of course it's not a troll. I posted my experience with the system they're complaining about, with pointers to solving their problem. Facts and logic included, politely.

The fact that Microsoft TrollMods are a predictable response to any comparison of Linux and Windows doesn't make any such comparison merely "designed for nothing but predictable responses". In fact, your message was among the design goals. Another was responses saying why Windows performance is now as good as, or better than, Linux, in those specific cases.

Also predictable, but still disappointing, is the total absence of those responses. Windows zealots would rather TrollMod to suppress competition than actually compare and compete.

Re:Parent perhaps not Troll

[Anonymous Coward]

Maybe the mods were looking for something that isn't an OS war flamefest response, but instead rather something useful like not wiping clean one's HD when said HD may contain files that one wants on it because they use it and/or have paid for it. You know, just a thought that maybe mods have what is known as common sense instead of evangelical advertisement.

Re:Parent perhaps not Troll

[Doc Ruby]

And maybe you're an Anonymous Coward who can tell the difference between an OS war flamefest response, and a response actually pointing out that the OS is the key to the solution. Even pointing out how to keep the old OS, pointing out how the old HW is also the problem.

You know, an Anonymous Coward actually able to understand the info in a post, rather than just jumping into OS flamewar mode when seeing a point about OS choice affecting the task.

Oh, right - that's unheard of on Slashdot. I must have you confused with an actual person, not an Anonymous flaming Coward.

xxxcopy

[Tsagadai]

One more X in xxcopy and you have a program the man actually needs.

NSCopy

[megabyte405]

I use NSCopy for any decently-sized Windows File Sharing file transfer - it can copy a whole directory tree and throttle the speed down or up (to maximum "plaid") Just google for it, it's free.

If you want more speed, I'd say get FireZilla (an FTP client) and FireZilla Server (an easy to use FTP server), both open source and free. Set up the server on the "source" computer, and download as fast as you can! It will use the bandwidth much better.

One of the other suggestions about moving the hard drive would work too, but these are within your original constraint of improving network transfer performance. If you move the hard drive, use a text-mode (like xcopy) utility to copy from the command prompt - far faster than using Explorer.

Good luck!

No, archive WHILE moving!

[MarcQuadra]

It would be faster to 'tar' directly to the target machine. Tar will stream the output and is probably much more efficient than Windows Explorer.

Also, booting the machine from a Linux boot CD and mounting the drive read-only and using tar to move the files is probably the fastest method. I do this all the time to recover/backup machines at work.

boot gentoo install cd...
mount network share/external drive
mount local drive '-o ro'
cd /path/to/local/drive
tar -vcf ./* /path/to/remote/drive/backup.tar

Re:Many Major Problems:

[ArbitraryConstant]

Well, for the purposes of figuring out whether or not Firewire is a bottleneck, it's enough.

Ensure your DMA hasn't reverted to PIO mode

[tarpitcod]

http://winhlp.com/WxDMA.htm

I've seen it happen multiple times on my 2000 box. A 35 MB/sec drive drops down to about 4 MB/sec. It's worst when your DVD/CD drive is on the same controller. You throw in a bad disk and windows decides that flakey things are going on and reverts you back to PIO.

A real PITA - and a good reason for always putting optical drives on their own controller.

Re:Many Major Problems:

[GoRK]

It might be true if you are doing a sector by sector read, and then only if you have platters with high areal density (ie extremely recent drives). Windows file copy; however, does not optimize its disk access in any way and so you have lots of delays while moving from file to file. So, yeah, Firewire (400) might be a bottleneck if you are doing drive imaging or copying lots of very large files, but sustaining ~50MB/s during random access is still a pipe-dream on normal consumer drives at this point in time.

Re:Linux is Faster

[Doc Ruby]

Just because you are gay, Anonymous homophobe Coward, and you recommend Linux, that doesn't make it a "fag's OS". It does make you a repressed "fag" cruising Slashdot for humiliation, regardless of your OS choices.

Re:Linux is Faster

[Doc Ruby]

Moderation -2
    50% Troll
    50% Offtopic

Topic is "speeding up FireWire transfers". I have done that by using Linux instead of Windows. I told my experience in detail, including how to switch.

TrollMods love Windows, or maybe just going slow.

you don't need those old files anyway.

[mcmonkey]

I think the most obvious thing is the recycle bin

Well, then don't put anything you want to transfer between drives in the recycle bin.

I've done this type of copy--physically move the source drive into the computer with the target drive--many times with Windows 98, 2000, and XP. I've copied files and folders to and from 'My Documents', 'Desktop', 'Application Data', and other special folders. I've never had an issue with this approach.

Whatever.

Your calculations are off

[mnmn]

10% of firewire 400 is 40mbps. fast ethernet is more than that even with the collisions. So I dont know how you got faster transfers with firewire.

One option is to just pull the drive from the old machine and use it as the slave drive. I use this when moving large files. Another option is to have a gigabit card, now around $14 everywhere. Newer PCs already have gigabit cards. Just use a crossover cable if you wont buy the (also cheap) gigabit switch.

As far as firewire is concerned, I've never used it to transfer files since I've always had faster alternatives.

Why bother with a patch?

[twitter]

Firewire is crippled in Windows by default. You need the patch here to restore functionality.

Why not just boot Knoppix or some other CD with a driver that does not suck?

Re:Why bother with a patch?

[Anonymous Coward]

ROFL

System Restore?

[sottitron]

Okay, as much question as it is suggestion... Would turning off System Restore on the source and destination drives help out?

When I transfer files between computers...

[drinkypoo]

...I typically use ssh or rsh, depending on if it's local or not. If it is, there's no reason to blow the CPU time on [de]compression, and I use rsh. If you use ssh, it looks something like this:

initiated from files; location:
tar cvf - file1 file2..filen | ssh user@host '( cd /someplace ; tar xvf - )'

initiated from files' destination:
ssh user@host '( cd /someplace ; tar cvf - file1 file2..filen)' | tar xvf -

Not exactly a new trick but one that bears repeating. You get prompted for a password and then the transfer commences. Using something like this, you don't pause for confirmation in between files like you would using most other methods.

"Performance of 1394 devices may decrease..."

[The_REAL_DZA]

That seems like a pretty precise number, I wonder if there just might be 1395 or even more?



seriously, though, thanks for the pointer; I'll be (re-re-re-re-re-re-re...-re)patching my XP box tonite!

Linux with SCP

[SpaceLifeForm]

Boot both boxes under Knoppix (or other Live CD), and SCP the files.

Re:Linux with SCP

[Linux_Bastard]

This works good, Make sure to boot Knoppix with DMA option, and be sure to specify "-1" to force protocol 1 with the scp. It has a lower overhead and will give you better speed. Better yet is to rsync over ssh -1. I use this often to "clean up" failed copys without starting over.

Something like

rsync -auv -e "ssh -1" /files_here otherbox:/files_there

If the link is truly the constriction point, add a -C to the ssh options to inline compress/decompress. This is usually only a bonus when going over a wan.
YMMV

Re:Linux with SCP

[Anonymous Coward]

Given the network setup and speed requirements any secure protocol might be a bit overboard. There is no threat of 3rd parties looking in as its a px to px connection. You're transfering a huge amount of data very fast. much faster than you'd see on the internet. Having to encrypt and decrypt the data can cause a slowdown in the transfer.

FileSystem is the problem

[SanityInAnarchy]

First off, I'm not sure NTFS has a file allocation table. But more relevantly, there are two filesystems I know of that do lots of small files really well: Reiser4 and XFS. At least in the case of Reiser4, Microsoft could license it.

The key feature here is lazy allocation. It not only keeps your drive from getting as fragmented, it also means that when it does decide it has to write, it's writing all the files at once, and can make intelligent decisions like, write all the metadata out, then write all the files, then go back and trip the flag that says the transaction worked. That's something like 3 seeks for an arbitrary number of files (assuming all those files fit in RAM).

The disadvantage is, it takes a bit more CPU, and it takes quite a bit of intelligent guessing to figure out when it should start flushing, to keep from the all-too-frequenty problem of having a gig of ram full and having to flush it all at once, locking up much of the rest of the system until it's done. But, that seems to be getting better all the time -- my Reiser4 box, under heavy load, usually figures out how to flush often enough that it still has enough RAM free to keep buffering till the flush is done.

Math

[HunterZ]

If it's only using 10% of a 400mbps link, that would be 40mbps. How is that faster than a 100mbit ethernet link?

The disk _is_ near the limit

[cow-orker]

You cannot expect many tens of MiB/s when reading files, the sustained transfer rate is never reached in practice. Writing is slower, still. You're getting 5 MiB/s for essentially random reads and writes, and that's about all you can expect. Yes, that means hard disks are actually slow. They basically haven't gotten any faster over the last couple of years, only bigger.

Can it be sped up? No. dd or dump could read faster, but the former would shred the target filesystem and the latter couldn't speed up the writing. So let it copy overnight and be done with it.

Hard Drive limitation

[Gr8Apes]

He's finding out that hard drives move data at a fraction of their stated max speed when moving large amounts of data. The only way you're going to fill up a FW channel to its capacity is to have very fast devices at both ends. This generally means RAID 0, 5, 6 or some combination thereof like 50 or 60. Raid 10 is more for redundancy in case a disk dies than performance, I mention it for completeness and to support its use over straight RAID0 for the uninitiated.

Re:Hard Drive limitation

[Linux_Bastard]

Sorry, but you seem to have raid's 5 and 10 reversed in your post.
Only raid 0 is faster than 10 (without proprietary equipment). Raid 5 is quite slow and 6 is even slower, especially for writing.

http://www.pcguide.com/ref/hdd/perf/raid/levels/mu ltLevel01-c.html

general RAID info (pictures)

http://www.acnc.com/04_00.html

http://www.pcguide.com/ref/hdd/perf/raid/levels/in dex.htm

Re:Hard Drive limitation

[Gr8Apes]

First, I didn't intend to convey speed by sequence. Second, RAID 5 with 8 disks is certainly faster than RAID 0 or RAID 10 with less than 6 equivalent disks. (8 is the max supported in a single array on my controllers, and I have owned about 8 different types, although none from the last 2-3 years - ie, IDE/SATA)

Another item - RAID 1 is faster than a single drive for reads, if your controller supports "striped" reading off a mirror. Second, RAID5 with 3 disks is dog-slow, but this really depends upon your controller. High end controllers can, and do, offload most of this type of processing so the system never sees the RAID5 effects unless the I/O load increases to a point the buffer no longer can handle the load. (This implies that write back cache is enabled)

RAID 6 would require a minimum of 5 drives, and again, it depends upon your controller. As long as the controller can process faster than the I/O feed rate and your controller/array can handle the bandwidth, your system will never know the difference.

Now, if you're comparing specific numbers of drives and channels, it becomes interesting. RAID0 is the fastest, on a single channel, RAID 10 can actually be slower than RAID5 or 6. This has to do with duplicating every write call vs just writing parity. 6 should be the same as 10 for 4 drives on a single channel, unless you're running the cheapest controller ever.

Re:Hard Drive limitation

[Linux_Bastard]

"First, I didn't intend to convey speed by sequence"

Lets see...
          "--- very fast devices at both ends. This generally means RAID 0, 5, 6 or some combination thereof like 50 or 60.
          Raid 10 is more for redundancy in case a disk dies than performance"

Nope, no speed by sequence there.

          "Second, RAID 5 with 8 disks is certainly faster than RAID 0 or RAID 10 with less than 6 equivalent disks"

If this is true of your setup, then there is something deeply wrong with your controller setup.
Either you have a controller with a fast cpu and lots of cache + really really poor disks, or it doesn't really do raid 10.

          "I have owned about 8 different types, although none from the last 2-3 years - ie, IDE/SATA)"

Ok, now I'm getting the picture, you are a Hobbyist.
There's nothing wrong with that, but I design HA linux clusters, and have been testing and specing storage subsystems for 15 years.
Don't try to teach your grandmother how to suck eggs.

        "RAID 6 would require a minimum of 5 drives"

Sorry, its only 4

        "As long as the controller can process faster than the I/O feed rate and your controller/array can handle the bandwidth, your system will never know the difference"

This statement isn't even wrong. Um... go back to school, or if my guess is right, start.

        "RAID0 is the fastest, on a single channel"

only the truly desperate use more than on drive per channel in PATA

        "This has to do with duplicating every write call vs just writing parity"

Um.. only raids 3 4 and "7" do the "just writing parity" to the other drive scenario
Raid's 5 and 6 stripe the data and parity, with 6 striping 2 sets of parity. which is always going to write ~equally to the drives, and have more overhead than simple mirror and stripe, which is what raid 10 is.

Don't even get me started on disk recovery times. Raid 10 disk recovery beats raid 5/6 by several orders of magnitude. No other raid rebuilds with lower system impact than raid 10, without exotic hardware.

Re:Hard Drive limitation

[Gr8Apes]

There's nothing wrong with that, but I design HA linux clusters, and have been testing and specing storage subsystems for 15 years.

Well then - we're in the same ballpark. However, I'm not a pedantic asshole, err, bastard.

So, let's take you to school:

Lets see...
"--- very fast devices at both ends. This generally means RAID 0, 5, 6 or some combination thereof like 50 or 60.
Raid 10 is more for redundancy in case a disk dies than performance"

Nope, no speed by sequence there.

Let's see - I dropped RAID 3,4,30, and 40. Guess what, they're generally not used anymore. (Note - for your pedanticness, in general, ie, normally, ie, not specific use cases, they're not used anymore. I just hit someone's comma's in a sentence limit.) This is just a natural sequencing of numbers there.

"Second, RAID 5 with 8 disks is certainly faster than RAID 0 or RAID 10 with less than 6 equivalent disks"

If this is true of your setup, then there is something deeply wrong with your controller setup.
Either you have a controller with a fast cpu and lots of cache + really really poor disks, or it doesn't really do raid 10.

Why yes, I do have controllers with buttloads of cache AND a fast CPU. Why would I run enterprise systems on the cheap?

"I have owned about 8 different types, although none from the last 2-3 years - ie, IDE/SATA)"
Ok, now I'm getting the picture, you are a Hobbyist.

Not exactly, I dropped out of enterprise system hardware support about 5 years ago. Since then, I checked out the initial set of IDE RAID cards, and discovered they still didn't overcome issue #1 with IDE - namely, multiple I/O. This means IDE of any flavor sucks for performance. They're great as mass store devices though.

"RAID 6 would require a minimum of 5 drives"

Sorry, its only 4

That's true. Raid 5 only requires 3. Raid 5 with 3 drives, or raid 6 with 4, will suck eggs and isn't recommended. It's kinda like running windows with a minimum configuration - it'll work, but it will suck. Even with 4 and 5 drives respectively, it won't be great.

"As long as the controller can process faster than the I/O feed rate and your controller/array can handle the bandwidth, your system will never know the difference"

This statement isn't even wrong. Um... go back to school, or if my guess is right, start.

I'm guessing you had nothing sarcastic to comment on because it was beyond you. Perhaps DeVry really wasn't all it was cracked up to be?

"RAID0 is the fastest, on a single channel"

only the truly desperate use more than on drive per channel in PATA

Oh, I see. We're still in the wanna be professional arena: SATA/PATA. In case you didn't catch it above, IDE(PATA)/SATA suck.

SCSI is still where it's at, and RAID0 on a single SCSI channel works just fine and certainly isn't a sign of desperation. Perhaps you should start studying for the true enterprise league? After all, if you're claiming to represent enterprise customers and selling them PATA/SATA for performance, you're a fraud.

Um.. only raids 3 4 and "7" do the "just writing parity" to the other drive scenario
Raid's 5 and 6 stripe the data and parity, with 6 striping 2 sets of parity. which is always going to write ~equally to the drives, and have more overhead than simple mirror and stripe, which is what raid 10 is.

I'm well aware of what RAID 0,1,3,4,5,6,10,0+1,30,40,50, and 60 do. Probably more so than you by what's becoming evident in this posting.

BTW, in your scen

Re:Hard Drive limitation

[Linux_Bastard]

Well then - we're in the same ballpark. However, I'm not a pedantic asshole, err, bastard.

My friends call me asshole, feel free. I do think you are being pedantic though.

So, let's take you to school:

Always happy to learn.

Lets see...
"--- very fast devices at both ends. This generally means RAID 0, 5, 6 or some combination thereof like 50 or 60.
Raid 10 is more for redundancy in case a disk dies than performance"

Nope, no speed by sequence there.

Let's see - I dropped RAID 3,4,30, and 40. Guess what, they're generally not used anymore. (Note - for your pedanticness, in general, ie, normally, ie, not specific use cases,
they're not used anymore. I just hit someone's comma's in a sentence limit.) This is just a natural sequencing of numbers there.

3's are still in use, but they are for people stuck in particular hardware. They suck.

"Second, RAID 5 with 8 disks is certainly faster than RAID 0 or RAID 10 with less than 6 equivalent disks"

If this is true of your setup, then there is something deeply wrong with your controller setup.
Either you have a controller with a fast cpu and lots of cache + really really poor disks, or it doesn't really do raid 10.

Why yes, I do have controllers with buttloads of cache AND a fast CPU. Why would I run enterprise systems on the cheap?

You missed the point. for this to be true, your disks have to be truly awfull, way way out of scale to the controller.

"I have owned about 8 different types, although none from the last 2-3 years - ie, IDE/SATA)"
Ok, now I'm getting the picture, you are a Hobbyist.

Not exactly, I dropped out of enterprise system hardware support about 5 years ago. Since then, I checked out the initial set of IDE RAID cards, and discovered they still
didn't overcome issue #1 with IDE - namely, multiple I/O. This means IDE of any flavor sucks for performance. They're great as mass store devices though.

Getging better all the time.

http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAb stracts/tips0458.html?Open

Lots of enterprise class people are running fibre attached sata nowadays. Some business just can't see paying $1500 more for a drive to get that extra 15%

"RAID 6 would require a minimum of 5 drives"

Sorry, its only 4

That's true. Raid 5 only requires 3. Raid 5 with 3 drives, or raid 6 with 4, will suck eggs and isn't recommended. It's kinda like running windows with a minimum configuration -
it'll work, but it will suck. Even with 4 and 5 drives respectively, it won't be great.

Granted.
Raid 5 sucks for writes, is passable for reads, and career ending on the rebuild. And if it looses 2 drives, it's garbage
Over and over I've seen it. One drive fails and the hot spare rebuild kills another drive, leaving you praying you beat the 60% restore failure odds.

Raid 10 has no parity to rebuld. The hot spare rebuild is just a re-mirror, and only affects ONE other drive in the array, plus, you can loose up to half the drives at a time.
Just having the quick rebuild without significant storage performance degradation is worth the price of the other drives.
With raid 10, you find out you lost a drive when the log reports send you email.
With raid 5 you find out when the pager and phone go off, and your explaining to the VP that the system w

Re:Hard Drive limitation

[Gr8Apes]

I'm going to cut this down, as I'm not sure you're even going to read it, but there's always a chance. :)

"Second, RAID 5 with 8 disks is certainly faster than RAID 0 or RAID 10 with less than 6 equivalent disks"

If this is true of your setup, then there is something deeply wrong with your controller setup.
Either you have a controller with a fast cpu and lots of cache + really really poor disks, or it doesn't really do raid 10.

Why yes, I do have controllers with buttloads of cache AND a fast CPU. Why would I run enterprise systems on the cheap?

You missed the point. for this to be true, your disks have to be truly awfull, way way out of scale to the controller.

Actually, I didn't miss the point. If your controller is fast enough that you don't notice the parity calculation happening, then there won't be any effects to notice. Then it becomes an issue of speed. 7 actual writing drives (RAID5 with 8 disks) out-perform 6 writing drives as long as the channels aren't saturated. (On SCSI controllers. I acknowledge complete ignorance on the actual internals including backplanes on new SATA/IDE/Fibre controllers, but the numbers are telling me it's not good - your given numbers that is. We'll get to that.)

BTW, I agree with most, if not all, of your comments on RAID 10. If money is no object, RAID 10 is the way to go. I will make this one statement though, if you're seeing 80% degradation of performance during a RAID 5 disk rebuild, your controller sucks. 50% is bad. 30% is about the max acceptable.

This never happens anymore. Even pc's are getting 533 -1000 MB/s on the pci/pcix, and thats our of scale of even 4G fibre.

1 SCSI 2-4 channel card meets the 500 MB/s easily. Latency is much lower than a fibre network. Always has been. Multiple cards increase bandwidth greatly, much like additional cards/ports.

Lots of enterprise class people are running fibre attached sata nowadays. Some business just can't see paying $1500 more for a drive to get that extra 15%

Now, here's where fibre is better - large flexible storage accessed by multiple machines. However, show me where fibre is cheaper than SCSI. 10K SATA drives cost as much or more than 10K SCSI drives. There are no 15K SATA drives. SATA/Fibre controllers, last time I checked just a month or so ago, were as much or more than a decent SCSI controller. The only thing fibre allows is multiple connected systems and easier management. (These are big issues, but again, it depends upon what your system is designed for - pure performance - fibre isn't that hard to beat. Pure performance on large multi-terrabyte arrays, things start to become more interesting. I haven't delved into that recently enough to comment on the latest state of the art, but SATA fell far short 2 years ago)

Now here's another set of stats: in 97 I spent roughly $10M on hardware for my organization, and about $400M by direction in associated organizations. In 2000, I spent another $2M personally, and by direction roughly $50M. In 2002, I spent another $1M by direction. Roughly 25-50% of that number was for storage. Each system was sized to support a minimum of 1K concurrent users.

At another job, we ran with 35K concurrent users.

We used SCSI RAID in all cases.

As for my discussion previously about mirrors in RAID 10 (or just RAID 1) existing on a single controller - if the mirror exists across 2 controllers, your performance goes to shit. Striping can exist across multiple controllers without too much effect, though. Again, this assumes a half-way worth a crap controller.

"your controller/array can handle the bandwidth, your system will never know the difference"

This statement is i

Synctoy

[joeaic]

If you don't want to use the command line xcopy, then I suggest you download a copy of Synctoy from Microsoft.

Whitepaper: http://www.microsoft.com/downloads/details.aspx?fa milyid=49818CF1-2287-40EA-8A6F-57BD8695F23D&displa ylang=en

Download:
http://www.microsoft.com/downloads/details.aspx?fa milyid=E0FC1154-C975-4814-9649-CCE41AF06EB7&displa ylang=en

XP File & Settings Transfer

[sjipca]

Umm Yea instead of doing that you can use the Windows XP Files & Settings Transfer ON the CD just Insert your XP CD or just borrow someone else's and then use that tool to move all of your crap to a new computer using USB 2.0 Data Transfer Cord. (480 Mbps)

FIREHOSE tool

[Burz]

"FIREHOSE gives you a basic data transfer over multiple network devices supporting TCP/IP layers. Stripe multiple 100Mbit, Gigabit, 10 Gigabit, or firewire to give one humungous pipe for firehosing your gigabytes and gigabytes of data.

"Unlike RAID striping, FIREHOSE striping load balances the network devices so every ounce of bandwidth is utilized. Combine a 400Mbit firewire eth device with a 100Mbit eth device to get 500Mbits of power. Combine 10 100Mbit ethernet ports for a gigabit pipe. The number of devices which can be striped is limited only by imagination and budget."

http://heroinewarrior.com/firehose.php3

ROFL! (n/t)

[Elemenope]

n/t

Delayed

[Gr8Apes]

/. was not accepting my post last night. I have a response ready @ home. Will post tonight.