Use BitTorrent To Verify, Clean Up Files

jweatherley writes "I found a new (for me at least) use for BitTorrent. I had been trying to download beta 4 of the iPhone SDK for the last few days. First I downloaded the 1.5GB file from Apple's site. The download completed, but the disk image would not verify. I tried to install it anyway, but it fell over on the gcc4.2 package. Many things are cheap in India, but bandwidth is not one of them. I can't just download files > 1GB without worrying about reaching my monthly cap, and there are Doctor Who episodes to be watched. Fortunately we have uncapped hours in the night, so I downloaded it again. md5sum confirmed that the disk image differed from the previous one, but it still wouldn't verify, and fell over on gcc4.2 once more. Damn." That's not the end of the story, though — read on for a quick description of how BitTorrent saved the day in jweatherley's case.

jweatherley continues: "I wasn't having much success with Apple, so I headed off to the resurgent Demonoid. Sure enough they had a torrent of the SDK. I was going to set it up to download during the uncapped night hours, but then I had an idea. BitTorrent would be able to identify the bad chunks in the disk image I had downloaded from Apple, so I replaced the placeholder file that Azureus had created with a corrupt SDK disk image, and then reimported the torrent file. Sure enough it checked the file and declared it 99.7% complete. A few minutes later I had a valid disk image and installed the SDK. Verification and repair of corrupt files is a new use of BitTorrent for me; I thought I would share a useful way of repairing large, corrupt, but widely available, files."

Comments

Nice

[Goldberg's Pants]

Awesome idea. I've done this in the past with stuff. If a corrupt version was on one tracker, I'd save the files, get a new torrent and import the old files. Saves a lot of bandwidth wasting.

Re:Nice

[Anonymous Coward]

How exactly did you go about doing this?

Re:Nice

[Goldberg's Pants]

Okay, I had some AVI's and a bunch of them had issues. All I did was copy them out to a different directory, then find a GOOD torrent (with the same rips) then make sure the filenames match exactly. Chucked them in the directory and voila. It checks them all and uses what data it can that you already have and replaces the rest.

Done this with RAR archived stuff as well. (Multipart rars on torrents are retarded, but that's another issue entirely.)

Re:Nice

[ThePhilips]

I do not know what GP meant precisely, but I had similar experience.

Some game (very old RPG) was available on Overlord and on BitTorrent. Not sold anymore. Problem was that BitTorrent had only single seed which minuscule upload speed - in several day I have downloaded only few megs. I tried then Overlord and in few days I got the game almost complete - but another snag had hit me: whether by mistake or intentionally, file was poisoned and three parts couldn't be downloaded. I was ready to throw everything away - antique games interest me little (but friend was recommending it as milestone RPG I had to play). Then suddenly I was enlightened: I fed the incomplete ISO of game to BitTorrent. BT client happily announced something like 98% of file complete and in less than one night downloaded rest of the file.

Re:Nice

[gomiam]

It should be quite simple. Let's say torrentA leaves you with a corrupt/incomplete filesetA (one or more files, it doesn't really matter). Let's supose torrentB contains the files in filesetA, perhaps with different names in its own filesetB.

Ok, you load torrentB in your favorite Bittorrent client, and start it up. It will automatically create 0-sized files with the names in filesetB (at least, all clients I know do that). Stop the transfer of torrentB, and substitute the 0-sized files in filesetB with the corresponding files in filesetA (may require some renaming). As you restart torrentB, your Bittorrent client will recheck the whole filesetB, keeping the valid parts in order to avoid downloading them. Voilá! You have migrated files from one torrent to another.

Note: You should make sure that the files you are substituting in are the same files you want to download through torrentB or, at least, keep a copy around until you see that the restart check accepts most of their contents.

Re:Nice

[peipas]

Okay, my friend had some AVI's and a bunch of them had issues. There, fixed it for you. You're welcome.

Re:Nice

[empaler]

I assume you then continued seeding? :)

Re:Nice

[Anonymous Coward]

Stories about AVIs can be told in the first person. It's only when stories involve pirated TV and movies that it's only grammatically correct to speak of them in terms of friends.

Re:Nice

[CastrTroy]

Reminds me of the "PARS" I used to get off usenet. I think it was bacally a RAR split up into hundreds of pieces, with parity information in each of the files. You only needed to download a certain percentage of the files to reconstruct the original file. It was great, because often pieces of the file would go missing, or become corrupted somewhere along the way.

Re:Nice

[Drantin]

Most bittorrent clients I've used also allow you to create files the full size when you start a torrent, rather than 0-byte sized ones. Helps prevent fragmenting and finding downloads stopped in the middle because the hard drive filled up...

Re:Nice

[Random Destruction]

kind of. Par files are parity files, and you need one par file for each missing rar file.

Re:Nice

[X0563511]

And then, Par2 came along, and allowed more flexibility.

We still use them, on usenet anyways.

Re:Nice

[Christophotron]

no, actually that's how the old par system worked. In the newer, more advanced .par2 system, the individual .rar files are divided into "blocks" and each par file can recover a certain number of "blocks". It's much more advanced than the old par files that you are referring to, and it's similar to the hashing mechanism in bittorrent. I haven't seen any of the old-style pars in a long time.

For example, if you are missing a total of 3 blocks (one block from 3 different files) you only need to download a very small par2 file that says "+3 blocks" and it will repair the three missing blocks. Of course, if you are missing a lot more data, even entire files, you can get several of the larger "+128" par files and it'll repair everything (assuming there is enough parity data). Often you can even request additional parity blocks, but that's only necessary if you have a *really* crappy nntp provider.

Re:Nice

[Hes Nikke]

Done this with RAR archived stuff as well. (Multipart rars on torrents are retarded, but that's another issue entirely.) any idea why the multipart RAR torrents tend to have healthier swarms than single file torrents of the same content? it pisses me off!

Re:Nice

[i.of.the.storm]

I think it has to do with the way the "scene" releases things, they usually do it via multipart rars or something like that. I saw something to that effect in the comments on a torrent a while ago. I think the reason is that things in the "scene" get distributed in ways that aren't bittorrent, so the breaking up into pieces makes sense there. I'm still not entirely sure what the "scene" entails, and how they differ from the people that put the torrents up, so I don't know the whole answer to that.

Re:Nice

[maskedbishounen]

Usenet. All "files" (posts) are stored server-side, and folks generally have a fast pipe to their ISP (or other provider).

With multipart binary posts, a single file is split up between so many posts. Between fifteen to fifty, let's say. It's common for usenet providers to not receive all the posts, so folks are sometimes left with incomplete/corrupt files. Enter the small, spanned archive formats. It's quite common to see up to 10% parity per usenet posting, especially for large files. Small split set sizes make for easy reposting, as well.

In regards to the grandparent, this likely relates to why the said torrents are healthier. Folks who can bypass the leeching process and go straight to the seeding. The only other means of really sharing on such binary groups would be posting (or reposting) stuff for folks. Due to ever limiting server retention, a lot of the binary groups look down on heavy posting.

I gave up on usenet years ago, though; well, not really. My ISP gave up on it, and I was too much of a bum to pay someone for decent service. I would encourage anyone to check if their ISP offers usenet access if they're into P2P and don't like the "2P" part that much.

Re:Nice

[Anonymous Coward]

There is no reason to assume that an avi file infringes copyright. My ISP does that and I hate them for it. If I drop an avi in the webspace that I'm paying for, they'll delete it for me. I hate that.

Re:Nice

[Jarik_Tentsu]

Mmm, in my experience, Firefox's Download Manager occassionally leaves me with incompletely downloaded files - especially when they're big. Dunno whether this is a bad connection (Telstra, I wouldn't be surprised) or an issue with the actual Download Manager, but I don't get these isseues when using Free Download Manager.

Anyways, I've done this before for a different thing.

There was a rare file I was trying to get my hands on, which was fairly large, but corrupted. There was a torrent which had it too, but was giving out really slow speeds (like...1-2 seeders, 3-4 leachers who must've been on dial up or Telstra broadband...). So I HTTP downloaded the corrupt file, then used the torrent to fix up the last corrupted parts. Worked perfectly. =)

~Jarik

Re:Nice

[Firehed]

Well theoretically you could brute-force the reconstruction of a file of a known size with its checksum (and you can do it without a known size too, but having a starting point will speed things up a lot and should negate collision issues you may encounter on shorter checksums like md5 no matter how unlikely they are). It's not parity in any sense of the word, but it could still get you there.

Of course, it's not exactly the most time-effective way to "download" a file, but I've always wondered why I can't throw some spare cycles at a nearly-complete download whose seeds have dried up and just guess until I've got the working version. I've got 83% of a 55MB download right now, 14 or so files in said download, and I'm in that exact stalled scenario. AFAIK bit-torrent provides me checksums in some form or another... how long does it take to hash a 5MB file? I can't be bothered to do the math of how many possibilities that would be at this time of night, but it'll certainly go faster than continuing to download it from nobody.

Re:Nice

[incripshin]

I had a torrent before continually fail the checksum for one of the 'pieces'. I never thought to find a different tracker. I assumed there were a few bad eggs or malicious leachers. Thanks for the tip.

Re:Nice

[Jurily]

Me too. But I never thought about the endless possibilities here.
Just ship everything with a .torrent to verify.

(Wow, all the authorities we could annoy with one minor change!)

Re:Nice

[gomiam]

True, true. Then again, if you filled your hard drive, it's just not big enough. Go buy a bigger one ;-)

Talking more seriously, fragmenting isn't usually an issue if you don't use FATxx.

Re:Nice

[meza]

I can't be bothered to do the math of how many possibilities that would be at this time of night, but it'll certainly go faster than continuing to download it from nobody. Not quite so. Because if you do the math (and if mine is correct at this late hour) you would see that it actually takes a pretty long time(tm). Imagine if only 1MB was missing. You would have to calculate the hash of every single possible 1MB file, so that is 2^8000000=~10^2300000 files. If you had a computer that could, quite unrealistically, calculate one hash each clock-cycle at 1GHz that would still take you 10^2299991 seconds. As a reference the universe according to Wikipedia is roughly 10^17 seconds old.

Besides that there is the information theory problem too. If the hash is 128bit long then every 2^128th file will have the same hash. This might seem unlikely if you only compare a few files (such as all the files ever created by man) but compared to the 2^8000000 hashes we where going to calculate it is actually quite substantial.

Re:Nice

[de_smudger]

Look, they were AVIs of someone installing linux distros, you insensitive clod ;)

Re:Nice

[Xabraxas]

NTFS fragments pretty easily too.

Re:Nice

[Anonymous Coward]

(Multipart rars on torrents are retarded, but that's another issue entirely.) What seems to be new for the submitter and everyone else in this thread is the very reason why lot of releases are transmitted "as is" in bittorrent.

Sure its redundant to torrent a multipart archive, but you can download those rar way faster everywhere else and, if you have a problem or a doubt, check/repair them with bittorrent.

So no, that's not another issue, not even partially...

Re:Nice

[sexconker]

No, you can't.

A checksum is not unique.
A 32 MB file has 8388608 ways to generate the same 32 bit checksum.

Using "given" data to help narrow the search is a bad idea as well - there is no guarantee that the given data is correct, unless you have individual checksums for them. Bittorrent does do checksumming on each individual chunk (I believe), so you could narrow your search space to the size of the incomplete and missing chunks only. The existing data in incomplete chunks would be almost useless, since you don't know if that data is correct. But you can start your search assuming it's correct, (it probably is, mostly) and speed things up.

But the bottom line is that checksums are smaller than the data they verify. Much smaller. Consider a simple example of a 2 bit checksum on an 8 bit chunk of data. Our checksum simple counts the ones, and rolls over.

00000000 : 00 (0 ones)
00011010 : 11 (3 ones)
10110111 : 10 (6 ones - 110 is truncated to 10)

There are 64 ways to get any particular checksum.
2^8(data length) / 2^2(checksum length) = 2^6 = 64. And that's with us having 25% of our data duplicated in checksums.

A checksum is a check. It is not a guarantee nor is it a blueprint from which you can reconstruct the original data. In certain cases it would be feasible - if you're downloading a thesis about Skittles, and it's corrupted, you could perform a brute force search (like you described) on the (small - it's just text) data, and then sort the matches by # of times "Skittles" is present, then by the % of data that is ASCII. You then hand verify the top 20 results or so, and you'll probably have it.

The same could theoretically be applied to AVIs by enforcing the AVI frame structure (throw out checksum matches that don't generate valid AVI files), attempting to grab audio out of the generated files, and then doing a frequency analysis of the audio - rank the results in terms of % of audio that falls within normal listening ranges (since it's almost guaranteed that audio in an AVI will be compressed in a lossy format).

You could do analysis of the video frames and such too. But the bottom line is that it's a HUGE undertaking - just redownload the damned thing, pay for it, or write your own paper. If it's vital data then go ahead, brute force it and waste your life away.

PAR2 files are neat - they give you chunkettes of parity data at different offsets. This allows you to potentially patch holes in data and reconstruct the original files. WinRAR (and other programs) do give you the option to create a recovery record that's placed in the original RAR (or whatever format) files. The problem is that you then have to download the recovery data. With PAR files, you don't download them unless you need them. The downside is that availability then becomes a problem.

Re:Nice

[sexconker]

"A 32 MB file has 8388608 ways to generate the same 32 bit checksum."

Oops - Make that 2^268435424 ways to generate the same 32 bit checksum.

32 MB = 256 Mb = 268435456 b
268435456 b - 32 b = 268435424 b

Re:Nice

[operagost]

Any modern file system will fragment if you expand an existing file. It simply has no way to guess how big the file will get when it is created unless your application chooses the proper allocation.

To give you an extreme example, imagine a 100 GB volume which has no files. You create a 1 MB file, and your filesystem places it near the top. Now you create a second file, and your filesystem places it... well, it could place it anywhere except that first 1 MB, so let's say it places it right next to the first file. Uh oh, it turn out that you need to write 1 GB of data to that first file and extend it. Now you have two fragments.

Ok, let's assume our file system is magical and knows that you like to extend files to huge sizes. So it places the second file at the end of the disk, instead. Oops, you fooled you file system: this time, you wanted to extend the second file by 1 GB. There is no room to append to the end of the file, so a second extent is created somewhere else and linked to the second file. You have two fragments again.

This is why performance tuning requires that you anticipate data requirements and allocate space accordingly; for example, by setting the initial size of database files to one that should reasonably accommodate the data requirements for the foreseeable future (and not automatically shrinking the database down when records are deleted).

Re:Nice

[Goldberg's Pants]

Oh I'm fully aware it's Usenet and stuff like that. It just kills me that the people who release to the torrent sites can't spend just 30 seconds unarchiving it. It's not so bad with TV shows, but anything else where you may want to pick a choose (say one song from an album) you can't.

And before anyone leaps on me for downloading music, it's legal here in Canada currently due to the levies we pay on certain blank media etc...

Re:Nice

[spasm]

"... that would still take you 10^2299991 seconds. As a reference the universe according to Wikipedia is roughly 10^17 seconds old."

It's ok, I just edited wikipedia to make the age of the universe 10^2299991.

Oh, nearly forgot, ~~~~

Re:Nice

[clint999]

Yeah, it's fixed with the latest driver, but they had to disable most of the TCP offloading. I had the same problem on my NF4 board. I chucked the NV Active Armor firewall software and never had a problem since. http://techreport.com/discussions.x/9483

Re:Nice

[ThePhilips]

Good question.

I actually seeded the ISO until several other seeds appeared. ;)

IIRC, there were in total 5 people interested in the game world-wide.

My simple rule of home seeding is: seed only on open public limitless trackers and when there is a need for seeders.

Re:Nice

[Random Destruction]

ah neat. I guess its been a little while since I've nntp'd :)

What broken software were you using?

[evanbd]

TCP/IP provides data integrity guarantees. So, if your ISP wasn't mucking with your packets (and their checksums), either Apple was sending the wrong bits or your hardware or software was screwing with them. My vote is it's not Apple.

I suggest you diagnose your computer problems, rather than relying on BitTorrent to fix them for you.

Re:What broken software were you using?

[Dice]

I asked the same question. Wikipedia answered it.

Re:What broken software were you using?

[kcbanner]

Its networking - shit happens. Some of his bits got thrown out of a router somewhere as heat, or maybe a packet timed out and didn't quite make it.

Re:What broken software were you using?

[Anonymous Coward]

Those who have never developed P2P software might never understand why they all need to use strong checksums to detect data corruption, and why bad blocks actually do appear in the wild; frequently.

You'd be shocked - SHOCKED - at how much data gets corrupted routinely - by errant antivirus software, flaky network equipment, plain ol' line noise that the checksums don't detect (which will happen much more often than you expect, see also birthday paradox), or misbehaving routers who think that any occurence of 0xC0A80102 obviously must be an internal IP address and needs to be changed to your external one. Even if that's in the middle of a ZIP file. Oops.

Encryption actually aids this somewhat, as the same byte patterns don't get repeated, so if there's an errant IDS changing things for example, it tends not to fire the second time.

I've done this before for file repairs. Works a treat, but you sort of wish that torrent used a Merkle hash tree such as the modified THEX standard Tiger Tree Hash. SHA-1's so last century.

Re:What broken software were you using?

[Anonymous Coward]

safari, of course.

Re:What broken software were you using?

[complete loony]

The TCP checksum offloading on nForce 4 motherboards (I have one) were notorious for corrupting TCP packets and allowing them to be received by the application. That's the most likely kind of failure that would be able to reproduce this problem.

Re:What broken software were you using?

[Anonymous Coward]

It's obvious you have no clue how the Internet actually works. Shit happens, but the Internet is designed for it. Dropped packets cause retransmission, not corrupted data; the Internet drops packets *by design* and the entire system is designed around that. Flipped bits happen, but they are detected by multiple checksums which make it astronomically unlikely for corrupt data to remain undetected. Nope; if you receive corrupt data, the blame is squarely on some piece of software fiddling with your packets and changing the checksums to match. Maybe it's the crappy cheap NAT router, or the ISP's deep-packet-inspection P2P filter, or their (not so) transparent HTTP proxy. But whatever the cause, it's almost certain that software is to blame.

I'd bet $100 that if he did the same download over HTTPS, thus preventing software meddling of the packet contents, it would come out perfect.

Re:What broken software were you using?

[SanityInAnarchy]

It's obvious you have no clue how the Internet actually works. Shit happens, but the Internet is designed for it... Maybe it's the crappy cheap NAT router I'm fairly sure that's what GP meant.

Oh, and TCP checksumming isn't perfect.

Re:What broken software were you using?

[CastrTroy]

I had the same problem. What's really terrible is that I don't think they ever fixed the problem. That drove me nuts for a few weeks trying to figure out why all my downloads were corrupted.

Re:What broken software were you using?

[complete loony]

I'm pretty sure they fixed the problem a while ago. Download the latest drivers and you should be ok, assuming you still use the motherboard that is.

Re:What broken software were you using?

[Anonymous Coward]

Or maybe he used Firefox to download it.

In my experience, Firefox regularly mangles large downloads. I have never once downloaded a DVD-sized file with Firefox and had it come through uncorrupted.

I stopped using it a few months ago, though. Perhaps they've fixed it. I was using wget to get anything over 500 megs at the end.

Re:What broken software were you using?

[Skapare]

Flipped bits happen, but they are detected by multiple checksums which make it astronomically unlikely for corrupt data to remain undetected.

I actually saw this happen once ... the astronomically unlikely [1]. TCP accepted the corrupt packet. I'm sure it will never happen again. Fortunately, rsync caught it in the next run.

One problem I ran into once with a certain Intel NIC was that a certain data pattern was always being corrupted. TCP always caught it and dropped the packet. There was no progress beyond that point because of the hardware defect always corrupted that data pattern. Turns out there was a run of zeros followed by a certain data byte (I tried a different data byte and with different run lengths and those never got corrupted). What the NIC did was drop 4 bytes, and put 4 bytes of garbage at the end. I suspect it was a clocking syncronization error. I got around the problem by adding the -z option to rsync (which I normally would not have done with an ISO of mostly compressed files). Another way would have been to do the rsync through ssh, either as a session agent (like rsync itself can do) or as a forwarded port (how I do it now for a lot of things).

[1] ... approximately 1 in 2^31-1 chance that the TCP checksum will happen to match when the data is wrong (variance depending on what causes the error in the first place) ... which approaches astronomically unlikely. Take 1 Terabyte of random bits. Calculate the CRC-32 checksum for each 256 byte block. Sort all these checksums. You will find 2 (or more) data blocks with the same checksum (or a repeating pattern in your RNG). Why? Because CRC-32 has 2^32-1 possible states, and you have 2^32 random checksums.

But whatever the cause, it's almost certain that software is to blame.

Agreed. Since it is at least software's responsibility to detect and fix it, if the problem happens, the famous finger of fault points at the software.

I'd bet $100 that if he did the same download over HTTPS, thus preventing software meddling of the packet contents, it would come out perfect.

Your $100 is safe.

Re:What broken software were you using?

[homer_ca]

Yeah, it's fixed with the latest driver, but they had to disable most of the TCP offloading. I had the same problem on my NF4 board. I chucked the NV Active Armor firewall software and never had a problem since.

http://techreport.com/discussions.x/9483

Re:What broken software were you using?

[Tawnos]

TCP has a 16 bit checksum. That means there's a 1 in 2^16 chance of an error getting by the checksum. Let's assume, for a moment, that the packets were sent 1kb at a time (ethernet max is greater than this, but it's an easy number). In a 1.5Gb file (assuming base 10 throughout for simplicity), this means a total of 1,500,000 packets must be transmitted. Using only the TCP checksum, 22 of these packets would be corrupt, but allowed through. Even though there are additional checks at layer 2, the fact is that when dealing with large amounts of data, relying on TCP for data integrity is not enough.

Re:What broken software were you using?

[ben there...]

It's called NVidia ForceWare Network Access Manager (NAM) now on 680 and 780 boards. And it's still a piece of crap. At least 2 third-party products (Azureus was one) mention in their FAQs to uninstall NAM to avoid crashes. I had Azureus crashes all the time, prevented (sort of) only by going into Task Manager and setting the affinity to the second processor, until I uninstalled NAM.

Good motherboards, bad motherboard drivers.

Re:What broken software were you using?

[rdebath]

You'll lose your $100

The checksum on a TCP packet is only 16bit thats 1 in 65536. You can only get about 100Mbytes in 65536 packets!

The lower layers normally have better CRCs so this is rather unlikely to cause a real problem but if something is chattering the TCP CRC is likely to fail too.

BTW: it's also a rather poor redundancy check even for 16bits.

Re:What broken software were you using?

[rdebath]

Transparent proxies also kill large downloads; especially when the browser is not IE. I hear "not IE" also included IE7!

Re:What broken software were you using?

[lagfest]

One problem I ran into once with a certain Intel NIC was that a certain data pattern was always being corrupted. TCP always caught it and dropped the packet. There was no progress beyond that point because of the hardware defect always corrupted that data pattern.

I'm pretty sure I had this exact bug with World of Warcraft. The game would get stuck retransmitting the same packet over and over again. It could be fixed by turning off hardware checksumming on the NIC.

Re:What broken software were you using?

[obarel]

There is a chance of 1 in 2^16 to get a bad packet that produces the right checksum, but that doesn't mean that one in 2^16 packets will be corrupt (on average).

For a packet to be bad and undetected, two conditions must hold: it must be corrupt to begin with, and it must be lucky enough to produce the right checksum. The probability depends on the error rate of the connection as well as the weakness of the checksum.

In some cases you'd get a lot more errors (picking up the phone on a dial-up connection produces many undetected errors, a lot more than one in 2^16), and in some cases a lot less (running the server and the client on the same machine).

Re:What broken software were you using?

[Anonymous Coward]

That's only assuming that every packet you get is corrupted. If only a small fraction of packets are corrupted (as should be true), there will be 0-1 packets coming through corrupted and undetected.

Good for seeding stuck torrents, too

[b4dc0d3r]

If I happen to see a stuck torrent (many leechers, no seeds), sometimes I can find a good version of the file I already have - so I start the torrent, stop it, replace the single good file (sometimes you need more if the file is smaller than the part size), and upload a few Kb to finish the torrent. Then sit back and watch as everyone fills up.

Re:Good for seeding stuck torrents, too

[Therefore I am]

Hi Johnny Appleseed! Nice to see you back. Did those trees just get boring?

Re:Good for seeding stuck torrents, too

[Scaba]

Then sit back and watch as everyone fills up.

You really need a girlfriend or some hobbies or something...

Re:Good for seeding stuck torrents, too

[Anonymous Coward]

You wouldn't happen to have the last 300mb of the Top Gear episode I was trying to get last month, would you? I think it was Season 8, Disk 3.

Re:Good for seeding stuck torrents, too

[reiisi]

There's nothing wrong with occasionally providing a helping hand, especially if you know what you're doing and it therefore costs you less than the next guy (or me, in this case) who has only half a clue and might take a half hour to figure out how to do it.

Besides, for all we know, maybe he was waiting for his girlfriend to get off work, or maybe he was waiting for some pieces to come out of the kiln, or something.

Anonymous Coward

[Anonymous Coward]

Those of us who use BitTorrent for *ahem* illegal purposes have been doing this since the beginning. The only way to get rare and complete downloads was to take the files to other trackers and match them against another md5 to finish the download.

It's like getting parity files over on usenet to fix that damned .r23 file which is just a bit too short for some reason :)

Scheduling

[FiestaFan]

Many things are cheap in India, but bandwidth is not one of them. I can't just download files > 1GB without worrying about reaching my monthly cap, and there are Doctor Who episodes to be watched. Fortunately we have uncapped hours in the night I don't know about other bittorrent clients, but uTorrent lets you set download speed caps by hour(like 0 during the day and unlimited at night).

Re:Scheduling

[urbanriot]

Azureus also has an excellent scheduling plugin written for it - http://students.cs.byu.edu/~djsmith/azureus/index.php

I don't know about other bittorrent clients, but uTorrent lets you set download speed caps by hour(like 0 during the day and unlimited at night).

Re:Scheduling

[SeaFox]

BitComet has too since version 0.84 at least.

Re:Scheduling

[Anonymous Coward]

Real men use rtorrent.

And yes, it supports scheduled throttling and many other common tasks.

Re:Scheduling

[Anonymous Coward]

ktorrent also has a plugin to enable this.

!new

[gustgr]

For heavy BT users this tactic is very common, provided the file(s) you are willing to download is fairly well available from different sources.

Re:!new

[Llamalarity]

Have not done so myself, have read of this technique being used to "upgrade" a late beta version to the final release. As long as only a few files have changed it should work.

Re:!new

[SanityInAnarchy]

It's an older concept than that, even. Goes back to the strange Debian habit of using a tool called Jigdo -- it would provide essentially a recipe for building an ISO out of all the files needed, where the files were mostly available from standard Debian mirrors. ISOs were available from far fewer mirrors than standard Debian packages, you see.

So, you'd use Jigdo, and if all went well, it'd assemble a working image. But if a few packages couldn't be downloaded, you could always take your mostly-complete Jigdo file and use rsync with an rsync-capable mirror. (Or, more recently, BitTorrent on Ubuntu -- but that's another story.)

I don't think this tactic is very common, though, as most people seem to have no fucking clue how BitTorrent works. I've seen torrents with gigantic multipart RARs, with an SFV of those. Let's see... so, my torrent software is already checksumming everything, and RAR has a builtin checksum too, or at least, acts like it does (it says "ok" or not) -- and on top of that, there's an SFV checksum (crappy CRC32), too. Never mind that RAR saves you at most a few megabytes (video is already compressed), which, based on the size of these files, you'll spend more time unpacking the RAR than you would downloading the extra couple megs. Or that, once you unpack and throw away the RAR, you can't seed that torrent from the working video. Or that multipart anything is retarded on BitTorrent, as the torrent is splitting it into 512k-4meg chunks anyway.

Whoops, end of rant. Oh, by the way, that wasn't about me, it was about my friend. Wink wink.

Re:!new

[Anonymous Coward]

I don't think this tactic is very common, though, as most people seem to have no fucking clue how BitTorrent works. I've seen torrents with gigantic multipart RARs, with an SFV of those. Let's see... so, my torrent software is already checksumming everything, and RAR has a builtin checksum too, or at least, acts like it does (it says "ok" or not) -- and on top of that, there's an SFV checksum (crappy CRC32), too. Never mind that RAR saves you at most a few megabytes (video is already compressed), which, based on the size of these files, you'll spend more time unpacking the RAR than you would downloading the extra couple megs. Or that, once you unpack and throw away the RAR, you can't seed that torrent from the working video. Or that multipart anything is retarded on BitTorrent, as the torrent is splitting it into 512k-4meg chunks anyway.

People who aren't aware of the full situation often make this complaint. These multipart rar files are "scene releases".

First of all, scene releases are _never_ compressed; it's always done with the -0 argument, this makes is basically equivalent to the unix split program. If a file is to be compressed, it is done with a zip archive, and the zip archive is placed inside the rar archive. This is because rar archives can be created/extracted easily with FOSS software, but cannot easily be de/compressed. This was more of an issue before Alexader Roshal released source code (note:not FOSS) to decompress rar archives.

Second, people often have parts of, or complete, scene releases and they are unwilling to unrar them (often because it's an intermediary, like a shell account somewhere where law isn't a problem).

Third, people follow "the scene" and try and download the exact releases that are chosen by the social customs of the scene (I am not going to detail those here), thus, "breaking up" (ie, altering) the original scene release is seen as rude.

Fourth, the archive splitting is in precise sizes so that fitting the archives onto physical media works better; typically the archive size is some rough factor of 698, ~4698 and ~8500.

Fifth, archives are split due to poor data integrity on some transfer protocols (though this is largely historical nowadays); redownloading a corrupted 14.3mb archive is easier than redownloading a 350mb file.

Sixth, traffic of the size is measured in terabytes, with some releases being tens, or sometimes hundreds of gigabytes in size. Thus, there become efficiency arguments for archive splitting; effective use of connections, limited efficiency of software(sftp scales remarkably poorly, though that is beginning to change - not that sftp is used everywhere), use of multiple coordinated machines and so on. This is an incomplete list of reasons; it is almost as though every time a new challenge is presented to the scene, splitting in some way helps to solve it.

AC because I'm not stupid enough to expose my knowledge of this either to law enforcement, or to the scene (who might just hand me over for telling you this - it has been done). Suffice to say that this is more complex than you understand, and that even this level of incomplete explanation is rare.

Re:!new

[AnarkiNet]

As far as I know, the habit of uploading torrents composed of a multipart RAR is that the "source" of those parts usually comes from those sites that allow anonymous users to upload files up to a certain size. So the "true" pirates, the ones that do the actual release to web of the game/software/whatever, will upload it to a direct hosting site in small chunks. The torrent author is just not smart enough to upload *after* unpacking the RARs.

Re:!new

[i.of.the.storm]

Ah, thanks for the enlightening post. Someone mod this informative. I thought I saw a comment along those lines, presumably by a "scene" person, on a torrent once, but I didn't remember the exact details.

Re:!new

[Anonymous Coward]

Nobody follows the "scene releases are never compressed" motto as assiduously as you believe.

People are welcome to "follow the scene" as long as they play on FTP where using multi-part files are actually useful and efficient. Once they are on a torrent, the uploader can bloody well unrar, check the quality and then seed it. Other users who want to contribute to the torrent can _also_ unrar prior to seeding.

This way,

1. I don't have to keep a crappy duplicate rar-ed releases for seeding (which I seldom do, i.e, I stop seeding once I unrar) and an unrar-ed version for my viewing.

2. The uploader will actually check the damn files before uploading. Have you noticed how many torrents have corrupt RARs? Or are uploaded without the subs?

3. Some people argue that if a RAR is corrupt in a release, they can just create another torrent with a "fix" for it containing the offending RAR alone. While this might sound dandy, if you had unrar-ed in the first place, this would have never occurred.

4. And what the hell is with this anachronistic bullshit of splitting AVI files? Hardly anybody uses 700MB physical media nowadays. Yet, the tards continue to split the AVIs because "that's the way it's always been".

5. While we are at it, what is with people rar-ing shit twice? This is usually done for sub files.

It's completely maddening for large releases like DVD / BR.

Re:!new

[Anonymous Coward]

These multipart rar files are "scene releases".

Yes, I know, I have a friend in the scene. It's not such a retarded policy there, but only because they use FTP -- and the choice to use FTP was a retarded idea anyway.

First of all, scene releases are _never_ compressed; it's always done with the -0 argument, this makes is basically equivalent to the unix split program.

Then why are they bothering to rar them?

One thing I can do with split, that I can't do with rar (that I know of), is easily play them back:

cat foo.mp4.* | mplayer -

Another thing I can do with split is, if I know what sizes they were split to, I can generate identical files. I can't do that with a rar -- once I unrar, no way am I keeping some 4 gigs around to seed. I'll nuke the rars, and I won't be able to recreate them, so I won't be able to reseed, ever, without re-downloading it.

If a file is to be compressed, it is done with a zip archive, and the zip archive is placed inside the rar archive. This is because rar archives can be created/extracted easily with FOSS software, but cannot easily be de/compressed.

...wtf? Why not a multipart zip, then? Or gzip+tar+split? (gzip the file to be compressed, tar the whole structure, then split it.) I'd assumed the whole reason for using rar in the first place was there was some lazy bastard with WinRAR who thinks it's the best thing ever, and doesn't know anything about FOSS. (I can say that because I used to be that lazy bastard.)

Second, people often have parts of, or complete, scene releases and they are unwilling to unrar them (often because it's an intermediary, like a shell account somewhere where law isn't a problem).

I'm not really sure what this has to do with what we're talking about... If scene releases weren't rared initially, they wouldn't have to unrar them. And if they are, this only enters into it if that particular shell account is used to seed a torrent, which seems very likely to be against Scene rules.

Third, people follow "the scene" and try and download the exact releases that are chosen by the social customs of the scene (I am not going to detail those here), thus, "breaking up" (ie, altering) the original scene release is seen as rude.

Unraring before creating a torrent is rude to the scene? Even if all the files are left intact, and in exactly the same structure? (If you're that worried, put an sfv/md5sum inside the archive, so that people can verify that all the files are where they're supposed to be.)

It's also interesting in that from what I understand, Scene rules prevent you from being on anything p2p. So it's not as though anything done when it hits torrent directly affects Scene members.

And by the way, uploading gigantic, split RARs is considered rude on at least a couple of BitTorrent trackers.

Fourth, the archive splitting is in precise sizes so that fitting the archives onto physical media works better

Again, why not split or multipart zip? And in this case, are there that many people who download Scene torrents, who burn them to DVD rather than watching and deleting?

Seems to me that if burning is less common than watching and deleting -- or watching and uploading to Usenet -- then it makes sense to let the people who actually need to burn figure it out themselves. It should take them no more time to split it than it takes me to unrar it.

Fifth, archives are split due to poor data integrity on some transfer protocols (though this is largely historical nowadays); redownloading a corrupted 14.3mb archive is easier than redownloading a 350mb file.

Not entirely historic; see FTP. One wonders if it wouldn't be easier for the Scene to use rsync servers internally instead.

Sixth, traffic of the size is measured in terabytes, with some releases being tens, or sometimes hundreds of gigabytes in size. T

Re:!new

[Anonymous Coward]

Come to think of it I've noticed how serious 'sceners' take their releases. Filling their NFOs with various excuses for why they released their files after group X. I suppose the scene is filled with anal-retentive asshats that everyone has to thread carefully around. Male culture at its finest =)

Re:!new

[Anonymous Coward]

There is absolutely no reason to split files when sharing them over BitTorrent. None. Preiod. As has been mentioned above, it actually just makes things harder to use.

What you're saying is that files are split because thats the way it has always been, or because they have been shared using some other method than BitTorrent, and people are afraid of changing the files. That's just silly.

More likely is that the people who put the files up on BitTorrent are simply ignorant.

Posted as AC to protect my kneecaps :-)

Re:!new

[dk.r*nger]

First of all, scene releases are _never_ compressed; it's always done with the -0 argument, [...] This was more of an issue before Alexader Roshal released source code (note:not FOSS) to decompress rar archives. So, historical, and pointless. And anyway, just an excuse if there's any point in using RAR anyway. Let's see..

Second, people often have parts of, or complete, scene releases and they are unwilling to unrar them (often because it's an intermediary, like a shell account somewhere where law isn't a problem). So they should use BitTorrent. Run a seed on your [strike]compromised windows host[/strike] "shell account".

Third, [....] social customs of the scene (I am not going to detail those here), thus, "breaking up" (ie, altering) the original scene release is seen as rude. Oh, I think we're at the core of the problem. Pale teenagers in their mothers basements getting hurt feelings. I appreciate that someone will rip the Lost episodes in HD pretty much as they are being broadcast, and I actually look for some "group names" in the torrents I get - because they provide one file, not a RAR. In other words, provide what people want, and they will respect you for that. Make their life hard, and they will not care about your 1998 social customs. Like anything else in life.

Fourth, [...]fitting the archives onto physical media works better Yawn. 1998 called, they want their infrastructure back. Harddrives are cheaper than dirt. Five years ago "the scene" at my college exchanged 250 gb harddrives.

Fifth, archives are split due to poor data integrity on some transfer protocols SO USE BITTORRENT! It easier and faster and better and more fun, but of course less 'leet than using [strike]compromised windows hosts[/strike] "shell accounts"

Sixth, [...] Thus, there become efficiency arguments for archive splitting;[...]it is almost as though every time a new challenge is presented to the scene, splitting in some way helps to solve it. No, BitTorrent does ALL this for you. ALL of it.

AC because I'm not stupid enough to expose my knowledge of this either to law enforcement, or to the scene (who might just hand me over for telling you this - it has been done). Badass gangster!

Suffice to say that this is more complex than you understand, and that even this level of incomplete explanation is rare. What? Moving files around on the internet is "more complex" than we understand? It probably the simplest fucking thing there is. Let me put it very simple for you: 1) Multi-file RARs made sense back when people got their stuff from FTPs and newsgroups. 2) It's the past. It's pure nostalgia. Get over it. If you're not using your "scene" FTP servers as Torrent seeds instead, you're wasting your resources.

Re:!new

[Anonymous Coward]

GP here. Agreed; the slashdot analogy is excellent.

Re:!new

[Anonymous Coward]

Then why are they bothering to rar them?

GP here. It's because, back in the day, rar had parity+split archives, and that wasn't really available in any other format. Unix split includes no parity, and most other formats either weren't available, or weren't available in a way that could easily be scripted on the unix machines of the early 1990s.

Since then, while everyone acknowledges that another format would be better, the problem is inertia. This is the same inertia you see in all large organisations; there are a lot of tools that people use that would take a long time to rewrite, and the cost/difficulty of rewriting is large enough that the modest gains of a better format don't seem worth it.

Re:!new

[Anonymous Coward]

GP here; Bittorrent gives every peer in the swarm every other peer's IP address. Try and figure out out why that's not a good idea when you're uploading Academy Screener DVD's. This is why we use sftp (which bares zero relation to ftp, by the way).

Re:!new

[totally bogus dude]

I actually look for some "group names" in the torrents I get - because they provide one file, not a RAR. In other words, provide what people want, and they will respect you for that. Make their life hard, and they will not care about your 1998 social customs. Like anything else in life.

Firstly, if you use torrents than nobody in the "Scene" gives a flying toss about whether you respect them or not. I have nothing to do with the Scene, and even I know that. They are not ripping things for us, they're ripping things for themselves. We're feeding from their scraps, if you like.

Once you understand that, all the other arguments become moot. Yes, multi-part RARs in torrents annoys me as well, but the people making them aren't doing it for us. Most (all?) Scene members would much prefer their releases never ever made it onto BT or USENET. Telling them that you disapprove of their distribution practices is, well, hilarious. Like a bank robber telling the cops he disapproves of their regular patrols of the street with all the banks on it. Actually, it's more like a bank robber in the US complaining about a pre-school teacher in Japan because he doesn't like the colour of the crayons they use. Thanks for the input, but who asked you, anyway?

So you're left trying to convince the people who do upload to more public services to unrar before they upload. More power to you, and I wish you luck. But I think the mob has largely spoken on this matter, and the mob says: "I don't give a crap if I have to unrar it first, so long as it's a) complete and b) a fast download". The torrents with multi-part archives tend to be seeded better than those which contain the extracted file, and therefore more people download the multi-part; which results in more seeds on it, resulting in more people downloading it...

As for using BT in the Scene -- it's up to them, it's their resources and they can do what they want with them -- so the following is purely mental masturbation. I would think BT would make it harder to keep "safe" and maybe easier to infiltrate. Password-protecting the servers (assuming most BT clients and trackers even support such) is probably insufficient; you'd likely want a local firewall to ensure only other Scene members can connect to your client. Keeping such a list updated in a secure manner would be somewhat tricky, I think, and telling everyone else the IP address of every other member sounds like a no-go.

Re:!new

[Anonymous Coward]

AC because I'm not stupid enough to expose my knowledge of this either to law enforcement, or to the scene (who might just hand me over for telling you this - it has been done). Suffice to say that this is more complex than you understand, and that even this level of incomplete explanation is rare. I'm not sure you need to be that worried about the level of information you're exposing here - I have never been part of the scene, but I've read an article or two on it and it's easy enough to guess this much. I'm sure that if I already know it without trying, the scary law enforcement people already know as well.

Secondly, just because there is a reason for these files to start off as multipart rars, doesn't mean that there is any reason for them to stay as multipart rars. Sure, it makes sense to split things in the scene, but when some kid puts it up on bittorrent they should still unrar it first.

Re:!new

[ccguy]

scene releases are _never_ compressed; it's always done with the -0 argument

[citation needed]

Re:!new

[Anonymous Coward]

Yeah, the warez scene seems to have this reflexive distrust of Bittorrent.

One Mac warez board I frequent now and then simply REFUSES to allow any Bittorrent links. They get deleted and I think you can even get banned for posting them or asking for them. Rapidshare, etc is just fine, and they still have a major hard-on for this weird BBS-like protocol called Hotline, and its descendent KDX.

I've never seen any EXPLANATION of why they loathe Bittorrent; they just do. It is, in some way, simply not l337 enough for them. And if you don't like it, why, then, you're a lamer and/or a leech, and you're not good enough for their precious warez. Or something like that. I just shrug and get the serial files they provide.

Re:!new

[mariushm]

SFV is not related to BitTorrent, it was used a extensively with DC++, which had no means to verify file integrity.
I believe BitTorrent uses SHA1 hashes of chunks from the files, these hashes are stored in the .torrent file.

The multi RAR thing used by scene releases is done because it (theoretically) reduces the number of duplicate CRC32's and reduces the number of errors causes by transfer. It's much easier to re-download one 50.000.000 bytes file rather than downloading a 4 GB iso again.

You can't always rely on RAR's error recovery to repair something, there are lots of cases when it doesn't work, because the recovery information is usually about 2-4% of the archive, so in lots of cases the error may not be recoverable.

Summing it all up, the torrent hashes combined with the RAR's checksums are better than none.

Re:!new

[Anonymous Coward]

1) Multi-file RARs made sense back when people got their stuff from FTPs and newsgroups. Not even then. A sensible format would make it easy to losslessly move between the archive format/transport coding and the extracted files. If that was possible, they could store the actual files instead of the .rar files where it was technically possible to do so, and they could keep the .rar files out of any slightly modern transport protocol.

A sensible format would also not be overly complex for the task it was intended to solve, and it wouldn't have huge problems with required non-free software, but those are somewhat minor points in comparison.

Re:!new

[SanityInAnarchy]

SFV is not related to BitTorrent, it was used a extensively with DC++, which had no means to verify file integrity. What I'm saying is that many torrents include SFV files, which, as you've just demonstrated, is pretty pointless.

It's much easier to re-download one 50.000.000 bytes file rather than downloading a 4 GB iso again. Irrelevant to BitTorrent. It's no more difficult to re-download one 50 meg file than re-download the 2 megs out of that 4 gig ISO that happened to be corrupted.

You can't always rely on RAR's error recovery to repair something, there are lots of cases when it doesn't work, because the recovery information is usually about 2-4% of the archive, so in lots of cases the error may not be recoverable. It's not error recovery I'm interested in, it's error detection.

Re:!new

[SanityInAnarchy]

Yes, multi-part RARs in torrents annoys me as well, but the people making them aren't doing it for us. Most (all?) Scene members would much prefer their releases never ever made it onto BT or USENET. If that's the case, then why aren't the people who actually do put them onto BT bother to unrar the fucking file?

So you're left trying to convince the people who do upload to more public services to unrar before they upload. Are you assuming that the people who do upload also never actually watch the stuff? Even to verify it?

Seems to me they'd have already unrared it, so at that point, it makes much more sense to simply ditch the rar and make a torrent from the unrared file.

But I think the mob has largely spoken on this matter I think the mob really hasn't said anything.

I would think BT would make it harder to keep "safe" and maybe easier to infiltrate. No more so than any other technology they're using. They probably just don't know how to do it with BT yet.

Password-protecting the servers (assuming most BT clients and trackers even support such) is probably insufficient; you'd likely want a local firewall to ensure only other Scene members can connect to your client. Private trackers exist.

Keeping such a list updated in a secure manner would be somewhat tricky, I think Not particularly. Every decent BitTorrent client includes PeerGuardian-like functionality now. All you'd need to do is point its blacklist at some obfuscated https URL. Problem solved.

Assuming you even had to do that in the first place.

Re:!new

[totally bogus dude]

Seems to me they'd have already unrared it, so at that point, it makes much more sense to simply ditch the rar and make a torrent from the unrared file.

Well I can only speculate, but my guess is that a lot of initial seeders have obtained the release as a multi-rar on a remote system where they can't watch it; probably a shell on a server with fat pipes. Some may not actually have the ability to unrar on the remote system to at least verify it's intact. Or if they're using a system they're not authorized to use, they might feel that their bandwidth usage won't be noticed but if they start unrarring massive files someone's likely to get wise. Others may simply be trying to be the first to get the torrent up, and feel that spending a few minutes unrarring it may cost them that honour.

Regardless, my hypothesis is that when a new release first makes its way to P2P land, a small group of initial uploaders will have a big impact on the future of the torrent. If the multi-rar happens to be better seeded than the unrarred one early in the game, it's likely to continue to grow in popularity as most people will choose more active torrents in preference of less active ones.

Transferring the files from a well-connected remote server to an end-users' home connection takes time. Unpacking the archive takes time. Watching the video - even just skipping through it to check for obvious problems - takes time. During all that time, the multi-rar can already be being seeded, giving it a significant initial advantage.

I think the mob really hasn't said anything.

Okay, that's true enough. I guess it's more correct to say that the mob doesn't really care one way or the other, otherwise rarred torrents wouldn't be around.

Not particularly. Every decent BitTorrent client includes PeerGuardian-like functionality now. All you'd need to do is point its blacklist at some obfuscated https URL. Problem solved.

Except a blacklist is the opposite of what they'd want to use.

Remember, the Scene members tend to belong to well-known groups who are definitely of interest to law enforcement. Not only do they need to minimize exposure of the Scene as a whole to LE, but they also need to minimize the knowledge each individual has about any others. It's not reasonable to expect someone who gets caught not to cooperate in order to get a reduced sentence.

Also, distribution within the Scene is based on a credit system. It might be possible to retrofit such things into something like BT without having a centralised server, but it's an awful lot of work for little gain. The way they do things now works fine.

Wikipedia has some info on the subject.

Re:!new

[SanityInAnarchy]

Unpacking the archive takes time. Really, not much.

Now, your scenario of a remote server, with an illicit account, makes a bit more sense -- downloading, unraring, and then uploading, may take too much time. At the same time, I'd think it would be a lot less bandwidth to notice if they simply download the rars once, and then seed the torrent from home.

But unpacking the archive alone is only going to be a few minutes, and that's not going to be the deciding factor.

Except a blacklist is the opposite of what they'd want to use. D'oh! It seems like it would be simple to patch it for the converse, though.

Also, distribution within the Scene is based on a credit system. It might be possible to retrofit such things into something like BT without having a centralised server *cough* TRACKER *cough*

Yes, there is a centralized server. Private trackers operate on a credit system -- you can pull up a list of torrents you've contributed, and the tracker will track everyone's total bandwidth up and down, and ratio. Most of them will ban people who have too low of a ratio.

All this is available to the tracker itself -- including your IP address, etc. But that would be just as true of SFTP -- if you can't trust at least one central server with that information, it's game over anyway.

It may be possible to game the system, but I haven't really seen that done even in large communities, and this is going to be a much smaller, much more technically-inclined community. (Look at what they have to put up now with SFTP.)

it's an awful lot of work for little gain. I'd have to know more about how the actual members are distributed, but I suspect that it'd get rid of their reliance on a few centralized, high-performance SFTP systems. BitTorrent requires a lot less in the way of dedicated hardware, while not giving up that much control.

And then there are things like WASTE, and a few other systems I could point to. Thinking of developing my own. It just seems ironic and slightly infuriating that they're relying on perhaps the strangest technology as a whole.

Re:!new

[totally bogus dude]

Trackers only track what the client tells it about - they periodically tell it how much they've uploaded and downloaded. Obviously if you alter your client to report unbelievable statistics to the tracker people can and will notice, but if you keep it in the realm of believability you can cheat quite effectively. I've actually once been banned from a private tracker for uploading too much; apparently they weren't expecting anyone to upload at several megabytes per second, and it triggered an auto-ban on the assumption that the stats my client was sending were fake.

I think the big issue though is that the Scene (at least, as I understand it) is a bunch of very small, closed communities (where a "community" is probably a single FTP server and the people who have access to it), with tightly controlled conduits between them. While you see this is as being inefficient (and clearly it is, if you're just considering distribution of data), they see it as a feature. They specifically want a minimum of people to have access to multiple communities. So there is no "one central server" that everyone trusts, but multiple servers that a subset of the people involve trust.

Another problem is that BT is largely a "pull" system - you publish a torrent and people can go get it. Most of the Scene operates as a "push" system, with people uploading new content in order to get credit. This means you'd have to have a separate system in order to make the .torrent available to the community you want to upload it to, preferably in an automated manner.

Yet another problem is the fact that individuals are trying to get as much credit as they can, and sharing the upload burden is not an effective way to maximise your own credit. Since most people are going to be trying to do 1:1 transfers anyway, BitTorrent and similar protocols actually introduce inefficiencies in transfer - not to mention a heck of a lot of overhead and complication.

I think another difference between the BitTorrent communities and the "Scene" is that BT communities reward people for continuing to upload content someone else provided in the first place, whereas the Scene specifically only rewards people for contributing new content.

Re:!new

[SanityInAnarchy]

Trackers only track what the client tells it about - they periodically tell it how much they've uploaded and downloaded. Interesting. It doesn't seem like it would be too difficult to simply do some double-entry accounting on this, though -- you can't suddenly decide to "upload" a few gigs to nobody; someone else would have to report downloading an equivalent amount.

I think the big issue though is that the Scene (at least, as I understand it) is a bunch of very small, closed communities (where a "community" is probably a single FTP server and the people who have access to it), with tightly controlled conduits between them. While you see this is as being inefficient It's not so much the community/cell structure that I see as innefficient, it's that even in a community of that size, I see FTP as the absolute rock-bottom last choice I'd ever use. And BT can work on that small a scale.

Another problem is that BT is largely a "pull" system - you publish a torrent and people can go get it. Most of the Scene operates as a "push" system, with people uploading new content in order to get credit. People upload that new content to one central server.

So the only problem here is if nobody wants to download that content -- in which case, should you still get credit for it?

I think another difference between the BitTorrent communities and the "Scene" is that BT communities reward people for continuing to upload content someone else provided in the first place, whereas the Scene specifically only rewards people for contributing new content. Yet obviously, someone has to pay the bills of that single server. (In fact, from what I understand, there are membership fees which go towards paying that bill.)

So it makes sense to have at least some sort of a credit system for contributing bandwidth, which goes towards eliminating the need for a central server with tons of bandwidth and storage.

Oh -- it also seems like a good place to do Waste, though I'm not entirely sure. I know a lot more about BitTorrent than I do about Waste, and that's not saying much.

Any MD5s on Apple's page?

[CSMatt]

Are their even MD5 hashes on Apple's download pages for such large files? Jusging by how the article was written and the lack of hashes on the QuickTime and iTunes download sites, it doesn't seem like they even bother.

Re:Any MD5s on Apple's page?

[wizardforce]

I would think that the only reasonable thing to do would be to have md5s because they are such large files after all, they could be corrupted.

Re:Any MD5s on Apple's page?

[Anonymous Coward]

Yes, there are- though most of the latest ones are SHA-1 digests. They're not usually seen in the "public front page" download areas and aren't universal, but are generally present for the downloads for updates and security patches through links from the tech literature and developer sections.

Re:Any MD5s on Apple's page?

[Anonymous Coward]

Apple provides SHA-1 hashes on the download pages in the Support section.

Re:Any MD5s on Apple's page?

[Anonymous Coward]

Apple's disk image files contain a hash and they're verified when mounting the image.

Hardware Failure is your bigger concern

[Bazar]

One should be more concerned as to why your files are becoming corrupted.

I'd say its a safe bet that the files from apple.com are in perfect condition.

Which means it either became corrupted in transit to, or on arrival to your machine.

Which leads the question, is your memory defective
run memtest86 to check your memory.
http://www.memtest86.com/

Check if your Harddrives have SMART and are reporting anything. A disk checker would also be a good idea.

The other idea that springs to mind is if your behind some proxy with the above problems, although i doubt anyone would want to proxy a 1.5gig file.

Fact is, if files are being corrupted on your disk, its just a matter of time before something more important is hit by corruption.

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

could also be one's routers.

There was a problem w/ dlink routers back in the day that hit alot of p2p users. If you placed your machine in the dmz, the router basically did a search and replace on all packets replacing the bitstring representing the global address w/ the bitstring representing the local address. On large files, this didn't just hit in the ip header, but in the data as well corrupting it. If you didn't use dmz functionality, just port mapping, it worked fine, so if you were using bittorrent, you'd get repeated hash fails on some parts that would never fix, because bitorrent has no capability to work around that (as opposed to eMule's extensions)

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

probably not the case. I too got a bad download, on two of three different machines, but all are in perfect hardware condition. more likely that apple has multiple servers with different copies of the same disk image, one (or more) are corrupt, and you are just happening to get that exact image. after a few more tries, I got an image that worked perfectly.

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

Agreed -- it is probably not the poster's issue. For the first iPhone SDK beta it took me three attempts at download to get a valid disk image.

Re:Hardware Failure is your bigger concern

[log0n]

As per the topic, Bittorrent fixed the problems - didn't cause them - so a failing router is not likely the problem. 99% likely it's bad ram.

Re:Hardware Failure is your bigger concern

[cheesybagel]

Maybe, maybe not.

IIRC TCP/IP has a guaranteed maximum error rate of at least 10^-5 bits. Well, the thing is, 1.5 Gigabytes is over 10^10 bits in length. So even at such an error rate, it is not guaranteed that your file will arrive without bit errors.

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

What about the possibility of bad RAM in the router? :P

Re:Hardware Failure is your bigger concern

[icebike]

One should be more concerned as to why your files are becoming corrupted.

I'd say its a safe bet that the files from apple.com are in perfect condition.

Which means it either became corrupted in transit to, or on arrival to your machine.

Which leads the question: Is the OP or any of the seeders on Comcast?

Re:Hardware Failure is your bigger concern

[BobPaul]

As per the topic, Bittorrent fixed the problems - didn't cause them - so a failing router is not likely the problem. You misunderstood his comment; please read it again. In his story, bittorrent didn't cause any problem either--it identified a problem by use of the same mechanism (hash checks of file parts) that it solved the problem in the OP.

While I agree that bad ram is most likely the issue, it's still possible bad ram in a router or even something goofy going on in a router, such as the firmware bug described, could have caused problems. The bits were mangled before they were written to the disk. They could have been mangled by anything that processed those bits as they traversed from apple's website to his HD, including Apple's website and the HD itself. That embedded devices tend to be more reliable does not mean they don't break and do weird things sometimes.

Re:Hardware Failure is your bigger concern

[rastoboy29]

I have certain people who play my game who simply _cannot_ download from my website--although it works great for me and most others.

I generally suspect malware on their clients, but I don't know for sure and it has long baffled me, because it is not rare at all.  Something like 40% or so.  Surely the malware problem is not so bad that 40% of net users can't download a 130 MB file via http without corruption?

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

It is more likely due to corrupted data on the way during download especially that many non-western countries use satellite links (which are lesser quality as far as data transmission) instead of fiber optics.

This may be off topic:

ISP's can also use more bandwidth illegally i.e. without paying the gov or regulating authority for bandwidth; it is easier to cheat or hide their usage via a satellite link. Although I doubt that they do that in India, I can tell you that in Lebanon it is difficult to get e decent connection from the ISP's for VOIP or on-line FPS games. Latency via satellite is ~ 650ms vs 150ms via FO. It is a good thing that the gov has its own ISP (although more expensive) since the gov never uses but fiber-optics.

Re:Hardware Failure is your bigger concern

[SnEptUne]

As far as I know, TCP/IP does not guaranteed error rate, it is all statistic. It isn't unusal for large file to get corrupted over network cable, especially over crappy cable, such as those in a dial-up connection. The protocol just isn't made to handle high noise.

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

see

http://www.utorrent.com/faq.php#I_get_tons_of_hashfails_on_my_torrents_and_the_torrent_never_finishes

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

It also could be that it is being downloaded in web normal web browser. I have found that Firefox and IE corrupt stuff all the time. Actually, it usually is a result from a misconfigured web server not setting the MIME type correctly (eg. for binaries).

This is why I use wget or something similar when downloading really huge files. This also has the advantage of making restarting and continuing the download easier if it dies part of the way through.

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

I've had problems where people trying to connect to my site on port 81 couldn't, as their ISPs seemed to be doing something weird.

Re:Hardware Failure is your bigger concern

[i.of.the.storm]

Pretty sure they don't have Comcast in India...

Re:Hardware Failure is your bigger concern

[jweatherley]

Bad RAM would be surprising as the machine is stable. I'm more inclined to believe that it is a case of 'shit happens' with networking. As others have pointed out there is a not insignificant chance of an error slipping through with gigabyte sized files. I don't think the router is in the clear either. BitTorrent fixed the problem, but the protcol will keep trying until it gets the correct data. Safari, which is what I used for the original download, only gets one chance.

Re:Hardware Failure is your bigger concern

[BobPaul]

Lucky bastards.

Re:Hardware Failure is your bigger concern

[jweatherley]

It's cable from India to the outside world. We were cut off a while back when some ships started dropping their anchors in the wrong place in the Arabian Sea.

Re:Hardware Failure is your bigger concern

[i.of.the.storm]

Hah, funny. But if the internet situation in India is anything like that in Bangladesh, they've got a lot of crappy ISPs that sell "broadband" which is pretty much dialup-speeds through ethernet connected to a local hub somewhere. My family's from Bangladesh and we visit every summer or so, and this last summer they had their new "broadband" which was exactly the same speed as their dialup they had earlier, but through ethernet. It was through a proxy server too, which made it really annoying to configure programs and things. But I guess having internet 24/7 (or in their case 12/7, they paid I think 20% less to have internet from 9PM to 9AM instead of the full 24 hours because my aunt and uncle are surgeons who are working most of the day) is nice, as well as not tying up the phone line. And now my cousins are learning IMing from the google talk client I left installed on their computer, and getting exposed to the internet in general, which is probably more good than bad.

Of course, India might actually have decent ISPs since it's about 10 times more populous than Bangladesh (which itself is 1/2 the population of the US despite being slightly larger than Iowa).

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

A few years ago, I wrote some software for patching and updating a rather large software installation on multiple clients. Even on a LAN, we saw approximately 1 bit error for every 4GB of data transfered over raw TCP/IP. Errors happen.

True reliability over TCP requires strong checksums on top of the weak error-correction provided by the protocol. The bottom line is that HTTP and FTP aren't really suited for transferring more than a few megabytes of data without assistance.

But there is a standard for solving this problem: Metalinker. This defines an XML-based standard for block checksums and multiple sources of a source file. Now if people would actually use it to distribute all their large files...

Re:Hardware Failure is your bigger concern

[icebike]

Bringing this back on topic....

The point of my post was to remind people that Comcast in the USA has been inserting reset commands into Bit Torrent (as well as other P2P protocols) as a way of "managing the network" (what ever the hell that means).

If the seeders were on comcast, some of the problem getting complete downloads could be due to that practice.

Of course Comcast both 1)denies they were doing this and 2)has promised to stop.

Re:Hardware Failure is your bigger concern

[noidentity]

The other idea that springs to mind is if your behind some proxy with the above problems, although i doubt anyone would want to proxy a 1.5gig file.

What's this about his behind?

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

The undeniable truth that nobody wants to see, yet it is staring them in the face, is that if files are not being corrupted on his hard drive, then he doesn't have to worry as much.

Re:Hardware Failure is your bigger concern

[nguy]

That's basically extrapolating from the TCP checksum. But if the raw error rate is so high that you get a TCP error rate of 10^-5, then there is something else wrong with the link.

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

You should definitively check your ram.

I downloaded 10 times the developer tools from the apple site and each time with a different checksum.

I finally understood that the issue was with my newly installed RAM.

Re:Hardware Failure is your bigger concern

[Hognoxious]

We were cut off a while back when some ships started dropping their anchors in the wrong place

How do you know it was the wrong place? Perhaps they were intent... BRB, somebody at the door.

Re:Hardware Failure is your bigger concern

[__aailob1448]

This has happened to me using a linksys router a few years ago. Hexadecimal values equivalent to my internal IP (and another nonexistent internal IP, oddly enough) would get corrupted on large files consistently until I figured it out.

No such problems on the Tomato firmware I'm using now :)

Re:Hardware Failure is your bigger concern

[Anonymous Coward]

I had a similar problem. I once downloaded a Kubuntu DVD image 3 times. And yes, I was in India, where the bandwidth and speed constraints kill.

I ran the memory test and found that one of my 256MB RAM had problems. The 512MB second DIMM was intact. I started working with reduced memory.

I was just wondering, why any one did not have a tool to download chunks of the old file to verify the downloaded files. I started writing one. Now, that I know Torrent can help us do that, why bother?

Thanks for sharing.

Re:Hardware Failure is your bigger concern

[Srin Tuar]

But TCP does error checking per packet, not for the whole file at once. (so once per 1460bytes or so)

its unlikely that TCP is the culprit here, but with a file that big, there are many many places where things could
go wrong, and a single bit error is all it takes to mess things up in a compressed file.

I would bet on the HTTP client/server software being involved.

Re:Hardware Failure is your bigger concern

[cylcyl]

If you combine this with the Chinese cyber-attacks on India, it would make a lot of sense. Maybe Chinese routers are injecting random packets on data going to India?

the new dr. who sucks...

[jjeffries]

But Torchwood is usually pretty good, imho.

Re:the new dr. who sucks...

[Anonymous Coward]

You must be gay. Most of the new torchwood is guys kissing guys or girls kissing girls (not that I have a problem with the 2nd part).

And the new Dr Who is hit and miss, but there are a lot of really good episodes.

Re:the new dr. who sucks...

[sp332]

Are you kidding? Sure, some episodes are slow or don't really work, but the second episode of the first "series" (that's "season" in the US) of the new Dr. Who is in my top five favorite sci-fi TV episodes of all time, including all the Star Treks and Babylon 5.

Re:the new dr. who sucks...

[mrmeval]

The first season is written by a schizophrenic that likes lesbian porn. Too many contradictory episodes and only two good ones. They need to get rid of the immortal, the generic asian chick and the geek, do something about the cop that doesn't know how to use a real gun and put a spine in her and flush the traitor down a toilet as organic residue. They really need to drown some of the writers.

They could be the first series to kill off all but one of their franchise characters and several of the support crew!

Re:the new dr. who sucks...

[zippthorne]

To be fair, very few British cops know how to use guns. At least, if the gun control advocates on my side of the pond can be believed.

Re:the new dr. who sucks...

[Kjella]

Oh please, some of the epsidoes like the whole John Hart ones are just incredibly poor. And they have the most absurd sexual relationships ever and I don't mean the gay thing. They go from deep kissing to completely psychotic let's kill each other mode in two seconds flat. And to let the guy who almost killed your crew and would have killed you not be immortal just go. I much prefer the "normal" characters over the man himself, the less of him the better. I'd like a good action/sci-fi/csi flick which it is at times, but Jack seems to be the "particle of the day" of the series.

Doctor Who is good fun, light entertainment. It's a guy flying aound in a blue police box and you're not supposed to take it so seriously, particularly since there's time paradoxes cropping up all over the place. And I think the series show I'm soooooooooo glad Star Trek didn't go with a time agency series, and why they should have kept it out of Enterprise too. You go from self-healing time to self-destroying time to being prevented from certain events to changing much bigger events to anti-time to changes that ripple through time slowly/quickly/not at all and you'll never be self-consistent.

It's fun for the odd episode but destroys the whole logic. For example now at the end of Stargate Atlantis, doctor McKey returned Colonel Shepard from the future - and only sent with him the location of where to find Teyla. WTF? He could have given him 25 years of science and technology, all wrapped up on the data crystal (the same kind that contain for example the entire replicator code...). Why didn't he? Because he "can't" use that solution. It's like the convienient non-interference with the timeline in Star Trek. It allows you to actually do a little time travel without creating so many issues.

Re:the new dr. who sucks...

[1u3hr]

The first season is written by a schizophrenic that likes lesbian porn. Too many contradictory episodes and only two good ones. They need to get rid of the immortal, the generic asian chick and the geek, do something about the cop that doesn't know how to use a real gun and put a spine in her and flush the traitor down a toilet as organic residue. They really need to drown some of the writers.

Obviously you didn't see the finale of series 2. Two of your wishes were fulfilled.

Torchwood is pretty silly (especially in supernatural episodes with ghosts, Death Incarnate, zombies), but still watchable.

Re:the new dr. who sucks...

[jweatherley]

The End of the World? Not a bad episode, but its main purpose was to demonstrate that the new Doctor Who has a decent budget. 'Blink!' from season three is one of the best episodes ever; classic series included.

Re:the new dr. who sucks...

[Anonymous Coward]

BAAAWWWW!!! Why can't shows made in another country be exactly like the ones made in my country and exactly how I'd make them! Life's not fair!

Fucking yanks. Go watch some xXx or whatever passes for entertainment where you live.

Been using bittorrent and rsync for this for years

[DiSKiLLeR]

I've used bittorrent for this purpose many times in years gone by.

Especially with our slow links, or worse yet, on dialup (if I go enough years back) in Australia.

Before bittorrent I would use rsync. That required me to download the large file to a server in the US on a fast connection, then rsync my copy to the server's copy to fix what is corrupt in my copy.

It works beautifully. :)

Good for game files too

[trawg]

We have been doing this for ages for certain high-demand games file that we mirror. While offering torrents for some of our download mirrors is only mildly useful (as we're in Australia we're trying to keep bandwidth on-shore to cut down international traffic, and BT doesn't really help this), it is extremely helpful for the VAST amount of users that appear to either have massively crazy Internet problems or are simply unable to drive a HTTP based downloader and resume downloads.

When a large number of users are having problems downloading or resuming a particular file, I simply create a torrent for them and give them some vague instructions about how to resume it and then generally I never hear from them again. They're happy because they don't have to download a 4gb game client again from scratch, they don't have to worry about resuming/corrupt downloads, and because its a torrent it probably feels like they're getting something for free that they shouldn't be.

Re:Good for game files too

[bagofcrap]

Sounds like a custom torrent tracker w/ geo-locating capabilities to keep things 'inside' would be handy. Legal issues aside I've often though it would be fun to write one. Imagine if, say, Comcast, Rogers or a large entity ran their own tracker, and only allowed IP's that were within their topology. You could probably actually hit their advertised 3 Megabit download speeds consistently.

This helps fill in pieces from Newsgroups.

[NFN_NLN]

Big deal, I do this all the time. It also helps when you're downloading files via Torrent and supplement with pieces from the newsgroups. This combination works well because newsgroups often have RAR'd binaries that are missing files. Find a similar package available on a Torrent site and fill in the missing files. Hell you can start the Torrent first and do a Force Check as you add each piece. Why not just download the whole thing via Torrent then? Well nntp is local and much faster... Had I known this was worthy of a slashdot submission I would have done it all long time ago.

Or synchronize with yourself...

[greerga]

For even more fun, if you have two differently-corrupted copies of a file and a torrent to go with it, then you can have BitTorrent stitch them together into a valid file without involving any third parties.

I used Azureus's internal tracker ability and two computers on a local network with the torrent modified to track on one of the machines, and one corrupted copy of the file on each.

Obviously only works if they don't have corruption in common, but it also doesn't require the original torrent file tracker to work anymore.

Re:Or synchronize with yourself...

[Saberwind]

Just yesterday I wrote a simple program to consolidate two partially-downloaded copies of a file that existed in two dying torrents (no complete copies existed in circulation), based on the premise that the files had runs of zeroes wherever blocks hadn't been downloaded. The result was a more complete copy, which I was then able to redistribute.

Re:Or synchronize with yourself...

[BobPaul]

but it also doesn't require the original torrent file tracker to work anymore More importantly, this can be done without access to the internet. The lack of an available tracker is already made unnecessary by DHT. Just on the internet, it sometimes takes a while for Azureus to find someone else with the file through DHT.

Re:Or synchronize with yourself...

[ben there...]

For even more fun, if you have two differently-corrupted copies of a file and a torrent to go with it, then you can have BitTorrent stitch them together into a valid file without involving any third parties. It would be cool if someone built a small utility to do just that, built off of something like cfv, which only does torrent (+sfv,crc,csv,md5,etc.) verification.

Torrents are really just fancy networked .par/.par2 files, but it would be nice to have a tool for torrent repairing that works as well as something like QuickPar does for newsgroup files.

Re:Or synchronize with yourself...

[Anonymous Coward]

You also need a torrent from an uncorrupted file I assume?

What a novel idea!!!

[WarJolt]

Using bit torrent for it's actual legal intended use. I love it!!!

I'm not a lawyer though. I just hope it doesn't violate apples NDA. Please please please follow the rules. Don't want to see you in prison or slapped with a large fine.

Bit torrent has received a bad reputation because of pirates. There are legitimate uses though. I do believe that doctor who episodes aren't public domain, so shame on you for that. Might want to be careful what you admit to on /.

Re:What a novel idea!!!

[Anonymous Freak]

I have never used BitTorrent to download anything that I did not already have the legal right to possess. That is basically what the OP did. He had a legal right to possess the file, just the technical inability to get it officially.

Most of my BitTorrenting is "100% legal" (Linux distros, UBCD, publicly released media (new NIN, Star Trek fan films,) etc.) Those I make sure to seed to at least a 2:1 ratio, often more. But some of it is to download items that I have the legal right to have, but do not have ready access to. (My closed-source OS install CD when I'm not at home, for example.) Those I make sure to stay as close to the line of legality as possible by, unfortunately, seeding as little as possible.

Re:What a novel idea!!!

[Anonymous Coward]

I just hope it doesn't violate apples NDA. Please please please follow the rules. zomg someone might violate copyrights on the intarwebs? that's unpossible!

p2p is here to stay regardless of whether anyone "follows the rules" or not. if at first you don't succeed just torrent the bitch and get it over with. why waste your time(/money/bandwidth) doing anything else?

Re:What a novel idea!!!

[Anonymous Coward]

you admit by your reasoning to be a cocksucker online which is worse than a commercial criminal or so would most noncocksuckers think

why should he be ashamed of his petty quasicrimes when you frivolously suck digital cock is what most people ask when they see your post

Re:What a novel idea!!!

[Anonymous Coward]

He's in India. I don't think he needs to worry about IP laws.

Re:What a novel idea!!!

[Anonymous Coward]

When will you use the apostrophe for its intended purpose? I'm gonna love the day when people learn that it's means IT IS.

Re:What a novel idea!!!

[Anonymous Coward]

Since when was piracy not a legitimate use of the shitty bittorrent technology?

Re:What a novel idea!!!

[Anonymous Coward]

SHUT THE FUCK UP

Re:What a novel idea!!!

[Hatta]

Most of my BitTorrenting is "100% legal"

And 50% of the time it works every time.

Re:What a novel idea!!!

[Anonymous Coward]

In some places downloading doctor Who or any other series or films or songs are 100% legal and legitimate, Mr RIAA.

Re:What a novel idea!!!

[Anonymous Freak]

I understand your quip, but my point is that most of my BitTorrenting is unquestionably legal, official torrents by the rights holders. I'm not saying that 100% of my BitTorrenting is legal, except for the not legal parts. I'm saying that 75% of the use is 100% legal. (The use of a percent was probably not the best choice, I now admit. Saying "Most of my BitTorrenting is unquestionably legal" would have been better.)

The remaining is what should be legal under fair use, but may not be. My example of downloading a non-free OS install disc is one. I have the legal right to possess the content, so while the letter of the law may being broken, the spirit is not. (And, for that matter, distributing Windows discs is probably 100% legal, since by Microsoft's own definition, the license lies with the Product Key code sticker, not with the disc.)

So most of my use is 100% legal, and the rest is what I would call "90% legal".

Here's one for you

[gEvil (beta)]

Hey guys, check this out! I just found out that you can send emails to multiple people AT THE SAME TIME by putting a comma between their email addresses! Pretty cool, huh?

Anyone know how to do this with USENET binaries?

[Anonymous Coward]

I can have 60% of a file downloaded but have BitTorrent only see 10%, I'm guessing because it's missing an article somewhere around there. Any client that zeroes missing file parts instead of simply not writing them? Is that possible? What about with par2 files?

Same for Rsync

[Anonymous Coward]

I used to buy Debian CD's from a Linux shop in Sydney Australia. A few times I'd get badly burned CDR's from them, so I would take an image of the bad discs with dd, then rsync that image to fix it and then burn the fixed image.

Worked perfectly every time. I'd rather use BitTorrent for that though. Probably be quicker.

Re:Invites to Demonoid

[Anonymous Coward]

Just for the sake of chaos, here are 5:
lcw82wrfd7vxyf6iuzq2i6l3kyrbos1lyykdd1bjxq9v5
6xeeoo52jhmaijrjodvhehaeqn3w70keuwxvajby
ky8cswluxe0jh2km2rw5tbpc37agdnogk32bq5r98
mfqtowgp6l2gial5leeardj1hw91lv9mey2rgc0s
xdnlqkijbc4fu105hil2jql3g8h9ri61uvtw3g
http://www.demonoid.com/register.php?with_invite=1

Re:Invites to Demonoid

[Anonymous Coward]

Thanks!

Re:Invites to Demonoid

[cyberzephyr]

Thanks a bunch. FYI /. users, i took the first one the other 4 are free!

simpler home-brew technique

[v1]

I wrote this bash script to do basically the same thing. It uses openssl (built into most unix and OS X in specific) to create 1mb check files basically the same as torrent files. Follow the instructions and its easy to fix a corrupt download from someone that has a good copy, with the minimum required data transfer. The person with the bad file runs option 1 to make the check file and sends that to the person with the good file. They run option 2 which identifies bad chunks and exports them, which they send back to the first person. Run option 3 and the exports are patched into their download and it's fixed.

Last time I used it, we repaired a 3.8gb transfer by exchanging 11mb of data. (the transfer had been resumed multiple times and apparently one of the transfers glitched its offset or something)

This is easier than BT because using BT can have a bit of a learning curve for seeding. Beta but appears stable. Feedback encouraged.

Re:simpler home-brew technique

[Just Some Guy]

The person with the bad file runs option 1 to make the check file and sends that to the person with the good file. They run option 2 which identifies bad chunks and exports them, which they send back to the first person. Run option 3 and the exports are patched into their download and it's fixed.

Isn't that almost exactly how rsync works?

Re:simpler home-brew technique

[v1]

I can't possibly come close to Rsync in 300 lines of bash. ;) But that's the general idea, yes. This is for people that are not totally afraid of terminal but don't know how to map ports, use rsync (safely) etc. My script statically uses 1mb blocks for example, and requires a lot of user interaction. But it does the trick.

Unfortunately something like this can't really count on BOTH ends of the file transfer being technically inclined.

Re:Invites to Demonoid

[Anonymous Coward]

Bottom 4 are already used. Didn't need to try the first one. Get it while it's hot!

Re:Invites to Demonoid

[Anonymous Coward]

Okay, to celebrate Demonoid's re-opening, here's some more:

f5mptgleeecic81ppcn2hzkjugrg4b4sdglrwe4e
enz2yuz1gsv17mpetil8ltsmq1e17cbtw11fc9uvoa
p6gzu1iguz4o0aep93l5h1jujwt13pg5q9wy5
5ubabvxlkj4z8jmr0iu8kreil7xcf7jkp2ia2252442
yubtly2w8ghvae5839faz5mmancawheh0vgf70merdm

I hope you verified the file

[cerelib]

This is not really anything BitTorrent specific, but good use of available tools. However, I hope you then checksum verified the completed file with an MD5 from Apple or somebody who has downloaded directly from them. While you probably weren't a target of an attack, you did download software from an unknown source. An attacker could download the SDK, insert malicious code, compute a new set of MD5 sums for the torrent file, upload to pirate bay or some tracker, and then seed the torrent expecting that nobody will attempt an external verification.

CRC

[the brown guy]

I had a shitty old hard drive that was failing CRC (cyclic redundancy checks) but the file I had downloaded was 4 gigs, and there were a few corrupt pieces, but by copying it to another hard drive, and replacing just the corrupt pieces I saved myself a shit load of bandwidth.

"Saves a lot of bandwidth wasting."

[Anonymous Coward]

Wouldn't the methodology just save a lot of bandwidth?
If you save the wasting, does that mean that you subsequently have to waste that bandwidth elsewhere, so that the entropy of the universe remains constant?
Would a sufficient quantity of hoarded bandwidth wasting go unstable, become a bureaucratic singularity, and emerge as a new government, complete with its own non-event horizon?
I worry about stuff like that.

Re:Invites to Demonoid

[Kredal]

Thanks! The top code worked for me, and the bottom one was already used... (:

What asshole tagged this '!news'?

[sootman]

OK, maybe not tonight-at-eleven news, but this is a totally clever hack, which is exactly what many people on Slashdot live for.

On a related note, I came up with a roundabout way to do something similar to help a friend who was having trouble moving large files. On the remote end, split the file into small chunks. Then md5 them all and save those results into a text file. Then, ftp them, and when they arrive, md5 them all again and compare your values to what's in the text file. If any don't match, re-download them; else cat them all together and you should be good.

I don't think this wouldn't have worked for the submitter, even if he knew someone with a known-good copy of the file, because I imagine these things work linearly, so if the bad part of the file was at the halfway mark, every chunk after that would have the wrong checksum. His method was very, very clever.

Re:What asshole tagged this '!news'?

[Free the Cowards]

To be honest, when I saw this story I was shocked it had shown up. I thought that using BitTorrent to repair mostly-whole files was obvious for this crowd. It's like "Using Water to Nourish Your Plants" showing up on a horticulturist site. If you know anything about how BitTorrent works then you should immediately realize that it will fix up mostly-good files for you.

The subsequent discussion has revealed that a large chunk of the slashdot population not only doesn't understand how BitTorrent works but doesn't even know about classic open source tools like rsync.

Re:What asshole tagged this '!news'?

[Anonymous Coward]

Agreed, I've been doing this routinely for years - going back to Kazaa and Getright on dial-up.

I didn't think I was doing anything particularly clever.

Re:Invites to Demonoid

[smart.id]

Got the second one, thanks.

Torrent Distribution Network - Results: Awesome

[erexx23]

I have been using Torrents for this very reason.

I was being required to copy sometimes 10-20GB of Virtual Machine Image Files from Server to PC or PC to PC on up 40 machines at one time.
This was taking way too long and copies were not perfect.
Restoration of VM images presented the same problem.
Updating a VM meant redistribution of the entire file to all machines again.

Using (Micro) Torrent and my own tracker changed all that.

I came up with the following solution using all available resources.
First I started by copying all images to workstations to a separate partition. (about 200GB of VM's.)
Then I created created my own internal Tracker and Web Page to host torrents.

The results were:
1. Extremely efficient use of all available network hard drive space.
2. Utilities every machine on the network to distribute the files.
3. Works extremely well restoring or redistributing the VM's to any one machine or several machines at once. (The more the better)
4. 100% accuracy in distribution.
5. The ability to quickly modify any one image on any machine, recreate the torrent(hash) and then update that image across hundreds of machines very quickly.
In other words, modifying a file only means that the machines only have to download the bits that changed not the whole image again.
6. With Micro Torrent any machine can be used as the tracker.
7. The Tracker is also the "master" file server, however any machine can be used to modifiy and upload a change
Just recreate and re-upload the new torrent replacing the old one. Remember that a torrent file serving network is Not a server centric file sharing system.

In the old days before BitTorrent

[Orion Blastar]

I used to download Linux ISO files directly from FTP or web sites.

Nothing upset me more than downloading an ISO only to find out that after I burned it to CD/DVD, it had CRC errors and random lockups during an install.

After BitTorrent with error correcting, the problem was solved. It works for other things as well.

Commercial software companies can offer ISO downloads via BitTorrent trackers and send the install CD Key via email. That way customers just burn the CD/DVD and install the key they got in email.

Some thing with media files, download via BitTorrent enter an unlock key you get via email when you bought it.

Business are stupid if they ignore the benefits of BitTorrent.

Even piracy doesn't hurt that much as most people want to try the software before they buy it. It is like kicking the tires before buying a car and taking it out for a test drive before signing the papers to buy it.

Re:In the old days before BitTorrent

[aeschenkarnos]

Even piracy doesn't hurt that much as most people want to try the software before they buy it. It is like kicking the tires before buying a car and taking it out for a test drive before signing the papers to buy it.

Nope, better. Kicking the tires and test-driving the car add tiny amounts of wear and tear.

Stupid Private Trackers, too

[feld]

No, I've seen this as a requirement for a few private trackers. It put me off on posting as I'm not going to waste my time.

Fix up jigdo file

[YoungHack]

I've used it to finish up the last 3% of a jigdo build when I was missing a file or two. Worked great.

Re:Fix up jigdo file

[Hatta]

Wow, someone actually uses jigdo?

How does this sit with RIAA?

[Fluffeh]

I wonder if you could legitimately argue that you were verifying the data in a personal backup of media that you had?

Unless I am mistaken, it is perfectly legal to make a backup of data that you own right? So, if you already own an item, would downloading it to have a backup be a legal thing to do?

And if that's the case, I wonder what the legal implications are in cases where the RIAA comes down on people who have been "participating in file sharing" activities.

Another nice tool for this: rsync

[swillden]

Assuming you can find a source that serves a known-good file via rsync, it's a very efficient way to fix up a damaged copy.

I once had to download a CD image over a dialup connection when I was at a client site in Mexico. I did the initial download via FTP, but it got corrupted and the MD5 sum didn't match the correct value. It had taken almost two full days to download the first time (over a weekend, so shipping a CD wouldn't have been faster), but rsync was able to find and correct the corrupted sections in less than five minutes.

Rsync is also an unbeatable tool for making incremental backups. I use it (rather, I use rdiff-backup, which uses rsync) to back up a server with almost 30 GiB of data, nightly, over a standard cable modem connection. Last night's, for example, took 57 minutes to run, found 527 changed files totaling 1.36 GiB of 26.2 GiB total. I don't know how much it actually downloaded, but I'm sure it was much less than 1.36 GiB.

Re:Moron

[Anonymous Coward]

Yeah, if I had a 30GB cap, I'd be just as pissed off and irritable as you are. But I don't. (Nelson voice): Ha ha!

ÂTorrent's Relocate feature.

[Anonymous Coward]

Latest ÂTorrent (1.8+) also allows you to point to a file name that is different on your hard drive. You don't need to worry about file names matching up any more if the bits are identical.

Re:Hardware Failure is your bigger concern - maybe

[erexx23]

I agree TCP/IP has problems with raw file transfers.

However, a Torrent system ensures the delivery of the file based on the files hash value.

This is very beneficial if an update or recovery of the original file needs to be made.
Simply recreate the torrent, upload the updated torrent.
Once the clients get the new torrent they only download the changes to that file.

For instance.
A 10GB virtual image file needs to be changed.
Make the changes needed, recreate the torrent, upload the new torrent.
Clients download the new torrent fot the same file.
Restart the download of that file to the same location.
The client makes a hash check.
This time according to the hash value only 12% of the file has changed.
Only the bits that need to be needed to match the hash are downloaded.

Not only that but because it an asyncronis file transfer across multiple machines, on a large network, the update occurs incredibly fast.

This works 100% of the time.

Since a central file server is not needed, any machine on the network can act as the tracker, hardware failure maybe is the biggest concern.
But then again a failed component is always inevitable.
With a torrent system corrupted data transmission is no longer a real problem.

Bittorrent backup and sync

[Anonymous Coward]

i've done all the tricks mentioned here as well. for a couple of years now, i've been wondering when someone would release a backup/archive/sync tool that utilized bittorrent tech...i'm still waiting...

The first rule

[tux0r]

The first rule of Usenet: don't talk about Usenet.

Re:The first rule

[pikakilla]

Second rule: dont mod up the person talking about the first rule.

Re:Anyone know how to do this with USENET binaries

[Ristol]

Most do, but you may have to enable it in the program's options.

What an evil ISP

[Anonymous Coward]

Seriously, get another ISP! I mean, what kind of a BOFH-run ISP only lets you download 1GB per *month*, except for night hours??? Whoever came up with that shouldn't be allowed more then 14k4 for ever!

Repairing files

[ant_tmwx]

Besides rsync & torrents, you can also repair files with metalinks, which require nothing extra on the server, and is not blocked like p2p in some places.

This is why so many distributions use them for ISO downloads, so you don't have to restart large downloads from the beginning.

!new

[gaurax]

I've been doing this with linux ISOs for quite some time. Never thought it could be unknown to anyone.

Yup

[mikiN]

...yesterday I used BitTorrent to repair an Ubuntu Studio iso that I downloaded from my local ftp firehose. The MD5SUMS mismatched, so I fetched the matching torrent file, fired up KTorrent and pointed it at the dir I downloaded the iso into. Only 1 block needed repairing, saving me a helluva long download.

... or zsync instead of rsync

[Anonymous Coward]

"zsync is a implementation of rsync over HTTP. It allows updating of files from a remote Web server without requiring a full download or a special remote server application. It uses a metafile, which is created on the server, to determine which parts of a file the user already has; it then downloads the remaining parts via HTTP."

Chance of CRC clashes is much higher

[tucuxi]

First, as rdebath argues, you only get 16 bits of CRC on TCP headers.

And furthermore, if you start calculating CRCs off random data, chances (>50%) are you will get a collision (two chunks of data with the same CRC) around the 256th try (this is known as the "birthday paradox" in criptography). Of course, to be really sure to get a collision you will need to try at most 65536 values; but you will reach a very high probability of clash much sooner than intuition may tell you.

See birthday attack for the math.

Re:Chance of CRC clashes is much higher

[swillden]

First, as rdebath argues, you only get 16 bits of CRC on TCP headers.

To be precise, you get 16 bits of CRC on TCP header, TCP data and a pseudo IP header, and another 16 bits of CRC on IP datagram header. That's really not very much.

And furthermore, if you start calculating CRCs off random data, chances (>50%) are you will get a collision (two chunks of data with the same CRC) around the 256th try (this is known as the "birthday paradox" in criptography).

Around the 301st try, to be more precise. But the birthday paradox really isn't relevant to the likelihood of an error getting through.

For an error to get through, what you need is a block that gets corrupted to a value with the same 16-bit CRC as the correct value had. Assuming CRC is a perfect hash function (it's not, but that's okay), this will happen, on average, once for every 2^16 corrupted blocks. Some of those undetected corruptions will hit the header and cause the packet to fail to be delivered, either because an address or port gets blasted, or because it hits the IP header and that CRC catches it. Most of them will hit the data because during a bulk data transfer, the headers comprise a small part of the packet.

So, how often will undetected corruptions happen, in practice? It depends on how often data errors occur. Usually, corruptions are relatively rare, because most link-layer transports add their own CRC (Ethernet uses a 32-bit CRC). If you get a bad link layer, though, that frequently corrupts packets, it won't take too long before you get one. Basically, you'll get undetected corruptions about 1/(2^16) as often as you get corruptions. So a really noisy link that corrupts one 1500-byte packet in ten will give you, on average, about one undetectable corruption per gigabyte.

Emule is better

[Anonymous Coward]

Use Emule for widely unavailable files.

Always Use Encryption

[CranberryKing]

Please people. It's very easy. Just go into your settings and look for something that says Protocol Encryption and say 'Enabled'. If everyone gets into this habit, we will all live in a far better world. In fact, encrypt any application (that traverses the Net) you can. Application layer is nobody's business but your own.

Re:Been using bittorrent and rsync for this for ye

[stevied]

Rsync also works nicely for "upgrading" CD images of beta Ubuntu releases to the final version, and for, say, making a Kubuntu Live CD out of the normal GNOME-based Ubuntu one. It has the advantage that it can spot blocks that have moved around in the new version but are still the same, even if they're no longer on block boundaries.

Re:!new, yawn, its not 1991

[cheekyboy]

I was sick of multipart files in 1991, ha!

All your points are solved by software, split rars are a hack on deficient protocols or routers that limit BW per tcp connection.

Oh and, what is it with these stupid long ass crap file names, S05E03-XDVD-HPEP-LOL-FUKME.avi

This is not 1972 cobol days dudes, if its unlikely to be a hit like friends, stick to one digit to seasons, S3, E03 is ok.
Kill the lame postfix acronyms, except sensible ones not in caps as they take more pixel, (dvd) or (ts) is smaller.

As gordan ramsey says, "you guys a fuking tossers, your a shit head".

TvShowName-S4ep23.avi is nicer, i always rename because they are TOO DAMN long on HTPC systems. Again, this aint 1200bps modem days. (they werent this bad btw)

Oh and another pet peve of mine to your so called elites, stop resizing 720 rips or tv shows to 604 or 624, if its done only to compress better, or
to play on PSP, then why should 90% suffer, 720 original is best on 42in LCDs. Stop resizing because you own a crap 12in crt. Or want to watch tv shows in a psp, cartoons are ok, but not good tv shows. Dont give me this 624 is ntsc in usa shit, only trailer trash own CRTs. If you can afford to download, you own an LCD. If you own a shit tv, well you'll get a better quality any way. Again, read my lips, 624 or 608 sucks 1980s style.

Par2 files, dudes

[m50d]

Or are they too 1.0 for the kids of today?

Other applications...politics?

[Shoten]

I tried using it on our current administration. It showed up as being 29% complete, but unfortunately nobody's seeding the uncorrupted parts that we're missing. :(

Re:!new, yawn, its not 1991

[deroby]

Funny how you complain about how bad others do your dirty work while you apparently save enough money because of it to watch your favorite shows on expensive hardware.

One really good thing about P2P Tech

[misterjava66]

I have also noticed that the P2P softwares as a group seem to offer excellent features in the area of moving files, large and small, and not corrupting said files, even in high noise/disconnect environments. Its a feature set that should make its way into webbrowers/common-downloaders, but seems to just not happen. Anytime I see a file to download and it is over 300MB, I'm like, "oh-boy this could be an adventure"
:-)

Birthday paradox does NOT apply here

[Skapare]

The birthday paradox involves a population in which finding ANY two (or more) of the same is considered a match. That does not apply to a TCP header checksum because the comparison needs to be made against ONE SPECIFIC checksum (e.g. the one the packet in question has). You get a packet and it has a checksum. You calculate a checksum from the data. Do they match or not when the data is corrupted? That's not a birthday paradox.

The birthday paradox DOES apply in cases where you want to create TWO packets with the same checksum, but it doesn't matter which checksum that is. You can create two messages with the same hash in the case of cryptography where there is a weak hash. But in the case of error checking, it's not about creating any pair of matching checksums; it's about creating one checksum that matches one you already have that you cannot change. In birthday terms, it's about finding someone in the population that has the same birthday as you do.

OK, it's 16 bits. My bad. TCP bad. But birthday paradox does not apply here.

Re:ftpdiff is a far better tool

[Anonymous Coward]

I hate to have to be the one to inform you but, contrary to what you believe, this is not Ann Coulter.

disk images containing the hash

[reiisi]

The security implications of that have always bothered me.

I wonder, does the current diskutilities app phone home to check the hash? Not that that provides more than a speed-bump for the middleman.

Of course, it is somewhat useful for checking file integrity for issues other than crafted corruption.

Apple's downloads?

[reiisi]

Actually, I'm thinking I may have just undone a perfectly good disk swap recently when the problem might have been at Apple's end.

I guess I need to test that disk.