That's "another compromise"... IP Spoofing hasn't a f***ing thing to do with DNS recursion. One can just as easily spoof your address in a non-recursive request.
heh... they tend to much more stable if there's traffic moving across them. Left idle for more than a few minutes and they tend to fall apart. One packet every 30sec works well for me.
And from the people who need that level of reliability, you can't get that from cisco either.:)
Hah. Sad, but all too often, true.
I've seen HA, RPR+, etc. work sometimes on some hardware. And then there's the ones where it does ever more inventively crazy things. Like... reloading every linecard/vip when it's explicitly not supposed to, "supporting" exactly 7 OIR events before crashing (maybe they fixed that by now), cbus complex's for "stuck output" in a single timeslot of one CT1 on a CT3 PA... And my personal favorite... master and slave RSP both in the active state following a crash -- each took over "their" half of the router.
[E]IGRP is a well documented protocol. The problem is in the patents and copyrights. People have written [E]IGRP stacks for other systems before, and Cisco promptly sued them into silence. (I've not heard of one of these in many years, btw.) OSPF is the open standard for this sort of thing; but, yes, eigrp is so much easier to setup.
HSRP is a deadend... See Also: VRRP (which is an open standard)
PHB's like to know where to place blame. But, they also don't want to throw their cash in the trash. Cisco's support is over priced just as badly as their hardware. For aging equipment, the support costs often exceed the cost of the hardware.
That's not to say Cisco is devoid of any neat toys. Unfortunately, all of their neat toys are found in the top line hardware that has heavily inflated prices.
That's the traffic to the outside world. The bits flowing inside the company are far higher and more important.
In my apartment, yes, 100Mbit is fine. I don't move a whole lot of stuff between machines; and when I do, there are other things for me to do instead of waiting. However, in an office, this is rarely the case. There's a whole lot of stuff moving between machines and people are dependant on these things to be productive. (all the sub-minute "waiting for word to load" delays add up at the end of the week.)
Wrong. Well, that's not how it's (legally) supposed to work. The LEA doesn't provision the tap, the TELCO does. The standard is very clear... very few people are supposed to have access to the CALEA terminal -- perferablly one; there MUST be both physical and electronic security. (i.e. in a locked room with the monitor pointed away from the windows, and sufficiently limited remote access.) If the telco is, in fact, giving access to "everyone" (esp. non-employees), then they are breaking a bunch of very big laws. (the problem here is confidentiality... everybody can see everything.)
[I know, because I've read the standard. I've been a "calea system administrator" responsible for setting up the system. And I've talked with several telco's around the country that have gone down the same path.]
The odd thing is... CALEA isn't all that complicated. It just requires someone to bridge two extremely camps. You have to understand both telco switching and internet routing. Unfortunately, the people who "speak 5E" don't know shit about the internet; and there are numerous physical bariers to learning 5E. (it's not like there are many 5E manuals floating around the internet. However, there is a phrack article.:-))
Hours. And on occasion, days. (it's best not to let my debian mirror get that far out of sync:-))
The economy of Tier1 ISP's is far more complex than I care to discuss. Besides, they aren't at issue. It's the smaller fish serving consumers (tier2/3 ISPs) that are at issue. They aren't big enough to play in the direct peered world -- certainly not for "free". If [cable company] wants a connection to Sprint, AT&T, MCI/Worldcom/UUNET/Verizon, then they will pay out the ass for it. Even when the interconnection is mostly on their own equipment (i.e. no "last mile" BS from some RBOC.) There are some reasonable players in the market, but they aren't seen as Tier1 guys -- at a former job, we had a DS3 from WCG that was ~4.5k/month and later a medium speed (115M) OC-3 for ~7k. (uunet wanted 30k, btw.)
POP and SMTP use well defined ports. And if you're doing it "correctly", the connection starts off as plain text and switches to encrypted following a STARTTLS. And anyway, what difference would it make if your POP/SMTP traffic were slowed to 5KB/s? Yeah, that's sorta suck, but it'd still work just fine. And they wouldn't throttle their own servers.:-)
(And technically, you can tell a difference... self signed cert vs. "valid" CA signed cert. But that's a lot more work than simply spying on packets.)
Nope. You're confusing "router" with firewalls, NAT, and other forms of connection tracking and stateful inspection. Most ISP routers don't do anything like that. The closest you'll find is flow caching which is just a fast routing process... it's optional and there are limits on how much memory it can consume. When an idle flow expires, that doesn't break the connection; the router will simply follow normal routing proceedures and build a new flow record for subsequent packets. Routers don't care what the traffic is; they only care about where they need to send it. (and in some edge cases, where it came from.)
That's not to say routers don't have limits. Every router has a maximum throughput both in packets per second, and bits per second. PPS is generally limited by routing speed -- how fast can the router determine where to send this packet. BPS is limited by the physical hardware -- how fast can bits be copied/signaled between interfaces.
One of the key reasons ISP's are resorting to throttling is shear bandwidth. ALL ISP's oversell their connectivity. (It's the only way to make money on the low end where consumer's live.) They don't have the capacity to give everybody their 5M down/384K up all the time. And it gets worse... for a cablemodem, a single channel is 30Mbps down and 10Mbps up (+/-). When one is saturated, the other is starved -- the ACKs have to get in there somewhere. So, that 5/384 setup can support 6 people down and 26 people up, at once, per channel. There are literally hundreds of modems on each channel. Honestly, I'm surprised anyone can max their connection for any measurable duration.
(DSL is just as bad, if not worse. Having worked for a DSL provider, I know it's worse.)
You're missing the point -- and ignoring the word port. Redhat is not building it. Redhat is not testing it. Redhat is not distributing it. And, Redhat is not supporting it. Someone. Else. Is.
All they share in common is the name ("fedora") and the underlying source. There's more to the creation of a port than simply running rpmbuild -ba * in the SRPMS directory. (if it were that easy, Redhat would still be doing it.)
And this is not entirely true. I've gotta call bullshit. They used to support PPC, but dropped almost everything but i386 long ago: v6.2 was the last release for sparc, v7.1 was the last release for alpha and ppc (v7.2 for alpha was released by DEC.) Feel free to correct any I've missed -- and yes, I'm ignoring ia64 and s390. Only VERY recently has PPC support resurfaced in Fedora.
Only their marketing people know for sure why. It sure as hell shouldn't have anything to do with intel based macs but most likely does... "If we want to support the intel macs, we also must support the ppc macs, too. (or at least appear to.)" While that makes perfect sense to marketing people, it's laughablly stupid on technical merit. After all, they claim to have dropped PPC in the first place because of "lack of demand." So, suddenly, intel based macs are going to bolster demand for a ppc redhat for the machines on which people already (apparently) don't want to run linux? *shakes head and walks away*
It's theoretically easy to get linux booted on an intel mac since it's the same fundamental platform as every other i386 system... install an EFI aware boot loader and kernel and the deed is mostly done. ('tho Apple could've done things to make it much more difficult.) Getting redhat running on a ppc mac requires building and testing every single piece of software for an almost completely alien hardware platform.
Off the top of my head I was suggesting adding 10% redundancy to the torrent *content* which would also increase the torrent file by roughly 10%
... and this is the very first time you state that AT ALL. (so, the answer is sorta "both and neither"... the FEC is distributed in the swarm with hash's in the torrent file along side the standard pieces.) If you're going to draft a new protocol, you really need to be exactingly clear with what you intend.
By adding a redundancy set, you would reduce the "rareness" of every piece in the data set and thus improve the chances of completing the torrent. It's just that simple.
Only on paper... instead of having to download 100% of the pieces, you only need download 91% [100/110] of the pieces. You are making a vein attempt to reduce the rarity of data pieces by piling on more pieces... But for this to be effective, every piece ("data" or "parity") must be of equal utility. Using "parity" pieces to fill in missing "data" pieces is an extremely expensive process -- far more so than just downloading 100% of the data. (it'll always better to download data pieces over "parity" pieces due to the complexity of the recovery process. (trust me, people don't download par files if they don't have to.)) Bram complains about the same "academic hand-wavy approach" to the complexity with avalanche's ECC. [I guess he's seen that Snafu comic, too.]
Also, adding redundancy to the swarm will not improve speed or longevity of a swarm. Downloading pieces is faster than recovering them in all but the rarest cases. Downloading 100 pieces will take however long it takes. In reality, it doesn't make much difference if it's 100 out of 100, or 100 out of 110. On paper, it follows a block with more distributed copies will download faster, but reality is far more complex... maybe you lucked up and asked a peer on a 28.8 modem for that piece, or all the peers with that piece are busy (i.e. "choking" you),... Life would be much simpler if "2 + 2" didn't sometimes come up "5". Having more than the necessary number of blocks to choose from makes it appear as if the 100th piece is effectively 11 pieces... ask for all 11 and use which ever one gets there first. However, in practice, no client would download 99 pieces and then "race" the last piece.
Ultimately without a seed -- at least one peer with the complete dataset -- or at least one complete set of data distributed throughout the swarm, the swarm will stall; the torrent is dead. Even with 500% redundancy, if no one has all of the data, then no one has all of the data. If there isn't a complete copy within the swarm, there won't be until someone brings in the missing pieces -- either as "data" or "parity". This is what limits the lifespan of a torrent. Redundancy has no effect on it; so FEC won't do anything to help you complete un-seeded torrents.
We can already do exactly what you suggest by including par/par2 files in with the data files. And I've repeatedly pointed out the catch-22 with this scheme... The only time "parity" pieces would be of any use is when there isn't a complete set of data pieces in the swarm. If there isn't a complete copy across the swarm, then there clearly isn't enough data and "parity" to recover a complete copy, and the "parity" pieces people have downloaded are meaningless until additional pieces arrive. [This assumes the "parity" is across all pieces ala par and par2. "parity" across sub-sets could, in theory, prove to be of some use, if significantly limited. (eg. recovery of a single file in a multi-file torrent)]
There's either a full copy in the swarm or there isn't. It doesn't matter how much redundant data you present to the swarm.
By all means, spend your every free waking moment designing and coding your FEC extension. The only way you're going to realize the futility of what you're proposing is to actually watch it not do what you expect. Unless you have a means of recovering more d
just a guess, but it's reverifying the data it downloaded. The status will show "seeding + checking". And yes, you can turn that off. It also sounds like you have friendly hash checking disabled... it won't pause at all between pieces.
You seem to have missed the paragraph above that where Bram states:
The central idea here is basically 'Let's apply error correcting codes to BitTorrent'. This isn't a new idea, everybody comes up with it. In fact I saw fit to mention that it's a dubious idea before.... While it is very cool, and very applicable to sending information across lossy channels, the case for using it in BitTorrent is unconvincing. (emphasis is mine)
And he's talking about Microsoft's Avalanche. He's pretty clear that error correction within bittorrent is quite unnecessary.
I'm fairly sure I have a good grasp of the core Bittorrent protocol
It's not so much about the protocol (for which there is no full specification as far as I've ever seen), but how bittorrent clients and trackers actually work... the actual dynamics and life cycle of a swarm. (so to speak.)
I am not suggesting a simple parity check or basic hamming code on top of the TCP transfer.
Indeed. You're suggesting turning a torrent file into what will mostly be a par/par2 file by placing a percentage of the torrent data as parity recovery blocks in there. Doing so will significantly increase the size of the torrent -- dwarfing the actual torrent by orders of magnitude. (I'll come back to that.)
Based on your response it appears that your understanding of Reed Solomon FEC is from PAR files.
Actually my understanding of FEC/ECC goes well beyond PAR. And technically, what you are describing is not "FEC"... it's parity data. You aren't "correcting errors"; you are "(re)constructing data you don't have." FEC is about being able to fix errors in the data as it's being received -- not so much about pulling data out of nothing. FEC comes before the data in a stream; ECC comes after... I'm just going to calling it "parity data" because that's what you're talking about.
... my suggestion to not distribute the FEC pieces when the there are more than a certain number of distributed copies visible or when you know a seed will always be available.... it should also be pretty beneficial for initial seeding.
Comments like this are why I don't think you understand how BT works... the site, or more specific, it's tracker(s), know almost nothing about what's going on in the swarm. The tracker isn't a BT client. So it's not part of any of the swarms it's tracking. Therefore it knows nothing at all about the distribution of data within the swarm. So, what's going to decide who gets a huge parity torrent and who get a tiny standard torrent?
With any given torrent, you'll either have to download the entire set of data or the equiv in data + parity (which presumablly you have from the torrent.) If you have enough to recover the complete set, then you recover the data and *poof* there's a seed in the swarm making the parity data an unnecessary waste. 'round and 'round we go...
It will, in fact, make no difference for the initial seeding as it will always be faster and computationally cheaper to download the pieces than to recover them. As a result, clients will prefer downloading to recovery whenever possible. (This is esp. true since the parity data had to come from somewhere... either download 250M of data and be done, or download 200M data + 50M parity and spend a half hour (+/-) recovering the 50M of missing data.)
I am *absolutely* suggesting an extension to the BitTorrent file format that would include entries for redundancy pieces, similar to the way PAR files are used... you can stop distributing the redundancy data...
Make up your mind... is the parity part of the swarm (to be downloaded like all the other pieces) or part of the torrent file? Explicitly state one, yet imply the other.
If it's part of the torrent, you're just wasting the site's bandwidth and possiblly opening the site to copyright violations -- they're no longer simply describing copyrighted content. Again, exactly how is the site supposed to know when to include parity data? Basically, if I have parity data or not will depend on when I download the torrent. If it's part of the swarm, why the hell would anyone spend time fetching the parity pieces when the actual data pieces are available? There are alot of issue with your poorly explained extension.
Now, let's talk about how BT works off the paper and in the real world... As an example, let's say we have a community of 5 peers (A-E). Peer A has content it wishes to share. Pe
you're using FEC to compensate for the losses due to unequal chunk distribution
Do you even know how Bittorrent works? (don't answer that, it's obvious you don't)
Again, you just don't get it. FEC. Is. Simply. Unnecessary. Lemme esplain...
"FEC chunks"? "initial seed"?... First, a word on terminology... (various BT spec docs beat this into your head) in BT a piece is the amount of data for which a hash exists. (the min. verfiable data) Each piece is divided into blocks that are requested from other peers -- with a tendancy towards all blocks of one piece coming from the same peer so you know who to ban if a hash check fails. An initial seed is the peer who first offers all pieces to the swarm -- the only one who starts out with 100% of the torrent. This is not necessarily the person who created the torrent, or published it to a site.
You seem to think a few bytes of FEC can magically regenerate a 256k piece out of nothing. At best, FEC would be useful in detecting and recovering from bit errors. (That is what it's designed to do, btw.) However, such errors do not occur within bittorrent swarms -- unless something horrible, and well beyond any error correction, occurs. Peers do not report a piece as being available until it has all the data and passes hash verification. At that point, barring hardware failures and coding mistakes, the peer has exactly what it's supposed to have. When it hands out the data for that piece, TCP/IP ensures the bits sent are the ones received; and the receiver is going to do the same hash verification.
So, let's summarize... peer A is going to ensure the validity of the pieces it has by hash verification before making them available to other peers. [data starts out being exactly what it's supposed to be.] peer B asks for blocks amounting to a piece which are transfered by TCP/IP, assembled and then verified. [data transported bit-for-bit, and verified to be what it's supposed to be.] Outside hardware errors (disk errors, bad network elements, etc.) or coding mistakes, it is theoretically impossible for there to be bit errors within a swarm where FEC would be of any value. Now, that's not to say hash failures do not happen; because they do. The most common reasons lead to errors well beyond what a small/simple FEC can handle: disk errors (512bytes or more disappear), applications changing file contents without the client noticing (mp3 id tags anyone), and (rare) mistakes in the hash verification process.
So given the multiple verifications and reliable transport, FEC is a waste of time.
I'm not sure what you mean by "FEC chunks". In order for any FEC data to be of use in recovering missing pieces it would have to be (nearly) the size of a piece. At that point, the torrent file ceases to be a torrent and becomes a par file. (and places torrent sites at greatly increased liability because the torrent file now contains "copyrighted material", if pseudo-encrypted) The whole point of the torrent file is to describe a set of data -- how to identify a set of files, not provide information on how to create it. (the tracker helps you find others with the same data so you can download it, fill in what you're missing, repair your data,...)
Calculating parity/FEC pieces and seeding those as well is laughably stupid. Put down the crack pipe a while and think about it... the initial seed offers all the data plus 50% more in (optional) par files. Now why the hell would anyone download the par files over the actual data or even in addition to the data? If the seeder is offering all the data, why does anyone NEED the extra parity data? If he/she doesn't stick around long enough to seed a complete copy of the data, what makes you think he/she is going to stick around long enough to seed enough data+parity for anyone to recover a complete copy? And once you have all the data, why
In a US court, maybe. NFOrce.nl is not IN the United States. And rather infamously, is not bound by our lame laws.
The BM/TBsource sites must have an upper limit or they fall over and catch fire rather quickly. It's a php tracker that's very heavy on database (sql) usage. There are any number of tricks and hacks batted around to speed things up, but php sucking data out of mysql always looses. That's why the Big Boys (tm) use other things.
As for the original problem of completing torrents... if you're not completing "0-day" torrents, then you're doing something very wrong -- like hopping on a torrent in the last hours of life (which borders on "leecher" behaviour, btw.) -- or are part of a seriously crappy community. I've never seen any issues with people not completing torrents. (unless they started their downloads just before the torrent expired... which is a leecher tactic in that the swarm will usually continue for some time after the tracker expires the torrent, thus downloading without being credited for it.)
Yes it does... do you boot windows just to read one web page and then turn the PC off? Start it and leave it running. Besides, almost anything of any size will take orders of magnitude longer to download than AZ takes to start.
Right. And Opera is dog ass slow because it takes a few seconds for it start. And IE is damned fast because it pops up as soon as I click the little blue 'E'.
Judge the application by actually using it instead of complaining about how long it takes to boot. It's actually very nice app once people put their java biggotry behind them. I have nearly 300 torrents loaded in azureus (covering several thousand files amounting to ~1TB of data) starting and stopping automatically as the swarms necessitate. It rarely uses more than 10% of a PIII 800 and never exceeds the VM's max. memory of 128M.
that doesn't mean that use of FEC (forward error correction) is a bad idea.
Actually, FEC is a bad idea anywhere there is a synchronos, bidirectional communications channel. It's far easier and expediant to simply ask for a block to be resent than consume the cpu and bandwidth to calculate and transmit error correction data that will be unused 99.9999% of the time. Bittorrent data transport is TCP/IP which is stateful, loseless transmission protocol. There is simply no need to add any additional error handling on top of a proven reliable transport. The only place where FEC is worth the effort is in async and/or one way communications channels where it's either impossible or infeasable to request retransmission. This is why USENET posters have been using PAR/PAR2 for years now.
there would be almost no impact on network bandwidth and the CPU overhead is negligible.
Both of these are also incorrect. FEC costs BOTH bandwidth and cpu time. Error correction codes take space - PERIOD. And it takes CPU resources to compute them. There's no such thing as "negligible CPU overhead"... it either takes CPU time or it doesn't; every instruction counts. (Get a hundred people throwing negligible crap into an application and suddenly it runs like crap.)
If you're having significant problems with bittorrent completions, I would suggest you find a quality "private tracker" community and stay away from the leaching havens like the old suprnova. And stop downloading poorly seeded torrents. (from someone who's sat on a torrent for 3 months waiting to finish it -- it finished after someone noticed it needed a seeder.)
DHT networking is a truly peer to peer protocol meaning you are slightly safer with your illegal downloading from the aut[h]orities.
WRONG!!!! In order for you to download content, you must be able to find other peers. And likewise, other peers must be able to find you. DHT does not magically make this requirement disappear. It's actually easier to find peers within DHT because there's no restrictions on accessing the swarm. With a private tracker, one must access that tracker to find the peers within the swarm. With DHT, anyone can find the peers for a swarm. DHT is more easily monitored making it much more dangerous.
The entire problem with BitComet was it's turning to DHT when the tracker was unavailable despite the torrent being marked as private. Some may call that a bug. But those that know bitvomit will suspect it was intentional...
You are completely mistaken about the reasons for a private tracker... illegal content is just as easily found on public trackers as well. The motive for a private tracker is fostering a community where people give back instead of take, take, take, and take some more. Remember suprnova, where there were swarms with thousands of peers yet the best anyone could download was a few kbps? Yet even on small "private"[*] trackers where swarms are just a few dozen peers (at best) download speeds were hundreds of kbps.
[*] "private" as in "registration required", but anyone can signup
I find its user interface clunky and pathetically slow, because it's java
Yet, it's UI is SWT which is almost entirely native code. Java is not what makes it slow; it only makes you think it's slow. If someone didn't tell you it was java, you'd likely never notice.
No, azureus is not "GUI only". It supports a text console interface, and has for several YEARS. However, as it's not the intended means of running it, it's not as straight forward to do. It can be managed either from it's console or via one of the web plugins (swing webui, or the newer azhtml plugin.) -- or the telnet plugin, but I would stay away from that thing.
If I started receiving bills from Sprint every time my Cingular phone called a Sprint phone, then we'd be looking at a valid comparison.
This is call "reciprocal compensation"... the telco originating the calls pays the telco terminating the call. It used to be 5cents/minute, or 2, something like that. Remediation is it's own industry.
Guess what. Bellsouth has been sued numerous times for refusing the payup their part while demanding everyone else pay them for terminating calls. That's right, Hellsouth wants to be paid for terminating calls but doesn't want to have to pay others for the same service. They're just stupid, greedy little f***s.
Slashdot Mirror
That's "another compromise"... IP Spoofing hasn't a f***ing thing to do with DNS recursion. One can just as easily spoof your address in a non-recursive request.
heh... they tend to much more stable if there's traffic moving across them. Left idle for more than a few minutes and they tend to fall apart. One packet every 30sec works well for me.
And from the people who need that level of reliability, you can't get that from cisco either. :)
Hah. Sad, but all too often, true.
I've seen HA, RPR+, etc. work sometimes on some hardware. And then there's the ones where it does ever more inventively crazy things. Like... reloading every linecard/vip when it's explicitly not supposed to, "supporting" exactly 7 OIR events before crashing (maybe they fixed that by now), cbus complex's for "stuck output" in a single timeslot of one CT1 on a CT3 PA... And my personal favorite... master and slave RSP both in the active state following a crash -- each took over "their" half of the router.
[E]IGRP is a well documented protocol. The problem is in the patents and copyrights. People have written [E]IGRP stacks for other systems before, and Cisco promptly sued them into silence. (I've not heard of one of these in many years, btw.) OSPF is the open standard for this sort of thing; but, yes, eigrp is so much easier to setup.
HSRP is a deadend... See Also: VRRP (which is an open standard)
PHB's like to know where to place blame. But, they also don't want to throw their cash in the trash. Cisco's support is over priced just as badly as their hardware. For aging equipment, the support costs often exceed the cost of the hardware.
That's not to say Cisco is devoid of any neat toys. Unfortunately, all of their neat toys are found in the top line hardware that has heavily inflated prices.
That's the traffic to the outside world. The bits flowing inside the company are far higher and more important.
In my apartment, yes, 100Mbit is fine. I don't move a whole lot of stuff between machines; and when I do, there are other things for me to do instead of waiting. However, in an office, this is rarely the case. There's a whole lot of stuff moving between machines and people are dependant on these things to be productive. (all the sub-minute "waiting for word to load" delays add up at the end of the week.)
Ya want to know how it works now.
:-))
Wrong. Well, that's not how it's (legally) supposed to work. The LEA doesn't provision the tap, the TELCO does. The standard is very clear... very few people are supposed to have access to the CALEA terminal -- perferablly one; there MUST be both physical and electronic security. (i.e. in a locked room with the monitor pointed away from the windows, and sufficiently limited remote access.) If the telco is, in fact, giving access to "everyone" (esp. non-employees), then they are breaking a bunch of very big laws. (the problem here is confidentiality... everybody can see everything.)
[I know, because I've read the standard. I've been a "calea system administrator" responsible for setting up the system. And I've talked with several telco's around the country that have gone down the same path.]
The odd thing is... CALEA isn't all that complicated. It just requires someone to bridge two extremely camps. You have to understand both telco switching and internet routing. Unfortunately, the people who "speak 5E" don't know shit about the internet; and there are numerous physical bariers to learning 5E. (it's not like there are many 5E manuals floating around the internet. However, there is a phrack article.
Hours. And on occasion, days. (it's best not to let my debian mirror get that far out of sync :-))
The economy of Tier1 ISP's is far more complex than I care to discuss. Besides, they aren't at issue. It's the smaller fish serving consumers (tier2/3 ISPs) that are at issue. They aren't big enough to play in the direct peered world -- certainly not for "free". If [cable company] wants a connection to Sprint, AT&T, MCI/Worldcom/UUNET/Verizon, then they will pay out the ass for it. Even when the interconnection is mostly on their own equipment (i.e. no "last mile" BS from some RBOC.) There are some reasonable players in the market, but they aren't seen as Tier1 guys -- at a former job, we had a DS3 from WCG that was ~4.5k/month and later a medium speed (115M) OC-3 for ~7k. (uunet wanted 30k, btw.)
POP and SMTP use well defined ports. And if you're doing it "correctly", the connection starts off as plain text and switches to encrypted following a STARTTLS. And anyway, what difference would it make if your POP/SMTP traffic were slowed to 5KB/s? Yeah, that's sorta suck, but it'd still work just fine. And they wouldn't throttle their own servers. :-)
(And technically, you can tell a difference... self signed cert vs. "valid" CA signed cert. But that's a lot more work than simply spying on packets.)
they tend to take up a lot of the routers memory
Nope. You're confusing "router" with firewalls, NAT, and other forms of connection tracking and stateful inspection. Most ISP routers don't do anything like that. The closest you'll find is flow caching which is just a fast routing process... it's optional and there are limits on how much memory it can consume. When an idle flow expires, that doesn't break the connection; the router will simply follow normal routing proceedures and build a new flow record for subsequent packets. Routers don't care what the traffic is; they only care about where they need to send it. (and in some edge cases, where it came from.)
That's not to say routers don't have limits. Every router has a maximum throughput both in packets per second, and bits per second. PPS is generally limited by routing speed -- how fast can the router determine where to send this packet. BPS is limited by the physical hardware -- how fast can bits be copied/signaled between interfaces.
One of the key reasons ISP's are resorting to throttling is shear bandwidth. ALL ISP's oversell their connectivity. (It's the only way to make money on the low end where consumer's live.) They don't have the capacity to give everybody their 5M down/384K up all the time. And it gets worse... for a cablemodem, a single channel is 30Mbps down and 10Mbps up (+/-). When one is saturated, the other is starved -- the ACKs have to get in there somewhere. So, that 5/384 setup can support 6 people down and 26 people up, at once, per channel. There are literally hundreds of modems on each channel. Honestly, I'm surprised anyone can max their connection for any measurable duration.
(DSL is just as bad, if not worse. Having worked for a DSL provider, I know it's worse.)
You're missing the point -- and ignoring the word port. Redhat is not building it. Redhat is not testing it. Redhat is not distributing it. And, Redhat is not supporting it. Someone. Else. Is.
All they share in common is the name ("fedora") and the underlying source. There's more to the creation of a port than simply running rpmbuild -ba * in the SRPMS directory. (if it were that easy, Redhat would still be doing it.)
And this is not entirely true. I've gotta call bullshit. They used to support PPC, but dropped almost everything but i386 long ago: v6.2 was the last release for sparc, v7.1 was the last release for alpha and ppc (v7.2 for alpha was released by DEC.) Feel free to correct any I've missed -- and yes, I'm ignoring ia64 and s390. Only VERY recently has PPC support resurfaced in Fedora.
Only their marketing people know for sure why. It sure as hell shouldn't have anything to do with intel based macs but most likely does... "If we want to support the intel macs, we also must support the ppc macs, too. (or at least appear to.)" While that makes perfect sense to marketing people, it's laughablly stupid on technical merit. After all, they claim to have dropped PPC in the first place because of "lack of demand." So, suddenly, intel based macs are going to bolster demand for a ppc redhat for the machines on which people already (apparently) don't want to run linux? *shakes head and walks away*
It's theoretically easy to get linux booted on an intel mac since it's the same fundamental platform as every other i386 system... install an EFI aware boot loader and kernel and the deed is mostly done. ('tho Apple could've done things to make it much more difficult.) Getting redhat running on a ppc mac requires building and testing every single piece of software for an almost completely alien hardware platform.
Off the top of my head I was suggesting adding 10% redundancy to the torrent *content* which would also increase the torrent file by roughly 10%
... Life would be much simpler if "2 + 2" didn't sometimes come up "5". Having more than the necessary number of blocks to choose from makes it appear as if the 100th piece is effectively 11 pieces... ask for all 11 and use which ever one gets there first. However, in practice, no client would download 99 pieces and then "race" the last piece.
... and this is the very first time you state that AT ALL. (so, the answer is sorta "both and neither"... the FEC is distributed in the swarm with hash's in the torrent file along side the standard pieces.) If you're going to draft a new protocol, you really need to be exactingly clear with what you intend.
By adding a redundancy set, you would reduce the "rareness" of every piece in the data set and thus improve the chances of completing the torrent. It's just that simple.
Only on paper... instead of having to download 100% of the pieces, you only need download 91% [100/110] of the pieces. You are making a vein attempt to reduce the rarity of data pieces by piling on more pieces... But for this to be effective, every piece ("data" or "parity") must be of equal utility. Using "parity" pieces to fill in missing "data" pieces is an extremely expensive process -- far more so than just downloading 100% of the data. (it'll always better to download data pieces over "parity" pieces due to the complexity of the recovery process. (trust me, people don't download par files if they don't have to.)) Bram complains about the same "academic hand-wavy approach" to the complexity with avalanche's ECC. [I guess he's seen that Snafu comic, too.]
Also, adding redundancy to the swarm will not improve speed or longevity of a swarm. Downloading pieces is faster than recovering them in all but the rarest cases. Downloading 100 pieces will take however long it takes. In reality, it doesn't make much difference if it's 100 out of 100, or 100 out of 110. On paper, it follows a block with more distributed copies will download faster, but reality is far more complex... maybe you lucked up and asked a peer on a 28.8 modem for that piece, or all the peers with that piece are busy (i.e. "choking" you),
Ultimately without a seed -- at least one peer with the complete dataset -- or at least one complete set of data distributed throughout the swarm, the swarm will stall; the torrent is dead. Even with 500% redundancy, if no one has all of the data, then no one has all of the data. If there isn't a complete copy within the swarm, there won't be until someone brings in the missing pieces -- either as "data" or "parity". This is what limits the lifespan of a torrent. Redundancy has no effect on it; so FEC won't do anything to help you complete un-seeded torrents.
We can already do exactly what you suggest by including par/par2 files in with the data files. And I've repeatedly pointed out the catch-22 with this scheme... The only time "parity" pieces would be of any use is when there isn't a complete set of data pieces in the swarm. If there isn't a complete copy across the swarm, then there clearly isn't enough data and "parity" to recover a complete copy, and the "parity" pieces people have downloaded are meaningless until additional pieces arrive. [This assumes the "parity" is across all pieces ala par and par2. "parity" across sub-sets could, in theory, prove to be of some use, if significantly limited. (eg. recovery of a single file in a multi-file torrent)]
There's either a full copy in the swarm or there isn't. It doesn't matter how much redundant data you present to the swarm.
By all means, spend your every free waking moment designing and coding your FEC extension. The only way you're going to realize the futility of what you're proposing is to actually watch it not do what you expect. Unless you have a means of recovering more d
just a guess, but it's reverifying the data it downloaded. The status will show "seeding + checking". And yes, you can turn that off. It also sounds like you have friendly hash checking disabled... it won't pause at all between pieces.
You seem to have missed the paragraph above that where Bram states: The central idea here is basically 'Let's apply error correcting codes to BitTorrent'. This isn't a new idea, everybody comes up with it. In fact I saw fit to mention that it's a dubious idea before. ... While it is very cool, and very applicable to sending information across lossy channels, the case for using it in BitTorrent is unconvincing. (emphasis is mine)
And he's talking about Microsoft's Avalanche. He's pretty clear that error correction within bittorrent is quite unnecessary.
I'm fairly sure I have a good grasp of the core Bittorrent protocol
... my suggestion to not distribute the FEC pieces when the there are more than a certain number of distributed copies visible or when you know a seed will always be available. ... it should also be pretty beneficial for initial seeding.
... you can stop distributing the redundancy data ...
It's not so much about the protocol (for which there is no full specification as far as I've ever seen), but how bittorrent clients and trackers actually work... the actual dynamics and life cycle of a swarm. (so to speak.)
I am not suggesting a simple parity check or basic hamming code on top of the TCP transfer.
Indeed. You're suggesting turning a torrent file into what will mostly be a par/par2 file by placing a percentage of the torrent data as parity recovery blocks in there. Doing so will significantly increase the size of the torrent -- dwarfing the actual torrent by orders of magnitude. (I'll come back to that.)
Based on your response it appears that your understanding of Reed Solomon FEC is from PAR files.
Actually my understanding of FEC/ECC goes well beyond PAR. And technically, what you are describing is not "FEC"... it's parity data. You aren't "correcting errors"; you are "(re)constructing data you don't have." FEC is about being able to fix errors in the data as it's being received -- not so much about pulling data out of nothing. FEC comes before the data in a stream; ECC comes after... I'm just going to calling it "parity data" because that's what you're talking about.
Comments like this are why I don't think you understand how BT works... the site, or more specific, it's tracker(s), know almost nothing about what's going on in the swarm. The tracker isn't a BT client. So it's not part of any of the swarms it's tracking. Therefore it knows nothing at all about the distribution of data within the swarm. So, what's going to decide who gets a huge parity torrent and who get a tiny standard torrent?
With any given torrent, you'll either have to download the entire set of data or the equiv in data + parity (which presumablly you have from the torrent.) If you have enough to recover the complete set, then you recover the data and *poof* there's a seed in the swarm making the parity data an unnecessary waste. 'round and 'round we go...
It will, in fact, make no difference for the initial seeding as it will always be faster and computationally cheaper to download the pieces than to recover them. As a result, clients will prefer downloading to recovery whenever possible. (This is esp. true since the parity data had to come from somewhere... either download 250M of data and be done, or download 200M data + 50M parity and spend a half hour (+/-) recovering the 50M of missing data.)
I am *absolutely* suggesting an extension to the BitTorrent file format that would include entries for redundancy pieces, similar to the way PAR files are used
Make up your mind... is the parity part of the swarm (to be downloaded like all the other pieces) or part of the torrent file? Explicitly state one, yet imply the other.
If it's part of the torrent, you're just wasting the site's bandwidth and possiblly opening the site to copyright violations -- they're no longer simply describing copyrighted content. Again, exactly how is the site supposed to know when to include parity data? Basically, if I have parity data or not will depend on when I download the torrent. If it's part of the swarm, why the hell would anyone spend time fetching the parity pieces when the actual data pieces are available? There are alot of issue with your poorly explained extension.
Now, let's talk about how BT works off the paper and in the real world... As an example, let's say we have a community of 5 peers (A-E). Peer A has content it wishes to share. Pe
you're using FEC to compensate for the losses due to unequal chunk distribution
...)
Do you even know how Bittorrent works? (don't answer that, it's obvious you don't)
Again, you just don't get it. FEC. Is. Simply. Unnecessary. Lemme esplain...
"FEC chunks"? "initial seed"?... First, a word on terminology... (various BT spec docs beat this into your head) in BT a piece is the amount of data for which a hash exists. (the min. verfiable data) Each piece is divided into blocks that are requested from other peers -- with a tendancy towards all blocks of one piece coming from the same peer so you know who to ban if a hash check fails. An initial seed is the peer who first offers all pieces to the swarm -- the only one who starts out with 100% of the torrent. This is not necessarily the person who created the torrent, or published it to a site.
You seem to think a few bytes of FEC can magically regenerate a 256k piece out of nothing. At best, FEC would be useful in detecting and recovering from bit errors. (That is what it's designed to do, btw.) However, such errors do not occur within bittorrent swarms -- unless something horrible, and well beyond any error correction, occurs. Peers do not report a piece as being available until it has all the data and passes hash verification. At that point, barring hardware failures and coding mistakes, the peer has exactly what it's supposed to have. When it hands out the data for that piece, TCP/IP ensures the bits sent are the ones received; and the receiver is going to do the same hash verification.
So, let's summarize... peer A is going to ensure the validity of the pieces it has by hash verification before making them available to other peers. [data starts out being exactly what it's supposed to be.] peer B asks for blocks amounting to a piece which are transfered by TCP/IP, assembled and then verified. [data transported bit-for-bit, and verified to be what it's supposed to be.] Outside hardware errors (disk errors, bad network elements, etc.) or coding mistakes, it is theoretically impossible for there to be bit errors within a swarm where FEC would be of any value. Now, that's not to say hash failures do not happen; because they do. The most common reasons lead to errors well beyond what a small/simple FEC can handle: disk errors (512bytes or more disappear), applications changing file contents without the client noticing (mp3 id tags anyone), and (rare) mistakes in the hash verification process.
So given the multiple verifications and reliable transport, FEC is a waste of time.
I'm not sure what you mean by "FEC chunks". In order for any FEC data to be of use in recovering missing pieces it would have to be (nearly) the size of a piece. At that point, the torrent file ceases to be a torrent and becomes a par file. (and places torrent sites at greatly increased liability because the torrent file now contains "copyrighted material", if pseudo-encrypted) The whole point of the torrent file is to describe a set of data -- how to identify a set of files, not provide information on how to create it. (the tracker helps you find others with the same data so you can download it, fill in what you're missing, repair your data,
Calculating parity/FEC pieces and seeding those as well is laughably stupid. Put down the crack pipe a while and think about it... the initial seed offers all the data plus 50% more in (optional) par files. Now why the hell would anyone download the par files over the actual data or even in addition to the data? If the seeder is offering all the data, why does anyone NEED the extra parity data? If he/she doesn't stick around long enough to seed a complete copy of the data, what makes you think he/she is going to stick around long enough to seed enough data+parity for anyone to recover a complete copy? And once you have all the data, why
... may be liable for contributory infringement
In a US court, maybe. NFOrce. nl is not IN the United States. And rather infamously, is not bound by our lame laws.
The BM/TBsource sites must have an upper limit or they fall over and catch fire rather quickly. It's a php tracker that's very heavy on database (sql) usage. There are any number of tricks and hacks batted around to speed things up, but php sucking data out of mysql always looses. That's why the Big Boys (tm) use other things.
As for the original problem of completing torrents... if you're not completing "0-day" torrents, then you're doing something very wrong -- like hopping on a torrent in the last hours of life (which borders on "leecher" behaviour, btw.) -- or are part of a seriously crappy community. I've never seen any issues with people not completing torrents. (unless they started their downloads just before the torrent expired... which is a leecher tactic in that the swarm will usually continue for some time after the tracker expires the torrent, thus downloading without being credited for it.)
Yes it does... do you boot windows just to read one web page and then turn the PC off? Start it and leave it running. Besides, almost anything of any size will take orders of magnitude longer to download than AZ takes to start.
Right. And Opera is dog ass slow because it takes a few seconds for it start. And IE is damned fast because it pops up as soon as I click the little blue 'E'.
Judge the application by actually using it instead of complaining about how long it takes to boot. It's actually very nice app once people put their java biggotry behind them. I have nearly 300 torrents loaded in azureus (covering several thousand files amounting to ~1TB of data) starting and stopping automatically as the swarms necessitate. It rarely uses more than 10% of a PIII 800 and never exceeds the VM's max. memory of 128M.
that doesn't mean that use of FEC (forward error correction) is a bad idea.
Actually, FEC is a bad idea anywhere there is a synchronos, bidirectional communications channel. It's far easier and expediant to simply ask for a block to be resent than consume the cpu and bandwidth to calculate and transmit error correction data that will be unused 99.9999% of the time. Bittorrent data transport is TCP/IP which is stateful, loseless transmission protocol. There is simply no need to add any additional error handling on top of a proven reliable transport. The only place where FEC is worth the effort is in async and/or one way communications channels where it's either impossible or infeasable to request retransmission. This is why USENET posters have been using PAR/PAR2 for years now.
there would be almost no impact on network bandwidth and the CPU overhead is negligible.
Both of these are also incorrect. FEC costs BOTH bandwidth and cpu time. Error correction codes take space - PERIOD. And it takes CPU resources to compute them. There's no such thing as "negligible CPU overhead"... it either takes CPU time or it doesn't; every instruction counts. (Get a hundred people throwing negligible crap into an application and suddenly it runs like crap.)
If you're having significant problems with bittorrent completions, I would suggest you find a quality "private tracker" community and stay away from the leaching havens like the old suprnova. And stop downloading poorly seeded torrents. (from someone who's sat on a torrent for 3 months waiting to finish it -- it finished after someone noticed it needed a seeder.)
DHT networking is a truly peer to peer protocol meaning you are slightly safer with your illegal downloading from the aut[h]orities.
WRONG!!!! In order for you to download content, you must be able to find other peers. And likewise, other peers must be able to find you. DHT does not magically make this requirement disappear. It's actually easier to find peers within DHT because there's no restrictions on accessing the swarm. With a private tracker, one must access that tracker to find the peers within the swarm. With DHT, anyone can find the peers for a swarm. DHT is more easily monitored making it much more dangerous.
The entire problem with BitComet was it's turning to DHT when the tracker was unavailable despite the torrent being marked as private. Some may call that a bug. But those that know bitvomit will suspect it was intentional...
You are completely mistaken about the reasons for a private tracker... illegal content is just as easily found on public trackers as well. The motive for a private tracker is fostering a community where people give back instead of take, take, take, and take some more. Remember suprnova, where there were swarms with thousands of peers yet the best anyone could download was a few kbps? Yet even on small "private"[*] trackers where swarms are just a few dozen peers (at best) download speeds were hundreds of kbps.
[*] "private" as in "registration required", but anyone can signup
I find its user interface clunky and pathetically slow, because it's java
Yet, it's UI is SWT which is almost entirely native code. Java is not what makes it slow; it only makes you think it's slow. If someone didn't tell you it was java, you'd likely never notice.
No, azureus is not "GUI only". It supports a text console interface, and has for several YEARS. However, as it's not the intended means of running it, it's not as straight forward to do. It can be managed either from it's console or via one of the web plugins (swing webui, or the newer azhtml plugin.) -- or the telnet plugin, but I would stay away from that thing.
If I started receiving bills from Sprint every time my Cingular phone called a Sprint phone, then we'd be looking at a valid comparison.
This is call "reciprocal compensation"... the telco originating the calls pays the telco terminating the call. It used to be 5cents/minute, or 2, something like that. Remediation is it's own industry.
Guess what. Bellsouth has been sued numerous times for refusing the payup their part while demanding everyone else pay them for terminating calls. That's right, Hellsouth wants to be paid for terminating calls but doesn't want to have to pay others for the same service. They're just stupid, greedy little f***s.