Slashdot Mirror
Usenet Encoding: yEnc
Motor writes "Anyone remotely interested in usenet binary newsgroups must have noticed the spread of yEnc. yEnc is an encoding scheme for usenet binaries which avoids the enormous (30-40%) bloat associated with the schemes currently in use - which all have to produce 7-bit data to stop ancient newsservers from choking. A good thing, surely? Well, not according to some people. The guy has some good points about yEnc and standards, but I can't help thinking that "standards" people have endlessly discussed better encoding schemes, and nothing has come out of it. yEnc may not be perfect, but it works and it's here - hence the rapid adoption. What do you think?"
In fact, Yenc will help pay-per-gigabyte Usenet users achieve a greater bang for their buck. Anything that saves money is a good thing!!
www.lonseidman.com
There are some proposals in thee XML and Web Services arena for dealing with some of the problems tha yEnc is skirting.
One, called DIME, is a MIME-like system that handles binary content, chunks, etc.
http://www.oasis-open.org/cover/dime.html
Forte released Agent 1.91 2 days ago with yEnc support. it looks like Mr. Nixon is fighting a losing battle.
Breaking MIME is not something I would (do) lose sleep over. People in the MIME community screamed at us when we had the temerity to introduce the text/html content type, rather than use application/binary. They were completely obstructionist when it came to insisting on 8-bit clean transport for HTTP. In the end we treated them as damage and routed around them. HTTP uses several headers that the MIME people villified.
The functional issues raised are significant and it would be good to see them addressed. In particular using the subject line is pretty lame. Either you want the encoding format to be completely independent of MIME or you don't. I think that MIME independence would be the better route since then it would be easier to move to a more modern protocol such as BEEP. But using magic numbers and MD5 inside the encoding does not seem like a bad move.
The more interesting 'meta-point' however is that tweaking the encoding format is only scratching the surface when it comes to fixing UseNet. The main problem with USEnet is that it still has to route every single article to every single node whether it is going to be read or not. While the flood fill routing was a good scheme when NNTP was developed and the number of nodes was small it is needlessly wasteful now that we have hundreds of thousands of NNTP servers, it is just not necessary to have that level of redundancy to route arround censorship.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
Check out Newspro, it's better than most binary grabbers.. I wouldn't use it for reading text newgroups, but XNews is good for that.. Newspro lets you combine headers from multiple servers. I download from my paid usenet server plus the @home servers and regularly saturate my connection at 500k/sec.. A buddy across town gets 700k/sec.
Newspro - http://www.usenetopia.com
XNews - http://xnews.3dnews.net/
Newspro is shareware, XNews is freeware.. They're both good, for different things.
Cheers,
Backov
In the law there is no overlap between theft and copyright infringement whatsoever.
And Pan's got it too. Tastes great, less filling!
I wonder if it's in CPAN yet...
Module Convert::yEnc (P/PN/PNE/Convert-yEnc-0.03.tar.gz)
Yep. Works for me!
Well then. When I put that page up, I honestly didn't expect many people to read it outside news.software.nntp and a few curious folks in alt.binaries.news-server-comparison. I certainly wan't expecting to get Slashdotted. Well, that's fine, except that the uproar might have waited a little bit.
In my essay, I state that what Usenet needs is "a better way to post Binaries". The next piece of the puzzle, of course, is to answer the question, "What IS a better way to post binaries?" I was thinking about finishing that page up tonight, but I am writing code at the moment instead.
So, when reading my comments, just keep in mind that, yes, I DO have some answers to that question, too. It's just that it's a bit of a more time-consuming question, so that page isn't done yet.
This time around, though, I will make sure to include a prominent warning to NOT run off and implement the ideas as quickly as possible, and to please not use all of Usenet as beta-testers. The idea that whatever gets done fastest is best just doesn't work for me. There were good reasons I didn't go and get people to implement my smaller encoding ideas when I first wrote the code. If only the yEnc implementor had continued where I left off rather than going down his rather misguided path...
All the comments are welcome. I've been getting some interesting email, too, of course. Many programmers of Usenet client software absolutely despise the thing and are quite annoyed at the amount of their time it is wasting. I guess it's just more of that never-ending divide between the users and the techies. So it goes.
yEnc is here, that's for sure. Now we just have to try to deal with it.
Jeremy
It is posted to alt.binaries.warez.ibm-pc
Subject: As Requested a UUencode version of Agent 1.91 with yEnc [1/2] - a32-191.exe
Message-ID: <q7Pm8.81615$rr6.1172226@news.webusenet.com>
See my other post in this thread. It's not all about dl speed or even drive space. Most serious usenet large binary downloaders use a premium news service, and they ALL have download/upload quotas and/or pay by the byte schemes. yEnc gets you 30% more for your buck on those servers.
If you think this is going to decrease the amount of data transferred by customers, you need to take off those rose-colored glasses. I suspect it won't affect that at all.
Smaller encoding helps us (the providers), it doesn't hurt us. (Besides, one look at the Supernews price list versus the competition will reveal that metered individual accounts are hardly the main focus.) With the customers the actual money comes from, less downloads would increase the margins, not decrease them. In fact, to a lesser extent, it would do so with the individual accounts as well -- the pricing on individual Usenet accounts (at almost all of the top providers) is such that the margins are lower at higher download levels. At the highest levels, it's nearly break-even. So less downloading would even be somewhat better there.
If you really think that I'm against a smaller binary encoding scheme, then you completely missed the point of my essay -- and you also must have missed the part about how it was my first experimental code, implementing smaller encoding, from which yEnc was hatched. If I truly wanted to avoid smaller encoding, why would I have implemented it in the first place? Why would I have done it in public and then sent the code to several people, explicitely releasing it into the public domain? You think I would have done that to prevent the spread of smaller encoding?
Had Jürgen picked up where I left off and did the thing right, I would be singing an entirely different tune.
Jeremy
... the implementation sucked.
I leech regularily from alt.binaries.anime and the related newsgroups. When the yEnc posts started coming in, I simply upgraded my newsreader to the newest version. But a LOT of people out there use Agent, and it was absolute pain to combine/decode all the yEnc posts that started popping up all over the place. The worst of it is that the yEnc posters were basically saying, "Start living in the present and upgrade". Nevermind that at the time that only yEnc-capable newsreaders were for Windows...
I mean, I don't know, but this sounds a lot like the OS wars that have been going on for quite some time. Some people simply don't want to have to switch newsreaders. Some people don't want to have to switch OSes. And that's fine, because it's a free world out there. I like the idea of yEnc (I get more out of my Easynews account), but I really don't think it should have been introduced so quickly.
~ Firecaster ~You just gotta love the speculation that runs rampant around here.
FYI, I am using a metered dialup account (actually ISDN, which isn't much faster, just more reliable). I cannot get unlimited, high-speed access (yet, I keep hoping). I pay well over $100 per month for Internet access at a fraction of the speed of peoples' cable modems and DSL lines, and it would be quite a lot more if I left it nailed up 24x7.
You, like several others here, seem to have somehow gotten the impression that I am opposed to smaller encoding. I can only conclude that someone is spoofing my web host's DNS and some of you are reading a different page than the one I wrote.
Jeremy
"Whatever kind of "confusion" this guy is referring to is nonexistent. yEnc downloading and decoding with the right software is as transparent as uudecoding. Promise. People don't have to relearn anything."
Well people do now have to think of yENC's longer lines. Where before my ISP newserver let me post 5000 lines of uue material now I only get 2280 with yENC. Longer lines under yENC. That's why Agent 1.91 has such a small line length as the default. Yes the payload is the same but you have to think a little different or just rediscover your servers limits.
As a "pretty active Usenet poster and binary downloader," I bet you know your stuff pretty well. You could switch to anything in a short time, I bet. That's great, so could I. You're not looking at the average user, though.
Since I put my page up, my mailbox is overflowing with messages from people who are agreeing with me, but for the wrong reason -- they are saying, yeah, right on, this yEnc thing sucks, I can't figure the thing out and I can't get half the binaries anymore, how do we get everyone to go back to the old way?
If you just look at the people who know what they're doing and know it well, of course you won't see much confusion. But a popular binaries group easily has several thousand times as many people downloading as it has posting, so I think you are grossly underestimating the level of confusion this is causing among the regular users.
I'm not worried about confusing them; it's inevitable. What I'm worried about is confusing them too many times in a row.
Jeremy
I don't know who was first and I'm sure this is not the only other tool/compressor but here in Belgium, we've been using a tool called Bommanews (b-news.sourceforge.net) for a while now because of rediculous upload-limits by the local cable-ISP. We're not allowed to run servers (ports < 1024 are blocked), have un upload-speed-limit of 128kbit/s and only 15% of our total traffic can be upload so some guy came up with this to allow easy distribution of files through the news-server. Because it's so popular, our ISP finally had to setup a 2nd news-server just for binary postings as the other one was overloaded (serves them well).