Slashdot Mirror
Usenet Encoding: yEnc
Motor writes "Anyone remotely interested in usenet binary newsgroups must have noticed the spread of yEnc. yEnc is an encoding scheme for usenet binaries which avoids the enormous (30-40%) bloat associated with the schemes currently in use - which all have to produce 7-bit data to stop ancient newsservers from choking. A good thing, surely? Well, not according to some people. The guy has some good points about yEnc and standards, but I can't help thinking that "standards" people have endlessly discussed better encoding schemes, and nothing has come out of it. yEnc may not be perfect, but it works and it's here - hence the rapid adoption. What do you think?"
In fact, Yenc will help pay-per-gigabyte Usenet users achieve a greater bang for their buck. Anything that saves money is a good thing!!
www.lonseidman.com
Forte released Agent 1.91 2 days ago with yEnc support. it looks like Mr. Nixon is fighting a losing battle.
I'm all for standardisation... but sometimes it takes _forever_ to get something standardized. If someone writes a better product, they generally don't want to wait for it to be declared a standard, especially with something like uuencoding which has been around as long as usenet, and isn't going to be replaced in a hurry unless someone comes out and waves a product around yelling "hey try this. it works better". Ogg Vorbis isn't a standard by any means. Hell, it is still on RC3. _but_ a lot of people are using it because it has far better sound compression than mp3. You don't hear people complaining that Vorbis has jumped the standardisation process do you?
Personally I can't see why we can't just send the data as 8-bit binary. uuencode and similar encoding formats should have died out with UUCP years ago, since there is no physical reason why 8bits can't be sent over the wire anymore.
I am seeing smaller binaries as a result of yEnc. This is fine. The problem is, my favorite binaries grabber has no idea what to do with the files, and won't even download them. I figured out how to make Agent download them, but A. I hate Agent (and don't understand why anyone likes it!) and B. the binaries don't always decode.
As I'm a lifetime lurker (well eight years, but it seems a lifetime!) I can only choose not to download yEnc encoded binaries. And no one will know! (my news server doesn't log downloads) It's all up to the posters to adopt or not.
Breaking MIME is not something I would (do) lose sleep over. People in the MIME community screamed at us when we had the temerity to introduce the text/html content type, rather than use application/binary. They were completely obstructionist when it came to insisting on 8-bit clean transport for HTTP. In the end we treated them as damage and routed around them. HTTP uses several headers that the MIME people villified.
The functional issues raised are significant and it would be good to see them addressed. In particular using the subject line is pretty lame. Either you want the encoding format to be completely independent of MIME or you don't. I think that MIME independence would be the better route since then it would be easier to move to a more modern protocol such as BEEP. But using magic numbers and MD5 inside the encoding does not seem like a bad move.
The more interesting 'meta-point' however is that tweaking the encoding format is only scratching the surface when it comes to fixing UseNet. The main problem with USEnet is that it still has to route every single article to every single node whether it is going to be read or not. While the flood fill routing was a good scheme when NNTP was developed and the number of nodes was small it is needlessly wasteful now that we have hundreds of thousands of NNTP servers, it is just not necessary to have that level of redundancy to route arround censorship.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
Despite its problems, XMODEM took off because it filled a need, just as yEnc does. Nixon's complaint that shrinking files by 35% won't make Usenet any smaller because people will just post more files is besides the point; it's like saying getting a 35% salary increase won't help your finances because you'll just buy more stuff with the extra money. Most people want that extra 35%, and Jürgen stepped up to the plate and delivered it.
Thankfully, as far as I know, nobody railed against Ward Christiansen the way Nixon does against Helbing. XMODEM's problems became obvious and the solution was to introduce YMODEM and then ZMODEM. XMODEM is still around, but its successors (and of course serial IP) have pretty much supplanted it. Ward's initial efforts are still deeply appreciated.
Yes there's the problem of legacy software, but a protocol that's only been around for a few weeks or months can't have that much of a legacy. The only programs that currently support yEnc are the ones whose maintainers react pretty fast to new developments, and those maintainers are likely to also quickly pick up any revisions/fixes to yEnc.
So the solution Nixon should be calling for is not a years-long bureaucratic standardization process that will get yEnc 1.3 entrenched while the standardization is happening. The solution is to fix yEnc's problems and release a new version as fast as possible, before the old version gets spread around too widely.
main(c,r){for(r=32;r;) printf(++c>31?c=!r--,"\n":c<r?" ":~c&r?" `":" #");}
In one sentence, standards ARE important because they allow for the most people to get the most benefit.
I work in an industry that relies heavily on standards, and my job deals specifically with standards. Making sure that WE follow standards, and making sure that other vendors follow standards.
Sure, they're slow to develop. But they're the best for interoperability, and that's crucial. In my line of work (for a major Mobile Phone System NSS provider), I have to deal with other providers that have to follow the same standars we do. That allows both of our products to communicate. This gives the end consumer (i.e., Cingular, Sprint, etc.,) the option to buy from different vendors. This forces us to make better products. This forces us to be more efficient. This forces our competitors to do the same thing. In the end, everybody wins.
The other alternative is what I see as the Micro$oft approach: Standards be dammed, I'm going to do it this way, and f*ck everybody else. It's the same approach that gives you security holes in your browser, because, well, who needs the standards?
I can't believe I'm reading comments like "well, it's here and it works so what's the problem?"
The problem is the future.
The problem is the inability to send an SMS from a CDMA service like Sprint to a GSM one like Voicestream. That's what happens when you blow off standards.
The problem is the inability to read an M$ Word doc that was sent to a Linux user.
Ignoring standards and going off on your own (especially, going off BADLY on your own) just divides us.
Good standards help us all. They give us better products. The lower costs.
CD-Rs. FireWire. PCI. countless others.
Besides, as the article begins by asking: Just what problem were they trying to solve?
Watch the Teaser Trailer for "The Lightning Thief" Her
It should be pointed out that this site, linked from yENC's own website, goes into more technical detail regarding the technical flaws of yENC. The fact that it's linked from yENC's own site is proof that the author is at least familiar with the concerns that people have with his implementation.
I personally still find it difficult to argue against the article author's point that THERE WAS NO RUSH to force yENC out the door in such an unpolished form. After so many years of waiting for something better, why ignore the recommendations of those you are trying to help?
< tofuhead >
It is still the dark of night.
Well then. When I put that page up, I honestly didn't expect many people to read it outside news.software.nntp and a few curious folks in alt.binaries.news-server-comparison. I certainly wan't expecting to get Slashdotted. Well, that's fine, except that the uproar might have waited a little bit.
In my essay, I state that what Usenet needs is "a better way to post Binaries". The next piece of the puzzle, of course, is to answer the question, "What IS a better way to post binaries?" I was thinking about finishing that page up tonight, but I am writing code at the moment instead.
So, when reading my comments, just keep in mind that, yes, I DO have some answers to that question, too. It's just that it's a bit of a more time-consuming question, so that page isn't done yet.
This time around, though, I will make sure to include a prominent warning to NOT run off and implement the ideas as quickly as possible, and to please not use all of Usenet as beta-testers. The idea that whatever gets done fastest is best just doesn't work for me. There were good reasons I didn't go and get people to implement my smaller encoding ideas when I first wrote the code. If only the yEnc implementor had continued where I left off rather than going down his rather misguided path...
All the comments are welcome. I've been getting some interesting email, too, of course. Many programmers of Usenet client software absolutely despise the thing and are quite annoyed at the amount of their time it is wasting. I guess it's just more of that never-ending divide between the users and the techies. So it goes.
yEnc is here, that's for sure. Now we just have to try to deal with it.
Jeremy
If, by any chance, you're transferring things over a modem (v42bis' lzw) or ssh vpn (zlib's deflate) or possibly other types of links, then you're probably not going to notice a difference anyway. The systematic encoding inefficiency that goes with base64 and uuencoding, results in a substantial lack of entropy that will be picked up on and exploited by good compression algorithms. Then end result won't be quite as good as having efficient encoding to begin with, of course, but it will be in the same ballpark. There's no way it'll be anywhere near a 33% difference.
This sounds like something that would have been useful 15 years ago before compression was widely used, and when people were still writing newsreaders. Now it looks like a waste of time and an excuse to get people to "upgrade" their software.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
One: yENC, when it was unveiled, did not really allow most conventional newsreaders any opportunity to adapt, til after the fact. This is akin to perhaps releasing zip files long before any archival software was actually available to open them... So do most of the folks using usenet for binaries get the opportunity to at least *choose* the way they do their downloads? Nope, they also are forced to adapt, or lose out...
Two: Loss in transmission... I've been downloading yENC attachments for the last month, and out of them, found over 50% loss/corruption in posting... Not due to retention/propagation either... Just files missing large chunks... Now this *could* be due to some problems on the senders' end, but it seems just a little *too* coincidental that almost all of the losses have occured with yENC uploads...
Just because you can mod me down, doesn't mean you're right. Shoes for industry!
In particular, there is no yenc RFC and yenc does not use MIME which is the agreed upon standard for encoding binary attachments. Yes, uuencode is a gross grandfathered format, but it is still 7 bit clean.
Releasing problematic improperly specified encodings that break internet protocols is not being a good citizen. "it works" is a poor justification. it does not work, and breaks compliant software.
-Kevin
But yEnc's bandwidth savings are real, which is a huge win for alt.binaries users. yEnc has been the most-requested feature for Pan over the last month. (0.11.2.90 supports it.) IMO yEnc is the format to use for multiparts right now.
Hopefully yEnc will motivate others to come up with a mime-friendly alternative encoding for Usenet. yEnc Considered Harmful is another yEnc opposition page that suggests mzip compression, but I haven't seen any public discussion of it yet.
If/when such a replacment comes along, Pan will support it too and add an are-you-sure dialog for yEnc postings.
> it is needlessly wasteful now that we have hundreds of thousands of NNTP
> servers, it is just not necessary to have that level of redundancy to route
> arround censorship.
I disagree entirely. Never underestimate the government's ability to stretch censorship to new levels.
Unless the very way NNTP servers operate is to gulp down and pass on each article for each newsgroup, the government would easily target those servers that spcifically carried groups or posts it doesn't like.
Pressure for news providers to drop certain groups began several years ago when the Vacco busts of people trading in child pornography led a news service to be criminally charged for the content of some groups and led other news servers in that state and elsewhere to drop gcertain groups thanks to their content. The charged news service took a plea even though they clearly would have won at trial or on appeal by claiming common carrier status, but hey, nobody wants to be the expensive test case.
Some may not see the problem with news servers being coerced by the government to drop those particular groups thanks to their contents, but the principle it sets is horrid. Certain "content owners" have of late been threatening to use the DMCA as a club to get news servers to drop groups which share TV shows and other such copyrighted material. If groups were more "localized" to a set of specific servers, or articles were localized to their originating servers, that would make it exceptionally easy for the DMCA to be used to require the "closure" of groups or removal of articles from USENET.
Furthermore, in this time of anti-terrrorist hysteria, the government has gotten away with the USA/PATRIOT mess already and is continually making some questionable choices. If it finds a newsgroup dedicated to dissent, or more spcifically dedicated to anti-globalism, for example, it cannot easily dstroy such a group because of the nature of USENET--the damage would be routed around by servers in other countries, even if every U.S. server could be forced to remove a group or article (not that they could be).
However, if the architecture of USENET were redesigned to localize groups or articles to subsets of servers--the likelihood of a government censoring USENET speech is magnified considerably.
It is the redundant architecture of USENET which will keep it free of censorship long after the WWW has been tamed--as it will be. Just look at the broiling mess within ICANN over officials trying to hand control of the WWW over to government-appointed reps. Eventually something like that will happen, and governments will cooperate with each other to make censorship in their mutual interests easier. Thanks to the architcture and nature of USENET, it will remain free and uncensored long after the WWW has fallen to censorship.
Just my 2 pence, though...
Chasing Amy
(We all chase Amy...)
"The more corrupt the state, the more numerous the laws"-Tacitus