Usenet Encoding: yEnc

Motor writes "Anyone remotely interested in usenet binary newsgroups must have noticed the spread of yEnc. yEnc is an encoding scheme for usenet binaries which avoids the enormous (30-40%) bloat associated with the schemes currently in use - which all have to produce 7-bit data to stop ancient newsservers from choking. A good thing, surely? Well, not according to some people. The guy has some good points about yEnc and standards, but I can't help thinking that "standards" people have endlessly discussed better encoding schemes, and nothing has come out of it. yEnc may not be perfect, but it works and it's here - hence the rapid adoption. What do you think?"

Comments

Intertia vs. Good Ideas

[ksw2]

The article points out some interesting points why yEnc shouldn't be adopted... none of which will probably keep the community from adopting it, however. If it's here, and being used, that is a whole lot more intertia than common sense can usually gain. Er, betamax, anybody?

Re:Intertia vs. Good Ideas

[Zeinfeld]

I have some sympathy with the article author, but not when it comes to the MIME issues. I have written plenty of IETF and other standards, I know the value of going through a standards process, however the IETF is not a place to do research, it is a place to standardise and improve existing protocols. The idea is that you start from code.

Breaking MIME is not something I would (do) lose sleep over. People in the MIME community screamed at us when we had the temerity to introduce the text/html content type, rather than use application/binary. They were completely obstructionist when it came to insisting on 8-bit clean transport for HTTP. In the end we treated them as damage and routed around them. HTTP uses several headers that the MIME people villified.

The functional issues raised are significant and it would be good to see them addressed. In particular using the subject line is pretty lame. Either you want the encoding format to be completely independent of MIME or you don't. I think that MIME independence would be the better route since then it would be easier to move to a more modern protocol such as BEEP. But using magic numbers and MD5 inside the encoding does not seem like a bad move.

The more interesting 'meta-point' however is that tweaking the encoding format is only scratching the surface when it comes to fixing UseNet. The main problem with USEnet is that it still has to route every single article to every single node whether it is going to be read or not. While the flood fill routing was a good scheme when NNTP was developed and the number of nodes was small it is needlessly wasteful now that we have hundreds of thousands of NNTP servers, it is just not necessary to have that level of redundancy to route arround censorship.

Re:Intertia vs. Good Ideas

[Spy Hunter]

I would argue that the adoption of a standard is a much better indication of its "goodness" than its technical features. yEnc has been adopted by lots of people because it solves problems that they have, therefore it is proven to be good. If someone fixes the flaws that this author talks about and makes a new scheme that works better, then it might get adopted. If it does, it will be because it solves real problems people have with yEnc. If it doesn't, that means that it is too much of a pain for people to switch and that the problems yEnc has are not that much of a problem for real users. I think this is probably the case. So you can't use filenames with double quotes. Big deal! Change them to single quotes or something! So one out of a thousand posts will be corrupted because of mis-recognized magic strings or something. Its not any worse than it was before, and the downloads are smaller! If the problems really are THAT bad, a solution will come and people will use it.

Re:Intertia vs. Good Ideas

[JordanH]

• Er, betamax, anybody?

Not really comparable to the betamax vs. VHS debate. I've not seen anyone arguing that the alternatives, uuencode or base64, are better than yEnc, just that yEnc has serious deficiencies.

Perhaps Mr. Nixon is arguing that yEnc is worse than some wholly theoretical alternative.

Some of Mr. Nixon's points do seem interesting, but if he is convinced that there is a better alternative to be put forward, he should get the code out there. Anything else is just sniping.

Re:Intertia vs. Good Ideas

[Rui del-Negro]

Betamax failed because it was a proprietary format. If Sony had allowed other companies to make it (like JVC did with VHS) it would still be around (20 years ago it was better than VHS is today). Betacam still dominates the broadcast world.

Now, yEnc looks like it was created by Microsoft. It's not a standard, it's a hack. The only way it can become a standard is by pushing it down people's throats and then using "public pressure" to force applications to support it.

To use a videotape analogy, it's like releasing magnetic tape reels after people had been using cassettes for years, just because the reels use slighlty lighter tape.

I hope yEnc in its current form is *not* supported by the industry. I think a company such as Forté could create a real standard using an encoding method similar to yEnc (it wasn't "invented" by yEnc's author, anyway). I think Agent's programmers, of all people, should know how hard it is to deal with these (non)standards, and could save themselves a lof of work in the future by making sure it's done right.

As it stands, yEnc is the same as UUEncode, only in smaller portions (actually it's worse, because you can sort of wrap UUE in MIME; you can't with yEnc).

Re:Intertia vs. Good Ideas

[Chasing Amy]

> it is needlessly wasteful now that we have hundreds of thousands of NNTP
> servers, it is just not necessary to have that level of redundancy to route
> arround censorship.

I disagree entirely. Never underestimate the government's ability to stretch censorship to new levels.

Unless the very way NNTP servers operate is to gulp down and pass on each article for each newsgroup, the government would easily target those servers that spcifically carried groups or posts it doesn't like.

Pressure for news providers to drop certain groups began several years ago when the Vacco busts of people trading in child pornography led a news service to be criminally charged for the content of some groups and led other news servers in that state and elsewhere to drop gcertain groups thanks to their content. The charged news service took a plea even though they clearly would have won at trial or on appeal by claiming common carrier status, but hey, nobody wants to be the expensive test case.

Some may not see the problem with news servers being coerced by the government to drop those particular groups thanks to their contents, but the principle it sets is horrid. Certain "content owners" have of late been threatening to use the DMCA as a club to get news servers to drop groups which share TV shows and other such copyrighted material. If groups were more "localized" to a set of specific servers, or articles were localized to their originating servers, that would make it exceptionally easy for the DMCA to be used to require the "closure" of groups or removal of articles from USENET.

Furthermore, in this time of anti-terrrorist hysteria, the government has gotten away with the USA/PATRIOT mess already and is continually making some questionable choices. If it finds a newsgroup dedicated to dissent, or more spcifically dedicated to anti-globalism, for example, it cannot easily dstroy such a group because of the nature of USENET--the damage would be routed around by servers in other countries, even if every U.S. server could be forced to remove a group or article (not that they could be).

However, if the architecture of USENET were redesigned to localize groups or articles to subsets of servers--the likelihood of a government censoring USENET speech is magnified considerably.

It is the redundant architecture of USENET which will keep it free of censorship long after the WWW has been tamed--as it will be. Just look at the broiling mess within ICANN over officials trying to hand control of the WWW over to government-appointed reps. Eventually something like that will happen, and governments will cooperate with each other to make censorship in their mutual interests easier. Thanks to the architcture and nature of USENET, it will remain free and uncensored long after the WWW has fallen to censorship.

Just my 2 pence, though...

Re:Intertia vs. Good Ideas

[Chasing Amy]

> ICANN's scope is the DNS system, upon which WWW, MAIL, NEWS, and everything else on the net currently relies.

Umm, that's the point.

He who controls DNS exercises immense influence on the Net, particularly the WWW. Giving control of the DNS system to delegates directly appointed by governments is a recipe for fostering censorship in the interests of those respective governments.

I.e., if DNS were in the hands of representatives appointed by governments, and some given websites were to be deemed undesirable--delete their DNS entries, and they go away. Poof. The instant DNS is controlled by governments, is the instant they begin implementing a system to instantly pull the DNS entries of websites which are "dangerous" and "patently offensive." Governments hardly ever agree on anything, but you can bet they'd agree to some deal-brokering--you support the "delisting" of our seditious sites, I'll support the delisting of yours.

USENET isn't as vulnerable because of its architecture. You can't just "delist" one newsgroup the same way you can delist one website.

If the ICANN higher-ups have their way and hand over the reigns to government-appointed reps, I guarantee you governments will take the opportunity to at least consider using their control of DNS to enable censorship.

Yenc is great!

[Mean_Nishka]

I for one am happy about Yenc's rapid adoption. My newsreader ( Xnews )software supports it, and I have noticed no difference in using Yenc over traditional binary encoding.

In fact, Yenc will help pay-per-gigabyte Usenet users achieve a greater bang for their buck. Anything that saves money is a good thing!!

DIME?

[leighklotz]

There are some proposals in thee XML and Web Services arena for dealing with some of the problems tha yEnc is skirting.

One, called DIME, is a MIME-like system that handles binary content, chunks, etc.

http://www.oasis-open.org/cover/dime.html

Agent

[Account 10]

Forte released Agent 1.91 2 days ago with yEnc support. it looks like Mr. Nixon is fighting a losing battle.

no no no

[SETY]

The author of the article seems to keep saying there isn't a problem with USENET encoding, but then goes onto complain that yenc shouldn't be used. He points out flaws in how this encoding scheme is implemented. fine fine fine.

There was a market for this thing, it spread like wild fire. It's too bad that no one made a better spec and program (the author aludes that there was planty of time to do this). yenc meets the "GOOD ENOUGH" criteria, thus it will be used, shitty, non-robust standard or not.

yENC

[arsaspe]

I'm all for standardisation... but sometimes it takes _forever_ to get something standardized. If someone writes a better product, they generally don't want to wait for it to be declared a standard, especially with something like uuencoding which has been around as long as usenet, and isn't going to be replaced in a hurry unless someone comes out and waves a product around yelling "hey try this. it works better". Ogg Vorbis isn't a standard by any means. Hell, it is still on RC3. _but_ a lot of people are using it because it has far better sound compression than mp3. You don't hear people complaining that Vorbis has jumped the standardisation process do you?

Personally I can't see why we can't just send the data as 8-bit binary. uuencode and similar encoding formats should have died out with UUCP years ago, since there is no physical reason why 8bits can't be sent over the wire anymore.

Re:yENC

[Rui del-Negro]

A "standard" is not the same as "a common thing", or even "something that most programs can read". For example, most programs can't read YCrCb TIFF files (Photoshop included), but these files do follow the TIFF specification, hence they are standard.

When you create an encoding method that is going to be used to transfer data across a network, you need to ensure that this method is compatible with everything along the way. When you send an e-mail with an Ogg file attached, this file is encoded in a way that enables the servers and the client at the other end to identify it, check its integrity, reconstruct it, process it, delete it independently from the rest of the message, etc. It doesn't matter what the file itself is (Ogg, MP3, TIFF, Doc, XYZ, etc.); these methods work with any file, regarldess of its type or contents.

8 bits can be sent over the wire, and in fact are sent over the wire. But to make sure the servers (and the clients) can tell where one file (or part of the file) ends and the next one begins, you need to "wrap" the data in a package that programs can understand. That's what MIME does. It says "this part is the message text in HTML", "this part is the message text in plain text", "that part is an image", "that part is an executable file".

Instead of using this "universal" packaging system, yEnc forces programs to look for specific strings and try to guess where things begin and end. And it has no mechanism for identifying individual parts in a multi-part post (again, programs must look at the text or message subject and try to guess).

Doing something right is usually not much harder than doing something wrong. And when people get used to something that's broken, they won't want it fixed.

Re:yENC

[JamieF]

It's interesting that it took this long for an ancient problem to be sorta kinda attacked, if not solved. Maybe this is because the Usenet infrastructure has changed to the point where 7-bit is silly, but why is this something that's just being addressed now? Don't tell me this is a brand new problem that the best minds of the news admin community couldn't have figured out by now.

One approach to short-circuit standards is to take the rogue approach that Netscape did, which is very different from the one folks keep accusing Microsoft of. That approach is to take an almost-suitable standard and extend it, in the same spirit, with the intent that your well-thought-out extension will be adopted later. Of course they did a less than perfect job, but the idea is sound: don't wait for the spec, lead it, while making it possible for it to remain open. That's different from the Microsoft approach of "Embrace, Extend [and Exterminate]" wherein you add undocumented proprietary features that lock you into a single vendor's solution. A hell of a lot of what's in the HTML 4 standard was released as a non-standard but documented extension by Netscape in 1.1 and 2.0. Some of that was bad (blink, and arguably frames), but when you judge Netscape, try to separate the new HTML tags from the bugs, because it's the bugs (and having to code around them) that make the web such tower of Babel.

So basically what I'm saying is, if MIME is almost there, CAREFULLY THINK THROUGH and implement the extensions you want, and implement it already. Or, pick something other than MIME. But, don't start from scratch and make mistakes that have already been learned from and solved. Build on the good decisions of the past. The real crime is solving the same problem, badly, over and over; getting ahead of a standards body while keeping intentions pure and decisions wise is not nearly as bad.

there is no real bloat associated with uuencode

[mmusn]

Uuencoded text will compress down to nearly the same size as its corresponding binary (or less, if the binary can be compressed). That kind of compression is now a standard part of modems, Internet protocols, and many file systems. Even the CPU overhead of compressing and decompressing that kind of data is negligible. If yEnc doesn't end up using less space on disk and doesn't end up using any less bandwidth than uuencode, indeed, "why encode" in yEnc and break a lot of software that expects USENET posts to be text-only?

Re:there is no real bloat associated with uuencode

[A Commentor]

Uuencoded text will compress down to nearly the same size as its corresponding binary (or less, if the binary can be compressed). That kind of compression is now a standard part of modems, Internet protocols, and many file systems.

I have yet to see any DSL or cable modem with compression. So for most of the heavy binary users, uuencode data will not be compressed. And on regular modems it won't be smaller than the the yEnc, since if, as you say, the binary can be compressed, then the yEnc will be compressed..

Even the CPU overhead of compressing and decompressing that kind of data is negligible.

Do you run a heavily used news servers to provide proof that the CPU overhead is 'negligible'.

Re:there is no real bloat associated with uuencode

[A Commentor]

Oh, come on, think. Even if your modem doesn't compress, you can get compression at the level of a tunnel or at the level of your news transfer protocol.

For 99% of the people, getting the tunneling software on your system will be easy, getting your news server to install software on their software will be alot harder...

Adoption of yEnc

[jonbrewer]

I am seeing smaller binaries as a result of yEnc. This is fine. The problem is, my favorite binaries grabber has no idea what to do with the files, and won't even download them. I figured out how to make Agent download them, but A. I hate Agent (and don't understand why anyone likes it!) and B. the binaries don't always decode.

As I'm a lifetime lurker (well eight years, but it seems a lifetime!) I can only choose not to download yEnc encoded binaries. And no one will know! (my news server doesn't log downloads) It's all up to the posters to adopt or not.

yEnc = XMODEM part deux

[phr2]

In the old CP/M days there was no standard way to transfer files over serial connections, except maybe Kermit. Kermit was slow because of its ping-pong protocol (no packet window--that was added later) and because it encoded binaries as printing characters. Ward Christensen invented XMODEM, which basically dumped the file through the wire as 8-bit characters, with very crude error checking and file headers. yEnc does something pretty similar for Usenet articles. It's a crude method for posting binary files as 8-bit characters instead of 6-bit characters. That of course cuts down transmission time considerably.

Despite its problems, XMODEM took off because it filled a need, just as yEnc does. Nixon's complaint that shrinking files by 35% won't make Usenet any smaller because people will just post more files is besides the point; it's like saying getting a 35% salary increase won't help your finances because you'll just buy more stuff with the extra money. Most people want that extra 35%, and Jürgen stepped up to the plate and delivered it.

Thankfully, as far as I know, nobody railed against Ward Christiansen the way Nixon does against Helbing. XMODEM's problems became obvious and the solution was to introduce YMODEM and then ZMODEM. XMODEM is still around, but its successors (and of course serial IP) have pretty much supplanted it. Ward's initial efforts are still deeply appreciated.

Yes there's the problem of legacy software, but a protocol that's only been around for a few weeks or months can't have that much of a legacy. The only programs that currently support yEnc are the ones whose maintainers react pretty fast to new developments, and those maintainers are likely to also quickly pick up any revisions/fixes to yEnc.

So the solution Nixon should be calling for is not a years-long bureaucratic standardization process that will get yEnc 1.3 entrenched while the standardization is happening. The solution is to fix yEnc's problems and release a new version as fast as possible, before the old version gets spread around too widely.

Re:yEnc = XMODEM part deux

[phr2]

But what if it takes a year before an "improved" standard comes about?

That's the point--right now, the only yEnc implementations are in programs maintained by people who jumped on things quickly. So by fixing yEnc right now instead of waiting a year, you avoid trapping users of the slower moving programs. The slower moving programs simply aren't using yEnc yet.

I agree yEnc should have been more carefully thought out before being released. But its problems are not obscure. They are obvious to people who have dealt with these issues before. I don't think years of study and a lot of iterations are needed to figure out how to fix yEnc's problems. It should be possible to fix them quickly and get the fixed version out there before too many people are using the old version.

Save /. Money...

[smoondog]

Hey, if this guy adopts it, he could save slashdot some money...

-Sean

Standards ARE important... here's why.

[gambit3]

In one sentence, standards ARE important because they allow for the most people to get the most benefit.

I work in an industry that relies heavily on standards, and my job deals specifically with standards. Making sure that WE follow standards, and making sure that other vendors follow standards.

Sure, they're slow to develop. But they're the best for interoperability, and that's crucial. In my line of work (for a major Mobile Phone System NSS provider), I have to deal with other providers that have to follow the same standars we do. That allows both of our products to communicate. This gives the end consumer (i.e., Cingular, Sprint, etc.,) the option to buy from different vendors. This forces us to make better products. This forces us to be more efficient. This forces our competitors to do the same thing. In the end, everybody wins.

The other alternative is what I see as the Micro$oft approach: Standards be dammed, I'm going to do it this way, and f*ck everybody else. It's the same approach that gives you security holes in your browser, because, well, who needs the standards?

I can't believe I'm reading comments like "well, it's here and it works so what's the problem?"
The problem is the future.
The problem is the inability to send an SMS from a CDMA service like Sprint to a GSM one like Voicestream. That's what happens when you blow off standards.
The problem is the inability to read an M$ Word doc that was sent to a Linux user.
Ignoring standards and going off on your own (especially, going off BADLY on your own) just divides us.
Good standards help us all. They give us better products. The lower costs.
CD-Rs. FireWire. PCI. countless others.

Besides, as the article begins by asking: Just what problem were they trying to solve?

Re:Usenet? Please go away!

[Sabalon]

Lets see...Napster is dead. Morpheus is dead. Morpheus part 2/gnutella is a zombie (perhaps it's just me - I have yet to have it actually pull down a file for me. It keeps telling me the other end is busy or something.) Even when they were alive, you get half way through and someone cuts you off...or you find out that 80M download that took a whole day was actually mislabled.

On Usenet...sure - you don't get to search and finding a file involves posting a request and hoping someone fulfills, but you get bandwidth - assuming you want to pay for it...you get files that are there (assuming you have decent retention.) and not dependant upon someone being online. And unless you have a crappy server, you don't get halfway through a download and someone decides to kick you off. And 99% of the time, what something is posted as is what it is.

As for the replication - well, there is no one point of failure. As well, you don't have one site getting the shit hammered out of it either. I pay $9/mo for usenet...I get three fast servers to choose from and some high GB limit.

If I had to go to the same server as everyone else, you'd have the same problem that moviefone had when star wars tickets went onsale online - DOA - all with nice corporate control of content.

Technical arguments against yENC, blah

[Tofuhead]

It should be pointed out that this site, linked from yENC's own website, goes into more technical detail regarding the technical flaws of yENC. The fact that it's linked from yENC's own site is proof that the author is at least familiar with the concerns that people have with his implementation.

I personally still find it difficult to argue against the article author's point that THERE WAS NO RUSH to force yENC out the door in such an unpolished form. After so many years of waiting for something better, why ignore the recommendations of those you are trying to help?

< tofuhead >

A few words from the original author

[Ruddygore]

Well then. When I put that page up, I honestly didn't expect many people to read it outside news.software.nntp and a few curious folks in alt.binaries.news-server-comparison. I certainly wan't expecting to get Slashdotted. Well, that's fine, except that the uproar might have waited a little bit.

In my essay, I state that what Usenet needs is "a better way to post Binaries". The next piece of the puzzle, of course, is to answer the question, "What IS a better way to post binaries?" I was thinking about finishing that page up tonight, but I am writing code at the moment instead.

So, when reading my comments, just keep in mind that, yes, I DO have some answers to that question, too. It's just that it's a bit of a more time-consuming question, so that page isn't done yet.

This time around, though, I will make sure to include a prominent warning to NOT run off and implement the ideas as quickly as possible, and to please not use all of Usenet as beta-testers. The idea that whatever gets done fastest is best just doesn't work for me. There were good reasons I didn't go and get people to implement my smaller encoding ideas when I first wrote the code. If only the yEnc implementor had continued where I left off rather than going down his rather misguided path...

All the comments are welcome. I've been getting some interesting email, too, of course. Many programmers of Usenet client software absolutely despise the thing and are quite annoyed at the amount of their time it is wasting. I guess it's just more of that never-ending divide between the users and the techies. So it goes.

yEnc is here, that's for sure. Now we just have to try to deal with it.

Jeremy

Re:A few words from the original author

[sparrowjk]

Your essay is the best summary I've seen so far of the reasons not to use yEnc. You have done a service to those of us who have been annoyed with yEnc -- now we don't have to explain it to anyone, we can just point them to your essay.

So, be it resolved that yEnc leaves much to be desired.

However, if yEnc is the impetus which actually gets the community moving toward implementing a good, solid standard, then it will have served its purpose. Perhaps if we had had yEnc 5 years ago, we would have a standard already. But we didn't, and now we must pay the piper.

Since people aren't going to give up the advantages of yEnc without a substitute, the priority going forward is clear: to develop a better standard. If it truly is better (and not simply another hack) then ensuring its wide adoption shouldn't be too much of a problem. If, however, people can't be persuaded to switch, so much the worse for Usenet -- but no point in dwelling on doomsday scenarios. As you say, the cat is out of the bag, and all we can do is damage control.

Re:Is usenet dead?

[MillionthMonkey]

I never post to it. (And after finding my ten year old posts to alt.drugs sitting in plain sight on Google, I doubt I ever will again!) But I do searches through newsgroups all the time when I'm looking for answers to obscure technical questions, or when I want to know if anyone's come across the same bug that I'm having trouble with at the moment. USENET might have more crap, but sometimes crap is what you want! You don't always want your search results to get choked up with corporate stuff. If you're doing Java programming for example, and you want to find out why class X doesn't work, a normal web search is difficult to control because all you get is bullcrap from Sun, and a zillion identical javadocs for class X that people leave around on their HTTP servers. I want to find out what people are complaining about, or whether anyone actually USES a certain API I'm considering. For getting a feel for what's going on in the field, without getting snowed under by marketing materials from a vendor, USENET is great. It isn't as corporatized.

Of course, the existence of alternate web based bulletin board systems like this one decreases its relevance for search purposes. And it's suffocating under the weight of all that spam. But USENET is still the biggest forum out there, and it's still the one that's the most easily searched.

Encoding efficiency isn't a BIG deal

[Sloppy]

If, by any chance, you're transferring things over a modem (v42bis' lzw) or ssh vpn (zlib's deflate) or possibly other types of links, then you're probably not going to notice a difference anyway. The systematic encoding inefficiency that goes with base64 and uuencoding, results in a substantial lack of entropy that will be picked up on and exploited by good compression algorithms. Then end result won't be quite as good as having efficient encoding to begin with, of course, but it will be in the same ballpark. There's no way it'll be anywhere near a 33% difference.

This sounds like something that would have been useful 15 years ago before compression was widely used, and when people were still writing newsreaders. Now it looks like a waste of time and an excuse to get people to "upgrade" their software.

Re:concretely: this is why you don't need it

[jridley]

See my other post in this thread. It's not all about dl speed or even drive space. Most serious usenet large binary downloaders use a premium news service, and they ALL have download/upload quotas and/or pay by the byte schemes. yEnc gets you 30% more for your buck on those servers.

Re:Consider the source

[Burdell]

Actually, Supernews is now part of Critical Path (the RemarQ name went away). And yes, Jeremy Nixon does work for Supernews. He also wrote Cleanfeed, the anti-spam program that Usenet admins everywhere to make news bearable to read.

The "he's against it because it saves bandwidth" argument makes no sense. If it saves users a little bandwidth, it saves Supernews many many times that much bandwidth, lowering their costs (which means they don't have to charge users as much to provide the same service). It also saves disk space, meaning Supernews doesn't have to buy new disks quite as soon. And a good bit of Supernews' business is in the corporate (outsourced ISP) service, which they don't charge by the gigabyte (they have speed caps, not monthly download quotas).

The problem is that any savings are just an illusion; this is just a momentary blip in the growth of Usenet. Since yEnc doesn't have the 100% market penetration that uuencode and MIME have, people are more likely to post binaries in multiple formats, causing storage and bandwidth needs to increase, not decrease.

Standards for Usenet encoding???

[roystgnr]

Can *anyone* look at the uuencoded, mime encoded, and other similarly mangled into 6bit, 70 character-per-line standards, and honestly tell me that Usenet was designed with binary file transmission in mind?

There are no Usenet binary transmission standards, just a few different hacks to make it work. If this guy's new hack makes it work better, good for him.

Good idea, but...

[Firecaster]

... the implementation sucked.

I leech regularily from alt.binaries.anime and the related newsgroups. When the yEnc posts started coming in, I simply upgraded my newsreader to the newest version. But a LOT of people out there use Agent, and it was absolute pain to combine/decode all the yEnc posts that started popping up all over the place. The worst of it is that the yEnc posters were basically saying, "Start living in the present and upgrade". Nevermind that at the time that only yEnc-capable newsreaders were for Windows...

I mean, I don't know, but this sounds a lot like the OS wars that have been going on for quite some time. Some people simply don't want to have to switch newsreaders. Some people don't want to have to switch OSes. And that's fine, because it's a free world out there. I like the idea of yEnc (I get more out of my Easynews account), but I really don't think it should have been introduced so quickly.

~ Firecaster ~

Two Problems:

[NeuroManson]

One: yENC, when it was unveiled, did not really allow most conventional newsreaders any opportunity to adapt, til after the fact. This is akin to perhaps releasing zip files long before any archival software was actually available to open them... So do most of the folks using usenet for binaries get the opportunity to at least *choose* the way they do their downloads? Nope, they also are forced to adapt, or lose out...

Two: Loss in transmission... I've been downloading yENC attachments for the last month, and out of them, found over 50% loss/corruption in posting... Not due to retention/propagation either... Just files missing large chunks... Now this *could* be due to some problems on the senders' end, but it seems just a little *too* coincidental that almost all of the losses have occured with yENC uploads...

Re:Seems silly to be complaining....

[khuber]

This is wrong thinking. NNTP is an internet protocol and yEnc should play by the rules of internet protocols. It does not and that's why it is bad. You can't break building codes to build your house and justify it because you saved a few bucks. "Hey...it seems to not collapse" "it hasn't caught on fire yet!"

In particular, there is no yenc RFC and yenc does not use MIME which is the agreed upon standard for encoding binary attachments. Yes, uuencode is a gross grandfathered format, but it is still 7 bit clean.

Releasing problematic improperly specified encodings that break internet protocols is not being a good citizen. "it works" is a poor justification. it does not work, and breaks compliant software.

-Kevin

Reminds me of Napster data format

[topham]

This just reminds me of the napster data format. Anybody ever read the reverse engineered specs? It's scary. It looks like it was designed by a monkey. And not a smart one.

yEnc sounds like a good idea, and a horribly bad implementation.

Jeremy's right, but it's too late now.

[Charles Kerr]

I'm one of the authors of the Pan newsreader and agree with Jeremy's analysis of yEnc. yEnc repeats many of uu's mistakes, so news clients have to search text/plain messages for =ybegin and =yend blocks instead of looking in the headers.

But yEnc's bandwidth savings are real, which is a huge win for alt.binaries users. yEnc has been the most-requested feature for Pan over the last month. (0.11.2.90 supports it.) IMO yEnc is the format to use for multiparts right now.

Hopefully yEnc will motivate others to come up with a mime-friendly alternative encoding for Usenet. yEnc Considered Harmful is another yEnc opposition page that suggests mzip compression, but I haven't seen any public discussion of it yet.

If/when such a replacment comes along, Pan will support it too and add an are-you-sure dialog for yEnc postings.

More like it used to be

[medcalf]

It used to be that someone did something useful, then the community, through use choices, adopted it as standard. Then, if there were flaws, these would be ironed out with an updated standard, usually all or mostly backwards-compatible with the original implementation. It's gotten to where new standards are useless, either because companies (like, say, RealNetworks or MS) refuse to submit their protocols/formats for public use/review, or because the standards committees (say, for Java (before it was pulled) or the W3C) argue for years without actually doing anything.

I, for one, am happy to see a useful format publically available.

Re:All of Mr. Nixon's points are easily refuted

[Anonymous Coward]

"Whatever kind of "confusion" this guy is referring to is nonexistent. yEnc downloading and decoding with the right software is as transparent as uudecoding. Promise. People don't have to relearn anything."

Well people do now have to think of yENC's longer lines. Where before my ISP newserver let me post 5000 lines of uue material now I only get 2280 with yENC. Longer lines under yENC. That's why Agent 1.91 has such a small line length as the default. Yes the payload is the same but you have to think a little different or just rediscover your servers limits.