Slashdot Mirror

yenc helps with the encoding overhead issues. It's widely used and most nntp servers support it.

all hail the power of usenet ! :)

seriously, usenet is a far better place to download "stuff". with the development of better encoding schemes, the overhead of a binary attachment is only a few percent. see http://www.yenc.org/ (specs are public domain) and http://www.yenc32.com/ (gpl'd utilities). add miracle-like recovery schemes, such as http://parchive.sourceforge.net/ (listed as gpl'd on their project page) and usenet is far from being obsolete.... it's just most old-timers may have forgotten about it and newbies don't know it's there.

plenty of usenet providers tout a "no downloads logged" policy... so download to your heart's content, fill up that hard drive.. just remember to wait for what you want to be posted, don't post requests for anything that may be 'questionable' as to legality... ;)

plus, if there is something not kosher about a binary that's been posted (virus, trojan, not what it was claimed to be, corrupt, etc), replies to that fact follow pretty quickly after the original posting.

while the idea of usenet being nothing but smut and spam are partially true, there are a lot of useful groups out there. even a bunch that fall into the useful AND legal category. :) :)

In fact you can still get yEnc encoded binaries from Google Groups. I'm sure they'll start discarding them eventually but if you made an encoding that isn't used much those messages would stay permanently.

many serial connections used characters 0-31 for signaling (with XON and XOFF being only the best-known) -- sending a file containing these characters might hang or abort your connection.

SMTP was never designed to handle binary files at all, which is why binary files are encoded into text, which increases their size by 30%.

yEnc has shown widespread acceptance in Usenet, I'd like to see it used as the de-facto format for SMTP mail. Electronic mail's "push" nature makes it extremely useful where FTP/HTTP is not (although IRC DCC is) and I'd enjoy having the pleasure of subscribing to mailing lists which send out multimedia or other forms of large content and having it delivered, just like postal mail, right to my desktop or a nearby ISP mail server. Who is with me?

As some of you might already have noticed, the author of yEnc responds to several of the points raised in his FAQ (scroll down to near the end of the page). His replies (to questions such as "Some people say that yEnc is badly designed and was rushed without proper discussion. Is this right?") seem quite realistic and sensible to me, and I get the feeling some of the accusations levelled at Jürgen in the article weren't entirely fair.

yEnc is yet another encoding method for binary files. Unlike Base64 or UUENCODE, it uses nearly all octet values, reducing the overhead from 33% to about 2%. Problems with yEnc No MIME or Back to the UUENCODE mess

A huge mistake is not to use MIME for yEnc. Let me explain why: In the pre-MIME era there was UUENCODE, which had several problems:

• There are no clear delimiters. UU-encoded files can start anywhere in a text message. As a result, some newsreaders incorrectly see attachments where there aren't any. This has been partly addressed by yEnc which uses =ybegin...=yend instead of begin...end. But there's still a chance that the markers appear in normal text.
• No labelling of file types (Content-Type in MIME).
• Charset recoding problems. Note that this is a much larger issue with yEnc than with UUENCODE. UUENCODE is only corrupted on gatways that can't handle some ASCII characters (esp. EBCDIC systems, which should now be extinct). For yEnc, every 8bit charset translation is fatal.
• No standard labelling of split messages. (This is addressed by yEnc, but only within the body.)

MIME provides a proven solution to these problems:

• It clearly labels all data out-of-bound, using Content-* headers. It clearly seperates text from binary data.
• Base64 uses an alphabet that most likely survives all charset translations.
• message/partial provides a standard way to split large messages. It even allows MTAs to split messages on the transport level (granted, that should NEVER happen on Usenet).

Back then, when UUENCODE was state of the art you could just cut and paste a UU-encoded file into your text message. With modern, MIME-aware newsreaders, this is no longer the case:

• Newsreaders might be tempted to apply Quoted-Printable (or Base64) encoding on the text. Yes, I've seen Quoted-Printable encoded UUENCODE attachments.
• Even if they can be convinced to use 8Bit (or Binary), they might suddenly do some charset recoding:
  • ISO-8859-15 and UTF-8 become more and more popular, especially due to the Euro Currency Sign. This means the charset has to be recoded from the systems native encoding (e.g. Windows-1252).
  • Even with plain old ISO-8859-1, DOS and OS/2 newsreaders have to recode from the DOS charset and Mac newsreaders have to recode from the Mac charset.
  Further, there's no way to include yEnc within UTF-8-encoded text, which is slowly becoming the worldwide standard: If you include the data as-is you'll have invalid UTF-8 sequences. (This also applies to other Multibyte Charsets, which are quite common in Far East countries.)
  (Note that this is no problem with UUENCODE or Base64, which only use an unproblematic ASCII subset.)

There are also some smaller problems which should be addressed:

• Using CRC32 as a checksum. There are much better hash algorithms like SHA-1 or MD5 available.
• Bad Extensibility and less features than MIME Content-*: Could be solved by integrating yEnc into MIME or vice-versa (e.g., Content-* headers could be allowed directly after =ybegin and before an empty line after which the binary starts.)

One of the solution, of course, is to embed yEnc into MIME. The first idea is to do that as a Content-Transfer-Encoding.
There have been some arguments against MIME, however, which should be addressed here.

The creation of new Content-Transfer-Encoding values is STRONGLY discouraged. (Quote from RFC 2045)

This is true and there's a very good reason for it: A lot of software needs to be updated to support it.
On the other hand, the situation is no better when MIME is not used: Users would still need new software. If the news client does not support the format, users can just export the message (nearly every newsreader can export a message in source format) and process it with external tools.
You have to ask whether a new, news-only encoding is a good thing. If yes, then it does not make a difference wheter it's embeded in MIME or not.

message/partial MUST not have binary content.

This is because it couldn't be recoded as neccessary on gateways. But with yEnc, recoding would only happen where the message would have been recoded anyway.
It only raises the question whether a news-only encoding is a good thing again. (With yEnc, recoding would only happen where the message would have been corrupted anyway.)

There's no per-part integrity checking for message/partial.

There is no reason why Content-MD5 could not be used on message/partial: RFC 1864 only disallows Content-MD5 on multipart/* and message/rfc822, not message/partial. (The reason for this is that these type can contain data that could be recoded at gateways. This would not happen with message/partial or, if it happens due to yEnc, the message would have been corrupted anyway.)

There's no easy way to find all parts from a single part (i.e. find the message ids of all parts) with message/partial

Neither is there with yEnc. All proposed solutions would both work with a non-MIME-yEnc and message/partial.

You can't use your standard newsreader by pasting the encoded data.

You haven't been scared enough by the yEnc-over-Quoted-Printable and Charset Fun chapter, have you?

Conclusion: yEnc as a MIME CTE is much better than yEnc without MIME. yEnc as a MIME Content-Type

The other solution, of course, would be to introduce yEnc as a MIME Content-Type. It then would have to use the binary 8bit CTE.

This would be a similar approach as used for application/mac-binhex40, which is also defined as a Content-Type. Also note that many compression and archive formats are a Content-Type.

As yEnc contains additional information (such as file name, markers, etc.) which a CTE usually does not, this seems to be the better solution.

Conclusion: yEnc should be a MIME Content-Type. Alternatives

Of course, one should ask what alternatives are there to yEnc (or any other super-Base64 encoding). Not using Usenet

Usenet is, even without the Base64 overhead, a horribly inefficient method to transfer large files. Because of the flood-fill mechanism, the articles are sent to all news-servers carrying the specified newsgroup, even if there's no user that wants the article there.

Peer-to-peer networks, such as Gnutella or Freenet are much more efficient and can transfer binary data as-is.

Conclusion: Binaries should not be sent over Usenet. This is not expected to happen any time soon, however. Use all MIME features

MIME already has most of the features necessary to send binary data over Usenet:

• An encoding: Base64.
• A standard to split messages: message/partial.
• An integrety check: Content-MD5 (MD5 is much better than CRC32.)
• Out-of-band labelling of data types.
• No mixing between text with charsets and binary data.

Of course, you would have to use all features provided by MIME to provide everything that yEnc provides (today):

• Use Content-MD5 on each embedded file.
• Use Content-MD5 on each message/partial
• Use Content-Disposition to transport file names and attributes.
• Use the number and total parameters on every part.
• Use the Message-ID convention described below.

Some features proposed for yEnc, such as assembling a file (message) from different sets of partial messages, could also be integrated into MIME - in a backward compatible way!

There is only one real argument agains MIME: efficiency. The Base64 encoding produces about 33% overhead.

Conclusion: MIME provides a proven solution to send binary data over Usenet. The only problem is efficiency. Link-level Compression

The 33% overhead can be more than nullified by using an compression method over bandwith-sensitive links.

There's a proposal for a MIME-aware compression scheme Assembly of partial messages

See my document about a Message-ID convention for partial messages. Conclusion

MIME already provides a solution to most problems that yEnc is supposed to solve. There's no reason to reinvent the wheel for yEnc, which causes some problems. Only the more efficient encoding remains. This problem can be addressed by introducing yEnc as a MIME compression scheme or by introducing a link-level compression/filtering.

Revisions
2002-03-19yEnc as MIME type would only require 8bit, not binary.

Some minor fixes. 2002-03-04Initial version.
Claus Frber <claus@faerber.muc.de>