Slashdot Mirror
Usenet Encoding: yEnc
Motor writes "Anyone remotely interested in usenet binary newsgroups must have noticed the spread of yEnc. yEnc is an encoding scheme for usenet binaries which avoids the enormous (30-40%) bloat associated with the schemes currently in use - which all have to produce 7-bit data to stop ancient newsservers from choking. A good thing, surely? Well, not according to some people. The guy has some good points about yEnc and standards, but I can't help thinking that "standards" people have endlessly discussed better encoding schemes, and nothing has come out of it. yEnc may not be perfect, but it works and it's here - hence the rapid adoption. What do you think?"
This is why it is so bad:
:-)
/do/ use outlook occasionally, and sometimes email attachments but only as a last resort.
(quote)
The problem is that doing [using) yEnc within MIME will break the current MIME specification, and very likely cause problems with existing software.You cannot add yEnc to MIME without first having two small changes made in the MIME specs. Bypassing the standards process would basically sabotage MIME by making it so that coding to the spec will produce software that doesn't work in the real world
(end quote)
Anyhow do we really need yet another damn standard for posting stuff. We're still having trouble with people posting html messages on usenet (Thanks outlook) - not to mention email...
Usenet really isnt the place for posting pics of you shagging your favorite goat near the railway lines, or the latest mpegs of enterprise etc.
In fact, lets kill off mime altogether, and get this multimedia crap out of our inboxes. Email/Usenet is for plain text, period.
Bah humbug.
okay so I
Uuencoded text will compress down to nearly the same size as its corresponding binary (or less, if the binary can be compressed). That kind of compression is now a standard part of modems, Internet protocols, and many file systems. Even the CPU overhead of compressing and decompressing that kind of data is negligible. If yEnc doesn't end up using less space on disk and doesn't end up using any less bandwidth than uuencode, indeed, "why encode" in yEnc and break a lot of software that expects USENET posts to be text-only?
Wait till it's done, then propose it as a standard like almost every other piece of the Internet. (or at least wait till it's close enough)...
I am seeing smaller binaries as a result of yEnc. This is fine. The problem is, my favorite binaries grabber has no idea what to do with the files, and won't even download them. I figured out how to make Agent download them, but A. I hate Agent (and don't understand why anyone likes it!) and B. the binaries don't always decode.
As I'm a lifetime lurker (well eight years, but it seems a lifetime!) I can only choose not to download yEnc encoded binaries. And no one will know! (my news server doesn't log downloads) It's all up to the posters to adopt or not.
What he's whining about is that it didn't fix every other problem in addition to overhead, and if anyone should actually bother making some huge new mime standard, now they won't have that carrot.
Obviously, if the rest of the problems were as big as he's trying to claim, yEnc would only be a minor setback for a new and more comprehensive standard, but the fact is that the 35-40% overhead of current standards is by far what's most annoying to usenet users. After we got PAR (parchive.sourceforge.net), reposts have been reduced drastically (except for pr0n and partly warez groups, where the dumb people with shitty servers rules).
Also, he's trying to say that because the increase in volume will outgrow the savings, there really is no savings. What kind of logic is that? Let's stop making processors faster, we'll just find bigger problems for them to be too slow for anyway, so what's the point?
After the introduction of PAR and yEnc, as a long time binaries downloader, I'll say the actual content of multimedia groups has more than doubled, and probably tripled, the last 6-9 months. That's progress to me.
I never post to it. (And after finding my ten year old posts to alt.drugs sitting in plain sight on Google, I doubt I ever will again!) But I do searches through newsgroups all the time when I'm looking for answers to obscure technical questions, or when I want to know if anyone's come across the same bug that I'm having trouble with at the moment. USENET might have more crap, but sometimes crap is what you want! You don't always want your search results to get choked up with corporate stuff. If you're doing Java programming for example, and you want to find out why class X doesn't work, a normal web search is difficult to control because all you get is bullcrap from Sun, and a zillion identical javadocs for class X that people leave around on their HTTP servers. I want to find out what people are complaining about, or whether anyone actually USES a certain API I'm considering. For getting a feel for what's going on in the field, without getting snowed under by marketing materials from a vendor, USENET is great. It isn't as corporatized.
Of course, the existence of alternate web based bulletin board systems like this one decreases its relevance for search purposes. And it's suffocating under the weight of all that spam. But USENET is still the biggest forum out there, and it's still the one that's the most easily searched.
Liberty in your lifetime
Discovery, that's why. Napster (Napster?, talk about dead!) and the like are great for finding a name you already know. By design, that's all it can do. Do a search for "early '80's Boston punk" for example and see what you get. This is where Usenet excels. Enthusiasts congregate into groups and upload files they enjoy, meaning a far better chance that I'll enjoy them too. I'm continually discovering great bands through Usenet that would remain unknown had I relied on name-search based file sharing.
One: yENC, when it was unveiled, did not really allow most conventional newsreaders any opportunity to adapt, til after the fact. This is akin to perhaps releasing zip files long before any archival software was actually available to open them... So do most of the folks using usenet for binaries get the opportunity to at least *choose* the way they do their downloads? Nope, they also are forced to adapt, or lose out...
Two: Loss in transmission... I've been downloading yENC attachments for the last month, and out of them, found over 50% loss/corruption in posting... Not due to retention/propagation either... Just files missing large chunks... Now this *could* be due to some problems on the senders' end, but it seems just a little *too* coincidental that almost all of the losses have occured with yENC uploads...
Just because you can mod me down, doesn't mean you're right. Shoes for industry!
Betamax failed because it was a proprietary format. If Sony had allowed other companies to make it (like JVC did with VHS) it would still be around (20 years ago it was better than VHS is today). Betacam still dominates the broadcast world.
Now, yEnc looks like it was created by Microsoft. It's not a standard, it's a hack. The only way it can become a standard is by pushing it down people's throats and then using "public pressure" to force applications to support it.
To use a videotape analogy, it's like releasing magnetic tape reels after people had been using cassettes for years, just because the reels use slighlty lighter tape.
I hope yEnc in its current form is *not* supported by the industry. I think a company such as Forté could create a real standard using an encoding method similar to yEnc (it wasn't "invented" by yEnc's author, anyway). I think Agent's programmers, of all people, should know how hard it is to deal with these (non)standards, and could save themselves a lof of work in the future by making sure it's done right.
As it stands, yEnc is the same as UUEncode, only in smaller portions (actually it's worse, because you can sort of wrap UUE in MIME; you can't with yEnc).
But yEnc's bandwidth savings are real, which is a huge win for alt.binaries users. yEnc has been the most-requested feature for Pan over the last month. (0.11.2.90 supports it.) IMO yEnc is the format to use for multiparts right now.
Hopefully yEnc will motivate others to come up with a mime-friendly alternative encoding for Usenet. yEnc Considered Harmful is another yEnc opposition page that suggests mzip compression, but I haven't seen any public discussion of it yet.
If/when such a replacment comes along, Pan will support it too and add an are-you-sure dialog for yEnc postings.
As some of you might already have noticed, the author of yEnc responds to several of the points raised in his FAQ (scroll down to near the end of the page). His replies (to questions such as "Some people say that yEnc is badly designed and was rushed without proper discussion. Is this right?") seem quite realistic and sensible to me, and I get the feeling some of the accusations levelled at Jürgen in the article weren't entirely fair.
The problem is that you've never actually tried what you're proposing.
I don't know if broadband modems support compression, mine certainly doesn't, or the compression sucks tremendously.
I made a simple test. I created a file with random data and uuencoded it. I then transfered these two files through my ADSL modem and measured the transfer times. The uuencoded file is 38% bigger, and surprise, surprise, takes 38% longer to transfer.
Maybe it would be a good idea to actually try using the things you talk about in the future.
A shame his article isn't better written. It seems Nixon has some good arguments as to how to make yEnc better, but is having trouble making them.
...
Nixon seems intelligent, but he doesn't want to understand that his problems, as an admin, are different from those of his users. To quote:
And the bandwidth savings? That's an illusion. A smaller encoding scheme gives us exactly one benefit: faster downloads and uploads for the users.
... and
The problem here isn't that we need a smaller encoding scheme, the problem is that we need a better way to post binaries on Usenet.
Well not really. What Nixon wants is a better way to distributed binaries on Usenet. Based upon the popularity, what the users want are faster downloads. That "exactly one benifit" seems to be pretty signifcant to them. As a admin, I'd guess that Nixon has pretty much high-speed access almost all the time. In that type of an enviroment, it's easy to forget the pain of life with a modem.
That disconnect is further demostrated by the iron-clad assertation that Usenet, in it's current form, is near perfect:
What was so broken? Nothing. Usenet was working just fine, and people were posting and downloading binaries just fine.
To me that conveyed a very unwelcoming aditude to anything that might rock the boat. Nixon feels everything is okay right, but it seems a lot of users seem to feel there is room for improvement.
More progress is likely to be made if the admins stop and understand why users are eger to adapt yenc instead of trying to dismiss as ignorant why they do so.
-Bill
SlashSig Karma: Excellent (mostly affected by moderatio
Incidentally, many people in alt.binaries.news-server-comparison (one of the places Jeremy hangs out) are very opposed to PAR files and say they cause an increase in bandwidth. I don't know if Jeremy's one of them.
An excellent idea would be to make a new format where yEnc and PARs are combined. Imagine this: get rid of RARs, PARs and SFVs altogether. Post a file as one big file and have 1% parity. Tbe parity will be for individual posts (not enire RAR archives like before). 1% is more than enough since a 500 message post will have 5 parity messages. Most incomplete files only have about 3 or 4 missed messages and the 5 will fix those INDIVIDUAL POSTS. If that isn't enough, a few more parities are generated.
I find this idea MUCH more important than silly nitpicks on implementation. Of course, this can eliminate yEnc from the face of the earth if this was implemented, simply because it makes downloading binaries much much easier and simpler.
I've emailed Jurgen about this, but his reason was that it'd make adding in this way too complicated for news client maintainers.
Those who want to push a MIME format should implement this and create a new standard, everyone will jump from the yEnc bandwagon onto this like they jumped from the UUE bandwagon to yEnc.