Usenet Encoding: yEnc
Motor writes "Anyone remotely interested in usenet binary newsgroups must have noticed the spread of yEnc. yEnc is an encoding scheme for usenet binaries which avoids the enormous (30-40%) bloat associated with the schemes currently in use - which all have to produce 7-bit data to stop ancient newsservers from choking. A good thing, surely? Well, not according to some people. The guy has some good points about yEnc and standards, but I can't help thinking that "standards" people have endlessly discussed better encoding schemes, and nothing has come out of it. yEnc may not be perfect, but it works and it's here - hence the rapid adoption. What do you think?"
The article points out some interesting points why yEnc shouldn't be adopted... none of which will probably keep the community from adopting it, however. If it's here, and being used, that is a whole lot more intertia than common sense can usually gain. Er, betamax, anybody?
In fact, Yenc will help pay-per-gigabyte Usenet users achieve a greater bang for their buck. Anything that saves money is a good thing!!
www.lonseidman.com
instead of breaking the standard, code it right, wait till it matures, use it then.
"It's such a fine line between stupid and clever" -- David St. Hubbins, Spinal Tap
There are some proposals in thee XML and Web Services arena for dealing with some of the problems tha yEnc is skirting.
One, called DIME, is a MIME-like system that handles binary content, chunks, etc.
http://www.oasis-open.org/cover/dime.html
I had scripts that automatically got some a.b* newsgroups, but after the invention of this bastardized yEnc piece of crap, all my scripts are broken, and I'm 2 months behind on data for our clients.
:) Only e-biz that's thriving still.
BTW, I work for a pr0n site
Forte released Agent 1.91 2 days ago with yEnc support. it looks like Mr. Nixon is fighting a losing battle.
NOTE: This is actually a question, not a troll
Does anybody really use usenet anymore? everytime i've poked around on my ISP's NNTP server, it seems to be filled with 90% spam, and non-spam posts seem to always be grossly offtopic. And no, i'm not just talking about the alt.binaries groups either.
This is not the greatest sig in the world, no. This is just a tribute.
There was a market for this thing, it spread like wild fire. It's too bad that no one made a better spec and program (the author aludes that there was planty of time to do this). yenc meets the "GOOD ENOUGH" criteria, thus it will be used, shitty, non-robust standard or not.
I'm all for standardisation... but sometimes it takes _forever_ to get something standardized. If someone writes a better product, they generally don't want to wait for it to be declared a standard, especially with something like uuencoding which has been around as long as usenet, and isn't going to be replaced in a hurry unless someone comes out and waves a product around yelling "hey try this. it works better". Ogg Vorbis isn't a standard by any means. Hell, it is still on RC3. _but_ a lot of people are using it because it has far better sound compression than mp3. You don't hear people complaining that Vorbis has jumped the standardisation process do you?
Personally I can't see why we can't just send the data as 8-bit binary. uuencode and similar encoding formats should have died out with UUCP years ago, since there is no physical reason why 8bits can't be sent over the wire anymore.
Uuencoded text will compress down to nearly the same size as its corresponding binary (or less, if the binary can be compressed). That kind of compression is now a standard part of modems, Internet protocols, and many file systems. Even the CPU overhead of compressing and decompressing that kind of data is negligible. If yEnc doesn't end up using less space on disk and doesn't end up using any less bandwidth than uuencode, indeed, "why encode" in yEnc and break a lot of software that expects USENET posts to be text-only?
I am seeing smaller binaries as a result of yEnc. This is fine. The problem is, my favorite binaries grabber has no idea what to do with the files, and won't even download them. I figured out how to make Agent download them, but A. I hate Agent (and don't understand why anyone likes it!) and B. the binaries don't always decode.
As I'm a lifetime lurker (well eight years, but it seems a lifetime!) I can only choose not to download yEnc encoded binaries. And no one will know! (my news server doesn't log downloads) It's all up to the posters to adopt or not.
I second this. Also, let's make it so that non-validating and just generally malformed XHTML documents are rejected by browsers, and can be filtered out of google. plain HTML =
I'd just love to turn on a "[ ] Reject non-validating pages" option in google and see the world wide web with new eyes :-)
Belief is the currency of delusion.
Despite its problems, XMODEM took off because it filled a need, just as yEnc does. Nixon's complaint that shrinking files by 35% won't make Usenet any smaller because people will just post more files is besides the point; it's like saying getting a 35% salary increase won't help your finances because you'll just buy more stuff with the extra money. Most people want that extra 35%, and Jürgen stepped up to the plate and delivered it.
Thankfully, as far as I know, nobody railed against Ward Christiansen the way Nixon does against Helbing. XMODEM's problems became obvious and the solution was to introduce YMODEM and then ZMODEM. XMODEM is still around, but its successors (and of course serial IP) have pretty much supplanted it. Ward's initial efforts are still deeply appreciated.
Yes there's the problem of legacy software, but a protocol that's only been around for a few weeks or months can't have that much of a legacy. The only programs that currently support yEnc are the ones whose maintainers react pretty fast to new developments, and those maintainers are likely to also quickly pick up any revisions/fixes to yEnc.
So the solution Nixon should be calling for is not a years-long bureaucratic standardization process that will get yEnc 1.3 entrenched while the standardization is happening. The solution is to fix yEnc's problems and release a new version as fast as possible, before the old version gets spread around too widely.
Pan supports yenc to name one, lots of *nix tools available to decode, encode and post. Don't want to change software? You can still use tin to tag all the parts save them to disk and use an extenal decoder. This was the only way before with many *nix news readers. So not much change really just a different filter to use! Yenc like another much loved binary tool "par files". Seems to have been available on *nix first not second.
Is it good is it bad? To late its here.
James
Hey, if this guy adopts it, he could save slashdot some money...
-Sean
What he's whining about is that it didn't fix every other problem in addition to overhead, and if anyone should actually bother making some huge new mime standard, now they won't have that carrot.
Obviously, if the rest of the problems were as big as he's trying to claim, yEnc would only be a minor setback for a new and more comprehensive standard, but the fact is that the 35-40% overhead of current standards is by far what's most annoying to usenet users. After we got PAR (parchive.sourceforge.net), reposts have been reduced drastically (except for pr0n and partly warez groups, where the dumb people with shitty servers rules).
Also, he's trying to say that because the increase in volume will outgrow the savings, there really is no savings. What kind of logic is that? Let's stop making processors faster, we'll just find bigger problems for them to be too slow for anyway, so what's the point?
After the introduction of PAR and yEnc, as a long time binaries downloader, I'll say the actual content of multimedia groups has more than doubled, and probably tripled, the last 6-9 months. That's progress to me.
In one sentence, standards ARE important because they allow for the most people to get the most benefit.
I work in an industry that relies heavily on standards, and my job deals specifically with standards. Making sure that WE follow standards, and making sure that other vendors follow standards.
Sure, they're slow to develop. But they're the best for interoperability, and that's crucial. In my line of work (for a major Mobile Phone System NSS provider), I have to deal with other providers that have to follow the same standars we do. That allows both of our products to communicate. This gives the end consumer (i.e., Cingular, Sprint, etc.,) the option to buy from different vendors. This forces us to make better products. This forces us to be more efficient. This forces our competitors to do the same thing. In the end, everybody wins.
The other alternative is what I see as the Micro$oft approach: Standards be dammed, I'm going to do it this way, and f*ck everybody else. It's the same approach that gives you security holes in your browser, because, well, who needs the standards?
I can't believe I'm reading comments like "well, it's here and it works so what's the problem?"
The problem is the future.
The problem is the inability to send an SMS from a CDMA service like Sprint to a GSM one like Voicestream. That's what happens when you blow off standards.
The problem is the inability to read an M$ Word doc that was sent to a Linux user.
Ignoring standards and going off on your own (especially, going off BADLY on your own) just divides us.
Good standards help us all. They give us better products. The lower costs.
CD-Rs. FireWire. PCI. countless others.
Besides, as the article begins by asking: Just what problem were they trying to solve?
Watch the Teaser Trailer for "The Lightning Thief" Her
Sig: What Happened To The Censorware Project (censorware.org)
The big savings on binaries is coming from .PAR files.
If you don't know what they are, then you haven't been on usenet for a while.
But essentially, it allows you to stripe you sets with parity so that you can lose up to "n" posts and the PAR programs can rebuild the missing pieces.
I believe this has helped the backbone tremendousl.
And Pan's got it too. Tastes great, less filling!
I wonder if it's in CPAN yet...
Module Convert::yEnc (P/PN/PNE/Convert-yEnc-0.03.tar.gz)
Yep. Works for me!
When you know it's a uuEncoded file, it should be very easy to write a speedy compression algorithm. Even just use a standard Huffman algorithm - it was originally designed to be implemented by hardware for storage devices (think tape drives) and wouldn't impose any real overhead on a modern server. I just can't see why we have to implement a new standard before it's ready.
Willy
There is no reason to despise magic strings. They work, and cannot ever occur in the user data. All yEnc magic strings start with =y, = being the escape character. Ctrl-Y does not need to be encoded, so yEnc is free to use =y for it's own purposes (e.g. =ybegin, =yend). Jeremy Nixon continues his misled rant...
No, using the subject line is not obviously a terrible way to determine filenames, segments, and anything else. I find it very convienent to know exactly what my yEnc files will be saved as, how big they are, and how many parts they are in inside the subject line. Nixon says "Sure, it works out most of the time, but it is imprecise and error prone (especially when spaces are used in filenames)" This is blatently false nonsense. Quotes reliabily allow clients to discern the filename. It's not "imprecise and error prone" by any stretch of imagination.
I give them that. Non-USASCII data in headers is a pain, and a large powerful organizational bodies needs to agree on a character encoding standard. Oh wait, they already did - Unicode!
False again. I've never had a filename containing quotes on my Windows box. If we expect newsgroups standards to reach everyone, we must use the lowest common denominator. Similar to how ISO9660 used 8.3 filenames, but on a higher level.
Which is exactly what the creators of yEnc intended.
They mean "AOL users" of course. Usenet hasn't had a new encoding format in 6 years, it's about time. Adopting this format should be as easy as switching from Napster to OpenNap to Morpheus to Grokster to Blubster and so on.
I don't blame him. Jurgen is a coder, not a politician. I would have done the same thing.
In short, yes I agree yEnc needs to be more polished. But the point is it works right now, and it's working great. It filled a gap in Usenet, itched a stratch to borrow an ESRism. Once yEnc is standardized as Y.32049 Annex D or whatever those standard organizations call it, we will use it. Until then, yEnc forever!
What do you think of MusicCity now?
While I quite agree to your "screw the luddites" idea, and I think yEnc is a progress, I'm not sure if HTML postings in usenet can be classified as a "progress", especially when this trivial idea does not need any imagination to accomplish. "Yay, we have HTML, let's slap it onto everything!!"
Not only isn't HTML postings an important "technical advancement", not everyone thinks HTML postings look better, either - surprise.
I, for one, deliberately killfile HTML postings and filter HTML emails because I don't need the funny colours that distract contents, and the security concerns associated with scripts, images, "runnable" postings and cookies.
Plus, every single piece of HTML posting and email I've seen are spam anyway. Why bother?
If you want to post HTML contents, there is a place it should be done and it is called "the web". I don't complain HTML postings in web pages because it is what the HTTP was designed for - and there are already many decent methods built in or around an application called "browser" to protect yourself from security hazards and spams. I cannot say so for newsreaders, at least yet.
The new modem standards apparently even have special compression modes for HTML, and if uuencode and base64 matter, modems and other compressors could recognize them as well.
That's great ... if you use a dial-up modem. Personally, I think any more incentive we can give to stamp out dial-up modems, the better.
Sometimes it's best to just let stupid people be stupid.
Personally I don't think someone who wants to be able to ssh into their shell account from work/etc and read USENET, rather than rely on (easily monitored) Google, or who doesn't want to run up a large phonebill downloading approximately 5-6x as much data as necessary, or who finds that there's no way of GUIfying the newsreader they've found works best for them, is a "luddite." I'd say they're using their common sense. And I'd say someone who describes them as a "luddite" and "living in the past" is someone whose respect for others is demonstrably lower than ought to be necessary for posting on Usenet. If there was a Usenet "licence", I'd argue that the name callers and bandwidth hogs ought to have their's revoked.
But YMMV.
Being on the Internet is about accepting standards. Some of those standards may not be perfect for you, but it's certainly not impossible to stick by them, and you can make your point easily enough and still follow them. Most people do. Get over it.
You are not alone. This is not normal. None of this is normal.
Not only isn't HTML postings an important "technical advancement", not everyone thinks HTML postings look better, either - surprise.
So should the web return to the days of pure ASCII? Should Slashdot disallow any HTML postings? HTML is useful. Look at how it used on Slashdot -- italics, boldface, links, etc. Not to mention that proportional fonts are infinitely easier to read.
I don't complain HTML postings in web pages because it is what the HTTP was designed for...
How can you possible be in favor of HTML on the web, yet not in favor of HTML in Usenet? Put it this way: If Usenet were invented today and they included HTML, could you honestly say it would occur to you to say, "you know what would make this better? ASCII only!"
I cannot say so for newsreaders, at least yet.
Depends on your newsreader. If you're using a Windows newsreader, it typically uses the IE COM component to display the HTML.
Sometimes it's best to just let stupid people be stupid.
So if straight ASCII is so great, why do you use HTML in your Slashdot posts?
Grumble mutter bah humbug
Your sig is particularly appropriate for your post.
Sometimes it's best to just let stupid people be stupid.
Lets see...Napster is dead. Morpheus is dead. Morpheus part 2/gnutella is a zombie (perhaps it's just me - I have yet to have it actually pull down a file for me. It keeps telling me the other end is busy or something.) Even when they were alive, you get half way through and someone cuts you off...or you find out that 80M download that took a whole day was actually mislabled.
On Usenet...sure - you don't get to search and finding a file involves posting a request and hoping someone fulfills, but you get bandwidth - assuming you want to pay for it...you get files that are there (assuming you have decent retention.) and not dependant upon someone being online. And unless you have a crappy server, you don't get halfway through a download and someone decides to kick you off. And 99% of the time, what something is posted as is what it is.
As for the replication - well, there is no one point of failure. As well, you don't have one site getting the shit hammered out of it either. I pay $9/mo for usenet...I get three fast servers to choose from and some high GB limit.
If I had to go to the same server as everyone else, you'd have the same problem that moviefone had when star wars tickets went onsale online - DOA - all with nice corporate control of content.
It should be pointed out that this site, linked from yENC's own website, goes into more technical detail regarding the technical flaws of yENC. The fact that it's linked from yENC's own site is proof that the author is at least familiar with the concerns that people have with his implementation.
I personally still find it difficult to argue against the article author's point that THERE WAS NO RUSH to force yENC out the door in such an unpolished form. After so many years of waiting for something better, why ignore the recommendations of those you are trying to help?
< tofuhead >
It is still the dark of night.
The key theme here is that people on usenet whine. A Universal Truth, as it were.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
1. Slashdot is only accessable from web browsers.
2. Slashdot is an HTML forum.
3. HTML is fine. Posting it to Usenet isn't. See comments about cost of downloading, newsreaders, standards, etc.
Stop setting up straw men.
You are not alone. This is not normal. None of this is normal.
Well then. When I put that page up, I honestly didn't expect many people to read it outside news.software.nntp and a few curious folks in alt.binaries.news-server-comparison. I certainly wan't expecting to get Slashdotted. Well, that's fine, except that the uproar might have waited a little bit.
In my essay, I state that what Usenet needs is "a better way to post Binaries". The next piece of the puzzle, of course, is to answer the question, "What IS a better way to post binaries?" I was thinking about finishing that page up tonight, but I am writing code at the moment instead.
So, when reading my comments, just keep in mind that, yes, I DO have some answers to that question, too. It's just that it's a bit of a more time-consuming question, so that page isn't done yet.
This time around, though, I will make sure to include a prominent warning to NOT run off and implement the ideas as quickly as possible, and to please not use all of Usenet as beta-testers. The idea that whatever gets done fastest is best just doesn't work for me. There were good reasons I didn't go and get people to implement my smaller encoding ideas when I first wrote the code. If only the yEnc implementor had continued where I left off rather than going down his rather misguided path...
All the comments are welcome. I've been getting some interesting email, too, of course. Many programmers of Usenet client software absolutely despise the thing and are quite annoyed at the amount of their time it is wasting. I guess it's just more of that never-ending divide between the users and the techies. So it goes.
yEnc is here, that's for sure. Now we just have to try to deal with it.
Jeremy
Either it is /.ed to hell or the server is having major problems (not that they are exclusive). A lot of people are having the download stop at 10-15%. All this for a simple 2 meg file. If someone already has it, throw it to a binary group and provide a link.
Cave, wreck, and deep diver.
I'm not sure if it is still true, but I know that Jeremy Nixon (the author of the article) worked at Supernews (now ReMarq) as one of their chief engineers. Not to be jaded, but it stands to reason that he would be against a technology that will decrease the data transferred by customers who pay by the gigabyte.
Stop setting up straw men.
No, it's not a straw man. Look past your "it's always been done that way" attitude, and think about it. Why is a Slashdot HTML forum "good" and a Usenet HTML forum "bad"? History is not a reason. Standards are meant to be enhanced.
Sheesh, some geeks are worse than PHBs on this issue. It reminds me of that shipping company commercial where the boss drones on about the fact that "we've always shipped this way. We will always ship this way" while the young employee stands there mocking him.
Why do I have a feeling that all the old farts are going to have to die off before this issue goes away?
Sometimes it's best to just let stupid people be stupid.
Liberty in your lifetime
If, by any chance, you're transferring things over a modem (v42bis' lzw) or ssh vpn (zlib's deflate) or possibly other types of links, then you're probably not going to notice a difference anyway. The systematic encoding inefficiency that goes with base64 and uuencoding, results in a substantial lack of entropy that will be picked up on and exploited by good compression algorithms. Then end result won't be quite as good as having efficient encoding to begin with, of course, but it will be in the same ballpark. There's no way it'll be anywhere near a 33% difference.
This sounds like something that would have been useful 15 years ago before compression was widely used, and when people were still writing newsreaders. Now it looks like a waste of time and an excuse to get people to "upgrade" their software.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
In the case of yEnc, someone found a problem and (Freely, as in public domain even) offered a solution if people wish to adopt it. People are adopting it. Progress is happening.
And then I hit:
And I can't help but think "wow - what a jerk." Why? Not because I don't have newsreaders and email clients capable of handing HTML based messages. Not because I dislike HTML itself (open standard - yay). But because HTML seems to be completely unneccisary in these environments.
Now... sure. There are some forums that might bennifit from it. Times where typefaces and bulletlists, etc. really add value to information. Maybe even times where some forms of information NEED this technology or it becomes very difficult to portray. But I rarely see it.
Instead, HTML based messages in Email and Usenet tend to increase the overhead with no real added value to the content. In some cases, they are used to attempt various shennanigans such as web-bugs (not to mention worms, etc). Little wonder tech-heads dislike it.
Change must happen. But when you run around trying to force change for change's sake rather than to solve real problems, then you simply become a problem yourself.
Because people with premium news services (AKA, ANYONE that's serious about large binary downloads, the people at which yEnc is aimed):
- have megabyte quotas, both upload and download
- pay by the megabyte for their downloads
Also, this saves space on the server hard drive. NO WAY are usenet servers compressing data on their hard drives. It's one of the most challenging situations for a hard drive, they're not going to wreck their performance by using compression. Having less data means more retention on the server.
I personally have a Newsguy extra account, grandfathered at 1GB per day. I EXHAUST MY QUOTA by about 10AM, most days. I still do, but by then I've gotten 30% more data.
It's not all about transfer speed. yEnc means people with download quotas get 30% more stuff per day.
And until you do, and perhaps even come up with practical solutions to these problems, I don't plan to take you seriously. I doubt anyone else does either.
You are not alone. This is not normal. None of this is normal.
In fact, lets kill off mime altogether, and get this multimedia crap out of our inboxes. Email/Usenet is for plain text, period.
Yes, because everyone who doesn't speak English should just shut up. You did realize that MIME is the only standard on how to send non-ASCII plain text across email, didn't you.
See my other post in this thread. It's not all about dl speed or even drive space. Most serious usenet large binary downloaders use a premium news service, and they ALL have download/upload quotas and/or pay by the byte schemes. yEnc gets you 30% more for your buck on those servers.
I was going to post a long technical piece on why what you're doing is wrong, but, essentially, it all boils down to the fact you're just a prick.
Let's have some moderation (not the select a category, but some moderate views) here. I agree that <i> and <b> can enhance a posting. What the other poster was commenting on, and which I agree (since I often connect via modem, often 28.8, sometimes even 14.4)is that excessive HTML is useless. It doesn't add to the content, it just gives it a little extra pizzaz, which is not always good.
-- Is "Sig" copyrighted by www.sig.com?
Thanks for the tip, but I've tried that too (in fact, they resolve to the same IP). The last line of the traceroute is:
16 so-1-3-0-0.SVCS-RTR1.RES.verizon-gni.net (141.156.255.46) 97.166 ms !A 97.214 ms !A 97.336 ms !A
Where the !A means access to the host is prohibited. (I'm going to have one hell of a time trying to convey this to a tech support moron.)
python -c "x='python -c %sx=%s; print x%%(chr(34),repr(x),chr(34))%s'; print x%(chr(34),repr(x),chr(34))"
Sheesh, some geeks are worse than PHBs on this issue.
You propose a method that many people can't read,
and while it's supposed to look better if you can read it, it frequently just looks like over-elaborate crap.
Geeks believe in content over packaging, at least more than your average person. HTML offers very little in an email or newsgroup environment, where messages are quickly written, but often come with huge overdone packaging. Geeks also like to use their computers via text links, and HTML is not friendly to that. Somehow it should not be surprising that they aren't happy with HTML in those enviroments.
So should the web return to the days of pure ASCII?
And when was that? You do know that HTML is an intregal part of the WWW, don't you?
How can you possible be in favor of HTML on the web, yet not in favor of HTML in Usenet?
Because the newsreader I use doesn't know what to do with it and the way a lot of the people who use HTML on Usenet do it, it doubles the length of the post.
Put it this way: If Usenet were invented today and they included HTML, could you honestly say it would occur to you to say, "you know what would make this better? ASCII only!"
For a medium that outside of the binaries newsgroups was designed to post text messages? Why wouldn't ASCII only be better for plain text? I actually think a lot of web pages have gone overboard and have worsened the bandwidth/content ratio considerably. There's nothing wrong with lean and mean.
If you're using a Windows newsreader, it typically uses the IE COM component to display the HTML.
While allowing other possibly malicious components to wreak havoc with your system - a problem that never happens with plain text messages, does it?
Go ahead. Post HTML on Usenet. But you should know if I read it, it's going to look like a hard to read text message with a lot of bracketed garbage and I'll be more likely to skip over your posts in the future.
I've heard this "is usenet dead" crap one too many times. Comp.lang.python isn't dead. Soc.religion.quaker isn't dead. comp.protocols.tcp-ip.ibmpc isn't dead (okay, so it doesn't have a lot of traffic, but the principals read it, which is what matters).
-russ
Don't piss off The Angry Economist
Can *anyone* look at the uuencoded, mime encoded, and other similarly mangled into 6bit, 70 character-per-line standards, and honestly tell me that Usenet was designed with binary file transmission in mind?
There are no Usenet binary transmission standards, just a few different hacks to make it work. If this guy's new hack makes it work better, good for him.
Discovery, that's why. Napster (Napster?, talk about dead!) and the like are great for finding a name you already know. By design, that's all it can do. Do a search for "early '80's Boston punk" for example and see what you get. This is where Usenet excels. Enthusiasts congregate into groups and upload files they enjoy, meaning a far better chance that I'll enjoy them too. I'm continually discovering great bands through Usenet that would remain unknown had I relied on name-search based file sharing.
Oh no! slashdot is beautifully rendered in a textual browser such as lynx and lightning quick over a 9600 baud cellular modem. If you never tried viewing this site under lynx, you may be surprised at the artistic detail of the formating.
Over to the left where the links take the shape of an ascii candle flame followed by more links presented in an intuitable format. Rob was generous leaving this site accessable to the more mature historical browsers.
I lost sympathy about here:
A smaller encoding scheme gives us exactly one benefit: faster downloads and uploads for the users. It is not going to make Usenet smaller. It is not going to allow servers to increase retention. Do you really think people aren't going to post more, if they can do it faster? Of course they are. They're always going to post more, with or without yEnc [...] big deal.
So effectively, what he's saying is, in effect: "this system changes nothing, and is of no benefit, except that it makes more data available on the Usenet and gives users faster uploads and downloads. So it's worthless."
This guy obviously hasn't had to use a metered dial-up account for a while. A 33% saving on transfer times is an enormous benefit. I feel quite insulted by the way he seems to think it's of no importance, as if my time and money aren't worth anything. "What's the rush" indeed! I'd happily tear up MIME and MD5 tomorrow if it would speed up my transfers by a third.
If yEnc is so widespread, it can only be because there's a demand for it. And if there's a demand for it, why the hell shouldn't programmers support it? Last time I checked, RFC's weren't enforced by law. The Net has seen a million non-standard hacks, and has, for the most part, assimilated the good ones and outlived the bad. yEnc is by no means the worst, and it brings real benefits to tens of thousands of people every day. I say leave it alone - or if you have to oppose it, at least oppose it constructively, for Christ's sake!
How do you reconcile putting in this assertion: [etc]
Because it's absurd to argue about features in an application (i.e., Slashdot, Usenet) based on what protocol it uses. Either a feature is useful, or it is not. You can't have it both ways -- either HTML is good for discussion groups (Slashdot OR Usenet) or it is not.
Sometimes it's best to just let stupid people be stupid.
If everyone is as anal as the poster of this story and cares about elegance and "pretty code" and whatnot, we all would not be using a x86 processor because its architecture is just "sooo damn ugly!!". Get over it. If it works, it works, does it really matter how anything is implemented if it does its job just perfectly fine?
:) Real life is not pretty, deal with it... stop living in fantasy world.
Personally, I will adopt anything that will make my life (not the developer's life) easier. yenc allows me to download 30-40% less, so I use it. Why? because if I don't use it, I won't be able to obtain the mp3s, the games, the warez, the pr0n, etc, etc, etc that I wanted
Most progressive newsreaders group multi-part binary posts together pretty well. That really isn't a problem at all. And the reason for the multi-partness doesn't have to do with a particular encoding scheme, but with the line number limit many servers impose on posts.
"(Man) tries to live his own life as if he were telling a story. But you have to choose: live or tell." --Sartre
... the implementation sucked.
I leech regularily from alt.binaries.anime and the related newsgroups. When the yEnc posts started coming in, I simply upgraded my newsreader to the newest version. But a LOT of people out there use Agent, and it was absolute pain to combine/decode all the yEnc posts that started popping up all over the place. The worst of it is that the yEnc posters were basically saying, "Start living in the present and upgrade". Nevermind that at the time that only yEnc-capable newsreaders were for Windows...
I mean, I don't know, but this sounds a lot like the OS wars that have been going on for quite some time. Some people simply don't want to have to switch newsreaders. Some people don't want to have to switch OSes. And that's fine, because it's a free world out there. I like the idea of yEnc (I get more out of my Easynews account), but I really don't think it should have been introduced so quickly.
~ Firecaster ~One: yENC, when it was unveiled, did not really allow most conventional newsreaders any opportunity to adapt, til after the fact. This is akin to perhaps releasing zip files long before any archival software was actually available to open them... So do most of the folks using usenet for binaries get the opportunity to at least *choose* the way they do their downloads? Nope, they also are forced to adapt, or lose out...
Two: Loss in transmission... I've been downloading yENC attachments for the last month, and out of them, found over 50% loss/corruption in posting... Not due to retention/propagation either... Just files missing large chunks... Now this *could* be due to some problems on the senders' end, but it seems just a little *too* coincidental that almost all of the losses have occured with yENC uploads...
Just because you can mod me down, doesn't mean you're right. Shoes for industry!
either HTML is good for discussion groups (Slashdot OR Usenet) or it is not.
One huge difference between Slashdot and Usenet is that Slashdot only permits a subset of HTML tags in posts. If Usenet HTML would limit itself to a similar subset, then it would be much more platable.
Also, Slashdot has already limited itself to those who can handle HTML, so why not permit HTML in posts? Usenet is and always has been available by plain text, so HTML gets much more scrutiny.
In particular, there is no yenc RFC and yenc does not use MIME which is the agreed upon standard for encoding binary attachments. Yes, uuencode is a gross grandfathered format, but it is still 7 bit clean.
Releasing problematic improperly specified encodings that break internet protocols is not being a good citizen. "it works" is a poor justification. it does not work, and breaks compliant software.
-Kevin
This just reminds me of the napster data format. Anybody ever read the reverse engineered specs? It's scary. It looks like it was designed by a monkey. And not a smart one.
yEnc sounds like a good idea, and a horribly bad implementation.
And how many "ancient" servers are really handling the (huge-traffic) binary newsgroups where yEnc encoding is appearing? I would have thought old servers were made for much lower news volume and couldn't handle the load.
I'm not a news guru though. I just read the stuff sometimes.
He didn't. He thought it had its place, but that place wasn't Usenet. Phenomincally easy to understand, you pretended - you lied, if you like - that he was arguing that HTML was an evil by itself.
*sigh* Of course, it has be a problem with me ("lied"??) -- it couldn't be that you're not understanding the point I'm making.
The point is that he recognizes the advantages of HTML on the web and on Slashdot, yet is unwilling to consider those advantages for Usenet. I turned the argument around -- if ASCII is so great for Usenet, then why not all ASCII for the web?
Both the web and Usenet are publication protocols. They just go about them in very different ways. I find it very useful to be able to italicize, boldface, embed links, have lists, etc in my publications, not to mention the advantages of proportional fonts.
Sometimes it's best to just let stupid people be stupid.
But yEnc's bandwidth savings are real, which is a huge win for alt.binaries users. yEnc has been the most-requested feature for Pan over the last month. (0.11.2.90 supports it.) IMO yEnc is the format to use for multiparts right now.
Hopefully yEnc will motivate others to come up with a mime-friendly alternative encoding for Usenet. yEnc Considered Harmful is another yEnc opposition page that suggests mzip compression, but I haven't seen any public discussion of it yet.
If/when such a replacment comes along, Pan will support it too and add an are-you-sure dialog for yEnc postings.
It used to be that someone did something useful, then the community, through use choices, adopted it as standard. Then, if there were flaws, these would be ironed out with an updated standard, usually all or mostly backwards-compatible with the original implementation. It's gotten to where new standards are useless, either because companies (like, say, RealNetworks or MS) refuse to submit their protocols/formats for public use/review, or because the standards committees (say, for Java (before it was pulled) or the W3C) argue for years without actually doing anything.
I, for one, am happy to see a useful format publically available.
-- Two men say they're Jesus. One of them must be wrong. - Dire Straits
The point is that he recognizes the advantages of HTML on the web and on Slashdot, yet is unwilling to consider those advantages for Usenet.
What's the major advantage to HTML on the web? You can link to other pages on the web, knowing they're there.
Now how would one use HTML to link to other posts on Usenet and be sure they're there? Could you write a post linking to several later posts you were planning?
No. The only thing you can really do with HTML on Usenet, seeing as binaries such as images are prohibited on text groups, is to present text. (Unless you're running malicious scripts.) And guess what? You don't need HTML to do that - you don't even need it to post web links, as one can just copy and paste them into a browser.
I turned the argument around -- if ASCII is so great for Usenet, then why not all ASCII for the web?
Because, dumbass, it couldn't be a web, then, could it? Maybe, before telling us how Usenet should work, you ought to learn how the web works first.
I'm glad you agree that Microsoft Word is proven to be one of the best programs in existence. =]
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
As some of you might already have noticed, the author of yEnc responds to several of the points raised in his FAQ (scroll down to near the end of the page). His replies (to questions such as "Some people say that yEnc is badly designed and was rushed without proper discussion. Is this right?") seem quite realistic and sensible to me, and I get the feeling some of the accusations levelled at Jürgen in the article weren't entirely fair.
"Whatever kind of "confusion" this guy is referring to is nonexistent. yEnc downloading and decoding with the right software is as transparent as uudecoding. Promise. People don't have to relearn anything."
Well people do now have to think of yENC's longer lines. Where before my ISP newserver let me post 5000 lines of uue material now I only get 2280 with yENC. Longer lines under yENC. That's why Agent 1.91 has such a small line length as the default. Yes the payload is the same but you have to think a little different or just rediscover your servers limits.
A shame his article isn't better written. It seems Nixon has some good arguments as to how to make yEnc better, but is having trouble making them.
...
Nixon seems intelligent, but he doesn't want to understand that his problems, as an admin, are different from those of his users. To quote:
And the bandwidth savings? That's an illusion. A smaller encoding scheme gives us exactly one benefit: faster downloads and uploads for the users.
... and
The problem here isn't that we need a smaller encoding scheme, the problem is that we need a better way to post binaries on Usenet.
Well not really. What Nixon wants is a better way to distributed binaries on Usenet. Based upon the popularity, what the users want are faster downloads. That "exactly one benifit" seems to be pretty signifcant to them. As a admin, I'd guess that Nixon has pretty much high-speed access almost all the time. In that type of an enviroment, it's easy to forget the pain of life with a modem.
That disconnect is further demostrated by the iron-clad assertation that Usenet, in it's current form, is near perfect:
What was so broken? Nothing. Usenet was working just fine, and people were posting and downloading binaries just fine.
To me that conveyed a very unwelcoming aditude to anything that might rock the boat. Nixon feels everything is okay right, but it seems a lot of users seem to feel there is room for improvement.
More progress is likely to be made if the admins stop and understand why users are eger to adapt yenc instead of trying to dismiss as ignorant why they do so.
-Bill
SlashSig Karma: Excellent (mostly affected by moderatio
As a "pretty active Usenet poster and binary downloader," I bet you know your stuff pretty well. You could switch to anything in a short time, I bet. That's great, so could I. You're not looking at the average user, though.
Since I put my page up, my mailbox is overflowing with messages from people who are agreeing with me, but for the wrong reason -- they are saying, yeah, right on, this yEnc thing sucks, I can't figure the thing out and I can't get half the binaries anymore, how do we get everyone to go back to the old way?
If you just look at the people who know what they're doing and know it well, of course you won't see much confusion. But a popular binaries group easily has several thousand times as many people downloading as it has posting, so I think you are grossly underestimating the level of confusion this is causing among the regular users.
I'm not worried about confusing them; it's inevitable. What I'm worried about is confusing them too many times in a row.
Jeremy
Shouldn't nVidia, Bezos or British Telecom be stepping in about now to claim a patent on encoding 8-bit data as 8-bit data?
In the very early 1990 I was using subnet, which was assimilated by usenet meanwhile, with uucp and wazoo.
Even way back then there was vital interest in more efficient filetransfer. uudecode simply sucked. Everybody agreed to this.
But what did happen in the last 12 years?
Nothing.
There are now more powerfull uudecode-implementations, but I havent really seen anything practical invention in the basics.
I totally agree that yEnc is a quickshot with none thinking at all and its weird and so on. But the usenet-gods havent done anything wort mentioning in the last 12 years and they will not do in the next five years. They lost it and are now crying for not getting asked. Sad fate, but graveyards are half full of indispencable people and half full of people who dispenced with them..
"Life is short and in most cases it ends with death." Sir Sinclair
you may be surprised at the artistic detail of the formating.
LOL...man...
7-bit died BEFORE THE PC- That's over 20 years ago. I seriously doubt there are any PDP11s still on the internet, and AFAIK nothing else felt the need to pack three 6-bit chaaracters into a 3 character file extension. Apart from that, it was a 16 bit machine and did 8-bit chars. Almost all other 7-bit hacks were dead before the PDP-11 was even launched. (except POCSAG pagers - now there is a seriously SAD protocol!)
The best answer is dotn compress for transmission - let the modem/imodem/NIC/why do it IF THERE IS A SUITABLE STANDARD, and if not, let the appropriate standards committee fix it, cos they have the tools for negotiation a compatible standard at run time.
Sent from my ASR33 using ASCII
Speaking as previous long time user of CBBS ( the birthplace of Xmodem ), I will point out that Xmodem enjoyed a great acceptance in the BBS community prior to adoption of support for Kermit. Early versions of Kermit were extremely inefficient compared to Xmodem, but it was the only game in town if you were interested in tranfers to and from MVS mainframes. Kermit quickly ruled the domain of inter-architecture tranfers, while XModem, and it's derivatives quickly dominated the PC domain.
:->
And years later, there was the legendary KA9Q code.
"To those who are overly cautious, everything is impossible. "
I don't know who was first and I'm sure this is not the only other tool/compressor but here in Belgium, we've been using a tool called Bommanews (b-news.sourceforge.net) for a while now because of rediculous upload-limits by the local cable-ISP. We're not allowed to run servers (ports < 1024 are blocked), have un upload-speed-limit of 128kbit/s and only 15% of our total traffic can be upload so some guy came up with this to allow easy distribution of files through the news-server. Because it's so popular, our ISP finally had to setup a 2nd news-server just for binary postings as the other one was overloaded (serves them well).
As a Deja.com shareholder, I followed the talks between Deja and Google last January very closely. I learned many interesting things about company strategy, usage patterns, and targeted markets. But one thing that I'll never forget is this one chilling statistic: 98% of binaries traffic on Usenet is pirated software or illegal pornography (by volume), and 93% by number of posts.
Ummm, you are familiar with the three kinds of lies right? Besides, pirated software and porn (whether legal or illegal) posts can take up hundreds of megabytes and are blown up by about 33% by less efficient encoding schemes like MIME and uuencode. This is why their "volume" is so high.
For the first time, I truly felt appreciative to Deja for all of the efforts they expended in keeping this illicit content off of their fine service.
Actually, it was quite easy for them. They just didn't save any binary content on their fine service so it was easy to keep the illicit stuff off.
And just for the record, whoever wrote up that 93% pr0n and warez posts statistic wasn't focusing just on alt.binaries.pictures.aviation or alt.binaries.pictures.astro. I have never once encountered anything like what you'd see on goats.cx (something I alas can't say about
And that brings me to my point: yEnc sucks. yEnc is a horrible standard. Just like uuencode, it forces you to track down all 56 posts that constitute the 1337 warez release or all 7 posts of your kiddie porn movie.
And just like uuencode, most serious Usenet users can use yEnc savvy newsreader like XNews or Agent to track down, join, and decode all 56 posts of your illegal copy of Bloatware 2002 automagically.
Does this
Yes I did know it was available and it use it, and this unix was my point, I obviously didn't make it clear enough. People were saying yenc was a windoz thing. I was trying to say it is not, as par is not. Windos users were the first to get clicky clients, not working impletementations.
What happens when the posting is about yEnc and includes a description of how the encoding starts and ends? Does it know that that's not really an attachment?
TWW
"Encyclopedia" is to "Wikipedia" what "Library" is to "Some people at a bus stop"
You don't have much programming experience, do you? Magic strings are a dire way to do anything. MIME has been around for quite a long time now and was invented as a solution to all the problems of magic strings in messages, the obvious one being how difficult it is to handle a message about the magic strings (Outlook still can't handle these). MIME makes sure that the strings are unique to the message, not the class of attachment.
Data about the message should all be in the header. This is so obious that it is hard to imagine what you think the header of a message is for.
No, using the subject line is not obviously a terrible way to determine filenames, segments, and anything else.
The subject line is for the subject. What the fuck are you smoking? Why not the From line or the X-Envelope line? Because that's not what they're for!
find it very convienent to know exactly what my yEnc files will be saved as, how big they are, and how many parts they are
Your client should tell you that from properly formatted headers; the Subject line is for the subject.
If we expect newsgroups standards to reach everyone, we must use the lowest common denominator.
So, no yEnc then. You are arguing that we can't change anything but that we should embrace any half-arsed new encoding that comes along
Which is exactly what the creators of yEnc intended.
I think you are confused about who the creators of yEnc was; one aim of any new encoding should be reliability.
I don't blame him. Jurgen is a coder, not a politician. I would have done the same thing.
Class-level magic strings belong in the museum along with line numbers in source code; Jurgen isn't much of a coder if he doesn't understand this.
But the point is it works right now, and it's working great.
It sort of works sometimes right now, if the spec had been put in as a MIME type it could have worked well all the time in a few months. As you say, Usenet has waited 6 years so why not a few months extra and get it right?
TWW
"Encyclopedia" is to "Wikipedia" what "Library" is to "Some people at a bus stop"
This character sequence, of course, can appear in any text message.
Inform yourself before propagating such nonsense!
Claus
Forte Agent _finally_ has yEnc support in the product (version 1.91.)
My guess is Outlook Express will follow.
People will not have their pr0n and warez messed up to the point that they cannot download it.
As for the "war" these two seem to be having let me boil it down in schoolyard terms:
Nixon: It's mine give it back!
Jurgen: I already gave it away!
This
What's the major advantage to HTML on the web? You can link to other pages on the web, knowing they're there.
Wha'choo talkin' 'bout, Willis? There is NOTHING about the web that guarantees a page "is there". In fact, the whole "linked" nature of the web is totally an illusion. All you're doing is marking a piece of text as something that should be inserted into your browser's address bar if you click on it. There's nothing magic about a link.
Now how would one use HTML to link to other posts on Usenet and be sure they're there? Could you write a post linking to several later posts you were planning?
Who cares about linking to Usenet posts? Just being able to link to web sites would be an advantage.
You don't need HTML to do that - you don't even need it to post web links, as one can just copy and paste them into a browser.
As you can on the web. Who needs links when you can just cut/paste into the browser? After all, that's all a link does anyway. The point is that it's a convenience, which is just as convenient in a Usenet post.
Maybe, before telling us how Usenet should work, you ought to learn how the web works first.
Perhaps you should take your own advice.
Sometimes it's best to just let stupid people be stupid.
Also.. are there hundredsof thousands of news servers nowadays? I think we are dealing with smaller numbers of larger servers now.. with news services like newsfeeds.com and such becoming increasingly popular hubs for usenet.
Just thought some of you Mozilla fans might be interested to know that there is already an open bug in bugzilla 119964 to support yEnc in Usenet postings.
Recent posting indicate that some of fellow Moz contributors may be taking this issue and fixing it ahead of the planned "future" date already assigned by the developer - if any of you can do this fairly quickly, i'm sure the rest of us MailNews users would appreciate your efforts!
Moderators need an additional choice: "Karma Whore" for people who cut-and-paste articles as their comments!
That could be because yEnc breaks the NNTP and MIME standards. Some (many?) NNTP and MIME-compliant servers won't pass yENC correctly.
DNA just wants to be free...
After all, the website has a decoder in "Pearl", but not "Perl."
"Alcohol, Tobacco, Firearms, and Explosives" should be a convenience store, not a government agency.
My ISP (cable) has a quite short retention period in the binary newsgroups (it's like they do it on purpose!) and many files are broken from missing multiparts. Integrating yEnc and PAR would solve that problem once and for all.
You're right, the main goal for Mozilla isn't selling more units to be profitable. However, in something like browsers, market share is indeed important. The more people who adopt Mozilla as their primary browser the better. Once companies start seeing that the number of IE users is decreasing as Moz users increase, they'll be more receptive to not using IE only features in their web design. This would be a win for everyone.
.
load "linux",8,1
HTTP was DESIGNED to transport HTML.
Actually, there's nothing about HTTP that's HTML specific, other than having a specific content-type.
The Usenet Protocol was DESIGNED to transport plaintext. Newsreaders that read HTML off postings do it by ugly hacks, that don't always work.
Roads were originally designed to carry horse traffic, but that doesn't mean we shouldn't adapt them as necessary when cars were invented.
If you can control your newsreader like you do with your browser, I'd not reject HTML-on-usenet as much.
Well, I can, since my newsreader uses the IE Browser object for displaying HTML. Still, I do sympathize with the fact that HTML can be abused on Usenet. I think that means we should have better newsreaders/browsers, not that HTML is intrinsically bad.
BECAUSE the design and implementation of the protocol undoubtedly would be DIFFERENT than the original Usenet so they can handle HTML INHERENTLY.
The only change that I can think of is that we wouldn't ship around messages with two copies, which would obviously be a plus. Still, it's not a perfect world and I think the advantages far outweight the disadvantages.
Sometimes it's best to just let stupid people be stupid.
You're comparing a seven year old version of Microsoft Word to the current version of StarOffice. How about either comparing Word 95 to other word processors available in 1995 or comparing Word XP to the current version of StarOffice?
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Has it ever been? I was downloading software posted to comp.binaries.apple (what's now called comp.binaries.apple2) as far back as 1989. If Usenet was ever text-only, that would've been before my time. :-)
20 January 2017: the End of an Error.
It is exactly my point - posting HTML on today's Usenet is analogous to putting cars on roads built 100 years ago.
Roads built 100 years ago were built for horses. Roads built today are built for cars.
While engineers have adapted roads and bridges to cars by making their surfaces smoother and increasing their capability of carrying heavier loads, the Usenet hasn't undergone such adaptation yet.
If I want to drive a car on 100-year-old road I can, for example, get an AWD and drives like there's no road. In this case, it is the car that's adapting for the old road. However, no similar adaptation has been gone through HTML yet. (HEY!!! It triggers something - now I think using WML on Usenet is the best thing)
I agree, that the advantage outnumbers the disadvantages. But for me, the disadvantages associated with security and eyesores associated with spams far outweighs the advantages. So you see, whether it is more advantageous or not, it is in the eye of the beholder.
Have a nice day.
What do you think of MusicCity now?
Who cares about linking to Usenet posts? Just being able to link to web sites would be an advantage.
... Why post HTML when you can post DIVX files of you talking and people can see you talk and stuff ... 90% of communication is nonverbal. You're making us miss so much!!
As someone said to me, "There is NOTHING about the web that guarantees a page "is there"."
The point is that it's a convenience, which is just as convenient in a Usenet post.
Some newsreaders actually change plain text urls to clickable links. Without inconveniencing the majority of Usenet readers by presenting a bunch of bracketed crap.
You know, you old school HTML people need to get with the program
Any good server should see a uuEncoded message and store it in compressed form on the server.
wow...
you've never run a news server have you?
I just can't see why we have to implement a new standard before it's ready.
Do you work in Redmond?
"We are not tolerant people. We prefer drastically effective solutions"