What's The Problem With USENET?

Sir_Winston writes: "For many months now, several of the larger news servers--at least, the "premium" news servers who sell access to individuals, not just to other businesses--have been experiencing severe problems with retention and sometimes with propagation as well. I can remember subscribing to services like Altopia and Newsfeeds because my local ISP's feed had retention of only ~2 days, but now several of the commercial news servers are down from 8-14 days to around 2-4 days. (some offer retention stats on their Web pages, and others can be gleaned from talking with customers). Talking with users, some seem to blame broadband connections for allowing users to flood USENET beyond reasonable capacities. Is this the case, or are there other considerations? Given that USENET isn't a truly distributed system, is it in for increasing problems as more people keep dumping massive binaries that may be better posted somewhere else? (For example, almost every mp3 posted to USENET can be found on Napster, often at higher bitrate). So, is there a problem, and if so how can it be addressed, and which USENET providers are still doing well at retention and completeness?"

Comments

Oh, really?

[wroot]

I don't know about the legal issues involved, but technically, you can connect to my University's news server without being affiliated with it and read/post through it. Go to university web sites near you and read the instructions for users. Most likely, all you'll have to configure will be the name of the news server (something like news.bla.edu)

Another obvious thing to do is to use deja (no binaries though AFAIK)

HTH

Wroot

Re:Death of Usenet Predicted!

[silverpelicanfeather]

I was on Usenet just today, used to post in NANUA and news.groups a lot, and was struck with how much things have cleaned up. A year ago, it was chaos, the UPA guys were just trashing groups right and left, took over NANUA completely, Burnore was running things and flame wars were everywhere. Now, either some people pushed off or died or something, the UPA seems to have gone somewhere else and there was some semblance of reasonable discourse. The binaries are the cruncher and most ISPs soon, if not already, are going to have to not propagate this immense bandwidth. But Usenet death has been constantly predicted since the outset, 1981?, and will probably outlive IRC.

USENET binaries of up to 1 month will be retained

[pcbsd]

We are adding storage to allow for up to 1 month of binaries on http://www.etin.com. Our current retention time is about 5 days. We retain non-binaries indefinitely.

Drop binaries

[ptomblin]

My site doesn't carry binaries. Neither do my main upstream sites. We have a very complete newsfeed and retain newsgroups in the big 8 for about 15 days.

By the way, alt may be a sewer, but there are a couple of islands of very high quality groups. Most of them are high quality because they are patrolled by a cadre of old-timers who attack anybody who tries to lower the standards.

Re:Drop binaries

[JordanH]

I remember lots of sites dropping binaries back in the late 80s (yes, I said 80s) because of all the MS-DOS executables and pictures being posted.

There was a time when a LOT of MS-DOS executables were posted. That died down. Archives like SimTel just posted summaries of new stuff and people found that they could use ftp instead. If you only had Usenet and Mail access, there sprung up the FTP Mailer services that would mail you something from an FTP archive.

I also agree that there's still a lot of high quality discussion on Usenet. Some of the comp.lang groups are excellent (.moderated for c and perl, thank you). The comp.sys and comp.os groups can have some good topical news and help.

Hey, if you're looking for some good free content sometime, cruise over to DejaNews and search for dmr@bell-labs.com, bs@research.att.com or mash@mash.engr.sgi.com. This is just name-dropping on my part as there are other non-famous posters to Usenet that have a interesting things to say. If you've been avoiding Usenet, surf DejaNews on some topic that interests you. I think you'll be surprised at what you might learn.

---

The problem is the Binary groups.

[Zachary Kessin]

I read about 6-8 newsgroups daily and I can say some of them like the aviation groups are great. And there are others which are just a useless waste (rec.food.veg) with so many trolls and such that usefull converstation is just imposible.

But I have been using usenet on and off since '91 and have no intention of giving it up.

The cure of the ills of Democracy is more Democracy.

Re: Use-Net, "real" system or not

[siliconowl]

The primary use for usenet these days, IMHO, is to be a place where the kiddies go so they don't annoy the people with a clue who left and moved onto real systems to get their stuff done....

I object to that sentiment in the strongest possible manner. I would count myself as a person with a clue and I find Use-Net extremely valuable. Among my regular haunts are the comp.sys.acorn hierarchy which is full of useful and timely news and information and I have yet to find a better way to get obscure questions answered. Other groups on my subscription list such as alt.books.pratchett carry high quality debate and discussion. The signal to noise ratio can become vanishingly small at times but to a large extent that's purely because one mans noise is anothers signal. Of course the same criticism could be leveled at any other similar system.

Re:The problem...

[damiangerous]

Current figures are more like 2GB/day, or they were last time I checked, at least a year ago.

Average daily traffic is around 175GB/day. It's broken 200GB on some peak days. It's been almost approximately a year since traffic broke the 100GB mark, and about 30GB the year before that, so I don't know what figures you were checking, unless that was a typo or you were referring to a non-binary feed. The last time a full Usenet feed was around 2GB was January of 96. That had already doubled by January of 97.

Too many crap groups

[frankie]

The easiest thing for most feeds to do is simply drop their alt.binaries.* groups

Agreed. alt.binaries.* is redundant and inefficient. Prune back problem groups and the rest work much better.

The problem with this simple and effective solution is that news providers (and ISPs in general) love to print group counts in their marketing. "Over 5 billion Usenet newsgroups! When you sign up for our service, we create a new group for you -- alt.cool.stuff.that.user.[foobar].likes !" etc.

For example, JHU has over 30,000 groups. They carry stuff like abg.* (Altenburg, Germany), uw.* (University of Waterloo), and realtynet.* (house selling). I know most of these groups have insignificant traffic, but it's symptomatic of the larger problem -- a willingness to carry every newsgroup ever created.

Too many groups is A BAD THING. For example, our feed has alt.education.*, k12.*, misc.education.*, and school.*, not to mention various local hierarchies and subgroups. If you have a pedagogy question, where the fuck do you post? If you want to hear other people's ideas, where do you read?

The clutter of excess groups make Usenet less valuable by segmenting the readership unnecessarily. Unfortunately the task of newsadmin is generally a minor duty tacked on to the schedule of a disinterested junior sysadmin who doesn't care about Usenet being actually usable.

Re:binaries on usenet.

[bluGill]

While I agree encoding is a problem, otherwise usenet is a good model for binaries.

Instead of tieing up the entire internet to get a binary, you tie up just one subnet at your ISP. The servers get the binary once, and from a site closer then the orginial ftp server.

Really it is a perfect model for binaries of use within a few days of publication, but not much use latter, that a large number of people want to see.

Saddly it doesn't work that way. Few ISPs run their own news server meaning you tie up the backbone for each binary. Worse, instead of binaries being of use only during a certian time, most are reposts every few weeks of something you don't want anyway. (Word'97 parts 1-35,37-50,51-74 - of 75 seperate parts)

A simple solution

[coyote-san]

There's a simple solution to this problem - run your own news server! I set aside a 4 GB partition for the comp.* hierarchy and I have a retention period of several months. I use "suck" to grab the articles from my cable modem's new server - that way I don't need to set up peering arrangements.

This isn't a maintenance-free solution - I don't believe your news server will properly process NEWGRP messages so you'll need to manually check that once a month or so, but it has several benefits:

<ul>
<li>You control the retention period,
<li>Your clients have extremely high responsiveness, since the server is on the same LAN and only serving local users, and
<li>You can run spam filters on the article spool.
</ul>

The importance of the last point can't be overemphasized if you're in the alt.* hierarchy. As an experiment I tried cleaning up some heavily polluted groups, and discovered that the bulk of the offensive spam is from a few <i>very</i> active spammers who are using a couple class-C IP address blocks. (I'm _S_U_R_E_ *Y*O*U* .W.O.U.L.D. /R/E/C/O/G/N/I/Z/E/ #T#H#E#M# ^I^M^M^E^D^I^A^T^E^L^Y^) A simple script that deletes any files containing a URL from these address blocks (in all three formats) does an incredible job at reducing the perception of spam.

Death of Usenet Predicted!

[jmaslak]

I can't resist... I know it has little bearing on the topic of what to do about the specific problems occurring with connectivity amoung Usenet nodes, but...

*Death of Usenet Predicted! Film at 11!*

I've been involved in Usenet and on-line communicications since about 1988. That doesn't make me an old-timer, but it does give me hopefully a little creadability.

In this time, I've seen people talk about how much trash has entered into Usenet, how spam is on the rise, how key backbone sites are dumping Usenet, etc. So far, Usenet has proven remarkably robust and resiliant to these kind of problems.

Yes, I browse Usenet with a very extensive SPAM filter. Just as in the old days, after about 500 messages in a thread, I tire of it and add it, also, to the SPAM filter. There are certain people who get the distinction of *plonk* as they are added to my kill files. But, life goes on.

In my opinion, Usenet is one of the last examples of community still left on the Internet (with some IRC channels and VERY FEW sites like Slashdot also there -- although I might argue that the community on slashdot isn't anything like the community on Usenet).

Adding to the problem

[Robert S Gormley]

If everyone followed your lead, you'd only add to the problem - in terms of news server availability. I'm sure most usenet server owners don't want you effectively downstreaming them without consent.

Simple Solution

[Anonymous Coward]

There are plenty of ways to distribute binaries on a peer-to-peer basis without cluttering usenet. Maybe it's time to drop binaries from usenet.

If enough usenet administrators decide to drop/cancel the binaries, usenet could be saved.

Broadband?

[Cyclopatra]

I could see broadband contributing to the low retention times for binary ngs, but it seems to me that most of Usenet, being text-based, wouldn't be as susceptible to being flooded by people with broadband accounts - after all, even with highspeed access, we can only type as fast as we did before, and that's pretty much the bottleneck for Usenet, even on a dial-up account.

What's more, Usenet-izens are much more strict about etiquette than anyone else on the Internet, as far as I've seen and a lot of Usenet rules of behaviour have to do with taking up as little bandwidth as possible.

Or maybe I've just been hanging out in the last bastions of politeness on Usenet...

Cyclopatra

"We can't all, and some of us don't." -- Eeyore

Re:Broadband?

[chuqui]

Usenet is ascii based, but not text based. something like 97% of a full usenet feed today is ascii-encoded binaries.

Re:Broadband?

[Paulo]

You *have* been hanging out in the last bastions of politeness on Usenet.

Most of the Usenet *groups* are text-based, but note that group != traffic. Last time I looked, binary groups consumed about 70% (if not more) of the total bandwidth in a typical news server.

Re:The problem...

[dougmc]

Here's a (now) amusing passage from a pretty out-dated book -

From O'Reilly's book "Managing UUCP and Usenet" (1990), page 154 :

"Before you even beging to think about installing Netnews, make sure that you have enough disk space. The volume of traffc on the net can be quite substantial (over 4 MB per day.) Since articles are usually left around for a week or two to allow users to read them, you need from 28 to 56 megs of free disk space."

Current figures are more like 2GB/day, or they were last time I checked, at least a year ago.

This traffic is almost all binaries. I'm currently running a news server for a company that I'm working for -- it's a SS2, 64mb ram, about 6gb of disk. It has a fairly full feed - most of the big seven groups (comp.*, rec.*, talk.*, sci.* etc.) and a few selected alt.* groups. No binaries. It has no problems keeping up with the incoming traffic and a few readers, and it has enough disk to keep most articles for at least a few days. I could upgrade it, but why? It's doing it's job just fine, even now ...

alt.* is a wasteland. Once you get away from it, Usenet is reasonably sane, even today. Which is a shame -- alt.* had such potential. at least until C&S got clever ...

Note that back when Usenet was designed, back in 1979, I'm not even sure T-1's existed. Back then, news and mail was shuttled around on dialup modems ( at 1200 bps or so) using UUCP and even the largest universities tended to not have connections to other places faster than 9600 bps.

It's all very different now ...

.uk

[smaugy]

In .uk, Claranews is regarded as one of the best pay-for news services (+ POP3 account). They just recently had to increase the cost from £20 to £29.99 *per year* to cover the cost of extra servers. It's still excellent value for money if you consider that Supernews costs £130 a year.

As for retention times, it's around 2 days in a.b.c.d.i., 2-3 in similar busy binary groups, 3-4 for "average" traffic groups, and 6+ (probably more than a week and a but; I haven't looked recently) for text-only groups - e.g. currently 7085 headers in comp.os.linux.misc & 11819 in .advocacy.

The news services provided with most ISPs seriously sucks - you'd be lucky to find any posts that have all the parts in binary groups.

HTH.

binaries on usenet.

[chuqui]

Usenet is the worst possible way to distribute binaries -- you are sending every byte of every messages to every computer everywhere, in case someone someone might want to look at it. It's the worse possible distribution model you could think of, especially since you have to encode it in 7bit ascii to boot, thus exploding filesize even further.

Want to fix usenet? do away with binaries. How? damfino. I wish you luck. As someone else said, with things like Napster, there's no reason for usenet binaries any more, anyway. Except for the "I have a hammer, so this must be a nail" problem. It's there, it's used, and nobody can say "no".

The primary use for usenet these days, IMHO, is to be a place where the kiddies go so they don't annoy the people with a clue who left and moved onto real systems to get their stuff done....

Re:binaries on usenet.

[Sir_Winston]

The encoding problems aside, there is one great advantage to USENET's kooky distribution model: data cannot be effectively blocked. It propagates like a virus, entering one server and then getting copied to nearby servers until it propagates all over the world. Now, hopefully systems like Freenet can handle that sort of problem, since they throw a technical solution at creating an uncensorable network without having to have the Rube Goldberg solution of distributing the data to every server, and provide a measure of anonymity and untraceability for both the posters and for the place(s) where the data are located. This sounds like a great solution for storing texts, but on the other hand it could never provide the the discussion qualities of USENET.

You might be surprised at how many fairly knowledgeable chums still poke around USENET. For example, if you want privacy and anonymity resources, there's no better place to go than USENET--the only things on the Web that can compare are those that rip off the FAQs and resources built up in some USENET groups.

The problem is certainly the binaries, though. Broadband connections have inspired too many twits to post hundreds of large files a day, which kills retention not just in binaries groups but in text groups as well since they're all typically stored in the same server pool. There are now binaries groups which were created specifically *for* broadband users, with no daily posting limits. Too many of these people fail to realize that what's posted in one group affects retention in all the others.

That's not to say that there shouldn't be binaries groups--I think there are some binaries groups in USENET which are great, because they're conversational and interactive, which you can't get from a Website or a Napster-type program. Some of them are very like the old BBS days, and it's nice to see "communities" not just file repositories.

For all the good stuff in some of the alt.security.* and comp.* and even occasionally in the alt.binaries.* hierarchies, I worry that a relative few people with broadband who are mostly posting stuff that would fit better elsewhere are going to drive USENET into the ground. It would be sad if USENET goes the way of BBSes--it'll go on, just like there are still a few BBSes here and there, but only as a shadow of its former self, catering mostly to nostalgics. That wouldn't be good.

The problem...

[Wakko Warner]

The problem with Usenet is that it was designed back in the '80s, when the only people on the Internet were researchers and college students and the fastest connection was a T-1. Typical lack of foresight: nobody expected this kind of bandwidth to be available back when colleges had 9600bps serial ports in every dorm room.

The easiest thing for most feeds to do is simply drop their alt.binaries.* groups (or at the very least, the binaries groups with the largest article counts); why bother having what effectively amounts to a local mirror of Napster? I fully expect to see most ISPs do something like this, if they haven't already.

The "nicer" providers might decrease the expire time on binaries groups to a day (I've seen half-day expires even) in order to raise expire times on other, more meaningful groups.

- A.P.

--
* CmdrTaco is an idiot.

Re:The problem...

[smaugy]

Quotes from Claranet (they have a news-service-only account rather than a normal ISP-with-access-to-the-news-server account) tech support:

Okay, but consider that a separate binaries service would require a LOT more equipment than the current service, and that as a consequence, costs (ie. prices) are likely to be MUCH higher.

Consider that at the moment there are 3 machines handling about 200 gigs a day. Add a separate binaries service, and you have to almost double the throughput of the system, and add special reader machines for the service (which do cost oh-so-much-money(tm).

But then I have no idea about why anyone would want to download binaries from USENET anyway... the web and FTP work just fine for me :))

--- next ---

> Do Claranet intend to keep up with it all
> for ever? Do you know what
> other ISPs think about the issue? When does
> the cost of keeping up
> exceed the benefit of doing so?


For the forseeable future, yes.

There are considerable economies of scale - the cost of running a news service depends to a large extent on the amount of news traffic rather than the number of customers so it might make sense for us to offer our news service to other ISPs, or alternatively buy our news service from another provider. Personally I favour the former option.

Re:The problem...

[Anonymous Coward]

>But then I have no idea about why anyone would
>want to download binaries from USENET anyway...
>the web and FTP work just fine for me :))

Then you aren't much of a pornhound.

Only amateurs surf the web for pr0n; there are some great 'bots that will keep track of a given newsgroup and download everything automatically, keeping track of what has been previously posted so you don't waste time downloading duplicates.

There's a lot of spam etc. but you can killfile a few domains and zap the rest browsing with ACDSee. (Is there an image browser under *nix as good as ACDSee??)

You can get gigs of porn in a real hurry on Usenet. Some of it is bad; some of it is reallyreallygood.

uhhh, pr0n

[Crow-]

Uhh dude, usenet is like the best source of pr0n ever. Don't do away with binaries until there is a good image swapping service.. scour exchange was cool but it was shutdown.

Binaries, and the UseNet

[haplo21112]

Ok, first off I will state that I run a Binaries only Usenet site. So if that troubles you just move on. Our site caters to the users who are looking for usenet binaries, and we filter everything else out(aka Spam, and text articles) and only present the pics, mp3's, fonts, etc. By the same token, we also offer usenet feeds tailored to the users need/desire. So if a user wants a binaries only feed, or a text only feed, we have that for them too. To my point, I have watched however over the years as the USENET has become hopelessly clogged and choked as more and more useless Crap gets dumped into it. In many many groups the actual material vs SPAM ratio is in the neighborhood of 10% vs 90%. Even worse many of the groups have messages that just send the user to a web site that actually contains the content of the message when they read them. I really would love to find a way to take back usenet(other than strict moderation, because lets face it a great deal of USENET seems to exist in the alt.* area these days where moderation is somewhat verboten), so that the groups are once again filled with real information, and content rather than nude pictures in rec.humor.funny, because someone is trying to advertise his web site. --Haplo Newsmaster www.wickednews.net

Re:The problem with USENET:

[MrPeach]

Maybe... People like you?

The problem with USENET:

[pb]

USENET is flooded with lusers and spam.

Now, anyone want to wager a guess as to what the problem with Slashdot is? ;)
---
pb Reply or e-mail; don't vaguely moderate.

Napster this, Napster that

[UnrefinedLayman]

It's all well and good that Napster exists, but anyone who says alt.binaries.* needs to be done away with needs a serious reality check: a very very small number of groups in the alt.binaries.* heirarchy is for MP3s.

1900 binary groups on my server and about 100 are for MP3s. You cannot, flat-out-cannot-I-don't-care-what-magic-tricks-voo doo-or-prayer-you-know, get everything from usenet on Napster, or anywhere else. In addition, the unchecked propagation of information through usenet servers is massively used as a distribution point of information. Want to kill usenet because of the network load? Go kill freenet--I hear it's a lot slower.

One word

[fortunetroll]

Trolls

They're not just on MUDs anymore.

Ours is a world of nuclear giants and ethical infants. -- General Omar N. Bradley

Re: NOT Adding to the problem

[coyote-san]

The original poster's problems, and mine, is not server availability, it's short retention periods. Nothing but disk space (either by using larger data spools or dropping newgroups) will solve that problem.

As for the issue of bandwidth consumed by this process, most people will only download the newsgroups they actually read. (I'm only grabbing all of comp.* because some friends asked for full archives for some research projects.) There is no additional overhead incurred in downloading articles that you would have downloaded anyway, it actually <i>helps</i> the availability because the access is spread out throughout the day instead of being clumped.

spammers, trolls, flooders, script-kiddies

[Anonymous Coward]

Part of the problem is that there are a lot more assholes who delight in destroying newsgroups and filling them up with garbage.

One of these characters is Gary Burnore. Some of you San Francisco Bay Area people might remember him as one who managed to piss off most of the ISPs around there and anyone heavily involved in database management. More of you might remember him as the one who harassed Jeff Burchell into shutting down the Huge Cajones anonymous remailer because someone fingered him for child molestation; a change for which he was subsequently convicted. In revenge, he had his now wife impersonate a lawyer to get the nonexistant logs of Huge Cajones shortly after Helena Kobrin harassed Jeff to get the same nonexistant logs. Jeff shut the remailer because of this.

Gary violated probation by leaving California for Raleigh, NC where he offers network services primarily to trolls, flooders, spammers, and other vandals and shields them from complaints. He and his syncophants have been known to threaten lawsuits, make physical threats, launch denial of service attacks, and make up stories to get critics kicked off their ISPs.

Usenet is attracting many more psychopaths just like Gary. They don't care how many people they piss off, nor are they concerned about possible legal consequences or retaliations.

Desert Rat