Slashdot Mirror
How Google Saved USENET
Masem writes: "Salon has a well-written article article on the recent revival of much of the USENET archives from '81 to '90 by Google. It mentions that much of the recovery was thanks to years of work in transferring data off 140-some 10" magnetic tapes (~120megs of data) to a more conventional format in order to recover much of the early posts. Even a reference to the previous Slashdot story is made." Update: 01/07 23:52 GMT by T : btempleton adds: "O'Reilly Network asked me to do an article on similar themes and rememberances of USENET history." Thanks, Brad.
--Chag
--Chag
/. is a commercial entity. goto slashdot.com
Yes, google saved the historical record of the USENET, but it needed not to save the USENET from anything else. USENET is alive and well.
Get your Unix fortune now!
I think I speak for everyone when I say "Thank you Google for arming me with the information contained in old USENet posts to bring up embarassing teenage posts to my friends!"
thelocust[dot]org
Google Groups is awesome, especially when searching for some obscure piece of hardware advice or settings.
I don't have to worry about getting and setting up a news client, and it's just one tab over from my default search engine.
Google did save USENET for me - though I never post, searching through all the linux and comp newsgroups is usually faster than looking up a HOWTO.
As a software developer, no matter what problem I run into, somebody else has already run into that problem and has asked my question and recieved an answer on groups.google.com. Whenever I get stuck on anything at all, it's the first place I run to. groups.google.com is the single most useful site you can point your browser (konqueror!!!) towards. I'm not sure how they make money over there at google, but what a great service they are providing!
Check out my podcast: DreamStation.cc Video Game Show
...how Google will make money off of this. They supposedly make money off licensing their technology (and presumable their collected data, as well.) No ads whatsoever. I applaud their dedication to that goal so far.
Groups.google.com seems like the kind of thing they're doing just becuase they can, though. I can't imagine there is much money to be made off the technology, because it's all text - the same search tech applies. So, as far as I can tell, there is no business reason to be doing this. it's a drain on resources with little to no return, except for (geek) community goodwill.
The conclusion I draw, then, is Google is in this just for the fun, challenge, or doing something for the community - maybe all three. Philantropy at its best. =)
± 29 dB
An interesting thing about these tapes: They stretch over time and can sometimes become unreadable because of that. There are times when, to extract the information on the tape, I would put a number of them in my freezer for an hour or so, then try again. Nine times out of ten that would actually work.
Another note about the article: I can still remember discussions with others who had modems about 1200 baud being just "too fast". The reasoning was that the average person couldn't read much faster than 300 baud. :)
years of work in transferring data off 140-some 10" magnetic tapes
;)
That means at least one person spent several DAYS PER TAPE???
Even punch tape 'd faster than that.
I was wondering what kind of backup googles uses now for all its info? What happens if one day a script kiddie breaks in and rm -rf / all the boxes? Do they have tape backups? How many etc. I also wonder how much they spend on it.
Google seems to be getting involved with a lot of things. It's nice to see that a group is not only trying to push the Internet forward, but also trying to preserve the past.
My other sig is an import.
I know this is a repeat but this is a great read. Dr. Gene Spafford's farewell posting. If you don't know who that is, look it up.
n ews.groups,soc.net-people
:-)
===
From: spaf@cs.purdue.edu
Newsgroups: news.announce.newusers,news.misc,news.admin.misc,
Subject: That's all, folks
Followup-To: poster
Date: 29 Apr 1993 19:01:12 -0500
Message-ID:
[ I originally was going to post nothing on this topic. I'm burned
out, and I don't want my fatigue to appear like I'm posting
self-indulgent garbage. However, several people have argued with
me, and convinced me that maybe I should make a statement to "end an
era," and as a piece of net "history." At the least, even if it is
perceived as self-indulgent garbage, it will fit right in with the
rest of the net. ]
There is a Zen adage about how anything one cannot bear to give up is
not owned, but is in fact the owner. What follows relates how I am
owned by one less thing....
About a dozen years ago, when I was still a grad student at Georgia
Tech, we got our first Usenet connection (to allegra, then being run
by Peter Honeyman, I believe). I'd been using a few dial-in BBS
systems for a while, so it wasn't a huge transition for me. I quickly
got "hooked": I can claim to be someone who once read every newsgroup
on Usenet for weeks at a time!
After several months, I realized that it was difficult for a newcomer
to tell what newsgroups were available and what they covered. I made
a pass at putting together some information, combined it with a
similar list compiled by another netter, and began posting it for
others to use. Eventually, the list was joined by other documents
describing net history and information.
In April of 1982 (I believe it was -- I saved no record of the year,
but I know it was April), I began posting those lists regularly,
sometimes weekly, sometimes monthly; the longest break was for 4
months a few years ago when I was recovering from pneumonia and poor
personal time management. (Tellingly, only a few people noticed the
lack of postings, and almost all the mail was "When will they come
out?" rather than "Did something happen?") As time went on, people
began to attach far more significance to the posts than I really
intended. It was flattering for a very short time, and a burden for
most of the rest; there is no telling how much time I have devoted
over the last decade to answering questions, editing the postings, and
debating the role of newsgroup naming, to cite a few topics. I really
tired of being a "semi-definitive" voice.
Starting several years ago, at about the time people started pushing
for group names designed to offend or annoy others, or with a lack of
concern about the possible effects it might have on the net as a whole
(e.g., rec.drugs and comp.protocols.tcp-ip.eniac) I began to question
why I was doing the postings. I have had a growing sense of futility:
people on the net can't possibly find the postings useful, because
most of the advice in them is completely ignored. People don't seem
to think before posting, they are purposely rude, they blatantly
violate copyrights, they crosspost everywhere, use 20 line signature
files, and do basically every other thing the postings (and common
sense and common courtesy) advise not to. Regularly, there are postings
of questions that can be answered by the newusers articles, clearly
indicating that they aren't being read. "Sendsys" bombs and forgeries
abound. People rail about their "rights" without understanding that
every right carries responsibilities that need to be observed too, not
least of which is to respect others' rights as you would have them
respect your own. Reason, etiquette, accountability, and compromise
are strangers in far too many newsgroups these days.
I have finally concluded that my view of how things should be is too
far out-of-step with the users of the Usenet, and that my efforts are
not valued by enough people for me to invest any more of my energy in
the process. I am tired of the effort involved, and the meager --
nay, nonexistent -- return on my volunteer efforts.
This hasn't happened all at once, but it has happened. Rather than
bemoan it, I am acting on it: the set of "periodic postings" posted
earlier this week was my last. After 11 years, I'm hanging it up.
David Lawrence and Mark Moraes have generously (naively?) agreed to
take over the postings, for whatever good they may still do. David
will do the checkgroups, and lists of newsgroups and moderators
(news.lists), and Mark will handle the other informational postings
(news.announce.newusers).
I'm not predicting the death of the Usenet -- it will continue without
me, with nary a hiccup, and six months from now most users will have
forgotten that I did the postings...those few who even know now, that
is. That is as it should be, I suspect. Nor am I leaving the
Usenet entirely. There are still a half-dozen groups that I read
sometimes (a few moderated and comp.* groups), and I will continue to
read them. That's about it, though. I've gone from reading all the
groups to reading less than ten. Funny, though, the total volume of
what I read has stayed almost constant over the years.
My sincere thanks to everyone who has ever said a "thank you" or
contributed a suggestion for the postings. You few kept me going at
this longer than most sane people would consider wise. Please lend
your support to Mark and David if you believe their efforts are
valuable. Eventually they too will burn out, just as the Usenet has
consumed nearly everyone who has made significant contributions to its
history, but you can help make their burden seem worthwhile in
between.
In closing, I'd like to repost my 3 axioms of Usenet. I originally
posted these in 1987 and 1988. In my opinion as a semi-pro
curmudgeon, I think they've aged well:
Axiom #1:
"The Usenet is not the real world. The Usenet usually does not even
resemble the real world."
Corollary #1:
"Attempts to change the real world by altering the structure
of the Usenet is an attempt to work sympathetic magic -- electronic
voodoo."
Corollary #2:
"Arguing about the significance of newsgroup names and their
relation to the way people really think is equivalent to arguing
whether it is better to read tea leaves or chicken entrails to
divine the future."
Axiom #2:
"Ability to type on a computer terminal is no guarantee of sanity,
intelligence, or common sense."
Corollary #3:
"An infinite number of monkeys at an infinite number of keyboards
could produce something like Usenet."
Corollary #4:
"They could do a better job of it."
Axiom #3:
"Sturgeon's Law (90% of everything is crap) applies to Usenet."
Corollary #5:
"In an unmoderated newsgroup, no one can agree on what constitutes
the 10%."
Corollary #6:
"Nothing guarantees that the 10% isn't crap, too."
Which of course ties in to the recent:
"Usenet is like a herd of performing elephants with diarrhea --
massive, difficult to redirect, awe-inspiring, entertaining, and a
source of mind-boggling amounts of excrement when you least expect
it." --spaf (1992)
"Don't sweat it -- it's not real life. It's only ones and zeroes."
-- spaf (1988?)
--
Gene Spafford, COAST Project Director
Software Engineering Research Center & Dept. of Computer Sciences
Purdue University, W. Lafayette IN 47907-1398
Internet: spaf@cs.purdue.edu phone: (317) 494-7825
===
Ye Gods!
The modern slashdot nerd trembles in the presence of those ancient USENET nerds of old
A 300 pound slashdot weakling is easily flung aside by the 500 pound USENET god. Who at slashdot keeps taped archives of every post for the nerds of future generations? Truly those were nerds.
i'm a tad concerned about the posts i made in the early 90's when i was an asshole know it all teenager coming back to haunt me... i wish google never uncovered those... i cringe when i read them now...
In a major university, and I decided to honor his
.. alt.emacs, here I come.
soul and follow his foot steps.
And now, thanks to google, I find myself battling
the flame wars he started.
Better go back and do him and VI and honor
I am sorry they will allow requestors to delete their own postings. While we might wish it otherwise, 10, 20, 50 years later, this may be the real historical value. To purge, seems the equivalent of having a letter to the editor removed from newspaper archives.
To those who feel like "they are walking around with their baby picture stapled to their forehead", we all mature. What I thought at 20, 30, and 40 show how I grew. What other archive in human history can provide the transitional opinions, discussions, and outright imbecilic flames wars?
While we would hate to have someone pull out our post in support of the flat earth theory, to act as though we all believed the earth was round is rewriting history. Convenient for us, but misleading to the future.
The question now becomes, what happens after Google and Slashdot, when the archive is tera-bytes large? Will it take 100 years for the next conversion?
The first "me too" post isn't until two years into the archive. I suppose that says something about the intelligence of the usenet demographic back then.
--
Scratch that. I found this page which tells you how to remove posts you have made.
Three tapes for rec.singles desperate
Seven for alt.swedish.chef.bork.bork.bork
Nine for comp.sci compiling late
One for Google's engine dark
In their Linux cluser where the shadows lie.
One engine to search them all, one engine to bind them
One engine to index them all and in the darkness find them
In Google's cluster where the shadows lie.
Carousel is a lie!
How Slashdot Saved Salon
Google has a history of doing a lot of things right, but I have my doubts about their new service: catalogs.google.com. It's a search engine for graphically scanned in versions of mail order catalogs! You type in sewing machine, say, and you get 3 views for each match: a scan of the catalog cover, a scan of the page, and a close up of the page, with the search terms highlighted in yellow.
It's so retrofuture weird! Like what someone on a C=64 in the 1980s might think a future of online shopping would look like...
SO YOU'RE GOING TO DIE: The Comic for Dealing with Death
I find it very interesting that in the last 10 years of USENET, it's traffic (and presumably use) have grown dramatically. However, the number of servers has, I believe, dropped equally dramatically. USENET was one of the most distributed systems I remember using, with it's shared-nothing, "flood-fill" algorithm.
Yet, as it scales up to more and more messages, it actually is becoming less distributed. A good lesson for all the futurists forcasting the rise of distributed systems...
You can read the article I wrote on the O'Reilly site
Has it been over a year since you last donated to the Electronic Frontier Foundation
While also saving the Usenet archives (public and widely dispersed information)..!
-------
Warning: Slashdot may contain traces of nuts.
Erm... Usenet is a PUBLIC system, any and all posts you make there are in the public domain.
The information is preserved for posterity, not for making money or other commercial exploits.
I can't really believe you think we'd be better off destroying information instead of preserving it!
So let me get it straight. 9 years of USENET posts occupy only 16.8GB of hard disk space?
You sure those 10-inch magnetic tapes weren't 1200MB or 120GB or something? Hell, a converted VCR using VHS as a backup medium can store like 100GB (saw one somewhere, I forget the link.)
Allot of the good gurus are moving over to slash ran message forums. Talking to a guy who is a perl guru, he has moved most of his perl help requests from usenet to Perl Mongers. I've been seeing this trend in the last few years, as independent subjects are moving over to a website based web forums. I even spend more time reading 5 mailing lists and a dozen message forums, and dont touch usenet anymore.
With these message forums and mailing lists not linked to a usenet group, there is a lot of wasted knowledge that is not shared. I would love to see a slash-mod or some type of mailing list enhancement that posts a overview or some kind of daily message post to usenet.
The whole idea of usenet was knowledge sharing, not binaries and spam ads. Glad google has saved usenet, but some effort needs start using it again.
Humm, Maybe Slashdot should enhance a usenet forum? Thou 5-20,000 posting a day on a usenet might be a little much. Maybe only 2+ posts make a moderated usenet group.
This is downright scary.
Nothing like looking through the archive to see an old post from a skilled sysadmin friend asking a basic question in the wrong group years ago.
Nothing like seeing delusional inane posts you wrote while in high school making you look like an utter twit.
Nothing like seeing old usenet posts from friends who have died years ago. This is just too creepy for words.
Aside from his good works in the terms of Usenet, David is the reason I am where I am today. 4 years ago, I was stuck in Perth, Australia and very bored. I was reading the student newspaper one day and saw an article about student exchanges. To cut a long story short, 6 months later I was at The University of Western Ontario.
:)
;)
I had looked over the courses they ran in Computer Science there, and saw one called "Unix and C". Being a bit of a geek and having used unix a *tiny* bit in my high school days, I thought it was be a cool one to take. David was the lecturer for this course. He had a lot of knowledge and passion for the subject, which is unsurprising considering his experiance with all manners of unicies. His classes for CS175a taught me a lot about Unix (and a little about C). I got 92% overall for the unit, an A+ and the highest mark I've ever got for any unit. The next semester I was at Western, I taught myself Perl, using an account on the CS Department servers and on the Reznet linux box a friend had
It was a unit for non comp-sci majors. CS Majors were expected to learn this stuff in a bunch of different classes.
Sadly, Western no longer offers CS175a - Unix and C. I feel it is a loss to the community as a whole, but at the same time, I understand that a one semester course in Unix and C probably isn't seen as too acedemic by many. Which I think is a shame. Too many universities turn out gimps fluent in one langauge, and one language only - Windows *shudder*. I think it sad that units to teach people how to click mice and use Word can get you acedemic credit, but Unix and C courses don't seem worthy enough to run.
When my time was up in Canada, I came back to Australia and while I finished my degree, I made money on the side doing CGI scripts in Perl. Then, when my degree was finished, I applied for a job as a System Admin at a department at The University of Western.. Australia. It was the first job I applied for and I got a callback the morning after I had a 70 minute panel interview. Due, in large part, to the stuff I had learnt in David's class, I passed the interview quite well.
Today, I am 22, earn over AU$40k, I get to play with lots of cool computing and network hardware, and I think it would be safe to say that if I hadn't taken that course with David, I wouldn't be where I am today. I suspect I would have been working as a security guard, making minimum wage, since my degree wasn't actually in Computer Science, but Security Studies. Thinking back, I'm pretty damn glad I did take it
David's homepage is here
Can you still download the archives? If so, where?
All that info would be incredibly useful!
What format do you think it would be in? Threaded text or database format or what? How would you read it or search it?
Also, what do they do with the attachments? Imagine THAT archive. Heh heh heh.
I found some 7-8 year old posts I made when I was a teenager. I can't believe how cocky I was, and how poorly I wrote. Very few people ever replied to my posts, and I now understand why. I even found a "me too" (well, almost) post from myself. Wow, that's scary.
I appologize to the whole slashdot community for my teen cockiness in the mid 90's. I didn't mean what I said the way I said it...at least looking back.
One good way to find your old posts is to search for your (old?) email address.
-Pete
Soccer Goal Plans
You posted your messages on an international network of servers that store messages and provide anyone with access to them. It's a little late to consider them secret. Why does it offend you that someone's storing them and providing anyone with access to them?
Google Groups is simply a very large and fancy news server that doesn't expire articles, and you implicitly granted permission for your articles to be stored on news servers by posting them in the first place.
My question is how Google determines whether someone is the real poster of a message. Can just anyone demand the removal of any message they don't like?
OK: how long before a presidential candidate's Usenet postings will be dragged out for the whole world (US) to see ?
Looking at the history, the first big Usenet spams came at exactly the wrong time- and it badly twisted the subsequent development of the Web.
Spam hurt Usenet by ruining it as a tourist destination right as mass tourism to the Web began. Long-time Usenet users couldn't recommend it to new Internet users ( "Really its a great place, just ignore the trash and the noise and don't give your name because you'll get a zillion ugly mails afterward" doesn't work as tourist advice). And for existing users, reading Usenet meant wading through muck, and then with address harvesting starting, a muck filled mailbox. Between this and the constant interruption of irrelevant ads, people were driven out, the extra traffic made Usenet a burden to ISPs, old users went elsewhere, new users never came. While the rest of the web exploded, Usenet started its long fade.
Arguing alternate history here, but if mass Spam had hit much earlier or later, the damage wouldn't have been as bad, both to Usenet or to the Web overall. Had it been much earlier, perhaps the cancelbots and other technology responses to spam would've been well developed by the time the mass tourism started. "let's ignore the problem and go somewhere else" isn't a solution when there is no 'else' to go to. Had it been much later, higher adoption rates for Usenet (as a % of all Web demand) would mean companies would need to take the Usenet model into account: people might've expected/demanded better spam solutions, more cross-website communications, and less walled-gardens. People would've been less likely to accept 'the only protection you'll get is to stop posting and come to our walled-garden web discussion group' as a solution. Ditto with the loss of shell accounts and open relays.
How can I post replies to articles that I find?
So, you are saying that USENET has changed from an informal discussion group to a searchable perpetual repository of technical support Q&As, plus a repository of background information on people who were foolish enough in the 1980s to post under their own names. I agree. The part I don't understand how you think that constitutes "saving" USENET. USENET didn't use to be much of an on-line community compared to some of the others, but it was a community. Once it became archival, anonymous, and searchable, that went away. Who, after all, wants their every word recorded and replayed into perpetuity?
The original SAIL users were contacted, one by one, and offered CD-ROM copies of their files. Where the original users permit, their files will be made publicly available. The permission process is still going on, but the result will be an archive of the early days of AI.
What happens to the archive when they're bought by someone else, or end up in bankruptcy court? Will it go the away of the online digital photo storing sites, vanishing one day without a trace, taking irreplaceable data -- data of immense academic historical interest -- with it?
Google should promise to donate the archive to the Library of Congress, do the transfer now, and make a social contract with the net community to turn over the reigns on this project if they're acquired or go out of business.
As a regular USENET poster, I'm gratified that you've found our posts useful, but please, please do consider participating yourself!
"But I don't know anything worth posting!" , I hear you cry. Well, for a start, since when has that stopped anyone on USENET, myself included! Besides, I'm sure everyone knows something about something, even if it's "only" mexican cooking (alt.food.mexican-cooking) and Italian manga (alt.italian.anime-manga).
Take the trouble to subscribe to a few groups and get involved. Keep them as lively discussion fora, not dusty historical archives and a spam collection!
I discovered USENET in 1992, and I've rarely gone away. It's definitely the most consistently interesting and useful part of the Internet, IMHO.
--
Hell, a converted VCR using VHS as a backup medium can store like 100GB (saw one somewhere, I forget the link.)
Assuming 9 Mbps of raw data (half the data rate of HDTV, because garden-variety VHS is nowhere near broadcast-quality), and assuming some heavy-duty error correction reducing effective data rate to 6 Mbps, VHS's SP mode records for 7200 seconds, giving 5 gigabytes on a tape at a bare minimum. (For comparison, a single-layer DVD holds about 4 1/2 GB.) If we go to EP mode, increase the bandwidth to S-VHS levels, and apply 3:1 text compression (common with deflation of large Latin-alphabet texts, especially containing quoted material), we may be able to store even more data per tape.
Will I retire or break 10K?