Slashdot Mirror
Library of Congress Offers Update On Huge Twitter Archive Project
Nerval's Lobster writes "Back in April 2010, the Library of Congress agreed to archive four years' worth of public Tweets. Even by the standards of the nation's most famous research library, the goal was an ambitious one. The librarians needed to build a sustainable system for receiving and preserving an enormous number of Tweets, then organize that dataset by date. At the time, Twitter also agreed to provide future public Tweets to the Library under the same terms, meaning any system would need the ability to scale up to epic size. The resulting archive is around 300 TB in size. But there's still a huge challenge: the Library needs to make that huge dataset accessible to researchers in a way they can actually use. Right now, even a single query of the 2006-2010 archive takes as many as 24 hours to execute, which limits researchers' ability to do work in a timely way."
Why does the federal government need to archive the useless information twitter calls tweets .. yet another huge wast of my money (being a taxpayer and all)
Is there limitation hardware or software? Where is the bottleneck?
Just give me a csv.
provide a limited version of the database with only some information from the tweets, so there's less data to search through? (of course, keep the full data in case a search depends on it)
I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
Just buy batches of 300 of those 1Tb flash drives in the article below and pass them out to the researches as needed?
You had better turn on indexing.
Some of the most important historical knowledge comes from things that people at the time wouldn't consider important. Things like grocery lists can help determine the diets and agricultural abilities of a culture at the time.
For an example I just made up: In the future, the presence or lack of traffic reports could, alongside legal/budget records, help a historian verify the spread/development of roadways.
Twitter could be a huge source of topics and a wealth of information for historians in the future.
They may conclude that we were all idiots. This too, counts as useful information.
So, just how many 'Libraries of Congress' are there in 300TB? ;-)
Does this mean that as the archives swell, the metric does also?
Where does this madness end?
Down With Slashdot BETA!!! I've been around the corner and seen the oliphant; you can only abuse me from your perspecti
It's illegal to make a copy of any of those other things though.
1 Library of Congress ~ 10Tb of data
Therefore, the database will be around 30 LoCs in size.
But, if we consider this database as part of the Library of Congress, we get a fixpoint problem..
"The Library's mission is to support the Congress in fulfilling its constitutional duties and to further the progress of knowledge and creativity for the benefit of the American people." (from its website.) No, I don't see how archiving Twits and tweets furthers this mission *at all.*
... oh, wait, sorry. That's in the FBI's mission statement.
It's not much of a step from there to archiving all the phone conversations of all Americans
Isn't Copyright just the worst?
300TB worth of tweets, which are basically very small text files? A single tweet, that uses all available character should only be 140 bytes. I just refuse to believe that there is 2+ trillions tweets out there, to make up 280+TB. Considering 1 billion tweets would be 140GB. (unless I'm failing massively at math here, which is quite possible.)
Look, I don't know about you, but we process hundreds of TB of data when we process genomes, using this fancy stuff called "databases", "hash indexing", and fancy software that may be hard for you to find like Perl, C, and various scripting languages.
It's fairly simple coding. Just build an index hash from keywords (which are all preceded by #), add another index by words (ignoring all the bit.ly and other web links), add a third index by @ reference (aka user names, which are really just a 20 character part of an SMS message), and go to town.
We do it every day.
Now, you've got a few extra complexities, we tend to use GACT and similar short codes, but we also have to add skips, nulls, misreads, ambiguities, so it's usually 8 symbols and you're looking at an extended ASCII power.
Still, you're getting obsessed with the size (which is nothing compared to a genome, and we have drives much much bigger than that).
Just do it and stop thinking it's "hard". It isn't. Buy a decent Perl book for Biochemistry or Genetics and get cracking. We wrote most of the code you'll need to build new libraries from.
-- Tigger warning: This post may contain tiggers! --
Why is parent modded to 0? Storing them on any number of services cloud services would be a lot cheaper than building their own system. Amazon and Google already host public datasets for researchers over 300tb. Hell, they could just agree to pay Twitter a service fee for data and keep offline tape backups. While we are at it, why not maintain a Torrent of each year?
Is there anything better than clicking through Microsoft ads on Slashdot?
What confuses me:
Percentage of Americans with Accounts:
Twitter: 13%
Facebook: 70%
So there is FAR less diversity, and extremely poor quality data, why did they not archive public Facebook posts instead?
I see it as, facebook hosts people who write articles, stories, poems, songs, music, pictures, etc. THAT is the point of the Library of Congress: Documenting and Preserving Culture. Not trying to datamine the history behind "WAT R U DOIN FRI GRRL?",
Yep, the ancient rubbish pit is often the most informative part of an archeological dig, however this is more along the lines of Samuel Peeps' diary. Four years worth of tweets is a bit over the top, IMHO a few random days and a few significant days would be all you really need. I have something similar at home, it's a large coffee table book that has one page of newspaper clippings for every month of the 20th century.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
All your meme are belong to us!
You are being MICROattacked, from various angles, in a SOFT manner.
They aren't. AC posts always start at 0. Welcome to Slashdot.
Researchers are hampered by all the CPU cycles going to FBI and CIA searches. (Makes me think of Person on Interest)
It has its place.
Thing is YOU don't get to define what future generations think of you and your civilization, if you want to help them to form an accurate view rather than just the image you want to portray then you need to leave some juicy rubbish dumps undisturbed, this is one such dump. I'd question the justification for the size of this particular dump but you make it sound like they are throwing out Mark Twain to make room for twitter. You know they have the resources to do both things at the same time, and that this project probably cost less than a single hand written Twain manuscript, right?
At the end of the day I see stuff like this as a GoodThing(TM). I'd much rather live in a society that over-values it's everyday trivia than one that under-values it's past, or worst still goes out of it's way to destroy it (Taliban). The Victorian English were the ones who started the drive for museums and preserving/understanding the past. Before that nobody really bothered about social heritage, it was all about family heritage. Egypt is a great example, they started looking after their own heritage after the British showed so much interest in digging up their monuments and taking the home with them. A large portion of the old city of Cairo is made from the lime that used to cover the pyramids (their sides were originally flat and white), all that's left is a little cap of lime on top of the largest one. Once the pyramid making fad and families had well and truly died, the people of Cario simply didn't care about a huge monument built by some long dead family, to them it was no more than a convenient source of building material that a long dead Pharaoh had left lying around.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
And what, exactly, is going to show our future kin if we archive all of Twitter? I'll tell you one thing, swallowing it is going to be very, very hard.
So how big, in Libraries of Congresses, is the archive that they're adding from Twitter, to said Library of Congress?
This tagline was transcoded to result in at least one smirk. If you experience failure to smirk, please consult your Gen
The average life of an inane conversation used to be maybe 15 minutes. I'm not sure the world is a better place for having extended that.
In the old days of USENET, conversation threads used to run for weeks, sometimes months, actually.
Not minutes.
of course, back then, we actually knew who everyone was, and could ping and finger them.
-- Tigger warning: This post may contain tiggers! --
A substantial number of posts are literal duplicates by known spambots.
You could store those separately as well as the Retweets (RTs).
Then, think about what typically gets posted.
Most might be something like 520,000 variations on "Touchdown!" or "That's gotta hurt!" during sporting events, or "It's snowing!"
A lot of the rest are probably repeats of what someone just said on Comedy Network or during a TV program. They will all be at about the same time in a region and be substantially the same thing, with 50,000 mispelt variations.
Add the ACs and it's a lot smaller than you think. Most of the rest of that are still duplicates of something somebody else wrote, but without attribution.
-- Tigger warning: This post may contain tiggers! --
300TB is about right. Twitter says they have 400 million tweets per day. Figure about 500 bytes per message with text, and metadata (source, destination, timestamp, flags). 400,000,000 msgs/day * 365*4 days * 500 bytes = 292,000,000,000,000 bytes.
Twitter offers a feed of 1 in 10,000 public tweets, so you can see how banal it is. I had a program monitoring that for a while, extracting links and evaluating them for spam. It's about as bad as you'd expect.
only 0.6 Libraries of Congress.
Right now, even a single query of the 2006-2010 archive takes as many as 24 hours to execute.
Why? Why does it take so long?
They talk about the hardware and software not being up to scratch, but many other companies seem to be able to process huge amounts of data quickly. Google, for one, seems to do it.
This post contains benzene, nitrosamines, formaldehyde and hydrogen cyanide.
Oh, right. I would like to point out my old-skoolish 6-digit uid!
Is there anything better than clicking through Microsoft ads on Slashdot?