Slashdot Mirror
How Do You Keep Track of Your Web-Based Research?
time961 asks: "I use the Web extensively to research a wide variety of topics (weird, huh?). However, much of the time I end up printing out web pages and filing them on paper, because that's the easiest way I know to say 'OK, that was interesting, I'll hold on to it until I actually do something about this topic'. Often, I'll run across something that seems relevant to a long-term project or interest and just want to grab it without even reading the details. Paper is OK for reading, browsing, and scribbling, but it's hard to search, it's heavy, and it's wasteful (and I yearn for a day when browsers can reliably print what's on the screen, instead of cutting it off at the margin because some designer doesn't understand layout!). How do others deal with organizing the results of browsing?"
Bookmarks and histories aren't the answer — they're not very good for searching, the UI isn't very good for, say, adding notes, and they don't work offline. Also, stale URLs are a huge problem — a key advantage of paper is that it doesn't randomly fade out in a few days (or decades), so a good solution would have to keep copies, not just references. I imagine something like a FireFox plug-in with a 'Remember This' button and some options for category, keywords, annotations, etc., but I'll bet there are more creative approaches, too."
First off, install a good PDF printer.
Just save your 'research' to a nice media server or something and then you can do the 'hands on' stuff once the missus has left for work innit.
File -> "save page as" -> "web page, complete".
You can either keep what you save in some sort of logical arrangement, or trust your handy desktop search engine to find it for you later (though that seems to reduce the problem back to finding the info in the first place, though at least you don't need to worry about the content going offline at some future date.
Just write to your ISP pretending to work for one **AA and you'll immediately get a complete list of your activities. As a bonus, you can also use that to terminate your subscription without the 2 mounthes notice.
Any time someone mentions how they don't like having papers around but want a hard copy, my response is immediately, print it to PDF! Your operating system should be able to do this :) Linux firefox, print to generic printer to a file named something.ps, then run ps2pdf on it, in just about every other GNOME app PDF support is built in to the print dialog. Mac OS X, well, you already knew you could save PDF (or save the preview, same diff) from your print dialog. Windows: www.sf.net/projects/pdfcreator is your friend - just don't install their toolbar (the existence of which makes me rather sad). Then, you've got the page (or whatever) archived in a nice, portable, paper-like file, and when desktop search is ready for the masses (if you're not on a Mac), you'll even be able to search it - much better than paper!
I recognize people by their sigs. Is that a bad thing?
You can find all the features in a nice list at the official homepage with tons of pretty screenshots. There's even a 50 page manual (PDF) created by Andrew Giles-Peters.
Even though development has seemingly halted since December 2005, it's still one of the most well rounded extensions for Firefox I've come across yet.
Perfect is the enemy of done.
wget is probably one of my favourite Linux command-line tools. All I need to do is wget -r http://www.doodahdoo.com/ and it saves a directory called doodahdoo.com and all the pages in it, as well as the images, and any embedded video and such. This is very handy, not only for getting a huge number of files (say my http backup server), but also for getting entire sites that I might have a use for in future.
At the moment, I have on order of 10GB just of websites, radio clips, and what have you that I have used for previous research. Not only that but I can also maintain a simple directory structure and never have to worry that that "firefox plugin" will still be compatible with version 4.765.
Another neat function is you can specify just a particular files (www.whatever.com/pic.jpg), or all the files with a particular extension *.jpg, or only the files in that directory. You can also use it to spider (limited) all the links on a site. Though be kind and don't do this too often, as I am sure it eats a lot of bandwidth.
The last (and greatest) thing, is it remains in a well-known and easily editable format.
Alternatively, I have also used a MediaWiki setup so that I could drop down notes for classes, or other interesting things in it, but this required substantially more overhead than wget.
I've been struggling with this myself, to a point. How about a personal wiki, such as Didiwiki, that runs locally?
I also save web pages as "Web Page, Complete". It now occurs to me that I should make a specific directory for those pages.
1 in 4 Maine children in struggle with hunger.
http://www.devon-technologies.com/products/devonth ink/
Using a good PDF exporter (I'm on OS X, so look elsewhere for free & easy ways to do this on Windows), DEVONthink will pretty much keep everything organized like a digital filing cabinet.
'Course, the cheapest version costs $39.95, but I can attest to the fact that this software WORKS (I got it heavily discounted in the MacHeist 2006 bundle).
I quite like Yojimbo http://www.barebones.com/products/yojimbo/
You can either save a "web archive", which is the web page incl. all graphics/css/etc., or a PDF of the page (nicely integrated into print services). Both document types are rendered inside the app and are searchable. Yojimbo has also tags and folders to keep things organised. And you can also save regular notes (formated and with images). Covers all bases.
When it comes to pure PDF, YEP http://www.yepthat.com/ is an excellent alternative. Kind of the iPhoto of PDF.
PDFCreator is a free open source pdf printer http://www.pdfforge.org/products/pdfcreator
CutePDF.
See my journal for slashdot ID's by year. Mine created in 2005. http://slashdot.org/journal/289875/slashdot-ids-by-year
// MD_Update(&m,buf,j);
Maybe something in Firefox one day that'll tell you that your bookmarking something again?
5 3
Ask and ye shall receive!
http://bookmarkdd.mozdev.org/
Or the Mozilla Addons page for it :
https://addons.mozilla.org/en-US/firefox/addon/15
XenoPhage
Technological Musings
It's a program that allows you to easily save a copy of just about anything (certainly anything on the web...) with links to the original and everything else. The notes are automatically stored in chronological order for browsing. You can also apply tags to your liking and it has full search capabilities as well. It's free for the regular version, if you want to import handwritten notes and have them be searchable as well there's a charge.
It's awesome and I think fits your needs exactly, or at least I use it to meet the needs you described and I've had no problems with it.
Now if I could just force myself to go back and do something with the research later...
P.S. There's a writer in The Atlantic named James Fallows who has a column on useful technology tools. That's where I first learned of Evernote. He had several other suggestions to fit the bill in that column and more generally, he's usually worth a read.
In my bookmarks folder I have a "Projects" folder.
Within my "Projects" folder I have an alphabetic listing of folders with each project's name.
If the project is small, I fill it directly with book marks. I do take the time to add notes, because if the URL does go stale, the notes will let me know what I'm now missing. More often than not, missing information can be replaced in the future with another URL that has the same or more up-to-date information. Additionally Google Desktop searches my bookmarks file, so I just double-click ctrl and can search via keywords that way.
This whole setup is a bit of a hack, but it's worked. I'm hoping either Firefox 3.0 will have a fantastic bookmark manager or a plug-in author creates something truly wonderful for the existing bookmark system.
In Opera you can select some text in a webpage, then right-click and select "Copy to note" (Shift-Ctrl-C). Notes are stored in a panel, and double clicking a note will load the webpage it came from. Handy.
I'm normally not a web 2.0 bandwagon type of person, but del.icio.us is probably the most useful thing for this that I have ever run across.
pros:
-tagging
-descriptions
-accessible from anywhere
-really simple to add to (with firefox plugin)
-searchable
cons:
-web pages are ephemeral
-del.icio.is itself could go away someday, and I'm not sure how to back it up locally
The best way to address the issue of web pages being ephemeral is to, as others have said, print to pdf. You mac people have it nice in this regard, but it is not hard to set up on windows or *ix.
I also mentioned that del.icio.us was searchable, but only the tags, titles, and descriptions. I fully expect google to someday roll out a similar service someday that lets you search through the pages you have tagged. That would be very useful.
I also like the suggestion of a personal wiki, but more for keeping track of little "tips and tricks" that I stumble upon rather than entire web pages.
Finkployd
Zoot http://www.zootsoftware.com/ may meet your needs.
Self awareness - try it!
At risk of getting modded down for recommending a Microsoft product here, you might want to look into OneNote 2007 (or one of the versions of Office 2007 that include it.)
It comes with a "print to..." driver so you can print to your OneNote notebook, and provides a good framework for organizing your notes, and you don't need to kill as many trees as printing to paper.
Another possibility is to get a PDF printer; you can either just organize your notes with file system folders, or if you want something a little bit more useful to track relations between different items, you can use something like PersonalBrain to for organization.
I use text file and the Firefox Copy URL + extension:
:: Firefox Add-ons9
Copy URL +
"The Copy URL+ extension enables you to copy to the clipboard the current
document's address along with additional information such as the document's
title, the current selection or both."
https://addons.mozilla.org/en-US/firefox/addon/12
It installs a context-menu, allowing you to copy any or all of page title, URL, and most importantly: the text currently selected.
At other times, I use bookmarks in a new folder specific to the subject. You can add keywords to bookmarks in FF.