Slashdot Mirror
Building a Fast Wikipedia Offline Reader
ttsiod writes "An internet connection is not always at hand. I wanted to install Wikipedia on my laptop to be able to carry it along with me on business trips. After trying and rejecting the normal (MySQL-based) procedure, I quickly hacked a much better one over the weekend, using open source tools. Highlights: (1) Very fast searching. (2) Keyword (actually, title words) based searching. (3) Search produces multiple possible articles, sorted by probability (you choose amongst them). (4) LaTeX based rendering for mathematical equations. (5) Hard disk usage is minimal: space for the original .bz2 file plus the index built through Xapian. (6) Orders of magnitude faster to install (a matter of hours) compared to loading the 'dump' into MySQL — which, if you want to enable keyword searching, takes days."
After doing all that, I think you may have missed your flight! :)
Most folk'll never lose a toe, and then again some folk'll...
Great Job, that is the power of open source for ya.
Now we need to work on porting that to over OS's and we will be set.
WulframII - Free Online Mutiplayer 3D Tank Shooting Game
This may seem like a stupid, trivial, and pointless project, but the programmer may have gained something from it that he could use later in something you don't feel that way about. If the programmer enjoyed doing it, that might have lead to a more productive coding session later in the day too.
Humans are slow, innaccurate, and brilliant; computers are fast, acurrate, and dumb; together they are unbeatable
I'll bite...Unfortunately, I don't have a basement, so therefore there are times that I am required to venture into the outer realm that happens to be heated by the big ball of gas known as Sol, as opposed to a pump ;P Seriously though, this is exactly what I have been looking for. What better way to show up your friends when they cry "You're wrong, google it!" knowing that there is no connection possible within twenty miles. Next time i'm drunk at the beach and someone wants to pretend to know the history of coffee harvesting, it's on.
Let us know when you're ready for prime time
"Up to now, installing a local copy of Wikipedia is not for the faint of heart: it requires a LAMP or WAMP installation"
I'm sorry it requires WHAT in order to display HTML pages?
"Right now (August, 2007) the file is a 2.9GB download, always available from here. I"
So an unmassaged file would fit onto a standard DVD. And a fully indexed file wouldn't take much more. Plus your equations could be images.
Building Cory Doctorow a better haircut.
Kick sand in their face!
Engineering is the art of compromise.
I'm on 14.4Kbps dial-up, you insensitive clod.
And that's no joke. Noisy phone lines suck. It could be worse; I could have been on an OLPC machine in Africa.
tasks(723) drafts(105) languages(484) examples(29106)
Why?
Ironically, You're already reading slashdot. You had just wasted your time.
You're reading it again, wasting more time ehh??....
But the point is, if programming an offline wikipedia makes you happy and you don't need the money then you would understand....
George W Bush
Is a dick head!!!!11
Disclaimer: Disregard the above post.
What's the point of it if there are no vandals or flame wars to make it interesting?
Combine this and one of the new E-ink ebook readers, make it pretty rugged, slap a solar panel on the back and man. . . you have something really close to a genuine hitchhiker's guide to the galaxy. Ah, I love where technology is heading =)
I'm perfect in every way, except for my humility.
compared to loading the 'dump' into MySQL -- which, if you want to enable keyword searching, takes days."
Do you mean searching takes days, or loading? Searching should be quick if you index the words. If you are duplicating a bunch of local clones of wiki, then simply copy down the raw MySql table data files rather than reload from delimited files etc. (One needs to make sure their version of MySql is compatible with the table file format.)
Table-ized A.I.
But....but....I thought MySQL was fast!
Use 'slashdot stuff' in the subject line in any email you send me if you want to get past the spam filter.
At least you'd have a line of young bucks waiting to crank the OLPC for a few minutes... -->interpret that however you'd like, it works.
And that doesn't happen offline? Only naive people like you need to be worried about reading Wikipedia.
There are bastards of every academic, social, and financial background.
Its really sad that i know both of those.
"Sic Semper Tyrannosaurus Rex."
My very serious question to you is how much better do you think things are at a "real" encyclopedia. They have many of the same problems, but they are just not public. "Real" encyclopedias can be just an inaccurate as the Wikipedia on many articles. For a quick first reference, Wikipedia is an ideal tool. Just be sure to take things with a grain of salt if you are not checking the sources for further information. Guess what though, the same applies to "real" encyclopedias too. One difference is that with "real" encyclopedias, you always lack revision information, and you often lack information about the sources used by the editors. (Some encyclopedias are better than others in that respect.)
Stylish sheet to fix many problems in Slashdot's D3: https://gist.github.com/801524
Because it might be useful to have something stored locally. I travel a lot with my laptop and I would like this. I would also appreciate the convenience of not having to fire up a web browser for wikipedia. You can search articles from the command line. You could also potentially write a better search feature, ie: bolt on some code to combine and summarize multiple related articles. The approach the guy used (a bunch of small bz2 files) is interesting and potentially useful. I'd say this was one of the better articles to hit Slashdot lately, and I'm glad I read it.
In Soviet America the banks rob you!
Uhh, why not just save yourself a few hours and download the html pages from here http://static.wikipedia.org/downloads/April_2007/e n/ and search with Google Desktop indexer http://desktop.google.com/.
For a change it's not just a link to a .tar.gz somewhere, but an actual article where he goes through what he did, and (more important) why he did things that way. Good reading even if you don't want an off-line Wikipedia.
Unlimited growth == Cancer.
It might defend on the topic/field in question. The articles you reference seem to be focused on tech stuff. I use wikipedia primarily for socioeconomic reference material, and find it in general to be pretty solid. There are places where the depth is limited, but it's definitely my first-reach resource as long as I have an internet connection - mainly because many of the specific things I'm after might not be in a general encyclopedia like Britannica - intertemporal equilibrium, hedonic regression, Edgeworth's limit theorem, the Bertrand paradox etc, etc.
A-Bomb
tasks(723) drafts(105) languages(484) examples(29106)
Yikes, defend = depend
A-Bomb
"ollusion to ruin people's lives when they run afoul of admins, corrupt editors doing and getting favors from the head honcho himself, pet pages that end up with incorrect information, speculation, or specious reasoning, and a general air of arrogance and groupthink reinforcing an internal idea that they can do no wrong."
Plus, I hear that's all trippled in the last six months!
Worthington's Law is the only economics anybody needs to know.
After all, I am strangely colored.
Yes, the paper encyclopedias are missing all the anime trivia. Christ, its embarassing to see "references in pop culture" sections which just spell out every geeky guy stereotype. I dont know why those people dont get banned. Everything in existance has an anime reference. That is unsettling.
Realize that some of the greatest things done by humankind were from doing "pointless projects" as you call them. Prime numbers for instance were studied by mathematicians just for fun, and now look, they're used for cryptography. Try doing your banking without them.
Complex numbers originated from something "useless" like trying to solve the quartic polynomial in radicals...try building a bridge without them. In fact all of science is built upon people going in random tangents doing things they enjoy, discovering seemingly "useless facts" but most of it becomes useful *and* gives us an idea of the universe in which we live.
Only working on immediate practical problems is very shortsighted, and if mandated throughout the academic community, would mean the death of innovation and most discoveries.
"What lies behind us, and what lies before us are tiny matters compared to what lies within us." Ralph Waldo Emerson
It only takes days if you use the php import script to import the sql dump, which was not designed for importing the entire dump.
Use the ANSI C implementation, which takes about 20 minutes to convert the XML to SQL and then takes a few hours to import into MySQL. Please not that you need a properly configured MySQL server in order to efficiently run a local copy of Wikipedia, which must have at least 8GB of ram.
http://meta.wikimedia.org/wiki/Xml2sql
python -c "x='python -c %sx=%s; print x%%(chr(34),repr(x),chr(34))%s'; print x%(chr(34),repr(x),chr(34))"
I would be concerned that Slimvirgin and the other intelligence agent(s) might not be able to revert and ban the edits I would be making offline. Maybe Jimbo can give them authority to come rough me up at home and beat my lcd with a hammer.
9 43254
http://yro.slashdot.org/article.pl?sid=07/07/27/1
They say to you that their hobby is painting/music/walking/repairing old car/gardening/making reduced model etc... And they seem to think that their hobby are perfectly acceptable. But as soon as you say you like to program stuff, they don't understand how this would be a hobby. They mostly fail to recognize that every one of us has something in common : the joy of act of creation. The fact that our hobby entail creating something immaterial and full of "logic" does not matter. It is still a joy.
C. Sagan : A demon haunted world:
http://www.amazon.com/gp/product/0345409469/
visit randi.org
is very very slow when you do it on a normal installation, the reason is MySQL comes with a "be nice to people who don't know what they are doing" setup. Go into the my.cnf and find the buffer settings, crank them up and restart the server. It can really do a lot (especially if you are running InnoDB which you of course are since MyISAM isn't a proper database).
For those who didn't know, Xapian, the search engine he used for this, is really awesome. Its very fast, stable, actively developed and packs some pretty impressive features. Its written in c++, but has bindings for Perl, Python, PHP, Java, Tcl, C#, and Ruby. If you need an embedded search function on a site, you should check it out.
I've used it for over 2 years on various sites and am really pleased with it.
Sam has one liberty, which he sacrifices for one security. Can you tell me what Sam has now?
TFA is:
1. Not a thinly-veiled attempt to advertise a crappy product
2. Not bashing Microsoft
3. Not about somebody who is trolling open-source (i.e. SCO)
4. Not about Bush taking away all our rights and ending freedom
5. Not about voting fraud and the end of democracy/America/the world
6. Not decrying Vista DRM and its ties to the MAFIAA
7. Posted on Slashdot
Furthermore, TFA is interesting and informative.
Am I in heaven?
I use Wikipedia, as I use the Internet, for geeky computer stuff and electronics tech, tools, and info.
I can't imagine ever taking it that seriously that I would use it for mainstream 'non-nerd' stuff.
Microsoft says legacy (serial/parallel) ports are bad. They don't obfuscate the hardware enough.
Unfortunately for Wikipedia, the quality or lack of it in competing encyclopedias does not resolve the problems in Wikipedia. I hope Wikipedia can work on these issues because I am seeing some of it too. I'm also seeing article rot being quite common, in that old articles deteriorate, and not really from a lack of good will either. Someone discuss the problem in a blog here: http://nonbovine-ruminations.blogspot.com/2007/02/ where-are-stable-versions.html
Beware: In C++, your friends can see your privates!
Can't help but assume there will be a cease and desist order in the /. headlines tomorrow.
-fb Everything not expressly forbidden is now mandatory.
I know that not everyone has a permanent connection to the net everywhere they go, but what is the point of storing a local copy of Wikipedia?
The beauty of it is that it is online and always up-to-date (wrong, or less wrong).
Trying to capture it locally seems to me to be like trying to print The Internet. By the time it's done spooling, it's out of date.
If it's an academic project, that's really cool, but I don't see a practical point to it.
Message contains 1 attachment: spam.gif
Dude, WP:1.0 wants YOU.
Wikipedia seems the best place for the author's "how to download and use offline".
Emerald Astrology
How is this different to moulin which is a fully interactive, offline version of the entire Wikipedia (without pictures) on a CD-ROM:
http://moulinwiki.org/l/en/
There's a one-button (for admins) export-the-whole-wiki-as-html feature in modern MediaWiki software installs...
But hey, two days and a few hundred lines of code is cool. You geek (verb). If we always took the easy way out we'd be using Windows and have committed suicide long ago.
MyISAM is very limited compared to other databases but at least it's a lot faster for some specific (useful) loads. InnoDB is not faster than other databases but is still rather limited. Ergo: use MySQL with MyISAM if your problem is a good fit to its capabilities, use another database (PostgreSQL, Firebird, MSSQL,
A PSP is very portable (fits in your sweater/backpack), hackable, and has up to 8Gb of storage. I have been dreaming for an year about porting wikipedia to it. Unfortunately I'm not familiar with the kind of programming needed and I could never find the time...
There's a serious problem with the article's way of treating the data that I didn't see addressed.
The wikipedia database file is one large bzip2'ed XML file which the author splits into blocks of 900k (bzip2's natural blocking) which he then parses for the "title" and "text" XML tags.
The problem with that approach is that some of these tags may well end up being split over block boundaries, so some articles risk being missed. EG:
END-OF-BLOCK: blablablabla...blabla[/text][othertag][ti
START-OF-NEXT-BLOCK: tle][sometag]blablablablabla...
So searching for "[title]" in boths blocks separately like TFA does will fail for one article.
(I've used square brackets instead of lessthans and greaterthans because slashdot won't let me use them.)
They need a "bad guy" on the block in order to redirect all the user hatred to them and the other (much worse) companies will pass by quietly. Like Microsoft and Novell.
Is anybody surprised that we are flooded with SCO stories?
Also companies are very afraid of stock-boom. With the SCO example they are trying to prove that stock boom can happen only if you are up against other -bigger- companies. While the fact is that stock boom is just around the corner for Microsoft and Novell; people have to simply alert the investors.
why are people saying this is useless? it's just what I need, and I was suprised when I went looking and couldn't find it when I first wanted it.
I cruise around in a sailboat. my longest passage was 35 days. how I would have LOVED to have been able to read wikipedia articles on that passage, even if they were a few weeks old. What do I care if an article is a few weeks old? 35 days at sea I'd have read a paper encyclopedia if I had one, but my boat isn't big enough to carry the weight of a paper encyclopedia. It sails like shit as it is from how many books I have stuffed in my V-berth.
and sometimes I'm on some random little island with no internet access for a periods of time... hanging out with a bunch of other sailors, and of course we get into discussions that leave us wishing we could go google something.
Even in Bora Bora, they had internet but it was 24$/hour, on crappy old computers! this would have been great!
and now! now I'm in China! They block parts of wikipedia. yeah I can setup and SSH tunnel when I happen to have internet access available, but how great it is to have a local (though somewhat outdated) copy of wikipedia, including any blocked articles!
sounds great!
Are there any projects putting wp on java enabled phones? It would be pretty cool to settle those arguments any time, any place.
I just need an offline wikipedia now.
They're using their grammar skills there.
What would happen if a title element is split between two bzip files? Or the content of an article is split in the same way?
Well I agree, and I think there is a (near?) consensus in WP policy that such things are not useful, unless they are particularly notable, and should be deleted.
For every expert, there is an equal and opposite expert. - Arthur C. Clarke
Good thinking. I was just wondering the same thing about my PalmOS PDA. It has plenty of storage available. I wonder if the existing Python port would be sufficiently powerful to run this.
Sometimes I worry that I'll develop Alzheimer's disease, but no one will notice.
Tell me we haven't reached a point where friends will argue, know he location to the nearest cellular tower and debate simply because they know you can't Google it or look it up on Wikipedia... What did you do before the Internet? Carry around a complete set of Encyclopedias just to prove that your right all the time?
I'd find new friends. Seriously. NOTHING is that important.
Every time I start to have faith in humanity, I ruin it by driving to work between 7 and 8 am.
The thing is, a portable version of wikipedia has been already developed:
http://encyclopodia.sourceforge.net/en/index.html
for the iPod; also the Encyclopodia Ebook format (basically an indexed b2zipped articles or blocks), is far better suited for portable devices.
Now if any PSP/DS/Palm developer is reading this...
its much better to get a pocketpc phone and have it on that, then no matter where you are you can immediatly call bullshit on people trying to sound smart. i love calling people on that bullshit "10% of your brain" thing.
So Skulldilocks threw acid on the schoolchildrens' faces, cause somebody from the bible told her to do it!
he'll be better of using lucene for search, faster than mysql
The core idea for accessing bzip-compressed data is interesting.
The later execution, however... Perl, Python, PHP, Xapian, Django all together as the runtime, and add to this C code for the preparation (I might be wrong about the details, just skimmed), for such a small application.
The other poster who rejoices about how wonderfully old-skool TFA is, is obviously right. This kind of duct-tape Linux development feels badly sooo 90's and smells like maintenance, performance, installation and portability problems.
Keeping it all in just one of the scripting languages would make it much more serious. (Perl, or maybe Bash for the easiest installation?)
This project *is* of immediate practical interest to its creator.
Anyone with experience of sdict?
They offer a dictionary reader for various systems, including portable devices, and dictionaries including Wikipedia.
Unfortunately their Wikipedia dict is a old (January), but it seems like a good approach for laptops or other small devices. When I get an 8Gb SDHC I'm going to try it on my Nokia N800.
In a time honored tradition, he could start a flame war over a trivial aspect of the existing lolcatgenerator, then decide to fork the code. If you're going to do something useless, might as well piss as many people off as possible before doing something redundant.
The world is made by those who show up for the job.
Dang it! I was all set to throw a plank across the creek behind my house. Now I've got to get out my math books or die in the effort. Curse you!
The world is made by those who show up for the job.
While interesting, it's certainly reinventing the wheel. There are lots of methods for doing this found on the site itself ( http://en.wikipedia.org/wiki/Wikipedia:Database_do wnload ) including static content already marked up.
Also, as others have noted, the his choping the file into chunks means you're going to loose at least one article per chunk.
I'd implemented this with a compressed file system and maybe some symlinks. Happily, the static content is already there for the taking. Some find and grep on a file system should be enough to do a title search with little overhead. The web server ( for searches from a browser ) need be little more than a daemon, in perl, python, etc, you could do it in less than fifty lines.
I didn't see any posts on this so I thought I'd bring it up. I think the author took the long way around.
The author did some nifty hacking that resulted in the following stack of dependencies:
* Perl 5.8.5
* Python 2.5
* PHP 5.2.1
* Xapian 1.0.2
* Django 0.9.6
He cited not wanting to use a RDBMS since he's not writing to the database, just reading. I can give him that, but it seems like it caused more trouble that it's worth.
This leaves me wondering: why not just use Google Gears and be done with it. Sure, the hacking part would shift largely to the javascript side of things (would be mostly wiki conversion), but you'd have the other bits (web server, and storage) already worked out. All you'd have to do is slap together some little app to insert the XML data into the database.
I live in an area with fairly bad mobile signal - I'm always trying to look things up on Wikipedia but finding I can't. Fortunately my Treo 680 smartphone can take 8GB SDHC cards (http://en.wikipedia.org/wiki/Treo_680), so I could fit this on with room to spare for MP3s and photos, and future growth of Wikipedia. Very tempting, though I'd need to port it to something like Lua and GCC - obviously the porting would be fairly trivial by the time Palm releases its Linux-based Treos in early 2008...
I think the most interesting thing is how this guy admits MySQL sucks donkey balls, but still gets promoted to the front page.
Actually, the only heated intellectual debates I regularly get into are on camping trips, where cellular reception is very rare. Worse yet, my friends often expect me to have encyclopedic knowledge with which to settle such debates.
Conversations around a campfire can go anywhere.
Because I can
which is one of the greatest motivations for human advancements.
The phrase "more better" is acceptable English. suck it grammar Nazis
Preload this in OLPC
You have to buy a used PSP to be sure that you can hack it.
Not anymore! See the Lumines User Mode Exploit discovered in June. We got my son the game a month ago and he's gone crazy downloading home-brew games and emulators onto his PSP.
The database dumps are static, and until more recent dumps are available, they will stay the same. So if George is ok now, he will be ok 12 hours from now.
Yeah, TomeRaider costs money, which already marks it as a loss against the "thrifty" slashdot crowd ;) , but I think it's worth it.
http://www.tomeraider.com/
They provide Wikipedia versions that you can use with their e-Reader. I bought one because I can use it on my Pocket PC, and it's just awesome having the Wikipedia available instantly, anytime. It's the fucking Hitchhiker's Guide version 0.1, gyat damn. =)
Where's the moderation for +!, Awesome?
I [may] disapprove of what you say, but I will defend to the death your right to say it.
http://www.webaroo.com/ already has a tool for this. They have packages on numerous topics, including a Wikipedia package
From TFA:
"The result of the import process was also not exactly what I wanted: I could search for an article, if I knew it's exact name; but I couldn't use parts of the name to search; it was all or nothing. To allow these "free-style" searches to work, one must create the search index - which I'm told, takes days to build. DAYS!"
This seems kind of stupid to me...could he not have done an SQL query using the moral equivalent of 'select articlettext where title like '%thingiaminterestedin%' - thus meaning you don't need to know the exact title.
Uhm, so he used a offline copy of wikipedia with a search engine to search through it? Wow that's like .. one of the very reason xapian exists. All the hightlights except latex come from xapian. This is like
1. Download Wikipedia
2. Install Xapian
3. Make Xapian index wikipeida
4. ????
5. Slashdot!
This Chinese police. You under arrest for around going Great Firewall. Please to turn self in at nearest station.
Oops! Forgot to post anonymously, dangit!
(T>t && O(n)--) == sqrt(666)