Boiling Down Books, Algorithmically

destinyland writes "A year ago, Aaron Stanton harangued Google over his new project, a web site analyzing patterns in books to generate infallible recommendations. In March he finally finished a prototype which he showed to Google, Yahoo, and Amazon, and he's just announced that he's finally received a big contract which 'gives us a great deal of potential data to work with.' The 25-year-old's original prototype examined over 200 books, plotting 729,000 data points across 30,293 scenes — but its universe of analyzed novels is about to become much, much bigger."

Comments

Just one more errosion....

[zappepcs]

The difference between now and 100 years ago becomes more apparent each day. Then, owning books was a sign of affluence, of intelligence. Now? Everything is up to question, and should be. Analyzing books and other public material is just another step in putting intelligence out there for everyone, not just those that can afford it. I applaud it, and all the dangers it brings. Such hurdles are necessary, but we must assault them to overcome barriers that should no longer exist.

Re:Just one more errosion....

[Anonymous Coward]

Knowledge, not intelligence.

Re:Just one more errosion....

[Anonymous Coward]

"not just those that can afford it."

Shit Bud, you make it sound like it's the 1200s. Books ARE cheap. Books are just another thing to compete for your money; sometimes they win, sometimes they lose. Like with those bankrupt families that have a 50" plasma screen and a couple Navigators in the driveway. They've made their choices. Personally, I've chosen books. No need to assault anything or anybody; there are no barriers other than our own (assuming you're a white male, of course).

Say, you aren't one of those I-want-everything-given-to-me-for-free computer type people, are you? Well, if you are, fuck off.

Re:Just one more errosion....

[blahplusplus]

What really hits a nerve with me is why the scientific community hasn't opened up all their journals for others to read. I imagine many retired and amateur scientists, engineers, hobbyists, etc, would have a lot of insight into many engineering and scientific problems and also make many discoveries as well. Intelligence is not limited to the credentialed, those of high status or currently employed, many discoveries happen simply by exposure to as many minds as possible, and finding connections and errors in others works..

Re:Just one more errosion....

[Strilanc]

While you're at it, complain about university students not making their books less expensive. Your beef is with publishers. They aren't the entire scientific community.

Re:Just one more errosion....

[dnwq]

The researchers publishing these papers typically don't get much more than citations - the money mostly goes to publishers like Elsevier. Blame them instead.

Re:Just one more errosion....

[Sir Holo]

blahplusplus: What really hits a nerve with me is why the scientific community hasn't opened up all their journals for others to read.

We scientists would absolutely love to have all of the journals opened up for free access to everyone. But, you see, the publishers own the copyright to our articles. The system requires us to give them the copyright, in order to get our stuff published. Then you, me, and everybody else has to pay to read recent research.

Thankfully, some established journals are going open-access.

That's very promising. But the fact remains that publishers such as Elsevier own the copyright to many decades-worth of scientific literature. And they're not about to give any of it away.

Re:Just one more errosion....

[blahplusplus]

"That's very promising. But the fact remains that publishers such as Elsevier own the copyright to many decades-worth of scientific literature. And they're not about to give any of it away."

Then I submit the scientific community creates a project website to buy the rights to these works, I've come up with many ways for funding such an endeavor. The barrier would primarily be geometric (population size vs amount of money each person could donate/give/invest in such a venture) and the attitudes of the people themselves.

Re:Just one more errosion....

[zappepcs]

I don't mean to throw stones, but books cost money, many people afford to be on the Internet, yet buying books has become old hat. When you can go on the Internet and get the latest information, books are ... well, a waste of money for the most part. The delay between discovery and publishing and reading is no longer tolerable, not in this throw away society. Look at some science fiction ideals... such delays are always intolerable. I will cite an event that is not even related to show that delay is not right: junteenth. It took several years for emancipation news to reach Texas. Is that right? The point is that information and knowledge should be universal, and instant. The great promise of the Internet was just that. If you wish to spend your nights reading information from 2+ years ago, that is your problem. The rest of us want today's information, and now. Good luck with the personal library.

Re:Just one more errosion....

[Skreems]

If you're talking about news, you're correct. But the original article is applying this to works of fiction, which still take at least a decade to go out of date (if not longer) despite the internet and the hard-on you appear to have for it. This "invention" is not about freeing information, it's basically a fancy way to mathematically calculate that if you like The Hobbit, you might also like The Lord Of The Rings. It might be beneficial to someone looking for more of the same, but it doesn't even seem to serve to further creativity since by design it will not recommend things that will expand your horizons, but will encourage people to stay with the safety of yet another rehash of something they've already read.

Re:Just one more errosion....

[smittyoneeach]

Or wisdom, for that matter.

Re:Just one more errosion....

[smittyoneeach]

If you wish to spend your nights reading information from 2+ years ago, that is your problem. The rest of us want today's information, and now. Good luck with the personal library.

It's getting to the point that you need a 2+ year filter just to dampen the noise in the signal.
And let's give a shout out to all of the library homiez. While I'm affluent enough to afford the occasional impulse book at the store with the built-in coffee shop, I do recall many an hour of random wandering in the public library in my youth.

Re:Just one more errosion....

[Chineseyes]

You do realize that doing something like this publicly would backfire in the worst ways imaginable? You would immediately increase the value of the works and some incredibly wealthy person or corporation may just buy everything out right in the hope that you pay him/her even more money than you had originally planned.

Re:Just one more errosion....

[Anonymous Coward]

Right! Intelligent people don't need to read books. They'll excel at whatever they do. Usually some kind of con artist.

Re:Just one more errosion....

[Anonymous Coward]

All scientists publish their papers both legitimately and illegitmately through an underground site. I imagine such men have the intelligence to do so.

There are many ways to do this and no, they don't have to be legal when you're dealing with commercial tyrrany.

Re:Just one more errosion....

[Anonymous Coward]

Intelligent people don't just "read books" they get their information from everywhere, even on a bank line they can still get information because they pay attention to everything, they don't have to struggle to learn something.. ask any Mensan if it's not the way they learn stuff. Sure there are some subjects that need deep study, but for the knowledge that comes in handy in everyday tasks... observation it's the way to go/. nd

Re:Just one more errosion....

[LilGuy]

I am. Especially when it come to information. And I almost always get it for free.

Except when I find it incredibly valuable, in which case I pay for a hard book copy.

Re:Just one more errosion....

[Wrath0fb0b]

I wish it weren't so (and I submit all my papers to http://www.arxiv.org/ as well to the journals), but the fact is, closed journals provide significant value both to the reader and to the submitting author. I'm not really trying to defend the system here, by the way, I'm just trying to explain what purpose it serves (and what an open alternative would have to match).

Referees and Peer-Review Referees are invaluable because someone has to objectively assess articles for basic scientific merit and rigor. The better journals can recruit referees for each submission that truly grok the subject matter and can often work very productively with an author. Quite a number of important advances are made and pitfalls avoided because a referee insisted that a researcher cover her bases before submission. Of course, nobody claims that PR journals are bullshit-free, but they are certainly far better than un-reviewed sources like arxiv.

This function is especially important for readers in multidisciplinary fields (myself included) that often read papers on subjects in which we are not expert enough to know what constitutes sound science. When I read about some group that has extracted and crystallized some protein, I'd like to know that someone competent at the relevant techniques has scrutinized their methods because I haven't the faintest clue (I'm a physicist by training, a biophysicist by necessity).

Prestige and Selection Another important function of the journals is to select articles by importance. If a paper makes Nature or Science, that's usually a good indicator that they've made an important advance. The benefits of this selection are twofold: first, readers can keep tabs on work at the forefront without wading through lots of papers. It sounds lazy, but most of us cannot read every paper that is published and are quite glad to outsource some filtering to the journals.

Secondly, it allows authors to demonstrate to people outside their immediate field what caliber work they've done. Even among people in the same department, it's not immediately clear what qualifies as a breakthrough work (as opposed to incremental work, which I don't trash in the least bit, but it's not really the same hat) -- prestigious journal cites are a good substitute, especially when the alternative is to either become an expert in the field or find one and ask.

Review Articles Most journals have an in-house staff to write articles reviewing the state of a particular field/technique/whatever. This is also an invaluable services because sometimes one needs a broad, textbook-level summary instead of a large number of discrete, deep papers on a topic. Given that science is done in small, insular little bits, it's natural that there is room for someone to aggregate and summarize those bits and put them into a larger perspective.

Editing Another thankless job (the snarky comments about the /. eds belie the fact that editing is hard work). Dupes are weeded out and researchers with poor language skills (especially when writing in an adopted language) are given help communicating their ideas. Confusing or unclear language is massaged back into form, figures are well-presented and well-labeled, text is formatted to be easy on the eyes, references are given in a standard form. These things count more than most /.ers realize (Knuth was on to something guys . . )

Access Brutal honesty, we don't really care about the access restrictions. Every university has license to pretty much all the major journals. We can get them from wherever with a quick login and so can everyone we know. Sorry, but that's the truth.

Re:Just one more errosion....

[shadowofwind]

What really hits a nerve with me is why the scientific community hasn't opened up all their journals for others to read. I imagine many retired and amateur scientists, engineers, hobbyists, etc, would have a lot of insight into many engineering and scientific problems and also make many discoveries as well.

I like your spirit and agree that there's a lot of really smart, creative people who aren't scientists. However...

One of the dispiriting things about science is how specialized most subjects have gotten. If you're not an expert in a field, its almost impossible to do anything. Even being an expert in a closely related field often isn't good enough. I don't think this is anyone's fault, its just the natural course of development. So I think the days of ameteurs accomplishing very much are behind us in a great many fields.

Another issue is that the people who fund scientists often aren't sufficiently literate to distinguish good science from phony science. Scientists are threatened professionally by people who peddle counterfeit knowledge, which has an effect on their fields similar to the effect a flood of counterfeit currency would have on an economy. So they try to protect themselves by controlling the validation of information and the supply of scientists. This legitimate desire is of course twisted by other less honorable inclinations. But there's a legitimate motivation there also.

Re:Just one more errosion....

[Sheafification]

As a member-in-training of the scientific community, I think you'll find that most scientists agree with you. Unfortunately the system right now is hard to break out of. You need to publish in a reputable journal for job evaluation and tenure purposes, but many reputable journals are under the thumb of the publishers.

In mathematics there have been several mass resignations of journal editorial boards in protest over the price. These editors usually then go on to form a brand new, cheaper journal in the same area. So some progress is being made. I can't say what has been happening in the other sciences though.

Re:Just one more errosion....

[Man On Pink Corner]

I dunno, man. Pretty much every point you covered is Wiki-able.

Re:Just one more errosion....

[Narpak]

...will encourage people to stay with the safety of yet another rehash of something they've already read.

Like most people since mass printing became possible. There are many authors whos work would give you great satisfaction, but who you will never read. Perhaps by at least giving people a good selection of tailored recomendation; the quality of that selection could hopefully improve.

The span of taste is wide and varied. More so than what any bookstore could provide (unless it is online). However when you take things online you encounter another problem; there is truly a vast (and growing) number of books avalible for purchase; trying to create a system for automated recomendation is a logical goal. Even if a system like that doesn't encourage reading things outside your established field of interest. If you arrive at a point where you need something different, a good system should be able to let you browse the top sellers, best reviewed and established classics of any genre. I have no doubt that after various tries, failures and breakthroughs, and as technology improves; consumers of litterature will be given a good online, digital tool for searching through databases and lists of material they might enjoy.

Re:Just one more errosion....

[Ethanol-fueled]

Dostoevsky and Tolstoy books(or any other books to include those written by Psychology's founding fathers and mothers), bought used from Amazon, are dirt-cheap and will teach you more about psychology than any single text will.

Re:Just one more errosion....

[Austerity Empowers]

...they don't have to struggle to learn something...

Not even mensa is that arrogant. If it's easy to learn and/or comes via the ether, it's probably trivial. Intelligent people have to work every bit as hard, we expect a whole lot more from them.

Re:Just one more errosion....

[Skreems]

I don't get this, though. The idea of "top sellers, best reviewed, genre classics" already exists, and this invention adds nothing to it. On the other hand, the idea of finding books you should read but don't know about seems a problem particularly poorly suited to an automated solution. This is what personal recommendations absolutely excel at, because no algorithm can gauge the cultural impact of a work of art, or the level of craft involved in its making.

Re:Just one more errosion....

[mrbluze]

Or wisdom, for that matter.

What about insight?

Re:Just one more errosion....

[Metasquares]

Basically. There's no advantage to observation over learning with a focused objective, but I think the key point is that learning is an unconscious process that is primarily carried out intuitively. You can direct your attention towards a subject and think a great deal, but you can't direct your intuition - all you can do is foster an appropriate environment. I've thought of it as a sort of receptiveness for new ideas (which I think are exogenous but are learned only after personal interpretation).

I would question whether this is how all of the gifted learn, however. I know a lot of gifted people who nevertheless think they can somehow coerce themselves to learn things through sheer conscious effort, without intuition ever taking over - people who make their primary goal thinking about something vs. understanding it, if that makes any sense. If they keep drilling enough, they eventually get whatever it is they were trying to learn, but it tends to take a long time, usually leaves them exhausted, and is swiftly forgotten. Unsurprisingly, these people have all ended up building small, narrowly focused knowledge bases.

That's not to say that learning difficult material is easy. It's a struggle irrespective of intelligence, and if learning a particular topic comes easily, there's always something harder. Until you intuitively understand something, conscious thought may be your only way of comprehending it - and if you don't intuitively understand a concept, thinking about it is hard work.

But that's just my own experience.

Re:Just one more errosion....

[ruin20]

In most things we evolve, not leap to new horizons. I find that most of the time I choose to read a book because I like it's similarities, I like the book because of it's differences. Like traditional sci-fi to apocalyptic sci-fi to steam punk to biohacking to cyberspace to crypto. I never would have read the Cryptomicon if I hadn't read I, Robot and can say today that I have a better appreciation for one from the other.

Typically the way we learn and get good at just about everything is that we go a little bit beyond where we're comfortable and we sustain an effort there. After a while our comfort level moves. Just like if I read enough on one subject typically I'll get caught up with a tangent subject and eventually move into that.

Re:Just one more errosion....

[Free the Cowards]

Much better to blame the researchers for not publishing in a more open medium. They're the ones who might actually change their habits, after all.

Re:Just one more errosion....

[Skreems]

Yes, that's quite true. The important thing in both, though, is that they're good, while this algorithm may just as easily recommend something absolutely terrible that happens to contain a lot of the same words and phrases unless it relies heavily on human input for that elusive quality assessment.

Re:Just one more errosion....

[Anonymous Coward]

it doesn't even seem to serve to further creativity since by design it will not recommend things that will expand your horizons, but will encourage people to stay with the safety of yet another rehash of something they've already read.

I would think such an algorithm could be used however most appropriate for stated needs. Want to read the same sort of thing? Look for works that match best. Want to expand your horizons? Limit the variables to certain kinds of similarities. I'm assuming a level of customization that may not exist for the end-user, but who knows?

Re:Just one more errosion....

[rubberchickenboy]

Weird...I was logged in but that posted as AC.

it doesn't even seem to serve to further creativity since by design it will not recommend things that will expand your horizons, but will encourage people to stay with the safety of yet another rehash of something they've already read.

I would think such an algorithm could be used however most appropriate for stated needs. Want to read the same sort of thing? Look for works that match best. Want to expand your horizons? Limit the variables to certain kinds of similarities. I'm assuming a level of customization that may not exist for the end-user, but who knows?

Re:Just one more errosion....

[Z34107]

I dunno, man. Pretty much every point you covered is Wiki-able. [Citation needed]

Re:Just one more errosion....

[gregbot9000]

It's getting to the point that you need a 2+ year filter just to dampen the noise in the signal.

Absolutely, I'd say 5 years minimum when it comes to fiction. I don't think I've read a book less then 5 years old in a decade. Unless the author has a proven track record, when someone comes telling me about how great a new book is I tend to wait until its passed the test of time. It's the only surefire way I've found to get past all the new releases and marketing hyped reviews, if people are still taking about it after five years you've got a winner.

Re:Just one more errosion....

[Anonymous Coward]

why not model the new open system after open-source? You publish for the sake of publishing and advancing the field, and all information is traded freely?

Re:Just one more errosion....

[vikstar]

There are a few places you can download publications for free. Pubmed and Citeseer usually have access to many papers for free download. Otherwise, sometimes authors put their own draft/pre prints on their websites.

many discoveries happen simply by exposure to as many minds as possible, and finding connections and errors in others works

Is this based on an actual study or your own conjecture?

Re:Just one more errosion....

[Virtual_Raider]

the idea of finding books you should read but don't know about seems a problem particularly poorly suited to an automated solution.

Er... -1,Wrong* : You don't seem to be considering the impact of statistical analysis and Very Large Sets of Data (C)(TM). It's becoming increasingly possible not only to know that 125K other people all over the world bought books B, C and D along with book A that you purchased, but now you can also index and analyse their content so it will be even easier to fine tune.

Imagine this: On the first iteration (first purchase) it can only out-of-the-blue recommend to you those books more consistently purchased along with the one you chose. But on subsequent transactions it can remember what you bought and compare the contents of the books. Now if you bought The Silmarillion, Kontakto and The Unfolding of Language over time, it would be possible to suggest that you read Shakespeare's works in their original Klingon once it realizes that you are equally interested in languages as in fictional civilizations.

I agree with you that the day an algorithm can make value judgements on the artistic merits of any work is still far ahead, but there was just recently a story about this FireFox plug in that sumarizes user reviews. Combine the two and...

* Didn't we have this conversation before, or is it just a popular .sig? If there was a "-1,Wrong" moderation, you would be told that the info is wrong but you would lose any insight provided by a direct reply of somebody that bothers to correct you AND post the right facts. With Slashdot being a discussion forum, it's on its best interest to actually promote discussion so you most likely will never see that mod option implemented.

Re:Just one more errosion....

[Anonymous Coward]

Agreed. It's not really rewarding that many of the publications from ACM and IEEE, to mention a few, require paid membership.

Re:Just one more errosion....

[ponos]

I don't mean to throw stones, but books cost money, many people afford to be on the Internet, yet buying books has become old hat. When you can go on the Internet and get the latest information, books are ... well, a waste of money for the most part. The delay between discovery and publishing and reading is no longer tolerable, not in this throw away society. Look at some science fiction ideals... such delays are always intolerable. I will cite an event that is not even related to show that delay is not right: junteenth. It took several years for emancipation news to reach Texas. Is that right? The point is that information and knowledge should be universal, and instant. The great promise of the Internet was just that. If you wish to spend your nights reading information from 2+ years ago, that is your problem. The rest of us want today's information, and now. Good luck with the personal library.

Well, you can access recent scientific articles for example on www.ncbi.nlm.nih.gov, read 10 (out of something like 1000++) and try to understand the subject that interests you. Or you can choose a reliable textbook and read that instead. It won't be 100% up to date, but it will often be easier to understand, probably more objective and will cover a bigger part of the literature than you could in a reasonable time frame.

So, if you're the kind of person that reads 100 scholarly articles just to implement something that has been in Knuth's 20-year TAOCP I admire your patience and perseverance. For the rest of us, textbooks are still extremely useful as a source of distilled knowledge. (No, wikipedia does not count as a textbook. I won't be choosing my patient's chemotherapy based on wikipedia any time soon.)

Then again, whether a textbook is online or on paper changes little. Many are available in both formats but the biggest part of the cost comes from the intellectual effort put in creating them and as a result they cannot be updated twice weekly. I would say that we're looking at a product that is at most 20% cheaper (often more expensive, too!) and probably reviewed a bit more frequently.

P.

Re:Just one more errosion....

[Anonymous Coward]

Strong encouragement for scientists who choose to publish in "open" journals would be a good start.

Re:Just one more errosion....

[Bastard of Subhumani]

Are all your posts full of pretention twaddle?

Re:Just one more errosion....

[umghhh]

That is fascinating - somebody came up with another way to dig patterns in mountains of data thus creating even more data to dig into and people claim it is intelligence, wisdom or knowledge and that everything changed. It is true of course. One big change between now and then (whenever that would be) is that today any ignorant connected to internet and equipped in basic reading skills is able to claim he posses all the knowledge of the world. Sadly the fact that more people have more and easier access to any information has changed one thing: they do not have to think for themselves they can read the answers on the web. In a sense nothing has changed though - reading with understanding is limited to the very few intelligent knowledgeable and (even bigger rarity) wise people. As for author of software in question - maybe knowledgeable fits him to some extent.

Re:Just one more errosion....

[kalidasa]

Have you EVER heard of a library? The journals are free to read; you just have to pay if you want it delivered to your doorstep or web browser.

Re:Just one more errosion....

[umghhh]

Let us think of it as a legitimate goal - to have all information instantly, allow them to come to your consciousness all at once, so that you can be aware of them all at once too, this state has been possible to achieve without google or amazon or even (oh my god is it possible?) internet - the substances used by people already thousands of years ago had the same effect. They are illegal in majority of places /. is read though so spending zillions on needed infrastructure and getting fat in the cellar while operating it seems to be a fair price to learn all the shit all at once.

Re:Just one more errosion....

[umghhh]

Not sure if that is true - they may include some algorithms that will introduce some small variation once in a while thus allowing the masses not to get scared by unknown but to proceed into new realms albeit slowly. Maybe this would be something tunable too? Whichever way they coded it such algorithms tend to work better with simple people. I do not have anything against simple people in fact I always aspired to be one but I got laugh attacks sometimes when similar tools in music realm try to guess what I could like.

Re:Just one more errosion....

[h4rm0ny]

What about insight?

What about synonyms?

Re:Just one more errosion....

[cpricejones]

Access Brutal honesty, we don't really care about the access restrictions. Every university has license to pretty much all the major journals. We can get them from wherever with a quick login and so can everyone we know. Sorry, but that's the truth.

This is simply not true. I work at a very large university, and it still amazes me to find that some electronic journals have not been purchased by the university. When I need these articles pronto, I must email friends at other universities. But what about smaller colleges? Enthusiasts? (I doubt there are that many biochemistry enthusiasts, but I'm sure there are a few who would love reading the new Methods in Enzymology or Nature, Science, Cell, what have you. The field needs these enthusiasts.)

Re:Just one more errosion....

[h4rm0ny]

Things become easier to learn when you have more context. If I tell someone about a chemical reaction, they may struggle to remember or understand it. If they are a chemist or have at least a grounding in the subject, they'll be able to slot the knowledge in with associated information and thus understand it and recall it more easily. An "intelligent" person has a wider array of contexts or the ability to quickly find an appropriate context. I am fairly intelligent, but I still don't recall who scored what goal in what match the way some people do. These people absorb such information because they have context as much as for any other reason.

Re:Just one more errosion....

[Agronomist Cowherd]

That's "pretentious twaddle", you ignorant twit.

Re:Just one more errosion....

[Illserve]

We did.

http://www.plos.org/

(not me personally, I had no role in this but as a member of the community I applaud)

Re:Just one more errosion....

[smittyoneeach]

"What's another word for thesaurus?" --Steven Wright

Re:Just one more errosion....

[Mayrel]

I wish it weren't so (and I submit all my papers to http://www.arxiv.org/ as well to the journals), but the fact is, closed journals provide significant value both to the reader and to the submitting author. I'm not really trying to defend the system here, by the way, I'm just trying to explain what purpose it serves (and what an open alternative would have to match).

I think you're talking about "open" in a completely different way to the parent. You seem to be using it in the sense of "accepts articles from anybody without any kind of review", whilst the parent was clearly talking about making the journals open "for others to read".

Referees and Peer-Review

Peer review is great, but you don't have to charge the readers of your journal to be able to review papers before they are published.

Prestige and Selection

Charging people to read the journal creates only a false sense of prestige. Genuine prestige arises from the quality of the articles in the journal. If Nature was made freely available tomorrow, do you think many scientists would stop submitting articles to it because of that?

Review Articles

Nothing prevents an editor writing a review article for a journal you don't have to pay to read.

Editing

Yes, editing is very useful, but just because people don't have to pay to read a paper, that doesn't mean it can't be edited.

Access

I think a paper you can read for free can just about match the accessibility of a paper you have to pay to read.

Brutal honesty, we don't really care about the access restrictions. Every university has license to pretty much all the major journals. We can get them from wherever with a quick login and so can everyone we know. Sorry, but that's the truth.

Ah, here's your real argument. You don't care that almost everyone can't read many journals for free, because you can.

You say that you're "just trying to explain what purpose it serves". So what are you saying here? The purpose of closed journals is to ensure that only the élite can read them?

Re:Just one more errosion....

[smittyoneeach]

I've been reading Neal Stephenson of late.

Re:Just one more errosion....

[ALB1]

Books are invaluable because they track a lot of things that happened before the Internet !

Remember those days when a bulletin board was an actual board, when a spreadsheet was in paper, or even when one would assume a program was written, not typed. And don't tell me Google libraries and other scanning projects will let me find that information with metadata, indexations, search engines and a coffee to go, because I won't believe you when told that I, with my weak thirty-years old memory and poor understanding of search engines complexity, will be able to find what piece of information I want from a 5 petabyte indexed database.

Let me take an example: I am currently reading a riveting history of aeronautics published in 1963... I could never find the same "insights into the future" (what will the Apollo spacecraft look like ? the gap in nuclear deterrence, the russians have a 100 megaton vector with a 2000 miles radius while the US is working on Polaris and Midgetman 500 kilotons and less than 1000 miles radius, etc.) in a new history of aeronautics... At best those info could be found in a very specialized study of the aeronautics in the cold war, but what else would be lost in time ?

That book, that knowledge, that history is so important to me, that I hope people in the future will find as valuable to have physical traces of the unfolding history of today in some other form than 60 gigabytes database dumps cassettes or a low resolution poorly scanned pdf book on someone's SSD in malaysia online from 1am to 5am.

I agree with you that purely informational straight-to-the-fact news post are better off in a database because the information is often in the title and its date. But there other kind of texts that should be intended to exist for the generations to come, and I think this information is better kept printed in a sturdy hard-cover book than in a scrambled binary stream somewhere on a hard drive, because no one will ever look there in a few decades.

Re:Just one more errosion....

[smittyoneeach]

What I find more fascinating than your observation is that there appears to be no filtration of noise from the signal.
Given a relatively free petri dish for information to slosh around in, there seems a shocking lack of condensation of real knowledge out of all the crap.
Wikipedia seems like a step of sorts in the preferred direction.

Re:Just one more errosion....

[hibiki_r]

No.

So are you telling me you've never seen someone get a great grasp of a concept that takes most people months to master? Or solve a design problem in a matter of seconds, while a team of 6 people, with just as much experience, have failed to find an acceptable solution for months?

Intelligent people don't have to work as hard to get the same results in many situations. That's the point of being intelligent. Some environments will ask more of intelligent people, but that's an issue of the environment, and those environments are typically a very twisted school or some parents trying to be evenhanded. After that, those environments go away. I've yet to see a workplace that asks more from an intelligent person than from a more average employee, just because they seem intelligent, and not because they have a position with more responsibility and a better salary.

Re:Just one more errosion....

[Anonymous Coward]

The difference between now and 100 years ago becomes more apparent each day. Then, owning books was a sign of affluence, of intelligence.

Now reading is only for retarded people, and people that want to... excuse me for a second, hahaha... "educate" themselves. The only *real* positive differences between now and 100 years ago, are a tiny increase in surgical technology and transportation technology. Just about *everything* else has gotten worse.

And, personally, I think it is sad that a guy can whine to big companies and get his way. There are *plenty* of people out there that would practically do anything to work for google (including me) and that have much better ideas then some newbie that couldn't program a *real* algorithm if his life depended on it. Honestly, I know a lot of people that could have done what this guy did (and *did* do the equivalent of what this guy did) when they were 15 years old. But did *any* of them get recognition??? No. They all worked there way to the top. And, then this hot shot comes along, learns to program two statements and thinks he should be rewarded for that??? This *really* pisses me off.

Re:Just one more errosion....

[Rubikon]

Actually, when making the comparison, it is generally untrue to say that "intelligent people have to work every bit as hard."

Intelligence is not innate knowledge of anything, it is potential, the ability to learn and understand. In one way, you can compare intelligence to an engine. An engine in a (non-hybrid) Toyota Corolla is, at the mechanical heart of it, very similar to that of a Bugatti Veyron. However, for each rotation of the crankshaft at the same RPM, the Bugatti produces FAR more power than the Corolla. Similarly, someone with an IQ of 140+ will "go farther" (given the same amount of time and effort) than someone with an average IQ, even if they both start with a zero knowledge set. This is not a value statement about the individuals, nor is it license to slack for the person with more intelligence, it is merely the acknowledgement of the fact that, to accomplish similar knowledge or proficiency, intelligent people generally do not have to work as hard.

</tangent>

Now, back to your regularly scheduled thread.

Re:Just one more errosion....

[Anonymous Coward]

Ok first recommendation systems are only as good as the input. Just because I BOUGHT a book/movie/toy/whatever does NOT mean I like that item. It also MUST take into account gifts. Just because *I* bought an item for a baby does not mean I have a baby and want to see baby formula. Also my tastes change over time. Just because I bought a book 2 years ago does not mean I want to see similar items today.

The BIGGEST problem these systems usually make is not taking into account items I dislike also, or overweighting recently added items/extraordinarily linked items. Because I bought 1 memory stick I am getting thousands or recommendations for every memory stick on the planet. Or my favorite, 'you bought game X, would you be interested in the system to play that game?' Well DUH I probably already HAVE the system. I probably am not interested in buying another one of the 20 different flavors of it. Basically new items seem to become the 'center' of the tree of recommendation.

I have used systems like this in the past. Maybe he does have a new and improved one. I couldn't even really tell from the linked articles what he was doing. It was a bunch of 'well it is related', and 'i cant really talk about it'. My guess is a web site recommendation system using his book recommendation system, hence the camping out on googles doorstep. Which brings in a whole new level of 'how will people game the system'.

The idea is cool but usually the implementation leaves something to be desired.

Re:Just one more errosion....

[Wrath0fb0b]

Peer review is great, but you don't have to charge the readers of your journal to be able to review papers before they are published.

Editors don't work for free.

Charging people to read the journal creates only a false sense of prestige. Genuine prestige arises from the quality of the articles in the journal. If Nature was made freely available tomorrow, do you think many scientists would stop submitting articles to it because of that?

Charging enables Nature to hire experts that can better assess the quality of submitted articles. If they didn't charge, the quality would slide and, yes, eventually, scientists would stop submitting articles.

Yes, editing is very useful, but just because people don't have to pay to read a paper, that doesn't mean it can't be edited.

Editors still don't work for free.

Ah, here's your real argument. You don't care that almost everyone can't read many journals for free, because you can.

Almost right. I accept that almost everyone can't read them because I don't think it's possible to have an freely accessible journal that provides the same benefits. If it were just a matter of flipping a switch and not changing the quality of the product, I would certainly do that. It's not.

You say that you're "just trying to explain what purpose it serves". So what are you saying here? The purpose of closed journals is to ensure that only the élite [sicwtf] can read them?

No, the purpose of the journal is to have experts that filter, scrutinize, assess-importance-of, and edit the papers so I don't have to do the drudge work. Just because you either didn't read those purposes or assert that they don't justify the cost does not mean you get to disregard what I wrote and assign some other nefarious purpose under the guise of "what are you saying here".

Re:Just one more errosion....

[Wrath0fb0b]

Charging people to read the journal creates only a false sense of prestige. Genuine prestige arises from the quality of the articles in the journal.

Quick addendum, I don't think you understood my meaning. The prestige comes from the quality of the articles that you've written, not acceptance into an elite journal. The point is that the elite journal provides a very useful proxy for assessing the quality of the work. A researcher's publication history provides a very good measure of the quality and focus of his work which would otherwise be difficult to gauge (it would require become expert in the field(s), reading all the articles and then judging their merit -- way too much work).

Re:Just one more errosion....

[Bombula]

it's basically a fancy way to mathematically calculate that if you like The Hobbit, you might also like The Lord Of The Rings

I agree with everything in your post, and would further add regarding the above quote that I'll believe it when I see it. Unless there has been some astonishing breakthrough in AI that I'm unaware of, this recommendation system will be based only upon the occurance of certain words. My guess is that this might work for a few specific genres of fiction (sci-fi searches for frequency, distribution and local context of the words "gravity" and "star"; fantasy searches for the words "sword" and "spell", etc), but is unlikely to work very well on general fiction or prose. If I like travel fiction or international mystery and intrigue, one book may be low-tech and set in Madagascar and another may be high-tech and set in Tokyo. My guess is that any system too dumb to understand plot, narrative and pacing (i.e. any current non-human system) is going to end up recommending other books about Madagascar or Tokyo instead of other travel fiction or internatioanl mystery novels.

I'm open to being surprised, but my guess is that it's going to be tough to beat the good old fashioned method for recommending books: look for other stuff by the same author, other stuff in the same genre, and award-winners.

Re:Just one more errosion....

[Free the Cowards]

And thanks so much for doing that. Once all researchers follow in your footsteps, I think that humanity as a whole will be a lot better off.

Re:Just one more errosion....

[bigstrat2003]

If you wish to spend your nights reading information from 2+ years ago, that is your problem. The rest of us want today's information, and now.

First: there is absolutely nothing wrong with reading information that is 2+ years old. Information itself does not become outdated. News may, but that's only a small portion of all available information. Unless we get some stunning development in calculus, for example, a student in 10 years will be just as well off reading a calculus book published today as a calculus book published 9 years from now. Furthermore, not all books are nonfiction books. Fiction is just as good whenever you read it... or are you saying that all those poor saps who still enjoy Shakespeare's work are just deluding themselves?

Second: as useful as the Internet is for rapid dissemination of information, printed media still has its place. It's generally far easier to organize and reference a personal-sized collection of books than it would be with electronic versions. Large collections are easier to manage electronically, but the average individual's collection is probably better off on dead trees. Books have the advantage of being more tactilely pleasing, as well: it feels a lot nicer to turn physical pages than to scroll down with your mouse wheel.

There's a place for both media; it isn't as simple as you paint it, where the enlightened use electronic media, and only Luddites use print.

Re:Just one more errosion....

[mpeg4codec]

It's a chicken and egg problem. Not surprisingly, most academic journals and conferences have a certain amount of respect associated with them. You can't go out and publish to OpenConference and expect anyone to pay it any heed, since literally anyone can host a conference or publish a journal. People will look at your CV and go ``well that's great that you have a hundred pubs to CreativeCommonsJournal, but the only one on here I'm going to take seriously is SIGCOMM.''

About the only hope would be to get some big names to either host or publish to some of these open conferences and journals to lend them some credibility and maybe even talk them up a little. However, even if you convince them to do it, it will be a slow process for a publication to gain any amount of respect in the academic community. I'm not saying it's impossible, it's just a huge undertaking.

I'd love to see a group of people start something like this, but remember: it takes big names, at least IMHO. I'm lucky that the field in which I work (internet-scale research), the conferences and publications are very open about sharing their work freely, and that seems to be the case in most fields of computer science. It's only the hard sciences that I've really noted a tendency to have to purchase articles. Probably something to do with the open-minded nature CS has always had toward intellectual property.

Re:Just one more errosion....

[Free the Cowards]

I hear you, and I know it's difficult. If it were easy, we wouldn't be talking about it, it would be done already.

I'm just saying that the change must come from the researchers, not the journals. The traditional journals have nothing to gain and everything to lose from going to a more open system, so looking to them for change is the wrong thing. The researchers are ultimately who decides what's reputable and what's not. It will surely take a long time, but if it's going to happen at all then that's where the change will come from.

Re:Just one more errosion....

[netmouse]

There are a fairly large number of journals that can be accessed through scientific associations, such as ACM or IEEE, which have huge digital libraries available to their members at relatively reasonable costs.

Re:Just one more errosion....

[Austerity Empowers]

Sure, but I know their secret, because usually I'm often one of them. The secret, is I've solved the problem already, some day before to research some curiosity I had. It may not have been that exact problem, it may not have even been in that field, but chances are, it was because I had invested work in solving something similar, just not when the audience was watching.

I've known a lot of really smart people, the only real differentiation was the level of curiosity. The more curious would spend time understanding (by reading, hypothesizing and verifying), and would in turn be better prepared for new/unexpected challenges later. By adulthood curious people who spent the time, resources and mentors to work out their questions can be quite impressive.

I wholly reject this notion that you can develop some incredible insight and problem solving skills by "drawing in mana" at the bank line. If you're solving a good problem in line at the bank, it's because you've already read and researched the subject before and you just need some time to mull the problem over and use the other side of your brain.

Re:Just one more errosion....

[Anonymous Coward]

The fact that it could be done is note the same as saying it will be done.

Re:Just one more errosion....

[acheron12]

Which open mediums offer peer review?

Re:Just one more errosion....

[Acer500]

How about +1 Wrong? (or Interesting/Wrong, Interesting but Wrong, Insightful but Wrong, etc..) :)

Re:Just one more errosion....

[ailnlv]

Aren't synonyms already out there for everyone?

Re:Just one more errosion....

[Free the Cowards]

I'm sure I don't know. If none exist, then certainly researchers ought to create one.

Re:Just one more errosion....

[PCM2]

If you wish to spend your nights reading information from 2+ years ago, that is your problem. The rest of us want today's information, and now.

YES! As another /.er posted to an earlier thread, god forbid we all keep working with an outdated value of pi. Imagine the egg on my face if I walked around assuming hydrogen only had one proton for the rest of my life, because I never checked the Internet.

Re:Just one more errosion....

[Irvu]

The problem is not with the researchers so much as the beuraucracies of univiersities and funding, and the problems of peer review.

Many universities, especially those outside the U.S. use metrics for rating their researchers that are weighted towords publications from Elsevier and others. England is especially bad about this. For that reason many scientists don't have much of a choice in that they are forced to publish there is go without pay.

So totally open spaces raise issues of what it means to be published and the cost of maintaining the system must be borne somewhere. For that reason alone some money must go to journal publishing. However it is very possible for that funding to come from something other than overpriced subscriptions.

Finally, and optimistically most scientists are recityfying this themselves. Most authors post copies of their academic papers online and make them available through Citeseer and other locations so even though the journal is costly the paper may still be obtained even if a few months later.

Re:Just one more errosion....

[Anonymous Coward]

"+0.5 Nice try" ;)

Re:Just one more errosion....

[pasha2891]

Parent post is spot on, hopefully this becomes more common knowledge and people start getting a better idea of how they learn.

Re:Just one more errosion....

[CAIMLAS]

I think you either fundamentally misunderstand what intelligence is, or have misrepresented said understanding in your post.

What you seem to be conveying as "intelligence", isn't; it's knowledge.

Intelligence will always* be limited, and there will be "haves" and "have nots", with a gradient between the two extremes. It's a subjective measurement, but the prevailing cultural understanding (and benchmark) is that common intelligence markers are determination, a drive to succeed and learn new things, and an inquisitive mind able to solve problems put before it. Likewise, the ability to form cogent, internally-consistent opinions (and new knowledge) from acquired knowledge has also been such an indicator.

Books themselves did not indicate intelligence. Books were expensive, yes - and it was commonly expected that someone with enough money to spend it on books was either poor in almost every other worldly respect, or was thoroughly established. The books themselves have always just been considered simple knowledge for a person to aggregate.

"Stupid" people will continue to take things at face value and not look deeper. They might utilize such technologies to provide themselves with better answers to life problems (helping them make better decisions), but it's not likely: stupid people have always done stupid things, and they will continue to be driven by the herd mentality and "social wisdom" - which we know as "superstitions" when applied to past eras.

As for hurdles and dangers, what exactly do you perceive? Stupid people will always make poor decisions, and providing them with a larger base of targeted/pertinent information from which to make said decisions hardly seems like it'd result in "dangers".

* Barring some sort of fortified nano-drink or brain augmentations we might have available in the future, that is. I'm not going to hold my breath.

Re:Just one more errosion....

[Man On Pink Corner]

Correct; the economics of the situation are what says it will be done.

Re:Just one more errosion....

[h4rm0ny]

And a nice way of saying 'euphemism' ?

Re:Just one more errosion....

[Bastard of Subhumani]

Yes, it serpently is!

Re:Just one more errosion....

[smittyoneeach]

"Wardrobe malfunction"

Newspeak

[RDW]

I love how the prototype version in the link gives a 98% match between George Orwell's '1984' and the text of the USA Patriot Act!

Re:Newspeak

[drinkypoo]

They're still working out that last 2% margin of error.

Re:Newspeak

[log1385]

From the FAQ:
"Does 1984 really match the U.S. Patriot Act?
No, that is an easter-egg. A bit of a joke on our part."

Re:Newspeak

[Pembers]

I thought they put that in as an Easter egg... the Patriot Act isn't a novel. Though some Eastern bloc countries allegedly used 1984 as a HOWTO, or a specification of an ideal government.

Re:Newspeak

[DittoBox]

Citations please. I'd love to know uses 1984 as a blueprint of sorts.

Re:Newspeak

[Hurricane78]

Since when do the US, Britain, Germany and France belong to the Eastern block? Or are you from a pacific island? ;)

P.S.: If yes: Can I come too? I'd do anything to get a decent government again.

Re:Newspeak

[smittyoneeach]

One man's joke is another /.er's pathetic little reality.

Re:Newspeak

[smittyoneeach]

You need an "if he" or "who" in that second sentence.

Re:Newspeak

[smittyoneeach]

I'd do anything to get a decent government again.

"Be thankful we're not getting all the government we're paying for." --Will Rogers

Re:Newspeak

[Pembers]

By "Eastern bloc" I meant the Soviet Union and its satellite states in Eastern Europe.

This page talks at some length about a Soviet dissident and his reactions to the novel. Basically, he found it hard to believe Orwell lived in Britain, not Russia. That doesn't (much) support my assertion that the Soviet government used the book as a blueprint (or at least, thought it had a lot of good ideas), but I did say "allegedly" :-)

Unfortunately for your hopes of moving to somewhere sane, I live in Britain.

Re:Newspeak

[DittoBox]

Hmmm, you're right. I generally think one or two sentences ahead of what I'm typing. I generally re-read what I type before sending or posting but apparently not this last time. Cheers mate!

Re:Newspeak

[Drinking Bleach]

Even easier: the Soviet Union was established in 1922, and "1984" was published in 1948.

Now unless you want to claim that the USSR had time-travel, that alone should be enough proof the book was not a blueprint of any sort.

Re:Newspeak

[kalidasa]

He fought in the Spanish Civil War, and saw what the Communist Party was pulling in Barcelona; and because he was a leftist (yes, Orwell was a leftist, but on the libertarian/anarchist side of the libertarian/authoritarian axis), he was very aware of what the British socialists were doing. So he didn't need to live in the Soviet Union to know what Communism was like.

Re:Newspeak

[CAIMLAS]

When Orwell wrote 1984, he looked at the prevailing trends in British culture as well as the encroaching and growing Russian culture of the day and tried to extend the trend to its ultimate (and exaggerated) conclusion. As he thought well on the topic, and was vary thorough at conveying the foundational ideas and methods behind such a system through his writing, it could have been used as a study guide.

Totalitarian dictators, such as Mao, Stalin, Tito, and so on all probably read his work, as well as the works of their philosophical ancestors - Hitler, Marx, Lenin, Mussolini, and so on.

We can see a pretty thorough culmination of such ideals in practice in the propped-up dictatorships in South America (both the ones put in place by the US historically, as well as the current crop of communist/leftist states), as well as throughout

Really, the practice has been going on for eons; applying modern technologies such as firearms, news media, and surveillance are all pretty new, and their applications to totalitarian rule took a couple decades to work into something usable by the State. Now, places like China and to a lesser extent Britain are honing and perfecting it to the point spoken of by Orwell, where the citizens (subjects) aren't even aware that they're living in such a controlled environment: traffic/security cameras on every corner; State controlled media, education, and entertainment; selective enforcement of obscure laws to keep people in line; and so on and so forth, all rolled into a comprehensive social, economical, and political nationalist ball.

a tool

[Anonymous Coward]

when i come to think of it, building a tool that rates the level of novelty in an idea would be good one. it would make the job of large companies or opportunity hunters easier as well..

cheers,
mbilgi

Re:a tool

[BootNinja]

might be a good tool to help the USPTO with their backlog.

Finally!

[Anonymous Coward]

I, for one, welcome our new book analyzing overlords!

It already exists.

[Ironchew]

The only infallible book recommendation has existed for nearly 2000 years now.

Don't want to read it? Heretic! But "translations" do exist for public convenience.

Re:It already exists.

[tomhudson]

The only infallible book recommendation has existed for nearly 2000 years now.

I call bullshit. Books didn't exist 2000 years ago, you ignorant clod!

Re:It already exists.

[Anonymous Coward]

What, the Bible? Not nearly 2000 years old. And if you're making a joke about its "infallibility", only a minority of unfortunately outspoken Christians actually believe that.

Re:It already exists.

[smittyoneeach]

Well ex-scrolls me, you codex-fancying fascist! ;)

Re:It already exists.

[tomhudson]

Actually, the honour of the first printed scrolls goes to the Chinese. Examples of scrolls printed using movable type (wood cuts of chinese ideographs) date to the 600's. the oldest know book is also Chinese, from 868.

They also invented toilet paper 1500 years ago ...

Re:It already exists.

[smittyoneeach]

That would be the oldest known _printed_ book, sir.
http://en.wikipedia.org/wiki/Codex notes "The basic form of the codex was invented in Pergamon in the 3rd Century BCE".

Re:It already exists.

[Anonymous Coward]

I've read it. Fantastic action story full of war, violence, rape and genocide. The second half wasn't as good though, I think the guy in charge must have mellowed out some.

It's less believable than Die Hard 4, but for some reason it's got quite the cult following. Personally I think LOTR was better, it's certainly more consistent.

If you already read, you don't need this...

[thereofone]

...and if you do not read, you won't want this.

Re:If you already read, you don't need this...

[thereofone]

Also, now that I've played with the "beta" a little I want to see the graphs for Finnegans Wake.

Re:If you already read, you don't need this...

[thomas.galvin]

I don't know about that. A lot of my books are like television is to other people: simple entertainment. There are times when I want to have my horizons expanded, or to learn something new and nifty. But there are other times when I just want to forget about everything that happened at the office today, and when I do, it's kind of amazing how often I pick up a book that involves a guy with a staff blowing things to smithereens. And if there's a tool that will point me to even more guys with staves blowing even more things to smithereens, I say bring it on. Sometimes, I just want something comfortable. And explosive.

Re:If you already read, you don't need this...

[vikstar]

Oblig. quotes from Ink and Incapability

J: Not this one, sir. It is a book that tells you what English words mean.

G: I *know* what English words mean; I *speak* English! You must be a bit
of a thicko.

E: Would this be the long-awaited Dictionary, sir?

G: Oh, who cares about the title as long as there's plenty of juicy murders
in it. I hear it's a masterpiece.

E: No, sir, it is not. It's the most pointless book since "How To Learn
French" was translated into French.

Re:If you already read, you don't need this...

[TuringTest]

Actually, the algorithm at Booklamp could be used to put to have your horizons expanded. According to the site's FAQ ("Can I have BookLamp find me books that are different than a certain book, if I want to?"), they plan a feature to find books just out of your comfort zone, to "Press your limits":

The idea is that, after the system has determined what your preferred range is, it can go to the edge of that range and select a new title. The goal is to find you a new book that you probably wouldn't read on your own, but is not so far outside your preference that you'll never make it through. Reward exploration. :) You can then adjust how far outside your preferred range you care to stray - mildly different, or total radical?

Re:If you already read, you don't need this...

[kalidasa]

Also, now that I've played with the "beta" a little I want to see the graphs for Finnegans Wake.

My GOD ... it's a Mandelbrot set!!!

Re:If you already read, you don't need this...

[netmouse]

I still think they're starting from the wrong angle by aiming for books that are similar rather than for books that someone who liked one book will also like. Their feature for finding something different sounds like it will require a user to specify how it should be different in a way that's potentially a lot of work for the user.

I'll believe it when I see it

[clarkkent09]

I am skeptical that analyzing the content of the books can lead to good recommendations, let alone "infallible". Two books can be very similar in subject matter and writing style and yet one can be great and the other one awful. The difference is just too subtle for an algorithm to figure out, though I hope I am wrong and it turns out that it works, it would be very useful. Same applies to movies and music as well. I always found "Customers who purchased this book also purchased...." section on amazon to be more valuable than my personalized recommendations

Re:I'll believe it when I see it

[drinkypoo]

You could save a lot of time by analyzing only the final chapter. I find that most books I pick up are okay until the end, at which point they make me wish I could go back in time and gouge my eyes out with crab forks to prevent myself from ever picking that piece of trash up (works like that not often being translated into braille.)

Re:I'll believe it when I see it

[wfstanle]

I wholeheartedly agree! Take for example two phrases which are equivalent...

"Eighty seven years ago our ancestors ..."

and

"Four score and seven years ago our forefathers ..."

They say the same thing but what a difference in eloquence.

Re:I'll believe it when I see it

[3waygeek]

I am skeptical that analyzing the content of the books can lead to good recommendations, let alone "infallible".

You're obviously not Catholic.

Re:I'll believe it when I see it

[Anonymous Coward]

Yea, and its impossible for a computer algorithm to identify the difference between them, because manputers only read in the saying, and not the actual letters or sentences.

Re:I'll believe it when I see it

[martin-boundary]

It always depends on which part of the statistical landscape the algorithm is good at modelling.

It may be that what makes a book great is hard to identify, but what makes a book really bad is much easier to identify. In that case, such an algorithm won't help with recommending high quality works for you to read, but it could be very useful in saving you from wasting your time with obviously bad books (ie it would help with initial triage).

Remember, there are a lot more bad books than good books, so if you had to go through all the books to find the good ones, then you'd spend most of your time just looking a bad books and rejecting them.

Re:I'll believe it when I see it

[MrHanky]

It's probably no less efficient than analysing email to check for spam. If you're interested in ch34p C0rel S0ftw4re, you may also have an interest in v1agra and rep1ika r0lex watches.

Re:I'll believe it when I see it

[Estanislao Mart�nez]

I am skeptical that analyzing the content of the books can lead to good recommendations, let alone "infallible". Two books can be very similar in subject matter and writing style and yet one can be great and the other one awful.

Who uses the algorithm makes a big difference. From the way you frame the problem, you're looking for "good" books, i.e., books you'll enjoy reading. But think of somebody doing academic research or looking for patent prior art--for them, one important task is to find all relevant references on a topic, good or bad, and sifting through them.

Re:I'll believe it when I see it

[Anonymous Coward]

As somebody who was raised Catholic, from my experience, you'll be hard-pressed to find Catholics that truly believe in Papal Infallibility as well, excluding missionaries.

Re:I'll believe it when I see it

[Rocketship Underpant]

I agree 100%. I suspect a more useful data mining system would use book *reviews*, mined from Amazon and all the other sites that post them. In addition to providing an overall barometer for quality, it could identify reviewers whose tastes run similar to your own, and use that as a starting point for recommendations.

Re:I'll believe it when I see it

[Anonymous Coward]

Then stop reading Neal Stephenson books. Every time I read one of his books I swear to all my friends that, for example, Diamond Age is the best book ever only to recant at the end.

Re:I'll believe it when I see it

[wik]

Question one: How does it rate Hemmingway?
Question two: How does it differentiate between Hemmingway and Imitation Hemmingway?

Re:I'll believe it when I see it

[chromatic]

[What] makes a book really bad is much easier to identify.

Is it part of a series of more than three books? If yes, it's probably bad. Is the lead character a Mary Sue? If yes, it's probably bad. Is the lead character an irresistible vampire, were-wolf, were-tiger, were-lemur, were-panda, were-hippo with magic powers? If yes, you should have known already not to read the Anita Blake books.

Re:I'll believe it when I see it

[wik]

The difference, of course, is rooted purely in the awkwardness of the speaker.

Re:I'll believe it when I see it

[martin-boundary]

I wouldn't know, but seeing as there's a beta referenced in TFA, maybe you should try it?

Re:I'll believe it when I see it

[DriedClexler]

Exactly ... which is why I read the summary as "Fast-talking kid talks fools out of their money."

Re:I'll believe it when I see it

[mepath]

I agree with you. What is the recommendation based on? Is it based on the content of the book, style of the book, voice of the book, message of the book, author, what readers also liked, etc. There are tons of things to consider when recommending a book. If there ever is a finalized product, I hope it's far better than "Customers who purchased this book also purchased..." feature of amazon.

Re:I'll believe it when I see it

[greyhueofdoubt]

No computer will be able to definitively sort books by 'quality' as that is very subjective even within groups of people with otherwise similar interests.

But-

You know, it's OK to read middling-to-bad books every now and then. How are you supposed to know that Infinite Jest is a fantastically well-written book if that's all you've read? Or maybe all you've read are Stephen King novels. What good is it for a S.K.-only reader to say that S.K. is a good author if all they've read are are S.K., Michael Crichton, John Grisham, and other mass-market paperbacks?

A user review saying, "all I read is S.K. and this is his best book yet" is useless to me. A review that says, "All I usually read is S.K. and I found Infinite Jest to be impenetrable, purposefully obscure, obfuscated, dull, and self-indulgent," would be more useful.

Maybe you use the ITMS to buy music/shows. The iTunes review system is useless- you will find countless reviews like, "5 stars: I haven't finished downloading the album yet but I already know I'm going to love it!"

The itunes rating system is kind of broken (or stupid people use it...) when the only people writing reviews for an album are people who would buy it even if it was a recording of the band sleeping. When every album is 4 1/2 stars, the star system is useless.

But that's my personal rant.

I've had amazon and friends recommend books that turned out to be bad- almost painfully bad- but I don't regret reading them. Among that same group of books were works of art that changed my life. And, like you said, there are only tiny differences among good and bad books.

I guess maybe we have two different mindsets- you believe in maximizing pleasure by seeking the known-good books/art/food/etc while I am willing to forgo maximum pleasure sometimes in order to better define what exactly gives me pleasure. It's like thinking that Big Macs are teh best without trying or even thinking about trying a steak or a salad.

No coffee yet, sorry.
-b

Re:I'll believe it when I see it

[CAIMLAS]

While I agree with you on the basic idea, when combined with (say) user book ratings, such a tool could be valuable in assessment. A non-insignificant part of what makes a book enjoyable throughout (aside from the non-categorical "human element" necessary for good fiction) are the things charted by this proof of concept.

Yet Another Pointless Dot-Com

[techno-vampire]

This is just another pointless project that's going to waste the time and skull-sweat of a good but unrealistic programmer. All he's going to have when he's done is the solution to a problem that doesn't, for all practical purposes, exist. Good writers won't need it because they know what to do and how to do it, so they won't use it. It will only be used by poor writers, who won't know how to put the suggestions into effect properly. It may, possibly, tell a writer where their book needs work, or where it's not interesting enough, but I doubt it. Most likely, all it will do is tell it where it's not like other successful books because it won't be able to recognize or take into account any originality. Even if its recommendations are right, a poor writer is highly unlikely to profit from them, because by definition a poor writer won't know which suggestions are good or the skills to take advantage of them properly. No, what a poor writer who wants to get better needs is either a good critique group or some friends who will act as beta-readers, telling him not only what doesn't work but why (Something, I might add, that I find it hard to believe this program could ever do.) and discuss things with the author until they understand each other. Mechanical criticism of literature can only result in mechanical literature, not good writing.

Re:Yet Another Pointless Dot-Com

[thereofone]

I don't think you read the summary, much less TFA.

Re:Yet Another Pointless Dot-Com

[techno-vampire]

As a matter of fact, I did. I read both. There was very little substance in the article, at least about the developer's idea. However, I do find it hard to believe that you can mechanize the study of good writing and come up with anything original.

Re:Yet Another Pointless Dot-Com

[Skreems]

Wasn't the "improve your writing" aspect an earlier project? I got the impression that the original project was as you described, but the new thing he's trying to partner with Google on is to take the same basic system and use it to recommend "similar to this book" type things in a storefront setting.

Of course all the problems you listed apply anyway. It's very easy to have a work with all the same pieces as a great work of art, but assembled in such a way that the derivative work is completely unsatisfying.

A great example of something similar that you can try today and watch as it fails miserably is Pandora.com. They categorize music by a number of different elements so they can recommend similar pieces. And their categorization is quite accurate; they correctly surmised after about five minutes that I enjoy symphonic rock with a mix of acoustic and electric guitars, obscure lyrics, complex themes, unusual rhythm patterns and interesting chord changes. They then proceeded to present me with one after another shitty Coldplay or Radiohead rip-off band who had every element down perfectly, but still managed to make amazingly bad music. I much expect this product to be the same thing but with books.

"If you like A Deepness In The Sky, why not try An Ewok Christmas: The Novel! They're both about humans meeting strange aliens, and spaceships, and computers, and why are you giving me the finger?"

Re:Yet Another Pointless Dot-Com

[ryen]

If you've RTFA you'd notice that the expected application is for recommendations to potential buyers of books. But i'm also skeptical on that front too...

Re:Yet Another Pointless Dot-Com

[greyhueofdoubt]

>>No, what a poor writer who wants to get better needs is either a good critique group or some friends who will act as beta-readers

I strongly, strongly disagree with using friends or family or anyone you know, frankly, to judge any of your written works. It causes negative emotions from contempt to feelings of betrayal.

People either give you glowing reviews and pick something basically at random to recommend that you change, like a character's outfit in a scene something equally foolish; or people are utterly ignorant of literature and pick apart your work by disagreeing with characters' personalities or actions.

I know I probably sound terribly cynical about this, but it's true. You will never get published if your work has only been reviewed by your mother, your gf, one coworker, and your best friend. None of those people will sacrifice their relationship with you to be honest (or might not be capable of a competent review) and you will never progress from the "I"m writing a book based on my (17-year-old) life because it's SOOOOOO interesting to other people" stage.

No, what writers need are good professors who can dish out honest and useful critiques. Than after school is done, you'll know if you are successful if your reviews, while they remain critical, start coming from people outside of academia, outside of your family, and from within the actual industry. You know, they people who have been in the business of publishing books for the past couple hundred years.

Again sorry about the rant, but I just absolutely refuse to read anything that friends/family have written. It may be good or bad- that part doesn't matter- but it is in my nature to pick things apart and I've noticed that that bugs the crap out of people. And if I was going to give a fluff review, I might as well not even read the thing. Constructive criticism come from knowledgeable people who are unafraid of your feelings.

>>Mechanical criticism of literature can only result in mechanical literature, not good writing.

Oh I forgot to add this earlier. If a writer has a very poor style mechanically, or spelling too awful for machine recognition, then I can assure you that they are a bad writer, period.

If a person rights, like, without any regard, for the differentses between speaken and righten works, and cant get a sentents to make sents let alone a paragraph, or can't stay in one person from sentents to sentents with no parallel structure? and I'm nnot saying that their havent been remarkable non-conformal works in the past? but those works were makde by peple who knew HOW to write well traditionly but chose to right different for artistic affect? because just writing poorely and hoping for a coherent plot is a pointless shot int he dark?

-b

Re:Yet Another Pointless Dot-Com

[techno-vampire]

I strongly, strongly disagree with using friends or family or anyone you know, frankly, to judge any of your written works. It causes negative emotions from contempt to feelings of betrayal.

People either give you glowing reviews and pick something basically at random to recommend that you change, like a character's outfit in a scene something equally foolish; or people are utterly ignorant of literature and pick apart your work by disagreeing with characters' personalities or actions.

If that's what you think, you have never encountered good beta-readers. Either that, or you're a "twenty-something" who still judges everybody by their own experiences. I, OTOH am a mature adult, and pick beta-readers not because they're my buddies, but because I trust them to be honest and objective. And, of course, when I read what they have to say, I leave my ego outside. I will agree with you on one thing, however: if you can't trust your friends and family to tell the truth, a stranger (professor or whatever) is better. I disagree, however, with your opinion that you can never get good writing advice from a friend. I know, because I've finished three novels (Not published yet, I'll admit, but I've gotten good feedback from a few agents.) and used trusted friends for help on all of them. Please note: the word "trusted" is hte key here. If you can't trust them to be honest, don't even ask them.

Re:Yet Another Pointless Dot-Com

[greyhueofdoubt]

>>I trust them to be honest and objective

You're right, and I would have added that except that I wouldn't think many people are friends/family with reliable critics or proof-readers. My mother does professional proofreading for a local publisher, but I don't know if I'd present my work to her unless it was non-fiction. But that concerns our personal relationship.

I guess the main gist of my comment was that _most_ people, when given the chance to critique a friend/relative's work, will err far, far on the side of 'nice' rather than risking the person's feelings. It's not so much a matter of honesty as it is about emotional dissonance between liking a person and maybe not liking their manuscript or the ideas presented therein. Additionally, it is very often the case that one's family and friends are not anywhere near the intended audience for your manuscript. If you are lucky enough to have a good friend who IS the audience and who is qualified to critique, then you should of course listen to them.

But therein lies the key to this: You pick people based on their qualifications, not their relationship to you. Maybe I should have just written that in my original comment instead of rambling away. If you have a friend who is the audience and who is competent to critique, then it matters not that they are your friend.

Anyways, your technique seems to be working for you, so I won't say there aren't exceptions.

-b

Re:Yet Another Pointless Dot-Com

[CAIMLAS]

Never underestimate the power of human stupidity.

To you and me, or anyone else, this might seem like a useless feature. But someone will see the business sense (or whatever they call it) in the idea; they'll buy it and put it online, maybe in an Amazon-like site. And people will use it, and it will be touted as a "success" - even if it's not all that useful or accurate.

algorithm bombing

[notgm]

how long before someone figures how to fool the algorithm, and we all start reading books about enlarging our genetalia, but in a classy way?

Re:algorithm bombing

[Anonymous Coward]

Please tell me how you feel about reading books about enlarging your genitalia.
Would that make you feel more like a man, or more like a woman? And please don't leave out any details.

Re:algorithm bombing

[alexo]

Please tell me how you feel about reading books about enlarging your genitalia.
Would that make you feel more like a man, or more like a woman? And please don't leave out any details.

What did this do to you? Tell me. And remember, this is for posterity, so be honest. How do you feel?

It already happened

[themushroom]

...considering the quantity of "classic" tripe that I had to read in high school and college. Who needs an algorithm when you have English teachers who follow flawed formulae?

A different book

[Hanyin]

What I wonder is: What happens if there's a different style of writing that's not accounted for? I hope they're not just marked down. What will it consider a good book, what's truly interesting and insightful or books that are made to sell like The Da Vinci Code? I can see how much easier it would be to identify which books well sell well but I fear that this will be its only use, and the less said about doing the same for movies the better.

Smart computer

[Trogre]

This computer should do fine, assimilating every book ever written. We'll just need to hire someone to periodically delete every Agatha Christie novel from its database.

Re:Smart computer

[smittyoneeach]

So, you'd punt Agatha and leave in Danielle Steele?

Re:Smart computer

[techno-vampire]

Of course you need to leave in Danielle Steele. Harald Robbins too. After all, you can't neglect the classics, can you?

Re:Smart computer

[celardore]

Who's Agatha Christie? Never heard of her...

Seriously though seeing that reference made me smile :)

Re:Smart computer

[smittyoneeach]

Just as every forest needs its http://en.wikipedia.org/wiki/Rhamnus_purshiana, so I suppose every library to need these two, too.

Is that a bug or a feature?

[smittyoneeach]

What really hits a nerve with me is why the scientific community hasn't opened up all their journals for others to read.

Do you know what you're saying? Do you really want to release possible Weapons of Intellectual Destruction on the world?
I look at the titles in the archives of
http://www.misq.org/
and I'm thinking that some of this stuff is best kept locked in the ivory tower.

Re:Is that a bug or a feature?

[AceofSpades19]

Information wants to be free

Re:Is that a bug or a feature?

[smittyoneeach]

Per the classic /. sig:
"Information wants to be anthropomorphized."

Solomon replies

[smittyoneeach]

Is there any thing whereof it may be said, See, this is new? it hath been already of old time, which was before us.

Ecclesiastes 1:10

An eHarmony.com matching books and people?

[littlewink]

You tell it what books you like and it finds other books that are similarly structured?

What's the big deal, except that Google has Google books?

Anyone could do this. There's plenty of narrative analysis software: the government's outpouring of our tax dollars to "protect us" since 9/11 has triggered every halfwit software development firm in the country to develop

• network relationship analysis software and
• text analysis software

and sell them to the local militia.

Who is Joe?

[mustafap]

There is one persistent son of a bitch on their forum, Joe, who seems to be their nemesis. I wonder what his angle is.

Other than that, I like their approach - involve the community *really* early on.

Apart from Joe.

Re:Who is Joe?

[Geminii]

It's a Joe-job :)

Copyright Infringment detection anyone??

[Anonymous Coward]

His prototype sounds in a way like Netflix's suggestion system for movies, where you vote your favorites and it'll suggest other ones based on your liking. But books are much more complicated, so I can see how his detailed analysis tool can really be the ultimate suggestion tool. I wonder if people will use this to discover copyright infringement on a new level. Hmm... my book and your book are a 99.5% match. Gee where did the .5% discrepancy occur. My character is a 19 yr old hobo, so is yours. My story is about him eventually becoming a successful company executive by pimping himself out to different high-powered women. My character's name is Matt, yours is Mike. Aha.

thhhpt!

[themushroom]

Whoever modded this to 'troll' never took the English classes I had. Yo.

Historic records, yes.

[jd]

I can see exceptional value in indexing, cataloguing and processing all articles in back issues of Wireless World (when it was still called that). There is an enormous wealth of information there on how radio technology improved, when and why. There is also a fantastic amount of information on how to achieve specific effects and how to restore old technology. Not to mention a few pieces on how to build a geostationary communications satellite.

Other old journals will likewise have a lot of valuable information in them. Archaeologists discover a lot through searching their own journals, discovering lost and forgotten reports of discoveries. Mathematicians routinely publish in arcane and super-obscure journals, making what is known far more extensive than what is known to be known.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Need more books?

[vrmlguy]

How about Project Gutenberg? They've got lots of books that have already been scanned.

Already been done...

[mspohr]

Country-western song generators...

http://www.baetzler.de/cgi-bin/country.pl

http://www.outofservice.com/country/

Feels backwards

[forgoil]

The best books I've read are the ones that broke away from what I had read before. The ones that gave me a new experience, and new view on things. I really really don't want to read copies of books I've already read. I want something out of the ordinary. I'll stick to my old methods if you, Mr. Algorithm, don't mind.

Re:Feels backwards

[TuringTest]

Except that, this algorithm can be used to find books that broke away from what you'd read before.

What?

[Anonymous Coward]

I'm an English Lit mayor, and before I became a programmer I use to work in a publishing house in NYC. And let me just say, that no good writer is not even going to give two cents to this project.

They already know their stuff. This portal is more geared towards writers with insecurities: a fourteen year old teenage boy trying to woo his girlfriend, as a plagirizer: "'Come, live with me and be my love,' ... and what next... let's see here... 'You are a phantom of delight.' Ingenious, I'm chortled in my joy."

Then again, I might be for a student who's trying to complete an essay without 'thinking.'

Re:What?

[tehcyder]

I'm an English Lit mayor, and before I became a programmer I use to work in a publishing house in NYC. And let me just say, that no good writer is not even going to give two cents to this project.

They already know their stuff. This portal is more geared towards writers with insecurities: a fourteen year old teenage boy trying to woo his girlfriend, as a plagirizer: "'Come, live with me and be my love,' ... and what next... let's see here... 'You are a phantom of delight.' Ingenious, I'm chortled in my joy."

Then again, I might be for a student who's trying to complete an essay without 'thinking.'

If you really are an English Lit major, the institution you graduated from should be closed down immediately.

wikipedia + unread ebooks = internet smart guy

[dodecalogue]

It's funny about digital books, I just don't read them. I love reading bound-paper books, always have, and it's not like I dislike the principle of digital books, but for whatever reason (tactile, olfactory?) I just can't read more than a couple pages. And it's funny too, browsing through people's digital-book folders, how often there's this type of person with a "philosphy" folder containing plenty of neitzsche, and maybe a couple other people who, after conversing a short bit, I find they are really looking forward to reading someday.*

* probably confusing disinterested grammar which could be used against me in a dashing rebuttal

Good for Artificial Intelligence

[foniksonik]

Anything that analyzes anything in a way that is useful for people is a step in the right direction for AI.

So much of human reaction is related to story telling and shared references where a behavior, event, activity or other interaction can be connected to something we've read or heard or done. It isn't only memory of such things, it is the internal summary we've made as well. If this software can create just such a summary which can be easily cross-referenced with other summaries... then it can help AIs create internal dialogue so that NEW experiences can immediately be summarized in relation to past learned information and categorized accordingly.

An example.... say your AI knows about Auto Mechanics, it then is exposed to a scenario where it needs to diagnose a boat engine problem... it can look at the components, compare it to the knowledge it already has and see that the boat engine is similar in design to an Auto engine. This will narrow down the set of knowledge it needs to utilize to move forward with a diagnosis.

What will be really cool is when they find similarities in patterns that we Humans just don't see.... due to our own close mindedness and need to compartmentalize. As in they will find biological equivalents to mechanical processes and vice versa....

Obligatory

[KalgarThrax]

Print's dead.

RTFA

[Anonymous Coward]

RTFA

Boiling down books?

[rts008]

just burn them instead...at least that way you can use them as a fuel source instead of using fuel to turn them into mush.

Great!

[vuo]

I'm delighted that the advertisers now know my taste in fiction. I'm pretty much the last person you should consult on that one.

"We entertain you, and you ARE entertained."