Slashdot Mirror
Boiling Down Books, Algorithmically
destinyland writes "A year ago, Aaron Stanton harangued Google over his new project, a web site analyzing patterns in books to generate infallible recommendations. In March he finally finished a prototype which he showed to Google, Yahoo, and Amazon, and he's just announced that he's finally received a big contract which 'gives us a great deal of potential data to work with.' The 25-year-old's original prototype examined over 200 books, plotting 729,000 data points across 30,293 scenes — but its universe of analyzed novels is about to become much, much bigger."
The difference between now and 100 years ago becomes more apparent each day. Then, owning books was a sign of affluence, of intelligence. Now? Everything is up to question, and should be. Analyzing books and other public material is just another step in putting intelligence out there for everyone, not just those that can afford it. I applaud it, and all the dangers it brings. Such hurdles are necessary, but we must assault them to overcome barriers that should no longer exist.
Support NYCountryLawyer RIAA vs People
I love how the prototype version in the link gives a 98% match between George Orwell's '1984' and the text of the USA Patriot Act!
when i come to think of it, building a tool that rates the level of novelty in an idea would be good one. it would make the job of large companies or opportunity hunters easier as well..
cheers,
mbilgi
I, for one, welcome our new book analyzing overlords!
The only infallible book recommendation has existed for nearly 2000 years now.
Don't want to read it? Heretic! But "translations" do exist for public convenience.
...and if you do not read, you won't want this.
I am skeptical that analyzing the content of the books can lead to good recommendations, let alone "infallible". Two books can be very similar in subject matter and writing style and yet one can be great and the other one awful. The difference is just too subtle for an algorithm to figure out, though I hope I am wrong and it turns out that it works, it would be very useful. Same applies to movies and music as well. I always found "Customers who purchased this book also purchased...." section on amazon to be more valuable than my personalized recommendations
Negative moral value of force outweighs the positive value of good intentions.
This is just another pointless project that's going to waste the time and skull-sweat of a good but unrealistic programmer. All he's going to have when he's done is the solution to a problem that doesn't, for all practical purposes, exist. Good writers won't need it because they know what to do and how to do it, so they won't use it. It will only be used by poor writers, who won't know how to put the suggestions into effect properly. It may, possibly, tell a writer where their book needs work, or where it's not interesting enough, but I doubt it. Most likely, all it will do is tell it where it's not like other successful books because it won't be able to recognize or take into account any originality. Even if its recommendations are right, a poor writer is highly unlikely to profit from them, because by definition a poor writer won't know which suggestions are good or the skills to take advantage of them properly. No, what a poor writer who wants to get better needs is either a good critique group or some friends who will act as beta-readers, telling him not only what doesn't work but why (Something, I might add, that I find it hard to believe this program could ever do.) and discuss things with the author until they understand each other. Mechanical criticism of literature can only result in mechanical literature, not good writing.
Good, inexpensive web hosting
how long before someone figures how to fool the algorithm, and we all start reading books about enlarging our genetalia, but in a classy way?
...considering the quantity of "classic" tripe that I had to read in high school and college. Who needs an algorithm when you have English teachers who follow flawed formulae?
Laughter is the Spackle of the Soul.
What I wonder is: What happens if there's a different style of writing that's not accounted for? I hope they're not just marked down. What will it consider a good book, what's truly interesting and insightful or books that are made to sell like The Da Vinci Code? I can see how much easier it would be to identify which books well sell well but I fear that this will be its only use, and the less said about doing the same for movies the better.
This computer should do fine, assimilating every book ever written. We'll just need to hire someone to periodically delete every Agatha Christie novel from its database.
"Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
Only Cops and Crooks Have Benefited From $2.5 Trillion Spent Fighting Trafficking.
The United States' so-called war on drugs brings to mind the old saying that if you find yourself trapped in a deep hole, stop digging. Yet, last week, the Senate approved an aid package to combat drug trafficking in Mexico and Central America, with a record $400 million going to Mexico and $65 million to Central America.
The United States has been spending $69 billion a year worldwide for the last 40 years, for a total of $2.5 trillion, on drug prohibition - -- with little to show for it. Is anyone actually benefiting from this war? Six groups come to mind.
The first group are the drug lords in nations such as Colombia, Afghanistan and Mexico, as well as those in the United States. They are making billions of dollars every year -- tax free.
The second group are the street gangs that infest many of our cities and neighborhoods, whose main source of income is the sale of illegal drugs.
Third are those people in government who are paid well to fight the first two groups. Their powers and bureaucratic fiefdoms grow larger with each tax dollar spent to fund this massive program that has been proved not to work.
Fourth are the politicians who get elected and reelected by talking tough -- not smart, just tough -- about drugs and crime. But the tougher we get in prosecuting nonviolent drug crimes, the softer we get in the prosecution of everything else because of the limited resources to fund the criminal justice system.
The fifth group are people who make money from increased crime. They include those who build prisons and those who staff them. The prison guards union is one of the strongest lobbying groups in California today, and its ranks continue to grow.
And last are the terrorist groups worldwide that are principally financed by the sale of illegal drugs.
Who are the losers in this war? Literally everyone else, especially our children.
Today, there are more drugs on our streets at cheaper prices than ever before. There are more than 1.2 million people behind bars in the U.S., and a large percentage of them for nonviolent drug usage. Under our failed drug policy, it is easier for young people to obtain illegal drugs than a six-pack of beer. Why? Because the sellers of illegal drugs don't ask kids for IDs. As soon as we outlaw a substance, we abandon our ability to regulate and control the marketing of that substance.
After we came to our senses and repealed alcohol prohibition, homicides dropped by 60% and continued to decline until World War II. Today's murder rates would likely again plummet if we ended drug prohibition.
So what is the answer? Start by removing criminal penalties for marijuana, just as we did for alcohol. If we were to do this, according to state budget figures, California alone would save more than $1 billion annually, which we now spend in a futile effort to eradicate marijuana use and to jail nonviolent users. Is it any wonder that marijuana has become the largest cash crop in California?
We could generate billions of dollars by taxing the stuff, just as we do with tobacco and alcohol.
We should also reclassify most Schedule I drugs ( drugs that the federal government alleges have no medicinal value, including marijuana and heroin ) as Schedule II drugs ( which require a prescription ), with the government regulating their production, overseeing their potency, controlling their distribution and allowing licensed professionals ( physicians, psychiatrists, psychologists, etc. ) to prescribe them. This course of action would acknowledge that medical issues, such as drug addiction, are best left under the supervision of medical doctors instead of police officers.
The mission of the criminal justice system should always be to protect us from one another and not from ourselves. That means that drug users who drive a motor vehicle or commit other crimes while under the influence of these drugs would continue to be hel
What really hits a nerve with me is why the scientific community hasn't opened up all their journals for others to read.
Do you know what you're saying? Do you really want to release possible Weapons of Intellectual Destruction on the world?
I look at the titles in the archives of
http://www.misq.org/
and I'm thinking that some of this stuff is best kept locked in the ivory tower.
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
Is there any thing whereof it may be said, See, this is new? it hath been already of old time, which was before us.
Ecclesiastes 1:10
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
What's the big deal, except that Google has Google books?
Anyone could do this. There's plenty of narrative analysis software: the government's outpouring of our tax dollars to "protect us" since 9/11 has triggered every halfwit software development firm in the country to develop
and sell them to the local militia.
There is one persistent son of a bitch on their forum, Joe, who seems to be their nemesis. I wonder what his angle is.
Other than that, I like their approach - involve the community *really* early on.
Apart from Joe.
Open Source Drum Kit, LPLC deve board - mjhdesigns.com
His prototype sounds in a way like Netflix's suggestion system for movies, where you vote your favorites and it'll suggest other ones based on your liking. But books are much more complicated, so I can see how his detailed analysis tool can really be the ultimate suggestion tool. I wonder if people will use this to discover copyright infringement on a new level. Hmm... my book and your book are a 99.5% match. Gee where did the .5% discrepancy occur. My character is a 19 yr old hobo, so is yours. My story is about him eventually becoming a successful company executive by pimping himself out to different high-powered women. My character's name is Matt, yours is Mike. Aha.
balance is struck, playing so it's users. Surprise would like to FreeBSD at about 80 chronic abuse of areL incompatible ITS CORPSE TURNED another folder. 20 my calling. Now I much organisation, turd-suckingly Were compounded Long term survival [amazingkreskin.com] end, we need you I know it sux0rs, to yet another Raymond in his to the original a previously parties, but here and enjoy all the Architecture. My HEAD SPINNING Why not? It's quick bben many, not the You should bring THE LATEST NETCRAFT To have regular politics openly. she had no fear
Whoever modded this to 'troll' never took the English classes I had. Yo.
Laughter is the Spackle of the Soul.
Other old journals will likewise have a lot of valuable information in them. Archaeologists discover a lot through searching their own journals, discovering lost and forgotten reports of discoveries. Mathematicians routinely publish in arcane and super-obscure journals, making what is known far more extensive than what is known to be known.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Comment removed based on user account deletion
How about Project Gutenberg? They've got lots of books that have already been scanned.
Nothing for 6-digit uids?
http://www.baetzler.de/cgi-bin/country.pl
http://www.outofservice.com/country/
I don't read your sig. Why are you reading mine?
The best books I've read are the ones that broke away from what I had read before. The ones that gave me a new experience, and new view on things. I really really don't want to read copies of books I've already read. I want something out of the ordinary. I'll stick to my old methods if you, Mr. Algorithm, don't mind.
I'm an English Lit mayor, and before I became a programmer I use to work in a publishing house in NYC. And let me just say, that no good writer is not even going to give two cents to this project.
They already know their stuff. This portal is more geared towards writers with insecurities: a fourteen year old teenage boy trying to woo his girlfriend, as a plagirizer: "'Come, live with me and be my love,' ... and what next... let's see here... 'You are a phantom of delight.' Ingenious, I'm chortled in my joy."
Then again, I might be for a student who's trying to complete an essay without 'thinking.'
It's funny about digital books, I just don't read them. I love reading bound-paper books, always have, and it's not like I dislike the principle of digital books, but for whatever reason (tactile, olfactory?) I just can't read more than a couple pages. And it's funny too, browsing through people's digital-book folders, how often there's this type of person with a "philosphy" folder containing plenty of neitzsche, and maybe a couple other people who, after conversing a short bit, I find they are really looking forward to reading someday.*
* probably confusing disinterested grammar which could be used against me in a dashing rebuttal
Anything that analyzes anything in a way that is useful for people is a step in the right direction for AI.
So much of human reaction is related to story telling and shared references where a behavior, event, activity or other interaction can be connected to something we've read or heard or done. It isn't only memory of such things, it is the internal summary we've made as well. If this software can create just such a summary which can be easily cross-referenced with other summaries... then it can help AIs create internal dialogue so that NEW experiences can immediately be summarized in relation to past learned information and categorized accordingly.
An example.... say your AI knows about Auto Mechanics, it then is exposed to a scenario where it needs to diagnose a boat engine problem... it can look at the components, compare it to the knowledge it already has and see that the boat engine is similar in design to an Auto engine. This will narrow down the set of knowledge it needs to utilize to move forward with a diagnosis.
What will be really cool is when they find similarities in patterns that we Humans just don't see.... due to our own close mindedness and need to compartmentalize. As in they will find biological equivalents to mechanical processes and vice versa....
A fool throws a stone into a well and a thousand sages can not remove it.
Print's dead.
RTFA
just burn them instead...at least that way you can use them as a fuel source instead of using fuel to turn them into mush.
Down With Slashdot BETA!!! I've been around the corner and seen the oliphant; you can only abuse me from your perspecti
I'm delighted that the advertisers now know my taste in fiction. I'm pretty much the last person you should consult on that one.
"We entertain you, and you ARE entertained."