Slashdot Mirror
Words That Speak a Thousand Pictures
venolius writes: "The New York Times (free registration required) has an article
on TextArc (created by W.Bradford
Paley), a site that "aids in the
discovery of patterns and and concepts in arbitrary text" (from the detailed
overview at TextArc). The site serves an applet that performs the task
(texts on which analysis is available include Alice
in Wonderland, Hamlet, and thousands
of others -made available by Project
Gutenberg-). The NYTimes article reports that Paley found that
"Dracula", which relies on a strong storyline had a few keywords
clustered hotly at the center, and that the metaphoric "Frankenstein"
generated a circle of 50 words of modest intensity that faded towards the edges.
"Portrait of the Artist as a Young Man" with evenly distributed key
words produces tight and round lines and "Alice in Wonderland"
produces loopier lines. Check it out! (the applet was tested on better
hardware, but I did well enough with 98/IE6/550MHz/64MB)"
alt.sex.stories.* And see what the results are!! :) What "patterns" might develop? The mind reels!!
...the one we already have, that is:
map connections between two words, concepts, or famous names
see a word's rhymes, synonyms, definitions
and I leave the rest to you.
Once again Project Gutenberg shows its beautiful face. If you haven't heard about it before, then read a Wired feature here. Michael Hart started the project years ago and he wants to digitize anything which is out of copyright. The uses are infinite (think of the blind who can fead texts to tactile printers, for example), which this story also shows.
Anyway, Hart is a big supporter of sensible copyrights (read the feature) and if you can spare the time, help him by digitizing your favourite book.
Hurra for Knark!
Although I only viewed one book, it came up with some interesting results. I'd be curious to know how similar an authors books are to one another... can this distinguish an author's style, or merely individual works.
I also imagine that a college professor might be interested to run this against term papers!
This is very nice looking.
Would make a really cool screen saver if it where in c and not java. Any volentears?
But now I must put on my "think like corp. hat"
Some publisher goes out and maps all the great books and compairs them with current best sellers. Coralate the patterns and then decide that Fromat X creates the best sellers that people buy. Now they refuse to print any book that does not fit their demo graphic of what they concider to be the next best seller.
Its only a matter of time befor these kinds of things are used like a DNA test to see weather a book has good "genes" or bad "genes".
I know it sounds like a conspearicy but I have seen corp.s do stranger things in attempting to repeat past successes. Just look at the movies. We are about to release Star Wars -2 in the name of working on a tried and true formula that started with the release of Jaws II. Did anyone else catch the Special on PBS (frountline i think) that talked about how Jaws was the birth of the end of original movies as we knew it?
Ascii artist &
This is exceedingly facinating. I've worked with word associations for computer authoring, mostly Markov chains of various lengths and phrase-structure stuff. While this takes works for human authors and works out from that, there are some very interesting concepts in here which may be useful in the other direction.
And on top, a wonderful way of displaying it, to catch the eye so the brain has time to engage. 8)
=Blue(23)
LITTLE GIRL: But which cookie will you eat FIRST? C. MONSTER: Me think you have misconception of cookie-eating process.
Point: Somehow if it uses *your* CPU it's different, but when Google's machines do all the work it's somehow OK. Next you'll be complaining about websites doing a DoS by forcing your browser to use CPU by rendering HTML and your TCP stack having to store a sliding window whenever you view a webpage. This selfish attitude is why all filesharing software must be redesigned to NOT allow anyone to kick uploaders. If you want to kick uploaders then shut down the filesharing App, but then you lose download karma.
My personal opinion: Due to heavy-client structure of the majority of machines, this applet uses the correct CPU (yours). Reduced ad revenues means that Internet companies can no longer afford massive server farms unless they require Cydoor installed on all client machines, alternatively they can decrease their equipment costs by delegating processing to client's (suerfers' browser's) CPUs. *nix systems, especially if apps are in a Java sandbox are heavily protected against most attacks, including process DoS attacks via kill -4, kill -9 and a scheduler+VM designed to stand up under heavy loads. Poor Microshaft Win9x people have to do Ctrl-Alt-Del which halts all processes while they look at the sceduler's contents, and it takes 30 seconds of a sucesful DoS for the offending process to be recognised as "unresponsive and kill -9'able" otherwise it is merely "kill -4'able". If you don't want them to use your CPU, then don't visit their website, disable JVM, disable HTML parsing in the browser (this takes CPU), read the raw HTML yourself, disable TCP stack, write packets by hand to minimise CPU usage or even bypass the CPU completely by setting the modem's line inputs yourself with a logic probe, then probe for response using oscilloscope and logic probe. Do PPP-IP-TCP by hand. That won't use your CPU.
Pop quiz hotshot, this is an Informative flamebait, what mod do you do, what mod do you do?
A caveman dreams of being us, the incalculable power and riches. We dream of being Q, then what?
TextArc would certainly be a useful tool for analysis of undeciphered languages and texts. Ventris certainly could have used this for Linear B. The only big limitations would be requiring a suitable sized text and having a consistent meaning to that text. As in, the Rosetta stone probably was not a long enough text to analyse this way.
So long and thanks for all the fish . . . !!!
Does anyone besides me think that this kind of technique could provide stronger protection in cases of source code piracy such as GPL violations, theft of codebase, etc.,? By generating visual patterns based on the occurrence of keywords (or even compiled bytecodes) a signature of a codebase could be generated that is still recognizable even after comments have been stripped out or subtle changes introduced. This could be immensely valuable in GPL infringement cases.
One of the coolest, and most useless things I've ever seen.
I like it!
You are probably thinking of LSA: (Latent Semantic Analysis), which was 'Invented' by former Bell labs researcher (and current U of Colorado psych. prof) Tom Landauer. He uses it to grade his papers, and others probably do as well. It uses the same principle that some search engines (e.g., excite) are based on, and essentially amounts to factor analysis on text. It maps every word in a text into about a 100-dimensional space, based on how often they co-occur in similar contexts. If you feed those factors into a clustering algorithm or and multi-dimensional scaler in order to present it graphically, you probably get something very close to this trick.