first step to building a NLP like search engine would be to map words to their respective subjects (or classification) - this has already been done with wordnet. then as you crawl the net you map the words found to your heirarchy, and you keep a running total of frequency of words on the document as well as the frequency on the net. Eventually, you can sift out the words that have little to no meaning (words that appear frequently typically have no meaning - the, a, and, but, etc...).
Now combine this with pagerank and social ranking and you can refine search results down pretty quickly. During my undergrad I was able to get really good results with this method but I needed more sites in my index to really see if it would work.
Essentially what happens is your queries start off broad and you refine the results down by providing more terms to search by that are associated with the line of queries. (This is how search engines like ask.com (teoma.com was the company that focused on this) work).
I work for the Florida Dept of Law Enforcement as a programmer and I was fingerprinted and had an extensive background check done. I had to list out any immediate family members that might show up as a flag before hand as well.
heh yeah it also asked the question if Abraham Lincoln was at the White House if his hand or foot was there with him. Because they started input data from encyclopedias and had not put in the data describing what a human was yet.
I also wanna say there was something about it posing a question about religion and langauges that was eventually used to write a master's thesis
Cyc is indeed a very interesting take on NLP but it still has a human element involved and until we can eliminate that from the equation the internet will continue to grow faster than we can process it.
not hardly...
they may use different languages to teach you a particular concept, but you definately do not need to be fluent in c/c++/java/lisp/ocaml/scheme/prolog/perl and brainfuck to learn programming theory. But learning language theory can help you learn the other languages.
theres a big and i mean BIG difference between computer programming and computer science.
Computer Science is not about programming. Its about theory: computational theory, programming language theory, and algorithms.
The government and businesses are very very interested in data mining.
The government can use data mining to assist law enforcement with tracking frauds, criminal activities, and even possibly stopping terrorist attacks.
Businesses can use data mining to analyze sensors which monitor temperature, barometric preasure and a ton of other measurements and then ues this information to figure out production procedures for that particular time period.
The possibilities for data mining are pretty much endless given you have the processing power to compute all the numbers.
Data mining does not just deal with data that is in a DB. Rather finding some pattern or correlation between sets of data that otherwise seem completely random or pointless given the context.
One of the main problems for data mining is privacy and until we can find a secure way to share data between DBs it will only hinder the advancement.
A really good book to read is Data Mining:Next Generation Challenges and Future Directions which discusses pretty much everything you need to know about data mining.
Also it is pretty much impossible for google to index all the information on the internet consider there is about ~110 billion webpages and it grows each year.
The end goal is to be able to return the single 1 page that a user wants that contains ALLLLL the information possible for him/her. Too bad for us these pages dont exist. So the logical thing to do is return all the pages that would satisfy this need, and just those pages - no more, no less. But what google attempts to do is, return ALL pages that it determines as relevent and rank them. This is why we get queries of 100,000+ for somewhat broad terms. It sucks and is a crappy way to do searching. Then on top of that their page rank algorithm is pretty much popularity based and you get even worse results.
Eventually there will be a search engine that can determine a fair ranking of pages without the need to check its popularity on the net.
Sounds like a really good business to start up. An online place where you can store personal information on protected servers. Have everything encrpyted when its uploaded and stored on servers. Then to retrieve the information you have to call or something.
er well it sounded like a good idea at first =)
The internet isnt contained within any countries laws and will be very hard to regulate what is put on it.
Google's system is completely automated and at the rate they crawl webpages and index them it would be IMPOSSIBLE to stop one of their crawlers from picking up some piece of personal info on the net.
Its just one of the flaws of the internet. Yes things will be put on there that you dont like. But if you arnt willing to live with what you said then dont say it in the first place.
Slashdot Mirror
'OK, if I decide to do this, I'm gonna need an unlimited supply of Xena tapes, and Hot Pockets'.
Couple hundred gallons of arsenal will do the job nicely... http://www.cdms.net/LabelsMsds/LMDefault.aspx?pd=70&t=1,2,3,4
Timber companies like rayonier are using the waste from the paper and saw mills to create cellulose and fuel biomass plants.
Robots could kill override the 3 laws by using the Zeroth law which is used to save humanity in the Foundation Series - "A robot may not harm humanity, or, by inaction, allow humanity to come to harm." http://en.wikipedia.org/wiki/Three_Laws_of_Robotics#Zeroth_Law_added
How many trees have you planted? At last count my family has planted almost half a billion....
first step to building a NLP like search engine would be to map words to their respective subjects (or classification) - this has already been done with wordnet. then as you crawl the net you map the words found to your heirarchy, and you keep a running total of frequency of words on the document as well as the frequency on the net. Eventually, you can sift out the words that have little to no meaning (words that appear frequently typically have no meaning - the, a, and, but, etc...).
Now combine this with pagerank and social ranking and you can refine search results down pretty quickly. During my undergrad I was able to get really good results with this method but I needed more sites in my index to really see if it would work.
Essentially what happens is your queries start off broad and you refine the results down by providing more terms to search by that are associated with the line of queries. (This is how search engines like ask.com (teoma.com was the company that focused on this) work).
I work for the Florida Dept of Law Enforcement as a programmer and I was fingerprinted and had an extensive background check done. I had to list out any immediate family members that might show up as a flag before hand as well.
it inferred that most people were famous
heh yeah it also asked the question if Abraham Lincoln was at the White House if his hand or foot was there with him. Because they started input data from encyclopedias and had not put in the data describing what a human was yet.
I also wanna say there was something about it posing a question about religion and langauges that was eventually used to write a master's thesis
I suggest reading What computers still cant do.
0 82900-3367853?v=glance&n=283155
http://www.amazon.com/gp/product/0262540673/103-5
Cyc is indeed a very interesting take on NLP but it still has a human element involved and until we can eliminate that from the equation the internet will continue to grow faster than we can process it.
not hardly... they may use different languages to teach you a particular concept, but you definately do not need to be fluent in c/c++/java/lisp/ocaml/scheme/prolog/perl and brainfuck to learn programming theory. But learning language theory can help you learn the other languages.
theres a big and i mean BIG difference between computer programming and computer science. Computer Science is not about programming. Its about theory: computational theory, programming language theory, and algorithms.
The government and businesses are very very interested in data mining. The government can use data mining to assist law enforcement with tracking frauds, criminal activities, and even possibly stopping terrorist attacks. Businesses can use data mining to analyze sensors which monitor temperature, barometric preasure and a ton of other measurements and then ues this information to figure out production procedures for that particular time period. The possibilities for data mining are pretty much endless given you have the processing power to compute all the numbers. Data mining does not just deal with data that is in a DB. Rather finding some pattern or correlation between sets of data that otherwise seem completely random or pointless given the context. One of the main problems for data mining is privacy and until we can find a secure way to share data between DBs it will only hinder the advancement. A really good book to read is Data Mining:Next Generation Challenges and Future Directions which discusses pretty much everything you need to know about data mining. Also it is pretty much impossible for google to index all the information on the internet consider there is about ~110 billion webpages and it grows each year.
I recently had a job interview where we got on the subject of WoW for like 5 minutes =p
was pretty weird
The end goal is to be able to return the single 1 page that a user wants that contains ALLLLL the information possible for him/her. Too bad for us these pages dont exist. So the logical thing to do is return all the pages that would satisfy this need, and just those pages - no more, no less. But what google attempts to do is, return ALL pages that it determines as relevent and rank them. This is why we get queries of 100,000+ for somewhat broad terms. It sucks and is a crappy way to do searching. Then on top of that their page rank algorithm is pretty much popularity based and you get even worse results. Eventually there will be a search engine that can determine a fair ranking of pages without the need to check its popularity on the net.
im gonna patent common sense
rofl
they have a Data center in ATL and one in Virginia (IIRC)
Sounds like a really good business to start up. An online place where you can store personal information on protected servers. Have everything encrpyted when its uploaded and stored on servers. Then to retrieve the information you have to call or something. er well it sounded like a good idea at first =)
I have the v1200 and love it. Its easy to work with, sturdy, cool, and nice looking. My next comp case will definately be another lian li
The internet isnt contained within any countries laws and will be very hard to regulate what is put on it.
Google's system is completely automated and at the rate they crawl webpages and index them it would be IMPOSSIBLE to stop one of their crawlers from picking up some piece of personal info on the net.
Its just one of the flaws of the internet. Yes things will be put on there that you dont like. But if you arnt willing to live with what you said then dont say it in the first place.