Google and Yahoo only survived the crash because they figured out how to make money at what they do.
Companies that didn't figure this out used up all their venture capital and went out of business.
If people want services like Flickr to exist, they should hope that their owners find a way to monetize them. At the same time, the companies need to find a way to do this without ruining what drew people to the services in the first place.
Re:Wow, open source search engines.
on
Lucene in Action
·
· Score: 1
Before the web, a "search engine" was a piece of software that provided a way to search a collection of documents efficiently. The usual method is to create an inverted index, a data structure analogous to the index in the back of a book, in which you can look up a word and get back a list of all the documents containing the word. There are also a set of standard techniques for ranking the results, for example based on statistics about the distribution of words across documents.
Lucene is a search engine in that sense. It's a library that you build into your application to give it indexing & search capability. You can use it to search files on your computer or whatever you want -- you write the application. For example, Lookout uses Lucene to make all your Outlook mail searchable.
In contrast, Google, Yahoo, etc. are services that use a variation of indexing and search technology specialized for the scale and particular characteristics of the web. It's possible to build a Google-style search engine on top of Lucene -- that's what Nutch is.
I was really impressed that the Newton had shape-recognition in the early 90s, until I learned that Ivan Sutherland's Sketchpad system had this capability in the mid 1960s.
There was a great panel at this year's CHI conference featuring Sutherland and others who worked at Lincoln Labs in the 1950s and 60s. Bill Buxton talks a bit about this on his web site. At the panel, they showed videos of Sutherland drawing with a light pen and having the shapes be recognized and squared up. Other demos included a graphical programming interface (20 years before NeXT's) and user-trainable gesture recognition. Given the computers these guys worked on (look at the pictures), the work they did was amazing.
Google has a well-known preference for scientific, academic, and generally geek-oriented sites. I'd guess these sites are more likely to use Apache. I'm no fan of MS, but I think the only bias here is about what pages the creators of each engine thinks are high value.
Incidentally, Google has its own financial incentive to be biased: to direct traffic to sites that have Google ads.
Apple actually developed the indexing technology even earlier than Mac OS 8.5 and Sherlock. It was code-named V-Twin, officially named "Apple Information Access Toolkit," and demonstrated at the Apple developer's conference in 1996 or 1997.
The original designer of V-Twin was Doug Cutting, of Excite, Lucene, and Nutch fame.
Sherlock's "Find by Content" feature -- the one that crawled your files slowly -- was one application of the technology, but V-Twin was used for many other things over the years. I believe both Spotlight and the SearchKit are based on updated versions of this same infrastructure.
As for why it didn't catch on in 1998: The old (pre-X) Mac OS didn't support multitasking very well (so indexing was intrusive), and disks were a lot smaller (so people didn't need search as much as they do now).
Yahoo hasn't used any Google technology for over a year. Specifically, Yahoo replaced Google search with its own last February. Since then, Yahoo's share of the search market has actually increased. The latest figures from comScore show Google handling 35% of search queries and Yahoo handling 32%.
This is not correct. Google, like Yahoo, includes sponsored links above their search results, as well as on the side. Here's an example. See the big blue box that says "Sponsored Links"? The ads do or don't appear on different queries depending on whether an advertiser has bid on that query. Their placement is based on how successful they are. (In other words, if you make enough money for Google, you get to have your ad shown above the search results.)
According to Accoona's technology page (titled "Accoona and Artificial Intelligence"), their AI system offers three advantages: (1) The ability to find results that don't contain all the query terms; (2) the ability to count some words more than others; (3) the ability to blend results from different sources.
#1 is called "scored OR" and has been around since the 1960s. In fact, every major web search engine used to use this approach until the web got really big.
#2 is called term weighting, and has also been around since the 1960s. These days it's generally not exposed to users, but most search engines have this capability under the hood.
According to the cited article, "original reporting has not been an issue yet, as the submitted articles relied on external sources." So basically, the people who are writing the current news articles merely watched CNN or listened to BBC or read Yahoo news, and paraphrased someone else's reporting. This strikes me as a total waste of effort. Why wouldn't I want to read the original story from someone who was actually there?
A distributed news-gathering organization -- a kind of open-source AP or Reuters -- is an interesting idea. But until they actually have reporters all over the world, I'll keep getting my news from institutions that do.
A couple of people have mentioned something in the discussion about their Grade Point Average. I've occasionally heard the suggestion that it's better to get really good grades at a mediocre school than mediocre grades at a really good school. This is totally untrue.
In my 10 years as a manager of software engineers in both Fortune 500 companies and startups, I have never heard of a single example where a hiring decision was made based on grades or GPA.
The only possible exceptions *might* be if your wonderful grades won you some important prize or honor (e.g. Phi Beta Kappa) that demonstrates that you were way ahead of most of your peers. And even then that would just be a teensy factor that might get your resume sorted higher in the pile.
In fact, when I see people put their GPA on a resume, it might even have a slight subconscious negative effect -- it makes me think they don't know what's important in a real job. Sort of like when people list their height and weight.
(BTW, I say this as someone who had a high GPA from a top-rated school.)
I was at Apple (Computer) when they settled the previous case (1991?). What we were told at the time was that the settlement involved Apple Computer paying Apple Corps a huge chunk of money so they could CONTINUE to "be in the music business" (which at the time meant having machines that could be used to create music). Now it sounds like they agreed to NOT "be in the music business." Does anyone know what that agreement actually said? Was it ever made public?
Many people commenting on this issue say "If you don't want Google to read your mail, don't sign up." That assumes that the only person who has a potential privacy issue is the recipient of the e-mail. My problem is on the other end: when I SEND someone e-mail I don't want someone else to read it. Why should I compromise my privacy so you can get a bigger mailbox?
Nielsen says, "We'll use half the storage space to index all our information so that we can search it instantly. Good riddance, snoozy Outlook search." This is his prediction for the future? Uh, where has he been for the past few years? Instantaneous indexed search of local content has been available to the mass market since Apple's Find-by-Content feature of Sherlock in 1998, and in many lesser-known products earlier. And there have been plenty of mail clients that do full-text indexed search, again starting in the 90s and continuing with Bloomba today.
Incidentally, it doesn't take anything like half the storage space to do this.
I'm old enough to remember when George Lucas said that Star Wars (the original movie) was going to be one of NINE parts. I think he said this around the time Empire came out.
This story is also told in a book I read -- I think it was the introduction to an edition of the script. Lucas conceived this whole saga in his head: fall of the old republic, rise of the rebellion, creation of the new republic. Each was going to be one movie. He decided to make the middle one first, but as he was writing, he realized that one alone was too long, and it became three parts. So he realized that the whole series would have to be three trilogies.
When he came back to work on Episode I, Lucas started talking about a SIX part series.
This account of whole language is wrong on several counts. First of all, whole language, correctly taught, relies on a variety of strategies, including phonics. Secondly, whole language has a basis in developmental psychology, i.e. what we know about how kids learn. Thirdly, it actually works. I have ten years of experience as a parent aide in a school that uses whole language, and the results are phenomenal.
What usually happened in places where whole language "failed" is that a bunch of bureaucrats ordered all teachers to teach this way, without giving them proper training. So the approach gets distorted, mistaught, and misunderstood, and then (surprise!) fails.
Slashdot Mirror
Google and Yahoo only survived the crash because they figured out how to make money at what they do.
Companies that didn't figure this out used up all their venture capital and went out of business.
If people want services like Flickr to exist, they should hope that their owners find a way to monetize them. At the same time, the companies need to find a way to do this without ruining what drew people to the services in the first place.
Lucene is a search engine in that sense. It's a library that you build into your application to give it indexing & search capability. You can use it to search files on your computer or whatever you want -- you write the application. For example, Lookout uses Lucene to make all your Outlook mail searchable.
In contrast, Google, Yahoo, etc. are services that use a variation of indexing and search technology specialized for the scale and particular characteristics of the web. It's possible to build a Google-style search engine on top of Lucene -- that's what Nutch is.
There was a great panel at this year's CHI conference featuring Sutherland and others who worked at Lincoln Labs in the 1950s and 60s. Bill Buxton talks a bit about this on his web site. At the panel, they showed videos of Sutherland drawing with a light pen and having the shapes be recognized and squared up. Other demos included a graphical programming interface (20 years before NeXT's) and user-trainable gesture recognition. Given the computers these guys worked on (look at the pictures), the work they did was amazing.
Incidentally, Google has its own financial incentive to be biased: to direct traffic to sites that have Google ads.
Sherlock's "Find by Content" feature -- the one that crawled your files slowly -- was one application of the technology, but V-Twin was used for many other things over the years. I believe both Spotlight and the SearchKit are based on updated versions of this same infrastructure. As for why it didn't catch on in 1998: The old (pre-X) Mac OS didn't support multitasking very well (so indexing was intrusive), and disks were a lot smaller (so people didn't need search as much as they do now).
Yahoo hasn't used any Google technology for over a year. Specifically, Yahoo replaced Google search with its own last February. Since then, Yahoo's share of the search market has actually increased. The latest figures from comScore show Google handling 35% of search queries and Yahoo handling 32%.
This is not correct. Google, like Yahoo, includes sponsored links above their search results, as well as on the side. Here's an example. See the big blue box that says "Sponsored Links"? The ads do or don't appear on different queries depending on whether an advertiser has bid on that query. Their placement is based on how successful they are. (In other words, if you make enough money for Google, you get to have your ad shown above the search results.)
On the other hand, Yahoo has had an "Also Try" query suggestion feature for ages.
#1 is called "scored OR" and has been around since the 1960s. In fact, every major web search engine used to use this approach until the web got really big.
#2 is called term weighting, and has also been around since the 1960s. These days it's generally not exposed to users, but most search engines have this capability under the hood.
#3 is found in any metasearch engine.
So how is this AI, exactly?
A distributed news-gathering organization -- a kind of open-source AP or Reuters -- is an interesting idea. But until they actually have reporters all over the world, I'll keep getting my news from institutions that do.
The only possible exceptions *might* be if your wonderful grades won you some important prize or honor (e.g. Phi Beta Kappa) that demonstrates that you were way ahead of most of your peers. And even then that would just be a teensy factor that might get your resume sorted higher in the pile.
In fact, when I see people put their GPA on a resume, it might even have a slight subconscious negative effect -- it makes me think they don't know what's important in a real job. Sort of like when people list their height and weight. (BTW, I say this as someone who had a high GPA from a top-rated school.)
I was at Apple (Computer) when they settled the previous case (1991?). What we were told at the time was that the settlement involved Apple Computer paying Apple Corps a huge chunk of money so they could CONTINUE to "be in the music business" (which at the time meant having machines that could be used to create music). Now it sounds like they agreed to NOT "be in the music business." Does anyone know what that agreement actually said? Was it ever made public?
Many people commenting on this issue say "If you don't want Google to read your mail, don't sign up." That assumes that the only person who has a potential privacy issue is the recipient of the e-mail. My problem is on the other end: when I SEND someone e-mail I don't want someone else to read it. Why should I compromise my privacy so you can get a bigger mailbox?
Incidentally, it doesn't take anything like half the storage space to do this.
I'm old enough to remember when George Lucas said that Star Wars (the original movie) was going to be one of NINE parts. I think he said this around the time Empire came out.
This story is also told in a book I read -- I think it was the introduction to an edition of the script. Lucas conceived this whole saga in his head: fall of the old republic, rise of the rebellion, creation of the new republic. Each was going to be one movie. He decided to make the middle one first, but as he was writing, he realized that one alone was too long, and it became three parts. So he realized that the whole series would have to be three trilogies.
When he came back to work on Episode I, Lucas started talking about a SIX part series.
So whatever happened to the last three parts?
This account of whole language is wrong on several counts. First of all, whole language, correctly taught, relies on a variety of strategies, including phonics. Secondly, whole language has a basis in developmental psychology, i.e. what we know about how kids learn. Thirdly, it actually works. I have ten years of experience as a parent aide in a school that uses whole language, and the results are phenomenal.
What usually happened in places where whole language "failed" is that a bunch of bureaucrats ordered all teachers to teach this way, without giving them proper training. So the approach gets distorted, mistaught, and misunderstood, and then (surprise!) fails.
Just to clarify, Nestle is a Swiss company, not American. This particular slezoid marketing campaign can't be blamed on U.S. corporations.