Slashdot Mirror

Microsoft will buy somebody who are far ahead of Google in Enterprise search.
I bet that 1st candidate will be http://fast.no/ and it is not very expensive
(for Microsoft)

AllTheWeb's FTP searcher is okay, but unfortunately they seem to be all too happy to yank sites from their database when they get whined at about copyrights. I liked it better when it was still ftpsearch.fast.no... bah, humbug. Anyway, for further searching satisfaction, see filesearching.com and a directory of FTP search engines-- many of which are based on the original ftpsearch server, whose source code is available.

what about those folks in Norway? They did just the same thing two years ago and nobody noticed.

The directory tree in each package could also contain a mixture of different architectures so that one could share most of the package's files (e.g., man pages) among platforms and only keep separate files where needed (e.g., executables). The scheme allowed you to categorize the files into any granularity you wanted. You could for instance have one exeuctable for m68k, one optimized for Pentium4, another optimzed for 486, and a generic x86 one (if you had, say, a PentiumIII). The system made sure that the symlink which suited you best was used.

There were also, of course, possibilities to specify files only valid for certain OSes and OS revisions, and one could specify files which should be valid only within one certain domain, or on one certain host.

Another feature was the ability to keep multiple versions of the same application around, and mark the different versions as stable, release, beta, etc. This allowed you to specify hosts or domains which only should run stable software, which should be able to run beta software, and so on.

In addition to keeping multiple versions of the binaries around, the system also could keep multiple build frameworks around. The build framework could contain configurations for various OSes and platforms, and allowed applications to be more easily installed on various architectures.

Of course, there was also a package config file indicationg the name of the package, short info, dependencies, etc.

For distribution one could create a tree-like structure where the whole or a subset of the packages and binaries for the packages were automatically mirrored to hosts in the network. This allowed you to have most of the packages located on the servers while the clients which had only limited diskspace only had a limited set of packages (e.g., shells) installed locally.

There were also add-ons to automatically configure various environment variables on the hosts, configre mailcap/mime-type files, emacs configuration entries, info-file entries, etc.

The whole thing worked wonderfully (once you as a sysadmin learnt how to use it properly). It automatically propagated new applications throughout the system once you installed them and generally made life easier for the sysadmin. Installing software on new machines was also pretty painless. You just configured your new client system, fired up a single command, and all applications suitable for the system were installed and configured for you.

Yesterday, I posted a reply telling about a Norwegian company (Fast), which have developed the search engine called AllTheWeb. AllTheWeb seems superior to any other search engine on the net when it comes to the hardware being used, as they are using specially developed hardware for searching through huge amounts of text (and other media).

digitoday.no (Norwegian only) today reported about further enhancements of the AllTheWeb search engine. I have tried to do my best in translating some of the article into English;

- Fast will soon change to a new and improved crawler which will find three times as many web pages. That way, Fast will soon cover the whole Internet.
...
Fast estimates that the web today consists of billions of page, but by removing duplicates and "garbage" the number will decrease dramatically. They estimate that their search engine will cover 1.8 billion web pages before christmas.
...
One of the biggest improvements is the ability to index dynamic pages. Dynamic pages are web pages you can only access by pushing a button, choosing something from a menu, or filling out information in a form.
...

The whole article (in Norwegian) can be read here. I'm not a translator, and my English is pretty bad, so you are warned. :-)

• Cache: Means that we are able to visit a site after it's been slashdotted.
• Relevance: Google's "relevance technology" is great. Find related sites, and find only pages related to your query. :-)
• Not only web pages: Google doesn't only search for web pages, but also PDF files and images. More search engines should have had features like that.

There's been one available for a long time at multimedia.alltheweb.com, brought to you by the fine folks at Fast Search and Transfer ASA.

...now yahoo is going to be 'powered by' google, and hotbot is 'powered by' lycos...

Actually, Lycos has dropped their original search technology, and moved to using the Fast engine (read the press release.)

Taken the complaints many people have that an HTML page can not always be rendered very nicely into WML, I tried to find out search engines that searched exclusively for WML pages. I found at least four different engines. Two of them seem to be more or less bogus (Waply and WapWarp). The best one seems to be FAST Wap Search. Are the any other good WAP search engines out there?

Well, they say they honor ROBOTS.TXT However your post suggests differently. You ought to e-mail them and find out. Their robots policy is stated here: http://www.fast.no/f aq/faqfastwebsearch/faqfastwebcrawler.html

But www.fast.no seems to be running Linux. Anyway...according to the Uptime List, FreeBSD has much higher uptimes than Linux. Looks like it is the choice of the folks that don't reboot. I think those are mainly to be found in commercial environments like this one. Quite funny - a search for my nick/handle only finds results on /. and [fm] :)

Check out this
http://www.fast.no/product/fastpmc.html

gaute

-- We plunge for the slipstream the realness to find

FAST is a norvegian company that specialize in fast algorithms for search and also transfer of video and images. Since FAST is a research company they are interested in selling their technology, not productifying it themselves. All the web is a marketing site for FAST technology and is probably going to be closed down after an eventual sale (according to norvegian newspapers).

The company has a couple of strange fellows working for them, there is former archeologist, medical doctors and statisticans. Guess a combination of their speciality fields spurred a really strange, but fast, algorithm.

Hope that will answer som questions.

By the way, the URL to fast is www.fast.no.

The specs on the search engine are available at http://web.fast.no/product/search/d et.asp?id=34.

The press release doesn't exactly scream it out, but the search engine is actually just a little bit of software stuck on top of some pretty neat custom hardware. They call their chip the FAST PMC (Pattern Matching Chip), and their server is just your average (well, sort of average) high end server, with a buttload of those chips stuck on PCI cards.

The specs on the PMCs are available at http://web.fast.no/product/PMC/det.asp ?id=52.

FAST claims 100 MB/sec throughput on each chip, and each card has its own RAM (from 8 MB to 2 GB). The chips actually run at 100 mHz each, and even have support for RegEx matching (slightly limited).

From the specs:
A typical configuration will contain 4 to 8 plug-in cards per search node, and 16 or 32 chips on each card.


Overall, I'm pretty impressed - putting search capabilities into hardware is a pretty good idea, especially since so much of a modern processor is geared toward things like Floating Point calculations, which doesn't help text searching at all.


Scott Severtson
Software Developer
Auragen Communications
scotty@auragen.com

The specs on the search engine are available at http://web.fast.no/product/search/d et.asp?id=34.

The press release doesn't exactly scream it out, but the search engine is actually just a little bit of software stuck on top of some pretty neat custom hardware. They call their chip the FAST PMC (Pattern Matching Chip), and their server is just your average (well, sort of average) high end server, with a buttload of those chips stuck on PCI cards.

The specs on the PMCs are available at http://web.fast.no/product/PMC/det.asp ?id=52.

FAST claims 100 MB/sec throughput on each chip, and each card has its own RAM (from 8 MB to 2 GB). The chips actually run at 100 mHz each, and even have support for RegEx matching (slightly limited).

From the specs:
A typical configuration will contain 4 to 8 plug-in cards per search node, and 16 or 32 chips on each card.


Overall, I'm pretty impressed - putting search capabilities into hardware is a pretty good idea, especially since so much of a modern processor is geared toward things like Floating Point calculations, which doesn't help text searching at all.


Scott Severtson
Software Developer
Auragen Communications
scotty@auragen.com