A FTP-Search engine like mp3.lycos.com (former Fast FTP-Search) has a crawler checking all ftp-servers and offer found links ! If the links are nonworking, the delay between crawler and search-engine-view is to long -> make more crawler-runs and you get not so many broken links.
Because FTPSearch has thousends of ftp-servers the could not check every server hourly. Thats the problem - it`s the problem of FTPSearch !
As an example you will find more mp3`s on a search engine who has 3 days old links than an engine with 1 day old links.
Slashdot Mirror
Great comment !
A FTP-Search engine like mp3.lycos.com (former Fast FTP-Search) has a crawler checking all ftp-servers and offer found links ! If the links are nonworking, the delay between crawler and search-engine-view is to long -> make more crawler-runs and you get not so many broken links.
Because FTPSearch has thousends of ftp-servers the could not check every server hourly. Thats the problem - it`s the problem of FTPSearch !
As an example you will find more mp3`s on a search engine who has 3 days old links than an engine with 1 day old links.