Amazon Bots Cause Grief For Associate Web Sites

theodp writes "Amazon Associates and Web Services developers are crying foul over the hammering they're taking from ill-behaved bots that Amazon had subsidiary Alexa Internet dispatch to evaluate the 'quality and reliability' of their sites. Amazon fessed up and acknowledged problems exist, but points to recent Operating Agreement changes that not only give Amazon and any of its corporate affiliates the right to do so, but also to use unstated technical means to overcome any methods that are used to try to block or interfere with such crawling or monitoring. Interesting stance from the folks who called on the Senate to prosecute those who degrade the technical quality of service at web sites."

Comments

Redirection Limit for this URL exceeded.

[Jan0815]

I am not able to view any of the mentioned links. Keeps on redirecting between login and some other page.

Funny to see that someone complaining about abuse links to pages that do not work with Webwasher filtering.

Boohoo!!

[Ratface]

Given that many people still boycott Amazon for their stance on software patents, I guess that they won't be shedding many tears.

One could argue something about watching out for who your bed-partners are! Bear in mind that a company that has such a disregard for even their affiliates has to have a pretty poor respect for anyone else out there! Caveat emptor!

Re:The Alexa archiver -- you can stop that one.

[D iz a n k Meister]

Seems like Alexa sold Amazon a whole lotta nothing when they agreed to verify the links on AWS sites.

According to one of the posts here:

Again, I don't get how my links can be broken since Amazon is delivering the content.

Re:All they have to do

[valisk]

Thats why they should set it to max request 1 page per minute from any one site, but check out many thousands of sites during that one minute.
Robots have been around since the web started and it suprises me that the designers of this robot havent looked at previous design and good practice.
If any of you Alexia numbskulls happen to be reading this perhaps you could buy yourself a copy of HTTP the def. guide from O'Reilly, which has a tremendously clear explanation of what to think about to prevent your robots from destroying every site they visit that isn't sat on a T3 and Sun Fire w/ 64 CPUs and 64 GB ram.

This is a stupid idea!!!!

[tomblackwell]

There is no guarantee that the "formatted index of all links" is accurate, or up-to-date. Amazon wants to make sure that every single amazon affiliate link meets their criteria.

Your solution would work only for the intelligent and diligent and lucky. There are many Amazon affiliates who are neither.

Can someone clarify?

[callipygian-showsyst]

What's the deal here? It's hard to beleive this is malicious--probably just the result of Amazon hiring the cheapest possible kids to do the Perl hacking/crawling. If they hired more experienced professionals, they may have been able to crawl their affiliated sites better.

Amazon is crawling these sites so that they can be featured on their website. When you search for an item, Amazon lists the prices and availability from the associates--everyone wins.

It seems that Amazon is searching a bit too often--combined with some affiliated sites that have very s-l-o-w dynamic pages, which is causing some problem. It's hardly a crime that Amazon is commiting--after all they want the most accurate, up-to-the-minute information on their website.