I scrape AA.com too, but in a very different way technically and for a different reason.
I've developed a free program called MileTracker that consolidates users' frequent flier accounts. One of the accounts that I pull information from is of course at AA.com. I haven't received any complaints yet, but there are several reasons why I can think of that:
I do the scraping from the client. There is no central server involved at any point.
There are only a few hundred users at this point, each generating at most 2-3 requests a day. More likely they're only generating 1 request every few days.
I've been very careful to simulate an actual human user using a Web browser. There are some steps that I skip for performance reasons (such as downloading images which could be cached anyway), but I make sure to handle all cookies, redirects, Javascript, etc. I also throw in some delays and randomization.
Someone looking through logs would probably have a lot of trouble telling the difference between a real user and my "ghost" user.
The application really isn't harmful to AA. In fact, there's a good chance that I actually increase real traffic to AA.com amongst their loyal customers. When there is a change in a user's frequent flier account balance, I provide an easy way for them to actually login to AA.com so that they can view the details. If AA complains, I'd probably obey since I have more to lose than I do to gain. It's AA's customers that use MileTracker that would suffer.
I've been doing something similar for almost a year now using mike-XYZ@moonlightlabs.com. I'd guess that 80% of my junk mail is from my published ICQ email address, 15% of it is from my USENET address, and 5% of it is from my college alumni directory. I can only remember receiving 1 spam message that wasn't addressed to one of the 3 above addresses (and I've used hundreds of differents ones so far).
Slashdot Mirror
I've developed a free program called MileTracker that consolidates users' frequent flier accounts. One of the accounts that I pull information from is of course at AA.com. I haven't received any complaints yet, but there are several reasons why I can think of that:
- I do the scraping from the client. There is no central server involved at any point.
- There are only a few hundred users at this point, each generating at most 2-3 requests a day. More likely they're only generating 1 request every few days.
- I've been very careful to simulate an actual human user using a Web browser. There are some steps that I skip for performance reasons (such as downloading images which could be cached anyway), but I make sure to handle all cookies, redirects, Javascript, etc. I also throw in some delays and randomization.
Someone looking through logs would probably have a lot of trouble telling the difference between a real user and my "ghost" user.The application really isn't harmful to AA. In fact, there's a good chance that I actually increase real traffic to AA.com amongst their loyal customers. When there is a change in a user's frequent flier account balance, I provide an easy way for them to actually login to AA.com so that they can view the details. If AA complains, I'd probably obey since I have more to lose than I do to gain. It's AA's customers that use MileTracker that would suffer.
I've been doing something similar for almost a year now using mike-XYZ@moonlightlabs.com. I'd guess that 80% of my junk mail is from my published ICQ email address, 15% of it is from my USENET address, and 5% of it is from my college alumni directory. I can only remember receiving 1 spam message that wasn't addressed to one of the 3 above addresses (and I've used hundreds of differents ones so far).