How to prevent getting blacklisted while scraping Amazon

by | Oct 30, 2019 | Natural Language Processing | 0 comments

You have to be very careful with Amazon and follow the Amazon Terms of Use and policies related to web-scraping.

Amazon is quite good at banning IPs of the bots. You would have to tweak the DOWNLOAD_DELAY and CONCURRENT_REQUESTS to hit the website less often and be a good web-scraping citizen. And, you would need to rotate IP addresses (you may look into, for instance, crawlera) and user agents.

email: sales @ easycloudsolutions.com

phone: (626) 607-4250

address: 4733 Torrance Blvd #139, Torrance, CA 90503-4100