• Skip to primary navigation
  • Skip to main content

Easy Cloud AI

Natural Language Processing Consultants in Los Angeles, CA

  • Home
  • News
  • About
  • Contact
  • Show Search
Hide Search

How to prevent getting blacklisted while scraping Amazon

October 30, 2019 By admin

You have to be very careful with Amazon and follow the Amazon Terms of Use and policies related to web-scraping.

Amazon is quite good at banning IPs of the bots. You would have to tweak the DOWNLOAD_DELAY and CONCURRENT_REQUESTS to hit the website less often and be a good web-scraping citizen. And, you would need to rotate IP addresses (you may look into, for instance, crawlera) and user agents.

Filed Under: Natural Language Processing

Copyright © 2021 · Powered by Easy Cloud Solutions