'Python requests behind proxy blocked in AWS EC2 but not in local
I am trying to scrap a page on tickpick.com with AWS EC2. For this I use the python requests module. I was suspecting tickpick.com to block AWS IP addresses, therefore I set up a free proxy in order to mask the AWS IP address. I also tried setting up an user agent.
I'm having a hard time because the python script request is blocked by Cloudfare when run on AWS EC2 but not when it's run on my local machine (Windows 10).
import requests
URL_MAIN = 'https://www.tickpick.com/mlb/los-angeles-dodgers-tickets/?page=1'
proxies = {
'http': 'http://*.*.*.*:80' # free proxy
}
r = requests.get(URL_MAIN, proxies = proxies, headers={'User-Agent': 'Chrome'})
print(r.content)
I have no clue why it would run in local and not in EC2 as I use the exact same python script and proxy to connect. I don't want to use selenium because it is too slow for my application. Is there anything else I can do ?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
