'python web scraping IP blocked
I am trying to extract the source code of the html page. It was working fine before. but now the source web server wanted more evidence that I am NOT a bot. This is the error: your IP is blocked. My IP is NOT blocked for sure as I can still open the page manually via any browser. Do I need to change any parameters before making the request. Thanks.
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"}
req = requests.get(url, headers=headers)
url_content = req.content
url_content = url_content.replace(b'data-imgid', b'\ndata-imgid')
output_file = open('downloaded.txt', 'wb')
output_file.write(url_content)
output_file.close()
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|