'How to pass multiple proxies as a list but stop when it works?
I have "implemented" a way to work with proxies, and python's requests library. I'm scraping a website and after a while even though I wait like 20 sec after making the next request it blocks my ip so I made this proxy code:
import requests
import time
HTTP = [list of http proxies]
HTTPS = [list of https proxies]
def try_proxies(http_proxies, https_proxies):
for proxy_http_element in http_proxies:
http_proxy = proxy_http_element
yield http_proxy
for proxy_https_element in https_proxies:
https_proxy = proxy_https_element
yield https_proxy
test = [p for p in try_proxies(HTTP, HTTPS)]
for proxy in test:
print(f"{proxy}")
proxies = {
'http': proxy,
}
try:
time.sleep(10)
res = requests.get("site",
headers={
'User-agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0',
'Origin': "https://www.car.gr",
"Access-Control-Request-Method": "GET",
'Accept-Language': "en-US,en;q=0.5",
'Accept-Encoding': "gzip, deflate",
'Request-Domain': 'domain',
'Site': 'car',
"Sec-Fetch-Dest": "empty",
"Sec- Fetch-Mode": "cors",
"Sec-Fetch-Site": "same-origin",
"Te": "trailers",
'Connection': 'close'},
proxies=proxies, timeout=10).text
print("here")
with open("anothertes.html", 'w') as testhtml:
string = str(res)
testhtml.write(string)
# print(res)
except requests.exceptions.ConnectTimeout as CT:
print(f"well at last your tried: {CT}")
# def main():
# try_proxies(HTTP, HTTPS)
#
#
# main()
It works at some extend as it tried to connect but since it is on a for loop it won't stop where it should stop. Can someone help me please?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
