'Selenium request held on bot checking page
I’m attempting to connect to the below website using selenium but the browser is held on a page which says “we’re making sure you’re not a bot”, which I believe is a Cloudflare block.
url: https:// www .aston martin austin. com /
Despite the page saying that it will connect after 5 seconds the browser is held on this site for longer. Implementing a sleep period sometimes results in the site loading, however, on other occasions the browser remains on this page without reaching the site.
I’ve tested various approaches including using a variety of chrome options (see code for a full list), selenium-wire to match headers against those when loading the site manually, a while loop with multiple gets and wait periods in between (which occasionally works - why is this?), but no method that I can code so far has proved to consistently work time after time i.e. connect to the home page first time. I've also tried changing the $cdc variable in the chrome driver executable.
The undetected-chrome package is the one package that does seem to work, however, I’d like to learn and understand which change I can make to my own code to make consistent successful requests without the help of an additional package.
url: https:// www.aston martin austin. com /
Note – spaces should be removed from the URL when testing.
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
options = webdriver.ChromeOptions()
options.add_argument("window-size=1200x600")
options.add_argument("--headless")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option("useAutomationExtension", False)
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("--no-sandbox")
options.add_argument("--ignore-certificate-errors-spki-list")
options.add_argument("--ignore-ssl-errors")
options.add_argument("--allow-insecure-localhost")
options.add_argument("--ignore-certificate-errors")
options.add_argument("--enable-javascript")
options.add_argument("javascript.enabled")
options.add_argument("--lang=en-US,en;q=0.8")
options.add_argument("--incognito")
options.add_argument("start-maximized")
options.add_argument("Sec-Fetch-Site=cross-site")
capabilities = options.to_capabilities()
capabilities["acceptInsecureCerts"] = True
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36")
ser = Service(r"C:\Users\user\Documents\Selenium\chromedriver.exe")
driver = webdriver.Chrome(options=options,service=ser,)
driver.get(url)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
