'May I be rate limited by using Selenium for webscraping on websites?
I know there are rate limits for when you send get requests to a website, but does it apply to selenium? Can I get limited for that?
For example, in the above code:
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(r"C:\Users\path\chromedriver.exe")
driver.get("https://google.com/")
while True:
driver.find_element(by=By.CLASS_NAME, value="link")
Does the find_element() function count as a GET request?
Solution 1:[1]
Selenium driven WebDriver initiated Browsing Context if gets detected as a bot can be Rate Limited too.
Generally, Cloudflare rate-limits the visitor when the visitor traffic crosses the rate-limit thresholds which is calculated by, dividing 24 hours of uncached website requests by the unique visitors for the same 24 hours. Then, divide by the estimated average minutes of a visit. Finally, multiply by 4 (or larger) to establish an estimated threshold per minute for your website. A value higher than 4 is fine since most attacks are an order of magnitude above typical traffic rates.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | undetected Selenium |
