'Web scraper in selenium is too slow and don't how to use other methods
Hi so this is the code i wrote it gets domain names and their prices it works perfect but is too slow and i want to use scrapy or beautiful soup but don't know how to do it
from selenium.webdriver.common.by import By
urls = ["https://www.brandbucket.com/styles/6-letter-domain-names?page=1","https://www.brandbucket.com/styles/6-letter-domain-names?page=2"]
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'}
driver = selenium.webdriver.Chrome()
for url in urls:
driver.get(url)
names = driver.find_elements(By.XPATH, '//div[@class="domainCardDetail"]//span')
for value in names:
print(value.text)
Solution 1:[1]
Have you tried
from selenium.webdriver.support.ui import WebDriverWait
element = WebDriverWait(driver, 10).until(...)
to wait only until a specific element is loaded? See here
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | KingOtto |
