'Can't scape extended data when the page have "Load More" button
there's a page that I want to scrape data, but it has load more button to expand the data. With Selenium and Webdriver, I made a command as below to load all the data to scrape before it's to be done. But below command only scrape data before "Load More" which is 1st page of it. Anyone has idea why is doing like this?
from selenium import webdriver
import time
url = "https://www.hp.com/us-en/shop/plp/accessories/computer-monitors"
headers = {"User-Agent":"my computer"}
browser = webdriver.Chrome("C:\Python310\chromedriver.exe")
browser.get("https://www.hp.com/us-en/shop/plp/accessories/computer-monitors")
loadmore = browser.find_element_by_css_selector('#content > div.clearfix.vwa > div.product-results.product-results.left-menu-open > div.search-results > span')
count = 0
while count < 5:
browser.execute_script("window.scrollTo(0, document.body.scrollHeight)")
time.sleep(2)
loadmore.click()
time.sleep(2)
count+=1
import requests
import re
from bs4 import BeautifulSoup
res = requests.get(url, headers=headers)
res.raise_for_status()
soup = BeautifulSoup(res.text, "lxml")
prods = soup.find_all("a", attrs={"class":"product-title pdp-link"})
for prod in prods :
print(prod.get_text())
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
