'Problem with parsing on selenium, scrolling
there is a problem, I will explain the script of the code.
First, we go to the main page, then the for loop that goes to page 1, scrolls and parses the links to the photo (it doesn’t parse without scrolling), then goes to page 2, and there is already a problem. It scrolls to the bottom in a second. And I need it to scroll gradually, like on the first page. I throw the code:
driver.get(url=url)
n = 0
urla = driver.find_element(By.CLASS_NAME, "ipsPagination_pageJump").text
for page_number in range(int(urla.split()[3])):
page_number = page_number + 1
driver.get(url=url + f"page/{page_number}")
time.sleep(2)
imgs = driver.find_elements(By.CLASS_NAME, "cGalleryPatchwork_image")
for i in imgs:
n = n + 500
driver.execute_script(f"window.scrollTo(0, {n})")
time.sleep(0.2)
print(i.get_attribute("src"))
driver.quit()
I know the code is very bad and not optimized
Solution 1:[1]
to scroll gradually one element by one element you should use the following execute_script command driver.execute_script("arguments[0].scrollIntoView(true);", i)
Code:
for page_number in range(int(urla.split()[3])):
page_number = page_number + 1
driver.get(url=url + f"page/{page_number}")
time.sleep(2)
imgs = driver.find_elements(By.CLASS_NAME, "cGalleryPatchwork_image")
for i in imgs:
#n = n + 500
#driver.execute_script(f"window.scrollTo(0, {n})")
driver.execute_script("arguments[0].scrollIntoView(true);", i)
time.sleep(0.2)
print(i.get_attribute("src"))
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | cruisepandey |
