'Selenium Python only taking information from 32 out of 100 elements
I'm scraping data from a dynamic page which the element id's change as you scroll, I have it set to scroll down until it loads the last element and then the body of the page zooms out so all 100 elements are visible.
I then have 100 of these lines to collect the required data:
store1 = driver.find_element(By.XPATH,
"/html/body/div[1]/main/div/div/ia-dataview/krn-data-view-platform/div/div/div[3]/div/div[3]/div/div[2]/div[2]/div/div[2]/div/div[1]/div/div[1]/div/div").text
department1 = driver.find_element(By.XPATH,
"/html/body/div[1]/main/div/div/ia-dataview/krn-data-view-platform/div/div/div[3]/div/div[3]/div/div[2]/div[2]/div/div[2]/div/div[1]/div/div[2]/div/div").text
job1 = driver.find_element(By.XPATH,
"/html/body/div[1]/main/div/div/ia-dataview/krn-data-view-platform/div/div/div[3]/div/div[3]/div/div[2]/div[2]/div/div[2]/div/div[1]/div/div[3]/div/div").text
cost1 = driver.find_element(By.XPATH,
"/html/body/div[1]/main/div/div/ia-dataview/krn-data-view-platform/div/div/div[3]/div/div[3]/div/div[2]/div[3]/div[2]/div/div[1]/div/div[6]/div/div").text
I have the numbers changed that need to be up to 100 and it only collects the first 32 then stops. If I leave the chrome webdriver open and search the elements all 100 are visible so I'm not sure why it isn't collecting them all.
If anyone has any ideas that would be amazing!
A little edit, I've tried using some explicit waits and it seems when I use presence_of_element_located with the last element I need it goes through but only gets the text out of the first 32, if I use visibility_of_element_located it times out.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
