'list index out of range when running Selenium
I am new in coding. I have a question. I tried to run selenium for scraping data with 13 pages in total. Unfortunately, after page 13, the loop is still running, and I don't know how to stop the loop. This is the error: list index out of range when running Selenium, please I need your help. Thank you !
This is the code that I make
txt = driver
.find_element(By.XPATH,'//*[@id="searchResultsCount"]').text
print(txt)
print(txt.split{' '))
def result_status(driver):
txt = driver.find_e1ement(By.XPATH,'//*{@id="searchResultsCount"]').text
current = txt.sp1it(' ')[1]
end = txt.sp1it(' ')[3]
return current, end, driver
def next_page(driver):
pg_elems = driver.find_elements(By.CLASS_NAME,'page-link') #<a href="#page-274"
nxt_elem = [x for x in pg_elems if x.text == 'Next‘][@]
nxt_elem.click()
time.sleep(2)
return driver
driver = next_page(driver)
results_df = pd.DataFrame()
# Put it all together (From Matt)
# Get current resuLts
current, end, driver = result_status(driver)
#Loop through resuLts
i=0
while current != end:
i += 1
if i%2 == 0:
results = driver.find_element(By.ID,'searchResultsArea')
results_html = results.get_attribute('innerHTHL')
temp = pd.read_html(results_html)[0]
results_df = pd.concat([results_df,temp], ignore_index=True)
results_df.to_csv("results.csv", index=False)
#Check Status
current, end, driver = result_status(driver)
print(current,'|',end='')
#Go to next page
driver = next_page(driver)
if i == 660:
break
results = driver.find_e1ement(By.ID,'overSearchResults')
results_html = results.get_attribute('innerHTML')
df = pd.read_htm1(results_html
And this is the error
IndexError Traceback (most recent call last)
~\AppData\Loca1\Temp/ipykernel_37444/2741504647.py in <module>
21
22 #Go to next page
---> 23 driver = next_page(driver)
24 it i == 690:
25 break
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
