'Problem with the code when running a loop with selenium in Python

I have to perform a web scrapping in the website https://portalbnmp.cnj.jus.br/#/pesquisa-peca.

  1. My goal is to select "Rio de Janeiro" in the field 'Estado"
  2. Send the key "" to the field "Nome"
  3. Search
  4. In the table that appears, I have to click in each row.
  5. Click "Emitir" in the next page
  6. Return to previous page and go to the process again for the next line of the table and so on.

My code bellow runs withou error when I run line by line, but in the loop I get all kinds of error. Stale, not clickable, not executable, etc. Some ideia of why this might happen?

for i in range(1, 11):
   
    element = driver.find_element_by_tag_name('p-dropdown')
    element.find_element_by_xpath("//*[contains(text(), 'Estado')]").click()
    element.find_element_by_xpath("//*[contains(text(), 'Rio de Janeiro')]").click()
        
    search = driver.find_element_by_name("nomePessoa")
    search.send_keys("")
    
    search.send_keys(Keys.RETURN)
         
    # row click 
    table = driver.find_element_by_xpath("//div[@class='ui-datatable-tablewrapper ng-star-inserted']/table/tbody")
    rows = table.find_element_by_tag_name('tr')
    
    rows.find_element_by_xpath("//tr[" + str(i) + "]/td[1]").click()
    
    # click 'Emitir'
    buttons = driver.find_element_by_tag_name("button")
    buttons.find_element_by_xpath("//*[contains(text(), 'Emitir')]").click()
    
    # return page
    driver.back()


Solution 1:[1]

When using Selenium try adding in checks to make sure the elements you are interacting with are loaded. In some cases you can add in explicit waits. (Try not to use methods like sleep() as it is strongly advised against per the documentation).

# import webdriver 
from selenium import webdriver 
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# get element  after explicitly waiting up to 10 seconds
element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, "p-dropdown"))
    )  # I would consider looking up by ID or class
element.find_element_by_xpath("//*[contains(text(), 'Estado')]").click()
... etc

This will make it so that you never click an element before it is loaded. Another thing to keep in mind with Selenium is that an element must be visible in order to interact with it. You can scroll to an element which will ensure it is visible by doing:

# example that scrolls to bottom of page
driver.execute_script("window.scrollTo(0,document.body.scrollHeight);")
# example that scrolls to a specific element
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(driver)
element = driver.find_element_by_tag_name('p-dropdown')  # just an example
actions.move_to_element(element)

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1