Message: element not interactable---Selenium Python

Question

I am a newbie to Selenium and wanted help with a roadblock that I can't seem to overcome. So, I wish to scrape average salary information from the following website:

https://www.glassdoor.com/Salary/Google-Software-Engineer-Salaries-E9079_D_KO7,24.htm

Since there are multiple experience levels, I need to iterate over all levels and collect their corresponding salary information. I need selenium to go over (i.e., click-through) all options and then execute a standard piece of code (that would collect the required information).

1. I first tried the Select method and used the following code:

select_code = Select(driver.find_element(by=By.XPATH, value=select_xpath))
print(len(select_code.options)) #to see the number of options
select_code.options[0].click()

But encounter the following error:

Message: element not interactable: Element is not currently visible and may not be manipulated

2. I then used the following solution (found on StackOverflow) but this also didn't work for all options (of experiences):

element = driver.find_element(by=By.XPATH, value=select_xpath)
driver.execute_script("arguments[1].click();", element)

It raises the following error for argument[1] (which I suppose is the way to move onto the second option):

Message: javascript error: Cannot read properties of undefined (reading 'click')

The full XPath to the experience dropdown is:

select_xpath = '/html/body/div[3]/div/div/div/div/div/div/div/div[1]/article[2]/div[9]/main/div/div/div[1]/div[3]/div/div[2]/div/select'

Can someone please help me with this (how can I iterate over all options)? Also, I wish to learn Selenium properly (not in bits and pieces from various places). Can someone also refer to some useful self-contained resource?

Thanks!

Answer 1

This is wrong:

element = driver.find_element(by=By.XPATH, value=select_xpath)
driver.execute_script("arguments[1].click();", element)

because element would be the first argument (arguments[0])

But really it's better to find a css selector that suits:

driver.execute_script("document.querySelector('article main select').click()")

That will avoid the round trip and prevent stale reference issues. Also note that nothing typically happens when you click on a select.