How can I scrape the position of the players in Selenium?

Question

If you visit this site, https://www.premierleague.com/players, you will be able to see a list of players with their positions and Nationality. I am only required to scrape the position but I am unable to

playerss = driver.find_elements(By.XPATH, '//*[@id="mainContent"]/div[2]/div[1]/div/div/table/tbody/tr')

for player in playerss:
    position = player.find_element(By.XPATH,'//[@id="mainContent"]/div[2]/div[1]/div/div/table/tbody/tr/td[2]')
    print(position)

This is what I have but it is not working. If anyone knows how to help, I will appreciate it.

Thanks!

Answer 1

This is how I was able to get the list of positions:

...
driver.get('https://www.premierleague.com/players')
positions = driver.find_elements(By.CSS_SELECTOR, 
                                 'table tbody tr td:nth-child(2)')

for pos in positions:
    print(pos.text)

if you have to use xpath:

positions = driver.find_elements(By.XPATH, '//table/tbody/tr/td[2]')

Answer 2

The position of the players are within:

<td class="hide-s">Midfielder</td>

where the <td> is having class as hide-s

---

Solution

To print the positions you need to induce WebDriverWait for the presence_of_element_located() and you can use either of the following locator strategies:

• Using text attribute:

  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr td:nth-of-type(2)")))])
  

• Using innerHTML:

  print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr td:nth-of-type(2)")))])
  

• Using innerText:

  print([my_elem.get_attribute("innerText") for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr td:nth-of-type(2)")))])
  

• Console Output:

  ['Midfielder', 'Defender', 'Defender', 'Forward', 'Forward', 'Forward', 'Defender', 'Goalkeeper', 'Forward', 'Defender', 'Defender', 'Defender', 'Defender', 'Defender', 'Midfielder', 'Defender', 'Defender', 'Midfielder', 'Defender', 'Defender', 'Defender', 'Midfielder', 'Goalkeeper', 'Midfielder', 'Midfielder', 'Midfielder', 'Defender', 'Midfielder', 'Forward', 'Forward']