How to scrape weight of player which is a hidden content using Selenium

Question

I have been trying to get the weight of this player, but it is only visible on the inspect element and not on the website, when I print the below, I get a blank result, Could anyone please help me with this?

Code trials:

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service

s = Service('/Users/karim/Desktop/chromedriver-2')
driver = webdriver.Chrome(service=s)
url = 'https://www.premierleague.com/players/71432/Rayan-A%C3%AFt-Nouri/overview'
driver.get(url)

g = driver.find_element(By.XPATH, "//li[@class='u-hide']")
print(g.text)

Answer 1

You need to do the following:

1. Click on accept cookies button.
2. Click on the close button that is on the top right-hand side.

Code:

driver.maximize_window()
wait = WebDriverWait(driver, 30)

driver.get("https://www.premierleague.com/players/71432/Rayan-A%C3%AFt-Nouri/overview")

try:
    wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Accept All Cookies']"))).click()
    print("clicked on accept cookies button")
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#advertClose"))).click()
except:
    pass

#print(wait.until(EC.presence_of_element_located((By.XPATH, "//li[@class='u-hide']"))).get_attribute('innerText'))
print(wait.until(EC.presence_of_element_located((By.XPATH, "//li[@class='u-hide']//div[@class='info']"))).get_attribute('innerText'))

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output:

clicked on accept cookies button
70kg

Process finished with exit code 0

Answer 2

The weight of the player is within:

<div class="info">70kg</div>

which is within it's ancestor:

<li class="u-hide">
          <div class="label">Weight</div>
          <div class="info">70kg</div>
    </li>
    

where the <li> is having class as u-hide

As the element is a hidden element using text attribute won't work here and you have extract either the innerHTML or the innerText.

---

Solution

To print the value of the Weight you can use either of the following locator strategies:

• Using innerHTML:

  print(driver.find_element(By.XPATH, "//li[@class='u-hide']//div[@class='info']").get_attribute('innerHTML'))
  

• Using innerText:

  print(driver.find_element(By.XPATH, "//li[@class='u-hide']//div[@class='info']").get_attribute('innerText'))
  

Inducing WebDriverWait:

• Using innerHTML:

  print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//li[@class='u-hide']//div[@class='info']"))).get_attribute('innerHTML'))
  

• Using innerText:

  print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//li[@class='u-hide']//div[@class='info']"))).get_attribute('innerText'))
  

• Console Output:

  70kg
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC