where havei gone wrong with this selenium for loop?

Question

I've read loads of other stackoverflow answers where they have all said the solution is to use find_elements instead, but I'm still getting nowhere with my code, can anyone help? I've tried doing my code similar to how I'd use beautifulsoup, thanks in advance!

Here's my code:

import selenium.webdriver as webdriver
from selenium.webdriver.support.ui import WebDriverWait, Select
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver import firefox
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import TimeoutException, NoSuchElementException, ElementNotInteractableException
import pandas as pd

user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:87.0) Gecko/20100101 Firefox/87.0'

FireFoxDriverPath = 'geckodriver.exe'
FireFoxProfile = webdriver.FirefoxProfile()
FireFoxProfile.set_preference("general.useragent.override", user_agent)
driver = webdriver.Firefox(executable_path=FireFoxDriverPath, firefox_profile=FireFoxProfile)
url = "https://www.coingecko.com/en/coins/all"
driver.get(url) 

rank = []
name = []
symbol = []
price = []
change1h = []
change24h = []
change7d = []
change30d = []
volume24h = []
supply = []
marketcap = []

results = driver.find_element(by=By.XPATH, value='//*[@id="gecko-table-all"]').find_element(by=By.TAG_NAME, value='tbody').find_elements(by=By.TAG_NAME, value='tr')

for result in results:
    rank.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[1]')).text
    name.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[2]/div/div[2]/a/span[1]')).text
    symbol.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[3]/span')).text
    price.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[4]/a/span')).text
    change1h.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[5]/span')).text
    change24h.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[6]/span')).text
    change7d.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[7]/span')).text
    change30d.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[8]/span')).text
    volume24h.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[9]/a/span')).text
    supply.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[10]/div')).text
    marketcap.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[12]/div/span')).text

df = pd.DataFrame({
    'Rank':rank, 
    'Name':name, 
    'Symbol':symbol, 
    'Price':price,
    'Change 1H':change1h,
    'Change 24H':change24h,
    'Change 7D':change7d,
    'Change 30D':change30d,
    'Volume 24H':volume24h,
    'Circulating Supply':supply,
    'Market Cap':marketcap
    })

df.head(50)

Here's the terminal output:

File "c:\Users\simon\Desktop\Selenium project\selenium-template.py", line 38, in <module>
    rank.append(result.find_elements(by=By.XPATH, value='//*[@id="gecko-table-all"]/tbody/tr[1]/td[1]')).text
AttributeError: 'NoneType' object has no attribute 'text'

Answer 1

You have a number of lines of the form

    rank.append(result.find_elements(...)).text

What this does is:

• finds some elements according to your XPath,
• appends this list of elements to rank,
• accesses the text attribute of the result of calling rank.append.

rank is a list, and calling .append on a list returns None, hence your error.

However, your fix is not as simple as just moving the .text inside the parentheses:

    # This won't work...
    rank.append(result.find_elements(...).text)

The next problem is that find_elements(...) (with an s on the end) returns a list. You can't call .text on a list. It's not clear how you would wish to handle there being multiple matching elements. Do you want to

• add the text of each element as a separate item to your list ranks?
• add a comma-separated string of the text of all matching elements to ranks?
• add only the text of the first element? (In which case, why are you using find_elements rather than find_element?)

Given that your variables have singular names, I'm guessing the last of these is what you're looking for. In that case, the easiest thing to is to write a function that returns the text of an element, substituting an empty string if no text is found:

def get_text_by_xpath(result, xpath):
    element = result.find_element(by=By.XPATH, value=xpath)
    return "" if element is None else element.text

# ...

for result in results:
    rank.append(get_text_by_xpath(result, '//*[@id="gecko-table-all"]/tbody/tr[1]/td[1]'))
    # and similarly for names, symbols, prices and so on

However, a further look at your code identifies another problem. For each iteration of your for loop, your variable result will contain one of the rows of a table. Then your XPath starts with //, causing the search for elements to begin again at the root of the page. Your XPath then selects the first row of the table, so if your code ran to completion without hitting any errors you would end up with the contents of the first row repeated once for each row in the table.

If your result object contains a <tr> element corresponding to a row from the table, you can get a table cell from that row with the XPath ./td[1]:

    rank.append(get_text_by_xpath(result, './td[1]'))