'Stale element web scraping table

I have been struggling on how do i get the text (the name and change) from td (the rows) on https://finance.yahoo.com/gainers to do that i have used selenium and iterate through the table and finding the tag tr then after that to get the text checked the tr tag for any td then printed the td text

structure of the site

<html>
    <body>
        <table>
            <thead>
                <tr>
                    <th>symbol</th>
                    <th>name</th>
                    <th>price</th>
                    <th>change</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>FMCCJ</td>
                    <td>Federal Home Loan Mortgage Corporation</td>
                    <td>4.7600</td>
                    <td>+28.40</td>
                </tr>
            </tbody>
        </table>
    </body>
</html>

it gives me the error stale element reference: element is not attached to the page document in the line print(''.join(rowtext)) and rowtext=(x.text for x in row.find_elements_by_xpath("//td"))

here is the whole code

table=driver.find_element_by_xpath("""//*[@id="scr-res-table"]/div[1]/table/tbody""")

for row in table.find_elements_by_xpath("//tr"):
    rowtext=(x.text for x in row.find_elements_by_xpath("//td")) 
    print(''.join(rowtext))

Solution 1:^[1]

The locators need to improve as well as we will have to give enough time to page to load. Please have a look at the below code:

from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait

PATH = r"/Users/your chromedriverpath/chromedriver"
driver = webdriver.Chrome(PATH)
driver.get("https://finance.yahoo.com/gainers")
driver.maximize_window()
WebDriverWait(driver, 40).until(EC.visibility_of_element_located((By.XPATH, "//*[@id='scr-res-table']/div[1]/table/tbody")))
size = len(driver.find_elements(By.XPATH, "//*[@id='scr-res-table']/div[1]/table/tbody/tr"))

for i in range(0, size):
    i = str(i + 1)
    Symbollocator = "//*[@id='scr-res-table']/div[1]/table/tbody/tr["+ i + "]//td[1]"
    nameLocator = "//*[@id='scr-res-table']/div[1]/table/tbody/tr["+ i + "]//td[2]"
    changeLocator = "//*[@id='scr-res-table']/div[1]/table/tbody/tr["+ i + "]//td[4]"
    symbols = driver.find_elements(By.XPATH, Symbollocator)
    names = driver.find_elements(By.XPATH, nameLocator)
    changes = driver.find_elements(By.XPATH, changeLocator)
    for symbol in symbols:
        print("--------------------Details from Row:  "+i+" ------------------------------")
        allValues = symbol.text.strip()
        print("Symbol: :" + allValues.replace('\n', '|'))
        for name in names:
            allNames = name.text.strip()
            print("Name:" + allNames.replace('\n', '|'))
            for change in changes:
                allChanges = change.text.strip()
                print("Change:" + allChanges.replace('\n', '|'))
    print("--------------------------------------------------")


driver.quit()

Output:

--------------------Details from Row:  1 ------------------------------
Symbol: :FMCCJ
Name:Federal Home Loan Mortgage Corporation
Change:+0.5400
--------------------------------------------------
--------------------Details from Row:  2 ------------------------------
Symbol: :ATASF
Name:Atlantia SpA
Change:+2.74
--------------------------------------------------
--------------------Details from Row:  3 ------------------------------
Symbol: :TABCF
Name:Tabcorp Holdings Limited
Change:+0.4199
--------------------------------------------------
--------------------Details from Row:  4 ------------------------------
Symbol: :EBR-B
Name:Centrais Elétricas Brasileiras S.A. - Eletrobrás
Change:+0.85
--------------------------------------------------
--------------------Details from Row:  5 ------------------------------
Symbol: :EPAM
Name:EPAM Systems, Inc.
Change:+28.40
--------------------------------------------------
--------------------Details from Row:  6 ------------------------------
Symbol: :IMPUY
Name:Impala Platinum Holdings Limited
Change:+1.40
--------------------------------------------------
--------------------Details from Row:  7 ------------------------------
Symbol: :KPLUY
Name:K+S Aktiengesellschaft
Change:+1.55
--------------------------------------------------
--------------------Details from Row:  8 ------------------------------
Symbol: :PSMT
Name:PriceSmart, Inc.
Change:+7.70
--------------------------------------------------
--------------------Details from Row:  9 ------------------------------
Symbol: :EURN
Name:Euronav NV
Change:+1.01
--------------------------------------------------
--------------------Details from Row:  10 ------------------------------
Symbol: :FLMMF
Name:Filo Mining Corp.
Change:+1.41
--------------------------------------------------
--------------------Details from Row:  11 ------------------------------
Symbol: :NEX
Name:NexTier Oilfield Solutions Inc.
Change:+0.79
--------------------------------------------------
--------------------Details from Row:  12 ------------------------------
Symbol: :ATASY
Name:Atlantia SpA
Change:+0.88
--------------------------------------------------
--------------------Details from Row:  13 ------------------------------
Symbol: :WBDWV
Name:Warner Bros. Discovery, Inc.
Change:+1.70
--------------------------------------------------
--------------------Details from Row:  14 ------------------------------
Symbol: :CZOO
Name:Cazoo Group Ltd
Change:+0.2200
--------------------------------------------------
--------------------Details from Row:  15 ------------------------------
Symbol: :RES
Name:RPC, Inc.
Change:+0.80
--------------------------------------------------
--------------------Details from Row:  16 ------------------------------
Symbol: :CTRA
Name:Coterra Energy Inc.
Change:+1.96
--------------------------------------------------
--------------------Details from Row:  17 ------------------------------
Symbol: :OXY
Name:Occidental Petroleum Corporation
Change:+4.12
--------------------------------------------------
--------------------Details from Row:  18 ------------------------------
Symbol: :WDFC
Name:WD-40 Company
Change:+12.37
--------------------------------------------------
--------------------Details from Row:  19 ------------------------------
Symbol: :FNMAT
Name:Federal National Mortgage Association
Change:+0.2100
--------------------------------------------------
--------------------Details from Row:  20 ------------------------------
Symbol: :SBSW
Name:Sibanye Stillwater Limited
Change:+1.12
--------------------------------------------------
--------------------Details from Row:  21 ------------------------------
Symbol: :FNMAL
Name:Federal National Mortgage Association
Change:+0.32
--------------------------------------------------
--------------------Details from Row:  22 ------------------------------
Symbol: :RLAY
Name:Relay Therapeutics, Inc.
Change:+1.99
--------------------------------------------------
--------------------Details from Row:  23 ------------------------------
Symbol: :KRYPF
Name:Kerry Properties Limited
Change:+0.1700
--------------------------------------------------
--------------------Details from Row:  24 ------------------------------
Symbol: :PIAIF
Name:Ping An Insurance (Group) Company of China, Ltd.
Change:+0.47
--------------------------------------------------
--------------------Details from Row:  25 ------------------------------
Symbol: :DISH
Name:DISH Network Corporation
Change:+1.95
--------------------------------------------------

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	QualityMatters

'Stale element web scraping table

Solution 1:[1]

Sources

Related Questions

Solution 1:^[1]