How to extract entire table using BeautifulSoup from webpage

Question

I am trying to extract the main table from this site, but i am able to extract only first name, not the others.

header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}


        
data= requests.get("https://etfdb.com/etfs/inverse/#etfs&sort_name=assets_under_management&sort_order=desc&page=1", headers=header)
soup= BeautifulSoup (data.content, 'lxml')
table = soup.find_all('tbody',)

for i in table:
    name = i.text
    print(name)

I am only getting SQQQ not the one's below.

Answer 1

Once you have found the start of your table, you could then find all the table rows by searching for <tr> elements. Then for each row extract all of the <td> elements.

For example:

from bs4 import BeautifulSoup
import requests

header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
data = requests.get("https://etfdb.com/etfs/inverse/#etfs&sort_name=assets_under_management&sort_order=desc&page=1", headers=header)
soup = BeautifulSoup(data.content, 'lxml')
first_table = soup.find('tbody')

for tr in first_table.find_all('tr'):
    row = [td.text for td in tr.find_all('td')]
    print(row)

This would display output starting:

['SQQQ', 'ProShares UltraPro Short QQQ', 'Equity', '--3x', '$2,768.14', '61.65%', '57,468,328', '$48.01', '7.65%', '24.77%', '14.88%', '-29.55%', '-95.71%', '-98.71%', '', '', '', '', '', '', 'Leveraged Equities', '2010-02-09', '0.95%', 'N/A', '$0.00', '2020-03-25', '$0.06', '0.00%', '0.00', '-2.36', '21', '87.81%', 'View', '40%', '20%', '1099', '$34.58', '$46.07', '$44.62', '$49.79', '64.40', 'View', 'View', 'View', 'View', 'View', 'View', '', 'A', 'C+', '', '', '', '', '5.81', '30.51%', '30.56%', '0.0', '0.00%', '0.00%']
['SH', 'ProShares Short S&P500', 'Equity', '-False', '$1,927.14', '11.75%', '19,136,312', '$15.22', '1.74%', '5.77%', '3.33%', '-11.92%', '-45.03%', '-54.71%', '', '', '', '', '', '', 'Inverse Equities', '2006-06-19', '0.90%', 'N/A', '$0.00', '2020-03-25', '$0.03', '0.00%', '0.00', '-0.90', '19', '88.02%', 'View', '40%', '20%', '1099', '$13.99', '$15.06', '$14.94', '$15.37', '64.28', 'View', 'View', 'View', 'View', 'View', 'View', '', 'A', 'B+', '', '', '', '', '5.81', '30.51%', '30.56%', '0.0', '0.00%', '0.00%']
['PSQ', 'ProShares Short QQQ', 'Equity', '-False', '$1,015.79', '18.93%', '20,123,440', '$12.88', '2.47%', '7.87%', '5.57%', '-7.80%', '-57.29%', '-69.16%', '', '', '', '', '', '', 'Inverse Equities', '2006-06-19', '0.95%', 'N/A', '$0.00', '2020-03-25', '$0.04', '0.00%', '0.00', '-0.92', '19', '86.79%', 'View', '40%', '20%', '1099', '$11.52', '$12.68', '$12.56', '$13.05', '64.53', 'View', 'View', 'View', 'View', 'View', 'View', '', 'A-', 'B', '', '', '', '', '5.81', '30.51%', '30.56%', '0.0', '0.00%', '0.00%']