'How to scrape an expandable table from investing.com using python beautiful soup or investpy?
https://www.investing.com/economic-calendar/initial-jobless-claims-294
As stated in the question, I tried to web scrape a data table from this link. However, I was only able to scrape the first few lines of the data until the "show more" button. Except for web scraping, I ve also tried investpy.economic_calendar(), yet the filtering parameters are so random so that I could not extract the jobless claim data directly. Could somebody please help me with this?
url = 'https://www.investing.com/economic-calendar/initial-jobless-claims-294'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
table1 = soup.find('table', id='eventHistoryTable294')
headers = []
for i in table1.find_all('th'):
title = i.text
headers.append(title)
mydata = pd.DataFrame(columns = headers)
table_rows = table1.find_all('tr')
#$df_side = pd.DataFrame(mydata)
#x = df_side.head(100)
for j in table1.find_all('tr')[1:]:
row_data = j.find_all('td')
row = [i.text for i in row_data]
length = len(mydata)
mydata.loc[length] = row
print(mydata)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
