'How to scrape an expandable table from investing.com using python beautiful soup or investpy?

https://www.investing.com/economic-calendar/initial-jobless-claims-294

As stated in the question, I tried to web scrape a data table from this link. However, I was only able to scrape the first few lines of the data until the "show more" button. Except for web scraping, I ve also tried investpy.economic_calendar(), yet the filtering parameters are so random so that I could not extract the jobless claim data directly. Could somebody please help me with this?

url = 'https://www.investing.com/economic-calendar/initial-jobless-claims-294'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
table1 = soup.find('table', id='eventHistoryTable294')
headers = []
for i in table1.find_all('th'):
  title = i.text
  headers.append(title)

mydata = pd.DataFrame(columns = headers)
table_rows = table1.find_all('tr')

#$df_side = pd.DataFrame(mydata)
#x = df_side.head(100)

for j in table1.find_all('tr')[1:]:
  row_data = j.find_all('td')
  row = [i.text for i in row_data]
  length = len(mydata)
  mydata.loc[length] = row

print(mydata)



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source