Trouble scrapping sports table - python

Question

I'm having a lot of trouble scrapping results off a sporting table in Python. I am new to scrapping and have tried everything I can find online. The website is https://www.nrl.com/stats/teams/?competition=111&season=2022&stat=38 and im just trying to get the team name and tries.

Any suggestions would be appreciated.

Answer 1

1. Make sure you have the good requirements: if you are using Anaconda go to Anaconda cmd line and type:

> pip install beautifulsoup4
> pip install requests

2. Now, You can try a scrapping library called beautifulsoup, you can specify the name of the div you want in the html source code of your website, with your link and the library catch all the data, example:

import requests
from bs4 import BeautifulSoup

#create variable page
page = requests.get('https://www.imdb.com/title/tt7286456/criticreviews?ref_=tt_ov_rt')
#create the variable soup and calling BeatifulSoup library for our web page
soup = BeautifulSoup(page.text, 'html.parser')
file = open("data.txt", "w") #don't forget create a data.txt file in the same repository of your file.py
#maping the div with class_="summary" with the function find_all()
for x in soup.find_all('div', class_='summary'):
    print(x) #print the data scrapped 
    file.write(x.text) #store the data in the file.txt
    file.write("\n")  
file.close()

More details in my Scrapping project https://github.com/mehdimaaref7/Scrapping-Sentiment-Analysis/blob/master/big_data.py

The data.txt file is optional, you can just use the variable x with print(x), to display the data you need to scrap.