'Scrape data from webpage with BeautifulSoup - How to append data to existing dataframe?

With the following code I try to scrape data from a website (reference: https://towardsdatascience.com/web-scraping-scraping-table-data-1665b6b2271c):

df = pd.DataFrame(columns=headings)
for i in range (102,158):
    URL = 'http://BuLidata.xyz/'
    URL_ = URL + 'B100' +str(i+1) + '.html'
    r = urllib.request.urlopen(URL_).read()
    soup = BeautifulSoup(r,'lxml')
    table = soup.find('table' ,attrs={'class':'abschluss'})
    body = table.find_all("tr")
    head = body[0]
    body_rows = body[1:]
    headings = []
    for item in head.find_all('th'):
        item = (item.text).rstrip('\n')
        headings.append(item)
    all_rows = [] 
    for row_num in range(len(body_rows)):
        row = []
        for row_item in body_rows[row_num].find_all("td"):
            aa = re.sub("(\xa0)|(\n)|,","",row_item.text)
            row.append(aa)
        all_rows.append(row)
    df1 = pd.DataFrame(data=all_rows,columns=headings)
    df.append(df1, ignore_index=True)

I 'intialized' the dataframe as an empty dataframe only with the correct column names and then tried to use a loop in order to loop over the data on the website. Partially it seems to work because df1 is the data of the last website link. But df is still the initialized empty dataframe. I am wondering what I did wrong here?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source