Python: Deleting duplicates from Dataframe

Question

When I output dfges, I get duplicates for a given element. For example, for Madrid the messages are all displayed three times each. Does anyone know a way to work around this problem or remove the duplicates? Is it possibly due to the upper For loop?

import ssl 
import smtplib 
import pandas as pd 
from GoogleNews import GoogleNews 
ssl._create_default_https_context = ssl._create_unverified_context
    
googlenews = GoogleNews() 
googlenews.set_encode('utf_8')     

Nachrichten = [
        df = pd.DataFrame(table_new) 
    nachrichten.append(df)

dfges = pd.concat(nachrichten, axis='index')
´´´

Answer 1

Seems you need a dfges.drop_duplicates(inplace=True). This will remove all duplicate rows. Note that two rows are considered duplicates only if they have the same values in each field (except the index).

See the doc for other details: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop_duplicates.html

Provide an example of your Dataframe dfges if you need more help.

Answer 2

As articles can be tagged with multiple locations e. g. "Munich" and "Madrid", hence appear multiple times in your df, you should detect and delete duplicates using drop_duplicates(subset=...) based on the 'Title' (and more criteria), as otherwise no duplicates are deleted add at all.

Your code could look like this:

...
dfges = pd.concat(nachrichten, axis='index')
dfges.drop_duplicates(subset=['Title'], keep='last', inplace=True)
print(dfges)
...