'Python pandas Filtering out nan from a data selection of a column of strings

Without using groupby how would I filter out data without NaN?

Let say I have a matrix where customers will fill in 'N/A','n/a' or any of its variations and others leave it blank:

import pandas as pd
import numpy as np


df = pd.DataFrame({'movie': ['thg', 'thg', 'mol', 'mol', 'lob', 'lob'],
                  'rating': [3., 4., 5., np.nan, np.nan, np.nan],
                  'name': ['John', np.nan, 'N/A', 'Graham', np.nan, np.nan]})

nbs = df['name'].str.extract('^(N/A|NA|na|n/a)')
nms=df[(df['name'] != nbs) ]

output:

>>> nms
  movie    name  rating
0   thg    John       3
1   thg     NaN       4
3   mol  Graham     NaN
4   lob     NaN     NaN
5   lob     NaN     NaN

How would I filter out NaN values so I can get results to work with like this:

  movie    name  rating
0   thg    John       3
3   mol  Graham     NaN

I am guessing I need something like ~np.isnan but the tilda does not work with strings.

Solution 1:^[1]

Simplest of all solutions:

filtered_df = df[df['name'].notnull()]

Thus, it filters out only rows that doesn't have NaN values in 'name' column.

For multiple columns:

filtered_df = df[df[['name', 'country', 'region']].notnull().all(1)]

Solution 2:^[2]

df.dropna(subset=['columnName1', 'columnName2'])

Solution 3:^[3]

df = pd.DataFrame({'movie': ['thg', 'thg', 'mol', 'mol', 'lob', 'lob'],'rating': [3., 4., 5., np.nan, np.nan, np.nan],'name': ['John','James', np.nan, np.nan, np.nan,np.nan]})

for col in df.columns:
    df = df[~pd.isnull(df[col])]

Solution 4:^[4]

You can also use query:

out = df.query("name.notna() & name !='N/A'", engine='python')

Output:

  movie  rating    name
0   thg     3.0    John
3   mol     NaN  Graham

Solution 5:^[5]

Inside query() pass column_name == column_name to keep the rows where column_name is not NA.

For your case:

nms.query('name == name')

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1
Solution 2	JacoSolari
Solution 3
Solution 4
Solution 5	rachwa

'Python pandas Filtering out nan from a data selection of a column of strings

Solution 1:[1]

Solution 2:[2]

Solution 3:[3]

Solution 4:[4]

Solution 5:[5]