How to filter list value in pandas

Question

I'm new to this site and to Python and this question may be a bit basic.

I have a pandas Series that contains a list of elements in each row and I am trying to filter each list to only keep the elements in a given list.

reference_list = [item_a, item_c]

index       answers
 1       [item_a, item_b, item_c]
 2       [item_c, item_d, item_e]
 3       [item_a, item_c, item_b]

The output I am looking for would look like this

index       answers
 1       [item_a, item_c]
 2       [item_c]
 3       [item_a, item_c]

So far I have tried for loops, pd.Series.apply(lambda x:) functions and comprehension lists but I did not get the result I needed.

If anyone could give me further insights on my mistakes I would really appreciate it.

Answer 1

You could try:

df['answers'] = df['answers'].apply(lambda x: list(filter(lambda y: y in set(reference_list), x)))

Answer 2

Used the 'np.in1d' function from numpy to filter.

import numpy as np

for i in range(0, len(df['answers'])):
    df.iat[i, 0] = np.array(df.iat[i, 0])[np.in1d(df.iat[i, 0], reference_list)]

Output

            answers
1  [item_a, item_c]
2          [item_c]
3  [item_a, item_c]

Answer 3

You need to either overwrite the column with the filtered column or create a new one:

import pandas as pd

df =pd.DataFrame( {"answers":[ [1,2,3,4],[1,4],[1,3,42]] })

# keep only odd elements, put them into "filtered"
df["filtered"] = df["answers"].apply(lambda v : [a for a in v if a%2==1])

print(df)

Output:

        answers   filtered
0  [1, 2, 3, 4]     [1, 3]
1        [1, 4]        [1]
2    [1, 3, 42]     [1, 3]

Answer 4

using set is another option:

reference_set = set(reference_list)
df['new'] = df.answers.map(set).map(lambda x: x&reference_set).map(list)

>>> df
'''
                    answers               new
1  [item_a, item_b, item_c]  [item_a, item_c]
2  [item_c, item_d, item_e]          [item_c]
3  [item_a, item_c, item_b]  [item_a, item_c]