pandas value_counts() with IntEnum raises RecursionError

Question

I got the following code to elaborate on my problem. I'm using python 3.6 with pandas==0.25.3.

import pandas as pd
from enum import Enum, IntEnum


class BookType(Enum):
  DRAMA = 5
  ROMAN = 3


class AuthorType(IntEnum):
  UNKNOWN = 0
  GROUP = 1
  MAN = 2


def print_num_type(df: pd.DataFrame, col_name: str, enum_type: Enum) -> int:
    counts = df[col_name].value_counts()
    val = counts[enum_type]
    print('value counts:', counts)
    print(f'Found "{val}" of type {enum_type}')


d = {'title': ['Charly Morry', 'James', 'Watson', 'Marry L.'], 'isbn': [21412412, 334764712, 12471021, 124141111], 'book_type': [BookType.DRAMA, BookType.ROMAN, BookType.ROMAN, BookType.ROMAN], 'author_type': [AuthorType.UNKNOWN, AuthorType.UNKNOWN, AuthorType.MAN, AuthorType.UNKNOWN]}
df = pd.DataFrame(data=d)
df.set_index(['title', 'isbn'], inplace=True)
df['book_type'] = df['book_type'].astype('category')
df['author_type'] = df['author_type'].astype('category')

print(df)
print(df.dtypes)

print_num_type(df, 'book_type', BookType.DRAMA)
print_num_type(df, 'author_type', AuthorType.UNKNOWN)

My pandas.DataFrame consists of two columns (book_type and author_type) of type categorical. Furthermore, book_type is a class inheriting from type Enum and author_type from IntEnum. When calling print_num_type(df, 'book_type', BookType.DRAMA) everything works out as expected and the number of books of this type are printed, whereas print_num_type(df, 'author_type', AuthorType.UNKNOWN) raises the error:

Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\abc.py", line 182, in __instancecheck__
    if subclass in cls._abc_cache:
  File "C:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\_weakrefset.py", line 72, in __contains__
    wr = ref(item)
RecursionError: maximum recursion depth exceeded while calling a Python object
Exception ignored in: 'pandas._libs.lib.c_is_list_like'
Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\abc.py", line 182, in __instancecheck__
    if subclass in cls._abc_cache:
  File "C:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\_weakrefset.py", line 72, in __contains__
    wr = ref(item)
RecursionError: maximum recursion depth exceeded while calling a Python object

What am I doing wrong here? Is there a workaround to get this error fixed? since I can't change the IntEnum type of AuthorType since it's provided from another library.

Thanks in advance!

Answer 1

See answer here

The main idea is that since x.value_counts() or counts in your function is itself a pandas Series, it's best to use .iat or .iloc when calling it, e.g, see iat docs

I think the easiest solution is to just use (x==0).sum(), or in your syntax:

val = (df[col_name]==enum_type).sum()

I put a minimal working example in the comments under your question so you can reproduce the problem/fix easily with the "x" notation.

Answer 2

What version of Pandas are you using? I realized after reproducing the error that upgrading Pandas (now on pandas-1.4.2) fixes the error, and the value_counts()[0] worked as expected.

run pip install --upgrade pandas