Pandas with lru_cache how to convert 'Series' objects to unmutable

Question

I know there are some similar question, but they all have not been solved.

I'm using python lru_cache in pandas project:

from functools import lru_cache
for df_ia in pd.read_csv(file, chunksize=n,iterator=True, low_memory=False):

    @lru_cache(maxsize = None)
    def myfunc(df_ia):
        print('my logic here')

error:

TypeError: 'Series' objects are mutable, thus they cannot be hashed

I checked some answers in stack overfollow ,I know the error comes about because I passed a tuple of Series rather than a tuple of column names/strings, but no answer is about how to solve this issue when read from a csv.

Any friend can help?

Answer 1

Just as an illustration of my comment. This

@lru_cache(maxsize=None)
def func(ser):
    return ser.sum()

ser = pd.Series(range(10))
func(ser)

doesn't work, as you have outlined:

TypeError: unhashable type: 'Series'

If you do wrap the series into a hashable class, for example like

class Wrapper:
    def __init__(self, ser):
        self.ser = ser
    def __eq__(self, other):
        return self.ser.equals(other.ser)
    def __hash__(self):
        return hash((tuple(self.ser.index), tuple(self.ser.values)))

and modify the function accordingly

@lru_cache(maxsize=None)
def func(wrapped_ser):
    ser = wrapped_ser.ser
    return ser.sum()

and then do

func(Wrapper(pd.Series(range(10))))
print(func.cache_info())
func(Wrapper(pd.Series(range(10))))
print(func.cache_info())

you'll get

CacheInfo(hits=0, misses=1, maxsize=None, currsize=1)
CacheInfo(hits=1, misses=1, maxsize=None, currsize=1)

But I'm not sure if that's suitable for your use case. Might well be that you must be more careful with the wrapping.

EDIT: Some additional words of caution: Mutable objects are by default not hashable, since the hash of an object shouldn't change over its lifetime. Series are obviously mutable. So if your function does change the underlying series, this is not a good idea. As long as the function just calculates some result based on the data in the series this might work.