'Pickling an lru_cache used not as a decorator

I've stumbled upon something strange. Basically, I use a library that takes the urllib.parse.urljoin function and wraps it in functools.lru_cache. But it applies the lru_cache "decorator" on it in-line - dynamically, like this:

from urllib.parse import urljoin
lru_cache()(urljoin)

It works fine, until I want to go for multiprocessing (concurrent.futures), which uses pickling. Then I get a pickling error.

This is a reproducible example I made myself:

import pickle
from functools import lru_cache

# Working pickle

@lru_cache()
def foo(n):
    return n**2

a = pickle.dumps(foo)
b = pickle.loads(a)
print(b(3))

# Not working pickle

def bar(n):
    return n**2

x = lru_cache()(bar)
a = pickle.dumps(x)
b = pickle.loads(a)
print(b(3))

Output:

C:\Users\dabljues\Desktop> python .\cache_example.py
9
Traceback (most recent call last):
  File "C:\Users\dabljues\Desktop\cache_example.py", line 23, in <module>
    a = pickle.dumps(x)
_pickle.PicklingError: Can't pickle <functools._lru_cache_wrapper object at 0x00000242624C2CF0>: it's not the same object as __main__.bar

So, as you can see, pickling works when the lru_cache is applied in a decorator style, it doesn't when it's applied inline on a function.

My question is: is there a way around it? I mean foo and x are both functools._lru_cache_wrappers, so why doesn't it work?

@Edit

Those are the __module__s and __qualname__s of bar with and without lru_cache:

import pickle
from functools import lru_cache

def bar(n):
    return n**2

x = bar
y = lru_cache()(bar)
print(x.__module__, x.__qualname__)
print(y.__module__, y.__qualname__)
__main__ bar
__main__ bar


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source