'calling joblib Parallel in a class method creates N instances of everything, instead of just function object

I tried to find answers to this elsewhere but couldn't locate it.


import numpy as np

class Item:
     def __init__(self,name):
        self.name = name
        # item attributes

     def plot(self):
        print("plotting")
        # matplotlib plot

class Video:
    def __init__(self):
        # load preprocessing
        # load pytorch model
        print("loaded pytorch model")
        self.items = []

    def process(self):
        # find items
        names_found = np.arange(7)
        for name in names_found:
            # do some processing
            self.items.append(Item(name))

Now say I want to write out plots in parallel

from joblib import Parallel, delayed

v = test.Video()
v.process()

Parallel(n_jobs=7)(delayed(item.plot)() for item in v.items)

but it outputs

# output
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"

"plotting"
"plotting"
"plotting"
"plotting"
"plotting"
"plotting"
"plotting"

as though it's loading everything in memory for each iteration. How do I parallelise just the plotting component here so it doesn't blow up memory etc. with N instances of the model.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source