'calling joblib Parallel in a class method creates N instances of everything, instead of just function object
I tried to find answers to this elsewhere but couldn't locate it.
import numpy as np
class Item:
def __init__(self,name):
self.name = name
# item attributes
def plot(self):
print("plotting")
# matplotlib plot
class Video:
def __init__(self):
# load preprocessing
# load pytorch model
print("loaded pytorch model")
self.items = []
def process(self):
# find items
names_found = np.arange(7)
for name in names_found:
# do some processing
self.items.append(Item(name))
Now say I want to write out plots in parallel
from joblib import Parallel, delayed
v = test.Video()
v.process()
Parallel(n_jobs=7)(delayed(item.plot)() for item in v.items)
but it outputs
# output
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"
"loaded pytorch model"
"plotting"
"plotting"
"plotting"
"plotting"
"plotting"
"plotting"
"plotting"
as though it's loading everything in memory for each iteration. How do I parallelise just the plotting component here so it doesn't blow up memory etc. with N instances of the model.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
