Memory issues with a list of lists [closed]

Question

I am having some memory issues and I am wondering if there is any way I can free up some memory in the code below. I have tried using a generator expression rather than list comprehension but that does not produce unique combinations, as the memory is freed up.

The list of lists (combinations) causes me to run out of memory and the program does not finish.

The end result would be 729 lists in this list, with each list containing 6 WindowsPath elements that point to images. I have tried storing the lists as strings in a text file but I can not get that to work, I tried using a pandas dataframe but I can not get that to work.

I need to figure out a different solution. The output right now is exactly what I need but the memory is the only issue.

from pathlib import Path
from random import choice
from itertools import product
from PIL import Image
import sys

def combine(arr):
    return list(product(*arr))

def generate(x):

    #set new value for name
    name = int(x)

    #Turn name into string for file name
    img_name = str(name)

    #Pick 1 random from each directory, add to list.
    a_paths = [choice(k) for k in layers]

    #if the length of the list of unique combinations is equal to the number of total combinations, this function stops
    if len(combinations) == len(combine(layers)):
        print("Done")
        sys.exit()

    else:
        #If combination exists, generate new list
        if any(j == a_paths for j in combinations) == True:
            print("Redo")
            generate(name)

        #Else, initialize new image, paste layers + save image, add combination to list, and generate new list
        else:
            #initialize image
            img = Image.new("RGBA", (648, 648))
            png_info = img.info

            #For each path in the list, paste on top of previous, sets image to be saved
            for path in a_paths:
                layer = Image.open(str(path), "r")
                img.paste(layer, (0, 0), layer)

            print(str(name) + ' - Unique')
            img.save(img_name + '.png', **png_info)
            combinations.append(a_paths)
            name = name - 1
            generate(name)

'''
Main method
'''
global layers
layers = [list(Path(directory).glob("*.png")) for directory in ("dir1/", "dir2/", "dir3/", "dir4/", "dir5/", "dir6/")]

#name will dictate the name of the file output(.png image) it is equal to the number of combinations of the image layers
global name
name = len(combine(layers))

#combinations is the list of lists that will store all unique combinations of images
global combinations
combinations = []

#calling recursive function
generate(name)

Answer 1

Let's start with a MRE version of your code (i.e. something that I can run without needing a bunch of PNGs -- all we're concerned with here is how to go through the images without hitting recursion limits):

from random import choice
from itertools import product


def combine(arr):
    return list(product(*arr))


def generate(x):

    # set new value for name
    name = int(x)

    # Turn name into string for file name
    img_name = str(name)

    # Pick 1 random from each directory, add to list.
    a_paths = [choice(k) for k in layers]

    # if the length of the list of unique combinations is equal to the number of total combinations, this function stops
    if len(combinations) == len(combine(layers)):
        print("Done")
        return

    else:
        # If combination exists, generate new list
        if any(j == a_paths for j in combinations) == True:
            print("Redo")
            generate(name)

        # Else, initialize new image, paste layers + save image, add combination to list, and generate new list
        else:
            # initialize image
            img = []

            # For each path in the list, paste on top of previous, sets image to be saved
            for path in a_paths:
                img.append(path)

            print(str(name) + ' - Unique')
            print(img_name + '.png', img)
            combinations.append(a_paths)
            name = name - 1
            generate(name)


'''
Main method
'''
global layers
layers = [
    [f"{d}{f}.png" for f in ("foo", "bar", "baz", "ola", "qux")]
    for d in ("dir1/", "dir2/", "dir3/", "dir4/", "dir5/", "dir6/")
]


# name will dictate the name of the file output(.png image) it is equal to the number of combinations of the image layers
global name
name = len(combine(layers))

# combinations is the list of lists that will store all unique combinations of images
global combinations
combinations = []

# calling recursive function
generate(name)

When I run this I get some output that starts with:

15625 - Unique
15625.png ['dir1/qux.png', 'dir2/bar.png', 'dir3/bar.png', 'dir4/foo.png', 'dir5/baz.png', 'dir6/foo.png']
15624 - Unique
15624.png ['dir1/baz.png', 'dir2/qux.png', 'dir3/foo.png', 'dir4/foo.png', 'dir5/foo.png', 'dir6/foo.png']
15623 - Unique
15623.png ['dir1/ola.png', 'dir2/qux.png', 'dir3/bar.png', 'dir4/ola.png', 'dir5/ola.png', 'dir6/bar.png']
...

and ends with a RecursionError. I assume this is what you mean when you say you "ran out of memory" -- in reality it doesn't seem like I'm anywhere close to running out of memory (maybe this would behave differently if I had actual images?), but Python's stack depth is finite and this function seems to be recursing into itself arbitrarily deep for no particularly good reason.

Since you're trying to eventually generate all the possible combinations, you already have a perfectly good solution, which you're even already using -- itertools.product. All you have to do is iterate through the combinations that it gives you. You don't need recursion and you don't need global variables.

from itertools import product
from typing import List


def generate(layers: List[List[str]]) -> None:
    for name, a_paths in enumerate(product(*layers), 1):
        # initialize image
        img = []

        # For each path in the list, paste on top of previous,
        # sets image to be saved
        for path in a_paths:
            img.append(path)

        print(f"{name} - Unique")
        print(f"{name}.png", img)

    print("Done")


'''
Main method
'''
layers = [
    [f"{d}{f}.png" for f in ("foo", "bar", "baz", "ola", "qux")]
    for d in ("dir1/", "dir2/", "dir3/", "dir4/", "dir5/", "dir6/")
]

# calling iterative function
generate(layers)

Now we get all of the combinations -- the naming starts at 1 and goes all the way to 15625:

1 - Unique
1.png ['dir1/foo.png', 'dir2/foo.png', 'dir3/foo.png', 'dir4/foo.png', 'dir5/foo.png', 'dir6/foo.png']
2 - Unique
2.png ['dir1/foo.png', 'dir2/foo.png', 'dir3/foo.png', 'dir4/foo.png', 'dir5/foo.png', 'dir6/bar.png']
3 - Unique
3.png ['dir1/foo.png', 'dir2/foo.png', 'dir3/foo.png', 'dir4/foo.png', 'dir5/foo.png', 'dir6/baz.png']
...
15623 - Unique
15623.png ['dir1/qux.png', 'dir2/qux.png', 'dir3/qux.png', 'dir4/qux.png', 'dir5/qux.png', 'dir6/baz.png']
15624 - Unique
15624.png ['dir1/qux.png', 'dir2/qux.png', 'dir3/qux.png', 'dir4/qux.png', 'dir5/qux.png', 'dir6/ola.png']
15625 - Unique
15625.png ['dir1/qux.png', 'dir2/qux.png', 'dir3/qux.png', 'dir4/qux.png', 'dir5/qux.png', 'dir6/qux.png']
Done

Replacing the actual image-generating code back into my mocked-out version is left as an exercise for the reader.

If you wanted to randomize the order of the combinations, it'd be pretty reasonable to do:

from random import shuffle

...

    combinations = list(product(*layers))
    shuffle(combinations)
    for name, a_paths in enumerate(combinations, 1):
        ...

This uses more memory (since now you're building a list of the product instead of iterating through a generator), but the number of images you're working with isn't actually that large, so this is fine as long as you aren't adding a level of recursion for each image.