'How would I split tens of the same task into threads/processes

I have this project focused around pathfinding. Those of you that are about to close it after reading the word 'pathfinding', do not worry, it really isn't about that.

Anyway what I wanna do is to split my map into chunks or clusters if you prefer. Now as a part of my map preprocessing, I need to find possible entrances into neighboring clusters and execute A* between all those entrances in one chunk and then cache that path and path length. You should see where this is headed by now. If I had 100 of clusters and in each cluster I would need to execute 6 A* algorithms, that would be lengthy. So what I had in mind is to divide this problems into threading to speed this process up. BUT what I also need is to cache all those entrances, all those paths and all those path lengths in my class.

Just a simple code from top of my head, it doesn't look 100% like that but the point is the same and any solution you would provide would be completely easy to use on my actual code:

class Pathfinding:
    clusters = list(Cluster)
    

#this is how one cluster would look (not to scale)
class Cluster:
    grid = [[0,1,1,0],
            [0,0,0,0],
            [1,1,0,1],
            [0,1,1,0]]
    entrances = list([from, to, path, length])


def find_paths_and_connect(cluster)
    for entrance in cluster.entrances:
        for neig_entrance in entrances:
            if neigh_entrance is not entrance
                path = astar(entrance, neigh_entrance, cluster.grid)
                pathfinding.cache_path(path, length)

def preprocess():
    clusters = grid.create_clusters()
    for cluster in clusters:
      find_paths_and_connect(cluster)

And I would like to have my preprocess function looking like this

def preprocess():
    clusters = grid.create_clusters()
    for cluster in clusters:
        thread = threading.thread(target=find_paths_and_connect, args=(cluster,))
        thread.start()
        some_thread_list.append(thread)
    for thread in some_thread_list:
        thread.join()

But this is actually slower than executing it withing main thread. I would like something like this, maybe processes but having it manipulate with the memory, so whatever happens, the class withing main thread gets updated with every cluster that was preprocessed in a thread or process, so basically something like passing it there by reference so the changes are actually visible and usable after it is all done.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source