'Restrict Processor Pool Executor access to functions with API calls

I have a small application that uses the Multi Processing Pool Executor to run a function in parallel. The executor maps a list which has a approx 2,400 objects in it to a function which essentially makes some API calls and does some calculations.

The problem I'm having is that there is an API endpoint where I call both the GET and the POST method of it in one process. The endpoint checks if an item already exists on the API Database. When multiple processes are accessing the endpoint the information from a previous process isn't being returned due to multiple processes accessing the endpoint at once.

The returned data isn't updated with the changes made by a previous process because a process is running the exact same code at the same time as another process, which causes duplicate items to be made when POSTing to an endpoint and causes information not to appear GETting an endpoint. I want to keep the speed of the ProcessorPoolExecutor, but I don't want it creating duplicates when accessing specific methods

I would normally just put the list in a for loop, which would solve the entire problem. However, due to the size of the list I have to do it this way as the list was originally taking 33 hours to complete an entire iteration. Putting it in a Process Pool, has improved the time by over 80% which is very significant.

My code is below any advice or help would be great:

''' main.py '''

def item_check(item, api_object):
    ''' Checks if item already exists in API (GET)'''
    if not api_lib.itemsearch(api_object, item.name):
        ''' Creates a product in the API (POST) '''
        api_lib.create_product(api_object, item, category_id)

def start():
    with concurrent.futures.ProcessPoolExecutor() as executor:
       executor.map(item_check, items)


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source