Python logging with an external module and multiprocessing

Question

I'm trying to use multiprocessing to run some numerical code using an external module, which makes use of python's logging module with the usual pattern:

import logging
logger = logging.getLogger(__name__)
logger.info("some message")

I would like each subprocess to have its own log file, which should contain any logging information from the external module produced under that subprocess. However, I find that the different subprocesses use the first subprocess's log file, or may at random use their own. Because the external module is fairly complicated, I have made this example which replicates the behaviour:

# test_module.py
import logging
logger = logging.getLogger(__name__)

class test_class:
    def __init__(self, x):
        logger.info(f'hello! {x}')

# test.py
def fun(x):
    import test_module
    import logging
    log_file = f'{x}.log'
    logging.basicConfig(level=logging.INFO, filename=log_file, filemode='w+')
    B = test_module.test_class(x)

if __name__ == "__main__":
    import multiprocessing as mp
    nprocs = 5
    with mp.get_context('spawn').Pool(nprocs) as pool:
        pool.map(fun, [x for x in range(10)])

This produces the following for 0.log

INFO:test_module:hello! 0
INFO:test_module:hello! 3
INFO:test_module:hello! 4
INFO:test_module:hello! 5
INFO:test_module:hello! 6
INFO:test_module:hello! 7
INFO:test_module:hello! 9

Some of the messages are contained variously in 1.log etc, at random:

INFO:test_module:hello! 1

My intention is for each log file to only contain its own message, for example 0.log should simply be as follows:

INFO:test_module:hello! 0

My understanding is that each subprocess should import the logging module after being created, and this should have no effect on the other subprocesses. And yet, many of them produce logging output in 0.log. I added the 'spawn' option in an attempt to ensure this is the case, as from what I understand it should produce a subprocess which shares nothing with the parent. Removing it and using the default ('fork', on my system) results in similar behaviour.

Answer 1

I would advice against using basicConfig and using __name__ in such a scenario.

Instead, set up the loggers and file handlers individually:

# test_module.py
import logging

class test_class:
    def __init__(self, x):
        logger = logging.getLogger(str(x))
        logger.info(f'hello! {x}')

#test.py
import test_module
import logging
import multiprocessing as mp

def fun(x):
    logger = logging.getLogger(str(x))
    logger.setLevel(logging.INFO)
    log_file = f'{x}.log'
    f_handler = logging.FileHandler(log_file, mode='w+')
    logger.addHandler(f_handler)
    test_module.test_class(x)

if __name__ == "__main__":
    nprocs = 5
    with mp.Pool(nprocs) as pool:
        pool.map(fun, [x for x in range(10)])