How to configure all loggers in an application

Question

Python's logging module lets modules or classes define their own loggers. And different loggers can have different handlers. Some of them may choose to log to a file, while some choose to log to, say, stdout.

Now my application uses several of these modules, each with their own loggers that have various handlers. Can I unify the logging behavior so that all logs go to a log file that I specified? In other words, is there a way to .config() all the loggers' handlers at once, from a single place?

Answer 1

From Logging HOWTO:

Child loggers propagate messages up to the handlers associated with their ancestor loggers. Because of this, it is unnecessary to define and configure handlers for all the loggers an application uses. It is sufficient to configure handlers for a top-level logger and create child loggers as needed. (You can, however, turn off propagation by setting the propagate attribute of a logger to False.)

Any handlers you add to the root logger will be used when child loggers create log entries.

import logging

root_handler = ...

root_logger = logging.getLogger()
root_logger.addHandler(root_handler)  # Will receive all log entries

# Meanwhile in a module...

import logging

logger = logging.getLogger(__name__)

logger.error(...)  # Will go to root_handler

Answer 2

Here are some good resources:

• logging best practice
• detailed description of how config_dict should look like

---

Briefly, (as far as I understand)

• The logging module provides hierarchical loggers meaning if the root logger (the logger you get with logging.getLogger()) is formatted in a certain way all the loggers with other names, (logging.getLogger("other_logger")) will be formatted the same (unless if you set propagate to False)

• The best practice for big projects as explained in the links above will be to define a logger configuration at the beginning of your package (i.e in __main__.py) and then just call

  logging.getLogger(__name__)
  

Example:

application:

src
??? animals
?   ??? __init__.py       
?   ??? dog.py
|   |?? cat.py
|   |?? fish.py
??? __main__.py

Inside main.py:

import logging
from logging.config import dictConfig

LOG_CONFIG = {
    'version': 1,
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'formatter': 'std',
            'class': 'logging.StreamHandler',
            'stream': 'ext://sys.stdout'
        },
          'detailed_console': {
            'level': 'DEBUG',
            'formatter': 'error',
            'class': 'logging.StreamHandler',
            'stream': 'ext://sys.stdout'
        },
        'std_fh': {
            'level': 'INFO',
            'formatter': 'std',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': 'logs/std.log',
            'mode': 'a',
            'maxBytes': 1048576,
            'backupCount': 10
        },
        'detailed_fh': {
            'level': 'WARNING',
            'formatter': 'error',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': 'logs/errors.log',
            'mode': 'a',
            'maxBytes': 1048576,
            'backupCount': 10
        }
    },
    'loggers': {
        '': {  # root logger
            'level':'NOTSET',
            'handlers': ['std_fh', 'console'],
            
        },
        'D.src': {
            'propagate': False,
            'handlers': ['detailed_fh','detailed_console'],
          
        },
         'src': {
            'propagate': False,
            'handlers': ['std_fh','console'],
          
        }
    },
    'formatters': {
        'std': {
            'format': '[%(levelname)s  - %(asctime)s - %(name)s::] %(message)s'
        },
        'error': {
            'format': '[%(levelname)s - %(asctime)s - %(name)s - %(process)d::module :%(module)s|Line: %(lineno)s]  messages:[ %(message)s ]'
        },

    }
}

logging.config.dictConfig(LOG_CONFIG)
rootL = logging.getLogger() # this is a root logger
srcL = logging.getLogger(__name__) # this is an `src` logger 
detL = logging.getLogger("D.") # this is 'detailed' 

def main():
    rootL.debug("hello from rootL")
    srcL.debug("hello from srcL")
    detL.debug("hello from detL")

Now, let's see how they print from main.py

>>> python -m src
>>> [DEBUG  - 2022-01-28 04:09:14,137 - root::] hello from rootL
>>> [DEBUG  - 2022-01-28 04:09:14,137 - __main__::] hello from srcL
>>> [DEBUG - 2022-01-28 04:20:45,153 - D.src - 36742::module :__main__|Line: 83]  messages:[ hello from detL ]

Now in cat.py, you do the same except you don't need to config it again.

import logging

rootL = logging.getLogger() # this is a root logger
srcL = logging.getLogger(__name__) # this is an `src` logger 
detL = logging.getLogger("D."+__name__) # this is 'detailed' 

rootL.debug("hello from rootL")
srcL.debug("hello from srcL")
detL.debug("hello from detL")

and I would recommend using the dict config instead of a file config for security measures (search here for eval()).