How to specify gunicorn log max size

Question

I'm running gunicorn as:

guiconrn --bind=0.0.0.0:5000 --log-file gunicorn.log myapp:app

Seems like gunicorn.log keeps growing. Is there a way to specify a max size of the log file, so that if it reaches max size, it'll just override it.

Thanks!!

Answer 1

TLDR;

I believe there might be a "python only" solution using the rotating file handler provided in the internal lib of python. (at least 3.10)

To test

I created a pet project for you to fiddle with:

1. Create the following python file

test_logs.py

import logging
import logging.config
import time

logging.config.fileConfig(fname='log.conf', disable_existing_loggers=False)


while True:
    time.sleep(0.5)
    logging.debug('This is a debug message')
    logging.info('This is an info message')
    logging.warning('This is a warning message')
    logging.error('This is an error message')
    logging.critical('This is a critical message')

2. Create the following config file

log.conf

[loggers]
keys=root

[handlers]
keys=rotatingHandler

[formatters]
keys=sampleFormatter

[logger_root]
level=DEBUG
handlers=rotatingHandler

[handler_rotatingHandler]
class=logging.handlers.RotatingFileHandler
level=DEBUG
formatter=sampleFormatter
args=('./logs/logs.log', 'a', 1200, 1, 'utf-8')

[formatter_sampleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s

3. Create the ./logs directory
4. Run python test_logs.py

To Understand

As you may have noticed already, the setting that allow for this behaviour is logging.handlers.RotatingFileHandler and the provided arguments args=('./logs/logs.log', 'a', 1200, 10, 'utf-8')

RotatingFileHandler is a stream handler writing to a file. That allow for 2 parameters of interest:

• maxBytes set arbitrarily at 1200
• backupCount set arbitrarily to 10

The behaviour is that upon reaching 1200 Bytes in size, the file is closed, renamed to /logs/logs.log.<a number up to 10> and a new file is opened.

BUT is any of maxBytes or backupCount is 0. No rotation is done !

In Gunicorn

As per the documentation you can feed a config file.

This could look like:

guiconrn --bind=0.0.0.0:5000 --log-config log.conf myapp:app

You will need to tweak it to your existing setup.

Answer 2

On Ubuntu/Linux, suggest to use logrotate to manage your logs, do like this: https://stackoverflow.com/a/55643449/6705684

Since Python>3.3, With RotatingFileHandler, here is my solution(MacOS/Windows/Linux/...) :

import os
import logging
from logging.handlers import RotatingFileHandler
fmt_str = '[%(asctime)s]%(module)s - %(funcName)s - %(message)s'
fmt = logging.Formatter(fmt_str)

def rotating_logger(name, fmt=fmt,
                      level=logging.INFO,
                      logfile='.log',
                      maxBytes=10 * 1024 * 1024,
                      backupCount=5,
                      **kwargs
                      ):
    logger = logging.getLogger(name)

    hdl = RotatingFileHandler(logfile, maxBytes=maxBytes, backupCount=backupCount)
    hdl.setLevel(level)
    hdl.setFormatter(fmt)

    logger.addHandler(hdl)
    return logger

more refer:

• https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler