Limiting size of salt-minion log file

Question

Without obvious cause, the /var/log/salt/minion file on my webserver will sometimes explode in size, quickly filling up my 250GB disk space. Looking at the logs when this happens, it is a process repeatedly complaining that there is no drive space. I can't get back to what happens before the drive space fills up. The end result is that the server is dead. It can't respond to web requests or do anything useful. I'm lucky that I can SSH into the server.

My solution is terrible. I have a cron job that checks the size of the file every 5 minutes. If it is bigger than 1GB, I delete the file. The result is that I really have no way to know what is causing the log file explosion in size. But, this is a production server. I can't let it go down for hours on end while I poke around.

Yes - I do have mirrored servers. All are RHEL7 running HTTPD24. All are truly mirrors of one another. All of them suffer from the random explosion in the size of the /var/log/salt/minion file. So, when one goes down, the load balancer will shuffle traffic to other servers.

What I want is a way to restrict the size of the log file. I've never done anything in the past with salt or minions. I didn't install it. It came with RHEL7. I've been googling for a couple weeks and I see nothing that explains what exactly sends logs to the minion file and how to tell it that I don't really need those logs. So, don't do it. I also can't find anyone else with this problem. So, I assume it is yet another "just me" problem. That is why I'm focusing on the direct solution: don't let the log file explode in size, fill up the drive, and kill the server.

Answer 1

To answer the surface-level question: adjust your logrotate config. Some RHEL-specific guidance.

The fundamental problem, however, is that you have not configured your production systems appropriately. Some common strategies to avoid this issue:

1. Separate partitions. So filling one does not kill the whole machine. A common arrangement is separate partitions for /, /var, and /var/log, but it depends on your applications.
2. System monitoring. So you can see your disk usage trends and get alerts before it's too late to do something. e.g. with Elastic, Nagios, Zabbix, etc.
3. External logging. So you can still look at past logs even if the servers are inaccessible or you deleted them locally. e.g. remote syslog or Elastic, etc.
4. Application configuration. So only the minimum log levels you need are enabled, and they are using appropriate file locations and outputs to work with the above.