'File Descriptors closing in forked subprocesses in Python
In a Python 3.8 application a parent process is responsible for creating child processes (workers) and forwarding messages to them via IPC (tasks).
Parent process has its own logger that uses FileHandler to write to a file main.log but it also opens one logger per child process with a unique file for each of them to log IPC activity and errors.
Child processes are created via system call fork. IPC is done via OS queues. See below diagram that illustrates the situation.
The problem: while it is beneficial to log IPC activity on the parent process side and keep it in a dedicated file, this opens a lot of File Descriptors (FDs) which are all inherited by each and every subsequent child process.
Question: how to manage these FDs? Manually find and close them in each child process on run()? Is there a better way?
Solution 1:[1]
You can pass close_fds=True to whatever function (e.g. run) you are using for starting a subprocess:
If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. Otherwise when close_fds is false, file descriptors obey their inheritable flag as described in Inheritance of File Descriptors.
Source: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
Solution 2:[2]
Set the close-on-exec flag for those file descriptors you don't want to carry over to the child processes. Either by setting the O_CLOEXEC flag upon opening, or later with fcntl set FD_CLOEXEC.
Before you ask, O_CLOEXEC != FD_CLOEXEC.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | md2perpe |
| Solution 2 | datenwolf |