'Dead lock in Popen while multiple threads running

I have 50 threads running this work:

mutex = threading.Semaphore(10)

def work(filename):

    command = ['python', filename]
    process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
        shell=False, encoding='utf-8', cwd=full_path)
    
    file_obj = django model object ...
    
    print(f"Process start process.pid:{process.pid} phone_id:{file_obj.phone_id}")

    while True:
        try:
            try:
               # mutex to avoid threads slow down main thread on too many i/o
                mutex.acquire()
                realtime_output = process.stdout.readline()
                file_obj.refresh_from_db(fields=['enabled'])  # read from database in sqlite
            finally:
                mutex.release()

            if not file_obj.enabled:
                print(f'{file_obj.phone_id} killing ....')
                process.kill()  # TODO: event.set() add process.wait() (until terminated)\
                process.wait()
                # break - will be in poll
                # the poll will stop the loop

            if realtime_output or process.poll() is not None:

                # process been terminated
                if process.poll() is not None:
                    if realtime_output:
                        print(f'process.poll :realtime_output {realtime_output}')
                    global_file.main.update_process(file_obj.id, '')
                    file_obj.save()
                    break

                # process having output
                if realtime_output and realtime_output != '\n':
                    msg = f'>>> {realtime_output}'
                    global_file.main.update_process(file_obj.id, msg)  # update in a global class the output

After a while some processes are hang and not showing any output / not running (the process generate log file that not have new lines on it).

I found many related articles but not sure what is proper solution for me I need to generate output "on-the-fly"

Very large input and piping using subprocess.Popen



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source