'Dead lock in Popen while multiple threads running
I have 50 threads running this work:
mutex = threading.Semaphore(10)
def work(filename):
command = ['python', filename]
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
shell=False, encoding='utf-8', cwd=full_path)
file_obj = django model object ...
print(f"Process start process.pid:{process.pid} phone_id:{file_obj.phone_id}")
while True:
try:
try:
# mutex to avoid threads slow down main thread on too many i/o
mutex.acquire()
realtime_output = process.stdout.readline()
file_obj.refresh_from_db(fields=['enabled']) # read from database in sqlite
finally:
mutex.release()
if not file_obj.enabled:
print(f'{file_obj.phone_id} killing ....')
process.kill() # TODO: event.set() add process.wait() (until terminated)\
process.wait()
# break - will be in poll
# the poll will stop the loop
if realtime_output or process.poll() is not None:
# process been terminated
if process.poll() is not None:
if realtime_output:
print(f'process.poll :realtime_output {realtime_output}')
global_file.main.update_process(file_obj.id, '')
file_obj.save()
break
# process having output
if realtime_output and realtime_output != '\n':
msg = f'>>> {realtime_output}'
global_file.main.update_process(file_obj.id, msg) # update in a global class the output
After a while some processes are hang and not showing any output / not running (the process generate log file that not have new lines on it).
I found many related articles but not sure what is proper solution for me I need to generate output "on-the-fly"
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
