'Python subprocess.Popen: non-blocking read of partial lines

My application launches a second application as a sub-process.

I use subprocess.POpen to do a non-blocking read of the second application's STDOUT and STDERR.

The code I inherited works fine to retrieve complete lines (those ending with a newline character). But I can't find any way of retrieving an incomplete line (not ending with a newline character).

To use a simple example, streamlink writes a lot of them, in the following formats:

File output.mp4 already exists! Overwrite it? [y/N] [download][output.mp4] Written 9.5 MB (36s @ 132.6 KB/s)

How do I modify the script below to retrieve un-terminated lines from the subprocess STDOUT and STDERR?

(For brevity, I removed the second half of the DoStuff() class, which neatly closes the subprocess when finished.)

#!/usr/bin/env python3
import os
import queue
import subprocess
import threading

class DoStuff():

    # Use streamlink (https://streamlink.github.io/) to download a NASA stream
    def __init__(self):

        url = 'https://www.youtube.com/watch?v=21X5lGlDOfg'
        cmd_list = ['streamlink', '--hls-live-restart', '-o', 'output.mp4', url, 'best']

        # Start the child process
        stdout_reader = PipeReader()
        stderr_reader = PipeReader()

        child_process = subprocess.Popen(
            cmd_list,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            preexec_fn=os.setsid,
            startupinfo=None,
        )

        stdout_reader.attach_fh(child_process.stdout)
        stderr_reader.attach_fh(child_process.stderr)

class PipeReader(threading.Thread):

    # Non-blocking read of STDOUT and STDERR
    def __init__(self):

        super(PipeReader, self).__init__()

        self.fh = None
        
    def attach_fh(self, fh):

        self.fh = fh
        for line in iter(self.fh.readline, str('')):
            if line == b'':
                break
            else:
                print(line)

# Let's go
DoStuff()


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source