Realtime output from a shell command in Jupyter notebook

Question

I'm telling jupyter to execute a python script:

!python build_database.py

When executed from the terminal, the python script prints the progress during execution. However, in the jupyter notebook, I get all output printed as a list of strings, AFTER the execution. Is there a way to see the output live?

Answer 1

It looks like it is not possible out of the box. The output handling of shell commands is buried deep in ipython internals.

One of solutions i would recommend is to create custom magic method based on code below.

Check this answer

Based on it i created a simple magic method that you can use:

from subprocess import Popen, PIPE, STDOUT

from IPython.core.magic import register_line_magic


@register_line_magic
def runrealcmd(command):
    process = Popen(command, stdout=PIPE, shell=True, stderr=STDOUT, bufsize=1, close_fds=True)
    for line in iter(process.stdout.readline, b''):
        print(line.rstrip().decode('utf-8'))
    process.stdout.close()
    process.wait()

Or Popen can be used as context manager. So the code will be a bit more readable and reliable. See docs

Popen objects are supported as context managers via the with statement: on exit, standard file descriptors are closed, and the process is waited for.

from subprocess import Popen, PIPE, STDOUT

from IPython.core.magic import register_line_magic


@register_line_magic
def runrealcmd(command):
    with Popen(
        command, stdout=PIPE, shell=True, stderr=STDOUT, bufsize=1, close_fds=True
    ) as process:
        for line in iter(process.stdout.readline, b""):
            print(line.rstrip().decode("utf-8"))

Usage:

%runrealcmd ping -c10 www.google.com

Above code probably could be written better but for your needs it should be perfectly fine.