Console warnings to file

Question

I am running a python script using pandas.read_csv to import a csv file. Pandas provides console warnings when it doesn't see what it expects such as:

Skipping line 163: Expected 41 fields in line 163, saw 42

How can I log this to a text file?

If I run the script from a command line, python > logfile.txt only the output of print shows up in the file, not the warnings.

Answer 1

Use stderr

In bash, there are 3 default streams:

STDOUT: Redirected by > or 1>, this is standard output from a program.

STDERR: Redirected by 2>, this is diagnostic output from a program.

STDIN: Input from console, use < to input it.

./prog 2> errorlog.txt >logfile.txt

If you want to redirect ALL output to STDOUT, use:

./prog 2>&1

Here is some more information: I/O Redirection

Answer 2

This line worked for me

python script.py >>script.out 2>&1

which appends STDOUT (1) to script.out file and redirects STDERR (2) to the same file (&1)

Note that output is appended, so you may want to call > script.out to reset the file beforehand, as in

> script.out; python script.py >> script.out 2>&1

Sources

• https://stackoverflow.com/a/876242/8462678
• https://stackoverflow.com/a/57272607/8462678 (accepted answer)