Windows encoding changed when redirecting output

Question

Hi I have the following python file 'test.py':

import sys
print(sys.stdout.encoding)
sys.stdout.reconfigure(encoding='utf-8') 
print(sys.stdout.encoding)

when I run

py test.py

I get:

utf-8
utf-8

but when I run

py test.py > test.txt

or

py test.py | Out-File -FilePath test.txt -Encoding ASCII

I get from test.txt:

cp1252
utf-8

---

Update: when I run following python code:

import sys, locale
print(sys.getdefaultencoding())
print(locale.getpreferredencoding())

I get:

utf-8
cp1252

---

Question:
May I know why this happen and what should I do so that the default encoding is utf-8 when redirecting?
Thanks

Answer 1

You are using Windows. This is happening because the Windows 7 console does not understand UTF-8. So when you display standard output, it needs to encode as something Windows can display.

Luciano Ramalho's book Fluent Python does a really good job explaining this.

Answer 2

May I know why this happen

Because the Python developers chose to do it like that. See the documentation:

On Windows, UTF-8 is used for the console device. Non-character devices such as disk files and pipes use the system locale encoding (i.e. the ANSI codepage).

What should I do so that the default encoding is utf-8 when redirecting?

Force the encoding. I've added the following to my programs that were plagued by this problem:

if os.name == "nt":
    sys.stdout.reconfigure(encoding="utf-8")

If you use stderr, you might want to reconfigure that as well.