Why is the output different with Python 2.7 than Python 3.10? [duplicate]

Question

$ /usr/bin/python2 simple.py 200 > out2.pbm
$ /opt/src/Python-3.10.1/bin/python3 simple.py 200 > out3.pbm
$ cmp out2.pbm out3.pbm
out2.pbm out3.pbm differ: byte 304, line 3

The python2 output is correct. The python3 output is incorrect.

Here is a correct .pbm output file.

simple.py is

import sys

w = h = x = y = bit_num = 0
byte_acc = 0
i = 0; iterations = 50
limit = 2.0
Zr = Zi = Cr = Ci = Tr = Ti = 0.0

w = int(sys.argv[1])
h = w

sys.stdout.write("P4\n%d %d\n" % (w, h))

for y in range(h):

    for x in range(w):

        Zr = Zi = 0.0 
        Cr = (2.0 * x / w - 1.5); Ci = (2.0 * y / h - 1.0)        
        
        for i in range(iterations):

            Tr = Zr*Zr - Zi*Zi + Cr
            Ti = 2*Zr*Zi + Ci          
            Zr = Tr; Zi = Ti               
            if Zr*Zr+Zi*Zi > limit*limit:
                break
                        
        if Zr*Zr+Zi*Zi > limit*limit: 
            byte_acc = (byte_acc << 1) | 0x00
        else:
            byte_acc = (byte_acc << 1) | 0x01
                
        bit_num += 1         

        if bit_num == 8:
        
            sys.stdout.write(chr(byte_acc))            
            byte_acc = 0
            bit_num = 0

        elif x == w - 1:

            byte_acc = byte_acc << (8-w%8)
            sys.stdout.write(chr(byte_acc))  
            byte_acc = 0
            bit_num = 0

What changed that could cause the different output?

Answer 1

I can't run this under Python 3.10.1 (Windows, 64-bit):

**Traceback (most recent call last):
  File ... simple.py", line 39, in <module>
    sys.stdout.write(chr(byte_acc))
  File  ... \lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\x80' in position 0: character maps to <undefined>

If I change the 2 instances of chr() to str() (so that it prints out a string representation of the byte's decimal value instead), it produces the same output under 3.10.1 and 2.7.11.

So you're getting burned by whatever default Unicode encoding scheme is used by your Python for sys.stdout under Python 3.

If I set an envar like so (syntax may differ under your OS):

set PYTHONIOENCODING=latin-1

then both Pythons produce the same output using chr().

One way

Here's one way to "fix it":

import sys
from sys import stdout

if hasattr(stdout, "buffer"): # Python >= 3
    def putbyte(b):
        assert 0 <= b < 256
        stdout.buffer.write(bytes([b]))
else: # before Python 3
    def putbyte(b):
        assert 0 <= b < 256
        stdout.write(chr(b))

and then change your code to use putbyte(byte_acc) instead of the current sys.stdout.write(chr(byte_acc)).

That's not quite enough, though. Writing to the internal buffer yourself also makes you responsible for buffer management across uses. After the current

sys.stdout.write("P4\n%d %d\n" % (w, h))

you also need to add

sys.stdout.flush()

to get the output string into the buffer before you add additional output bytes.