Python 3.73 inserting into bytearray = "object cannot be re-sized"

Question

I'm working with a bytearray from file data. I'm opening it as 'r+b', so can change as binary.

In the Python 3.7 docs, it explains that a RegEx's finditer() can use m.start() and m.end() to identify the start and end of a match.

In the question Insert bytearray into bytearray Python, the answer says an insert can be made to a bytearray by using slicing. But when this is attempted, the following error is given: BufferError: Existing exports of data: object cannot be re-sized.

Here is an example:

    pat = re.compile(rb'0.?\d* [nN]')   # regex, binary "0[.*] n"
    with open(file, mode='r+b') as f:   # updateable, binary
        d = bytearray(f.read())         # read file data as d [as bytes]
        it = pat.finditer(d)            # find pattern in data as iterable
        for match in it:                # for each match,
            m = match.group()           # bytes of the match string to binary m
            ...
            val = b'0123456789 n'
            ...
            d[match.start():match.end()] = bytearray(val)

In the file, the match is 0 n and I'm attempting to replace it with 0123456789 n so would be inserting 9 bytes. The file can be changed successfully with this code, just not increased in size. What am I doing wrong? Here is output showing all non-increasing-filesize operations working, but it failing on inserting digits:

*** Changing b'0.0032 n' to b'0.0640 n'
len(d): 10435, match.start(): 607, match.end(): 615, len(bytearray(val)): 8
*** Found: "0.0126 n"; set to [0.252] or custom:
*** Changing b'0.0126 n' to b'0.2520 n'
len(d): 10435, match.start(): 758, match.end(): 766, len(bytearray(val)): 8
*** Found: "0 n"; set to [0.1] or custom:
*** Changing b'0 n' to b'0.1 n'
len(d): 10435, match.start(): 806, match.end(): 809, len(bytearray(val)): 5
Traceback (most recent call last):
  File "fixV1.py", line 190, in <module>
    main(sys.argv)
  File "fixV1.py", line 136, in main
    nchanges += search(midfile)     # perform search, returning count
  File "fixV1.py", line 71, in search
    d[match.start():match.end()] = bytearray(val)
BufferError: Existing exports of data: object cannot be re-sized

Answer 1

This is a simple case, much like modifying an iterable during iteration:

• it = pat.finditer(d) creates a buffer from the bytearray object. This in turn "locks" the bytearray object from being changed in size.
• d[match.start():match.end()] = bytearray(val) attempts to modify the size on the "locked" bytearray object.

Just like attempting to change a list's size while iterating over it will fail, an attempt to change a bytearray size while iterating over it's buffer will also fail.

You can give a copy of the object to finditer().

For more information about buffers and how Python works under the hood, see the Python docs.

---

Also, do keep in mind, you're not actually modifying the file. You'll nee to either write the data back to the file, or use memory mapped files. I suggest the latter if you're looking for efficiency.

Answer 2

Found the solution. Mongodb wasn't started on my server. I feel stupid