memory consumption at encoding base64

Question

I have problems with memory consumption at my software using golangs lib encoding/base64

My software is splitting a videofile to separate images, (gocv mat) converting them to base64 string and saving it to file in json format.

During testing I found that the memory usage is piling up until the oom-reaper is killing the process.

Investigation with pprof showed that the encoding/base64 memory seems to pile up.

I did pprof snapshots after each image frame, and allocated mem of encoding/base64 is raising from 976.89kB(flat) to 4633.54kB(flat) shortly before oom-reaper was killing the process.

Beginning:
      flat  flat%   sum%        cum   cum%
  976.89kB 32.29% 32.29%   976.89kB 32.29%  encoding/base64.(*Encoding).EncodeToString
  512.50kB 16.94% 49.23%   512.50kB 16.94%  runtime.allocm
  512.20kB 16.93% 66.15%   512.20kB 16.93%  runtime.malg
  512.05kB 16.92% 83.08%  1488.94kB 49.21%  runtime.main
     512kB 16.92%   100%      512kB 16.92%  time.resetTimer (inline)
         0     0%   100%   976.89kB 32.29%  main.Process

End:
Showing nodes accounting for 6170.44kB, 100% of 6170.44kB total
      flat  flat%   sum%        cum   cum%
 4633.54kB 75.09% 75.09%  4633.54kB 75.09%  encoding/base64.(*Encoding).EncodeToString
 1024.41kB 16.60% 91.69%  1024.41kB 16.60%  runtime.malg
  512.50kB  8.31%   100%   512.50kB  8.31%  runtime.allocm
         0     0%   100%  4633.54kB 75.09%  main.Process

list shows me the code acoording to it:

(pprof) list encoding/base64
Total: 2.95MB
ROUTINE ======================== encoding/base64.(*Encoding).EncodeToString in /usr/local/go/src/encoding/base64/base64.go
  976.89kB   976.89kB (flat, cum) 32.29% of Total
         .          .    175:
         .          .    176:// EncodeToString returns the base64 encoding of src.
         .          .    177:func (enc *Encoding) EncodeToString(src []byte) string {
         .          .    178:   buf := make([]byte, enc.EncodedLen(len(src)))
         .          .    179:   enc.Encode(buf, src)
  976.89kB   976.89kB    180:   return string(buf)
         .          .    181:}
         .          .    182:
         .          .    183:type encoder struct {
         .          .    184:   err  error
         .          .    185:   enc  *Encoding

So in my golang code the according line of code was:

func Process(img gocv.Mat) ( myImage Images  ){

    detectImg, detectClass, detectBoxes := Detect(&net, 
                                           img.Clone(), 
                                           0.45, 0.5, 
                                           OutputNames, classes)
    defer detectImg.Close()

    // convert gocv.Mat to []bytes
    myImg , _ := detectImg.ToImage()
    myJPG := new(bytes.Buffer)
    jpeg.Encode(myJPG, myImg, &jpeg.Options{95})
    myBytes := myJPG.Bytes()


    // memory consuming
    encodedString := base64.StdEncoding.EncodeToString(myBytes)

// [...]

    return myImage

}

How can I release the memory of "encodedString" in this case that it does not pile up? (Update: Answers say this is not necessary and not possible)

Or is it maybe not my wrong coding, and the mem-leak is at the lib base64 ?? (Update: Answers say this is surely not the case)

Answer 1

To answer your questions:

How can I release the memory of "encodedString" in this case that it does not pile up?

You cannot and you need not. Unused memory is "freed".

Or is it maybe not my wrong coding, and the mem-leak is at the lib base64 ?

No, package encoding/base64 has no memory leak. (Chances are 0 that you detect a memory leak in a trivial function in the standard library of a garbage collected language.)

To guide you towards a solution:

Your application uses absurd amount of memory but that's because a) processing video and images is memory hungry and b) you seem to do nothing to keep memory low: E.g. you encode the whole image into a bytes.Buffer, then encode the whole bytes.Buffer to a string then work on that string and so on. You probably should encode the image into a stream, encode this stream to base64 and stream this output further to where it deposited. This is totally painless in Go as all these encoders work on io.Writers which can be chained very easily.