Reading an io.Reader of unknown size into bytes array in golang

Question

I have a file (say the descriptor is named file) opened via the os.Open() method so I want to read its contents into a bytes array.

I assume the approach would be to create the later

data := make([]byte, 10000000)

and then read the contents into it

n, err := file.Read(data)

My question is whether there is a more elegant/go-idiomatic way of going about this since by not knowing in advance the file size, I just pass a number I estimate would do (10000000) upon initialisation of the bytes array.

Answer 1

You can use the io/ioutil (up to Go 1.15) or os (Go 1.16 and higher) package. There is a helper function that reads an entire file into a byte slice.

// For Go 1.15 and lower:
package main

import "io/ioutil"

func main() {
    data, err := ioutil.ReadFile("path/to/my/file")
}

// For Go 1.16 and higher:
package main

import "os"

func main() {
    data, err := os.ReadFile("path/to/my/file")
}

In Go version 1.16 the io/ioutil function is still there for compatibility reasons but it was replicated in os. I assume that it will stay in io/ioutil for as long as Go has a version 1.xx because of the compatibility promise so you might keep using that one.

In case you have a file descriptor or any io.Reader you can use the io/ioutil package as well:

package main

import (
    "io/ioutil"
    "os"
)

func main() {
    f, _ := os.Open("path/to/my/file")
    data, err := ioutil.ReadAll(f)
}

Answer 2

The ioutil.ReadAll() is a wrapper for io.ReadAll() and it use predetermined size "512" bytes which enlarge as the read loops.

You can instead use the size from (*file.File).Stats().Size().

A more straightforward method would be using os.ReadFile() which automatically convert a file into []byte for you. It creates []byte with sizes from Stats().Size() method above before reading the file into byte.