Decoding JSON stream that isn't surrounded by array brackets?

Question

I'm trying to decode a JSON stream from a file. The file is newline separated JSON objects. But it's not formatted with array brackets surrounding the objects, so I can't figure out how to use encoding/json stream decoder.

I get the error not at beginning of value because there is no opening bracket.

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "strings"
)

func main() {

    // My data is missing opening and closing brackets around the jsonStream string.
    // [
    const jsonStream = `
        {"Name": "Ed", "Text": "Knock knock."},
        {"Name": "Sam", "Text": "Who's there?"},
        {"Name": "Ed", "Text": "Go fmt."},
        {"Name": "Sam", "Text": "Go fmt who?"},
        {"Name": "Ed", "Text": "Go fmt yourself!"}
    `
    // ]
    type Message struct {
        Name, Text string
    }
    dec := json.NewDecoder(strings.NewReader(jsonStream))

    // read open bracket -- this fails (no bracket)
    t, err := dec.Token()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%T: %v\n", t, t)

    // while the array contains values
    for dec.More() {
        var m Message
        // decode an array value (Message)
        err := dec.Decode(&m)
        if err != nil {
            log.Fatal(err)
        }

        fmt.Printf("%v: %v\n", m.Name, m.Text)
    }

    // read closing bracket -- this also would fail (no bracket)
    t, err = dec.Token()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%T: %v\n", t, t)

}

Removing the dec.Token() error checks doesn't work either. I guess the parser doesn't expect to see objects before it has seen an array open bracket?

Is there some workaround?

Answer 1

One suggestion is to use a pipe, and prepend and append the [].

reader := strings.NewReader(jsonStream)

pr, pw := io.Pipe()

go func() {
    pw.Write([]byte(`[`))
    io.Copy(pw, reader)
    pw.Write([]byte(`]`))
}()

dec := json.NewDecoder(pr)

You should add some error handling.

See https://play.golang.org/p/Oogu_NdAUyR for the whole thing.

Answer 2

I ended up doing this:

f, err := os.Open(fname)
if err != nil {
    // handle error
}
d := json.NewDecoder(f)
for {
    var v ValueTypeToUnmarshalTo
    if err := d.Decode(&v); err == io.EOF {
        break // done decoding file
    } else if err != nil {
        // handle error
    }
    // do something with v
}

The decoder appears to ignore newlines so there's not even a need to use the stream decoder. That's only if you actually have a JSON array.

Taken from part of this answer: https://stackoverflow.com/a/34388102

Thanks for the help!

Answer 3

There is a small difference in parsing discrete Json objects as stream or an array of Json objects.

In case of array, we need to get the token explicitly (one of [ or {)- that will make way for the Json parser to continue parsing the Json objects in the array. However, in the case of discrete Json objects stream processing, there is no need to look for the starting token. Check out the sample code that has both flavors.

https://go.dev/play/p/Zinv6CwNEPP