How to read an unformatted json stream and write to a formatted json file with minimal memory usage?

Question

I'm calling an API that return a response of type application/json. The response can be very tiny but it can be very huge, 500mb to 700mb. I would like to format the content (indentation, new lines, etc) and write the response to a json file with the help of System.Text.Json so the file can be read easely by an humain.

This code write the response stream directly to a file very efficiently on memory and speed but it doesn't format the json content. Since that the stream is directly written to the file, it doesn't take memory at all.

var response = await new HttpClient().GetAsync("an url", HttpCompletionOption.ResponseHeadersRead);
var responseStream = await response.Content.ReadAsStreamAsync();

using (var fileStream = File.Open(filePath, FileMode.Create))
{
    await responseStream.CopyToAsync(fileStream);
}

I tried this code to add the formatting but it doesn't seems right since that it use over 1gb of memory.

var response = await new HttpClient().GetAsync("an url", HttpCompletionOption.ResponseHeadersRead);
var responseStream = await response.Content.ReadAsStreamAsync();
var jsonDocument = System.Text.Json.JsonDocument.Parse(responseStream);
var jsonWriterOptions = new System.Text.Json.JsonWriterOptions()
{
    Indented = true
};

using (var fileStream = File.Open(filePath, FileMode.Create))
using (var jsonTextWriter = new System.Text.Json.Utf8JsonWriter(fileStream, jsonWriterOptions))
{
    jsonDocument.WriteTo(jsonTextWriter);
}

Is there a more optimized way, that use less memory, to deal with huge content?

Answer 1

It seems that there's no easy way to do it with System.Text.Json because you cannot use a Stream object with the System.Text.Json.Utf8JsonReader directly. To bypass this limitation, you need to put the file content into memory with the System.Text.Json.JsonDocument object so obviously, it will take up a lot of memory.

For now, with what a read on the web, the only solution that is memory efficient is to use Newtonsoft.Json library. Since that there's no parsing involved with this logic, no memory is used to make the conversion.

In this example, the stream come from an HttpClient response.

var response = await new HttpClient().GetAsync("an url", HttpCompletionOption.ResponseHeadersRead);
var responseStream = await response.Content.ReadAsStreamAsync();

using (var streamReader = new StreamReader(responseStream))
using (var jsonReader = new JsonTextReader(streamReader))
using (var streamWriter = File.CreateText(destinationFilePath))
using (var jsonTextWriter = new JsonTextWriter(streamWriter))
{
    jsonTextWriter.Formatting = Formatting.Indented;

    while (jsonReader.Read())
    {
        jsonTextWriter.WriteToken(jsonReader);
    }
}

This example show how to read an existing file

using (var streamReader = new StreamReader(sourceFilePath))
using (var jsonReader = new JsonTextReader(streamReader))
using (var streamWriter = File.CreateText(destinationFilePath))
using (var jsonTextWriter = new JsonTextWriter(streamWriter))
{
    jsonTextWriter.Formatting = Formatting.Indented;

    while (jsonReader.Read())
    {
        jsonTextWriter.WriteToken(jsonReader);
    }
}