ASP.NET return dynamic generated binary file without storing entire content in memory

Question

There are a lot of places on the internet that show how to return files but I have found none that will return dynamically generated binary data without storing the whole contents on memory. Maybe I should serialize my data using Json instead of protobufers.

Thanks to this question I was able to create something like this:

[HttpGet]
public ActionResult DownloadItems()
{
    // get 100K items from database as IEnumerable.
    IEnumerable<SomeObject> items = myDatabase.query("my query that returns 100K objects");

    // create memory stream where to place serialized items
    MemoryStream ms = new ();

    // write all serialized items to stream
    foreach(var item in items)
    {
         byte[] itemSerialized = item.BinarySerialize();
         ms.Write(itemSerialized,0,itemSerialized.Length);
    }

    // set position to the begining of memory stream
    ms.Position = 0;

    return File(ms, "application /octet-stream", "foo.bin");
}

This works well but I am loading 100K items into memory. My question is how can I return the same dynamically generated file without having to load all the items into memory?

I remember that the HTTP protocol returns something like this when returning binary files:

HTTP response headers
...

---------SomeGUID--------------

.. binary data

---------SomeGUID--------------

as a result I believe that having something like this will make it work (it has pseudo code):

[HttpGet]
public ActionResult DownloadItems()
{
    // get 100K items from database as IEnumerable.
    IEnumerable<SomeObject> items = myDatabase.query("my query that returns 100K objects");

    // write the begining of file (PSEUDO code)
    this.response.body.writeString("-----------------SomeGuid------------");

    // write all serialized items to stream
    foreach(var item in items)
    {
         byte[] itemSerialized = item.BinarySerialize();
         this.response.body.write(itemSerialized,0,itemSerialized.Length);
    }

    // set position to the begining of memory stream
    ms.Position = 0;

    this.response.body.writeString("-----------------SomeGuid------------");
}

I can install fiddler or any other proxy to see how the real binary transfer of a file looks like. But is there a build in way of doing that so I don't have to go through all that trouble?

Answer 1

Rather than trying to reuse File() / FileStreamResult, I would recommend implementing your own ActionResult and rendering the content to the response stream there.

public class ByteStreamResult : ActionResult
{
    private readonly IEnumerable<byte[]> blobs;

    public ByteStreamResult(IEnumerable<byte[]> blobs)
    {
        this.blobs = blobs;
    }

    public override async Task ExecuteResultAsync(ActionContext context)
    {
        context.HttpContext.Response.ContentType = "application/octet-stream";
        foreach (var item in blobs)
            await context.HttpContext.Response.Body.WriteAsync(item, context.HttpContext.RequestAborted);
    }
}

return new ByteStreamResult(items.Select(i => i.BinarySerialize()));

Or you could go one step further and implement a custom formatter.

Answer 2

I just created my own fake file stream for this to work:

public class FakeFileStream : Stream
{
    private readonly IEnumerator<object> _enumerator;
    private bool _completed;

    public FakeFileStream(IEnumerable<object> items)
    {
        if (items is null)
            throw new ArgumentNullException();

        _enumerator = items.GetEnumerator();
    }


    public override int Read(byte[] buffer, int offset, int count)
    {
        if (_enumerator.MoveNext())
        {
            var currentItem = _enumerator.Current;

            // deserialize item.  
            byte[] itemSerialized = currentItem.SerializeUsingDotNetProtoBuf();

            // this will probably not happen but it is a good idea to have it implemented.
            // if this is the case store data on memory and return it on the next read
            if (itemSerialized.Length > buffer.Length)
                throw new NotImplementedException();

            // copy data to buffer
            Buffer.BlockCopy(itemSerialized, 0, buffer, 0, itemSerialized.Length);
            return itemSerialized.Length;
        }
        else
        {
            _completed = true;
            return 0;
        }
    }

    // unused methods
    public override void Flush() => throw new Exception("Unused method");
    public override long Seek(long offset, SeekOrigin origin) => throw new Exception("Unused method");
    public override void SetLength(long value) => throw new Exception("Unused method");
    public override void Write(byte[] buffer, int offset, int count) => throw new Exception("Unused method");

    // Properties
    public override bool CanRead => !_completed;
    public override bool CanSeek => false;
    public override bool CanWrite => false;
    public override long Length => throw new NotImplementedException("Not needed");
    public override long Position
    {
        get => throw new Exception("Unused property");
        set => throw new Exception("Unused property");
    }

    // Implement IDisposable
    public override ValueTask DisposeAsync()
    {
        _enumerator.Dispose();
        return base.DisposeAsync();
    }
}

and My endpoint looks like this:

[HttpGet]
[ProducesResponseType(StatusCodes.Status200OK, Type = typeof(FileStreamResult))]
public IActionResult GetBinary()
{         
    // get some IEnumerable collection 
    IEnumerable<Foo> items = MyDatabase.MyTable.Find("my query");

    // create a fake file stream
    var fs = new FakeFileStream(items);

    return File(fs, "application/octet-stream"); 
}

Edit

Do not use sugested FakeFileStream. For some reason it gave me problems. Maybe because it was writting to the stream very few bytes.

Anywyas I was not able to do this from a controller. But I was able to do it using middleware. I had to do simething like this:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.UseHttpsRedirection();

// any previous middleware you may have

app.Use(async (context, next) =>
{
    var downloadFile = context.Request.Query["downloadFile"];
    if (!string.IsNullOrWhiteSpace(downloadFile))
    {
        context.Response.ContentType = "application/octet-stream";

        // get 100K items from database as IEnumerable.
        IEnumerable<SomeObject> items = myDatabase.query("my query that returns 100K objects");

    

        // write all serialized items to stream
        foreach(var item in items)
        {
             byte[] itemSerialized = item.BinarySerialize();
             await context.WriteAsync(itemSerialized, context.CancelationToken);
        }

        return;

  
    }


    // Call the next delegate/middleware in the pipeline.
    await next(context);
});

// etc rest of your middleware

app.Run();