IEnumerable emit as stream?

Question

Is there a way to take an IEnumerable<T> and emit it as a readable stream kinda like this?

private void DoTheThing(IEnumerable<Foo> foos)
{
    using (var myStream = foos.EmitAsStream(f =>
    {
        var line = JsonConvert.SerializeObject(f);
        return line;
    }))
    using(var streamReader = new StreamReader(myStream))
    {
        while (!streamReader.EndOfStream)
        {
            var line = streamReader.ReadLine();
            Console.WriteLine(line);
        }
    }
}

Obviously, there are some problems with this, for example it seems to imply a StreamWriter with none specified, but the idea would be that the stream reader would just yield enumerate through the IEnumerable under the hood, apply the transform delegate, and pull the results without making any new, potentially large objects in memory.

I have some very large enumerable objects in memory, and I need to push them to external places (like Amazon S3) which accepts a stream. I really can't afford to build a MemoryStream with the collection and send that; I can spool to disk and read from disk, but I'd prefer not to if I have the option, since it seems like an extra step.

Is this possible? If it's possible, is it practical?

Answer 1

The EnumerableToStream package will do exactly what you ask:

using EnumerableToStream;

using (var myStream = foos.Select(f =>
{
    var line = JsonConvert.SerializeObject(f);
    return line;
}).ToStream())

That ToStream() method is the one that does the magic. If you are interested in the implementation details, have a look at the source code.