C# Read large zip archive from non-seekable stream

Question

I've got a zip file provided in a stream that is not seekable. The zip file itself has a size greater than Int32.MaxValue. My goal is to read one of the entries contained in the zip file and output it to some target stream. Similar to this:

using var cs = new System.Security.Cryptography.CryptoStream (...);
using var z = new System.IO.Compression.ZipArchive (cs, ZipArchiveMode.Read);
var entry = zip.GetEntry("1");
entry.Open().CopyTo(...);

Because the stream does not support seeking, the System.IO.Compression.ZipArchive class tries to copy the stream into a backing MemoryStream. This fails because a MemoryStream has a size limit of 2GB and raises an IOException:

Stream was too long.

I would like to avoid downloading the entire stream into a file before reading the entry. Is there a way to do this? Using other libraries is fine with me.

It is possible to read the stream twice or thrice. My hope is that this suffices to locate the entry content so it can be decompressed.

Answer 1

This library appears to be what you (we) need: https://github.com/adamhathcock/sharpcompress

From their readme.md:

The major feature is support for non-seekable streams so large files can be processed on the fly (i.e. download stream).

Note that another option would be to use Pipelines: https://docs.microsoft.com/en-us/dotnet/standard/io/pipelines

I ended up using the standard deflate library. I think it would be best to use pipelines with either library you choose, probably zlibstream with pipelines would work.