NodejS Read/Write Binary Data to/from file

Question

Alright, so I have a string with binary data ("111011 10001 etc"), and I'm trying to save it to a file to them read it on another file using streams, the issue is that, the stream is cutting the data off (final binary number in chunk is cut off)

This is how I'm sending the data to the file (reading a file, encoding it with the golombRice encoder and storing it in a file, using chunks of data)

  const writer = fs.createWriteStream(
    `./encodedAndDecoded/encoded${filename}`,
    {
      encoding: "binary",
    }
  );
  const reader = fs.createReadStream(`./silesia/${filename}`, {
    encoding: "base64",
  });

  await new Promise((resolve, reject) => {
    reader.on("data", (chunk) => {
      writer.write(Buffer.from(golombRiceEncoding(chunk, 3)));
    });
    reader.on("end", () => {
      writer.end();
      resolve();
    });
  });

This is how I'm reading it (reading the encoded file, decoding it with the golombRice encoder and storing it in a file, using chunks of data, the issue with this is that the chunks don't have the full binary data because the stream cuts it)

  const writer = fs.createWriteStream(`./encodedAndDecoded/decoded${filename}`);

  const reader = fs.createReadStream(`./encodedAndDecoded/encoded${filename}`, {
    encoding: "binary",
  });

  await new Promise((resolve, reject) => {
    reader.on("data", (chunk) => {
      writer.write(Buffer.from(golombRiceDecoding(chunk, 3), "base64"));
    });
    reader.on("end", () => {
      writer.end();
      resolve();
    });
  });

Wonder if there's a way to make it read the data using streams, but without cutting a binary number? I don't mind if it reads x at a time, the issue is when it cuts a binary number, invalidating the data when decoding. Thank you

Answer 1

Your particular compression scheme that you're trying to decode looks like it's a variable byte scheme. So, if a chunk boundary doesn't perfectly line up with the size of a variable byte piece of compression, your decompression library will fail.

To decode something like this on the fly, you have to build logic right into the decoder that recognizes when it only has a partial piece of data and then buffers that to combine it with the next chunk of data that arrives. Because the compression is variable byte, you can't known where proper boundaries are without being part of the decompression logic.

Or, you can give up doing on-the-fly decoding and buffer all the compressed data into memory and then decompress it all at once when you have all the data (then you don't have any chunk boundaries to worry about).