'Write data as stream to SFTP using ssh2-sftp

I need to read data from MongoDB, transform it, and put it as a file to a remote SFTP server. As documents in the DB are quite large, I try to do this as streams. I have a solution that works but is not very efficient. Appreciate any ideas how to improve it. This is how it works so far:

  1. Read data from Mongo as stream
const aggregationStream = mongoConnection.db.collection.aggregate.stream();
  1. Create a new Transform stream to do some manipulations with the data
const csvStream = new Transform({ ... some csv transformation });
  1. Pipe both streams via require('util').promisify(stream.pipeline)
await streamPipeline(
  aggregationStream,
  csvStream,
);

So far everything works well.

  1. Put the data to SFTP:
await sftp.connect();
await sftp.put(csvStream, targetPath, { encoding: null });

And here is the problem. I make use of streams between Mongo and transform but the stream doesn't go further to SFTP as a stream. Instead, it waits while ALL the objects are sent from Mongo to the transform stream and starts writing to the SFTP only AFTER the csvStream finishes transforming Mongo data. Need ideas how I could stream data mongo -> transform -> sftp immediately.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source