'Write data as stream to SFTP using ssh2-sftp
I need to read data from MongoDB, transform it, and put it as a file to a remote SFTP server. As documents in the DB are quite large, I try to do this as streams. I have a solution that works but is not very efficient. Appreciate any ideas how to improve it. This is how it works so far:
- Read data from Mongo as stream
const aggregationStream = mongoConnection.db.collection.aggregate.stream();
- Create a new Transform stream to do some manipulations with the data
const csvStream = new Transform({ ... some csv transformation });
- Pipe both streams via
require('util').promisify(stream.pipeline)
await streamPipeline(
aggregationStream,
csvStream,
);
So far everything works well.
- Put the data to SFTP:
await sftp.connect();
await sftp.put(csvStream, targetPath, { encoding: null });
And here is the problem. I make use of streams between Mongo and transform but the stream doesn't go further to SFTP as a stream. Instead, it waits while ALL the objects are sent from Mongo to the transform stream and starts writing to the SFTP only AFTER the csvStream finishes transforming Mongo data. Need ideas how I could stream data mongo -> transform -> sftp immediately.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
