'Nodejs readstream, what value should I use for highWaterMark
I have a copy function that uses createReadStream and createWriteStream to give me progress events during a file copy:
createReadStream(source, { highWaterMark })
.pipe(
progress({ length: stats.size }).on('progress', (event) =>
subscriber.next({ ...event, type: 'fileStreamProgress', stats: { source, stats } })
)
)
.pipe(createWriteStream(destination, force ? undefined : { flags: 'wx' }))
.once('error', (err) => subscriber.error(err))
.once('finish', () => subscriber.complete());
I don't really know what highWaterMark is. My rough understanding is that it is chunk size. If I had infinite memory surely I should just set this to infinity? Or does the "chunk" only get sent to the next item in the path once it has all been read? I do see that copy speed gets higher the larger this number is but at a certain point it starts to get slower again.
I will mostly be copying video files that are about 20 -30 GB and wondering what highwatermark value will give me the best performance.
Thanks
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
