'Nodejs readstream, what value should I use for highWaterMark

I have a copy function that uses createReadStream and createWriteStream to give me progress events during a file copy:

        createReadStream(source, { highWaterMark })
            .pipe(
                progress({ length: stats.size }).on('progress', (event) =>
                    subscriber.next({ ...event, type: 'fileStreamProgress', stats: { source, stats } })
                )
            )
            .pipe(createWriteStream(destination, force ? undefined : { flags: 'wx' }))
            .once('error', (err) => subscriber.error(err))
            .once('finish', () => subscriber.complete());

I don't really know what highWaterMark is. My rough understanding is that it is chunk size. If I had infinite memory surely I should just set this to infinity? Or does the "chunk" only get sent to the next item in the path once it has all been read? I do see that copy speed gets higher the larger this number is but at a certain point it starts to get slower again.

I will mostly be copying video files that are about 20 -30 GB and wondering what highwatermark value will give me the best performance.

Thanks



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source