'NodeJS heapUsage unchanged when using fs.readFile vs streams
I've been learning about memory management in Nodejs and I'm trying to understand why the following two behaviors occurs:
PS: I'm using the following utility functions to help me print memory to console:
function toMb (bytes) {
return (bytes / 1000000).toFixed(2);
}
function printMemoryData(){
const memory = process.memoryUsage();
return {
rss: `${toMb(memory.rss)} MB -> Resident Set Size - total memory allocated for the process execution`,
heapTotal: `${toMb(memory.heapTotal)} MB -> total size of the allocated heap`,
heapUsed: `${toMb(memory.heapUsed)} MB -> actual memory used during the execution`,
external: `${toMb(memory.external)} MB -> V8 external memory`,
};
}
Part 1) fs.readFile with encoding vs buffers
When I do:
let data;
fs.readFile('path/to/500MB', {}, function(err, buffer){
data = buffer
console.log('Memory usage after files read:', printMemoryData());
});
I get the following output:
Memory usage after files read: {
rss: '565.22 MB -> Resident Set Size - total memory allocated for the process execution',
heapTotal: '11.01 MB -> total size of the allocated heap',
heapUsed: '5.66 MB -> actual memory used during the execution',
external: '524.91 MB -> V8 external memory'
}
Even though I'm storing the data in a local data variable/v8object, the heap isn't used.
But when I do add the encoding:
fs.readFile('path/to/500MB', {encoding: 'utf-8'}, function(err, buffer){
console.log('Memory usage after files read:', printMemoryData());
});
I get the following output:
Memory usage after files read: {
rss: '1088.71 MB -> Resident Set Size - total memory allocated for the process execution',
heapTotal: '535.30 MB -> total size of the allocated heap',
heapUsed: '529.95 MB -> actual memory used during the execution',
external: '524.91 MB -> V8 external memory'
}
Why does the heap get used here instead of in the first function call without an encoding? I don't even have to store the buffer in a local variable for the heap to be used. I also understand that after the next event loop tick in the second example the heap will be cleaned up. But this leads me to my next question in Part 2
Part 2) This part is the same as part 1 but with streams.
const readStream = fs.createReadStream('path/to/500MB');
let data;
readStream.on('data', (buffer) => {
data+= buffer;
});
readStream.on('close', () => {
console.log(printMemoryData());
});
I get the output:
{
rss: '574.57 MB -> Resident Set Size - total memory allocated for the process execution',
heapTotal: '692.75 MB -> total size of the allocated heap',
heapUsed: '508.72 MB -> actual memory used during the execution',
external: '7.97 MB -> V8 external memory'
}
Why the difference in behavior with streams in heap used in part 2 vs the first function without encoding in part 1?
They both have an increase in RSS, but only in the streams does the heap get used when I store the buffer in a local variable/v8 object.
Thanks for any feedback.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
