'how to read all the files in a s3 folder
The s3 folder contains large number of files(eg 1 million file), each file contains only one json record compressed with gz format. how to read all data/files one by one.
we have tried
const AWS = require('aws-sdk');
var s3 = new AWS.S3();
var LineStream = require('byline').LineStream;
var params = {
Bucket: 'xxxx',
params: 'xxxxx'
};
s3.headObject(params, function (err, data) {
var stream = s3.getObject(params).createReadStream();
stream.on('data', (data) => {
console.log('Served by Amazon S3: ' + data);
});
});
any other approach is also fine
thanks in advance Sundar
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
