'Writing a big JSON file slows whole program - and express app
I have a very large JSON file that updates sometimes, so I pack it and save it every few minutes with jsonpack.
The problem is, this process slows the entire website (express) to a halt while it does so. It often takes around 30 seconds before it serves traffic again.
Here is the snippet that does the saving:
setInterval(() => {
//if(db.sites.length > 250000){
if(db.sites.length > siteCap){
canqueue = false
}else{
console.log(`${db.sites.length} stored, (${noCrawl.length} noCrawl)`)
}
fs.writeFile("./db.txt",jsonpack.pack(db),()=>{console.log("done!")})
}, 600000);
Hopefully, there is a solution to it slowing down this much. The full code is on this GitHub repo if needed: https://github.com/codingMASTER398/Cheesgle (snippet is at line 203 at the moment)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
