'Writing a big JSON file slows whole program - and express app

I have a very large JSON file that updates sometimes, so I pack it and save it every few minutes with jsonpack.

The problem is, this process slows the entire website (express) to a halt while it does so. It often takes around 30 seconds before it serves traffic again.

Here is the snippet that does the saving:

setInterval(() => {
  //if(db.sites.length > 250000){
  if(db.sites.length > siteCap){
    canqueue = false
  }else{
    console.log(`${db.sites.length} stored, (${noCrawl.length} noCrawl)`)
  }
  fs.writeFile("./db.txt",jsonpack.pack(db),()=>{console.log("done!")})
}, 600000);

Hopefully, there is a solution to it slowing down this much. The full code is on this GitHub repo if needed: https://github.com/codingMASTER398/Cheesgle (snippet is at line 203 at the moment)



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source