Synchronized API request and writing to files

Question

I'm working on a node.js application that, once each week, will make a series of API fetch requests, and store the returned JSON into files.

Currently, the code iterates in a for loop, using each string in an array as a parameter for the request. For each parameter, it will request, wait for the returned JSON, write the JSON to a file, then move onto the next.

This takes time, and while this is only running once a week I'm interested in any ways I can speed this up. Is there any way I can simultaneously make fetch requests, but still write to individual files? When I've attempted this previously, the files are created but not written to.

Answer 1

Assuming you write each API call to a separate file, this can be constructed in an asynchronous manner. Whether that ends up speeding the performance or not depends on how much you end up saturating the various IO streams.

Here is pseudocode to demonstrate. I have made this a bit more declarative than it needs to be... ie., it could be more elegant than this, but by way of showing the overall pattern:

async function callToAPI(uri) {
  // have your api call logic here, resolve to desirable response. 
}

async function writeToDisk(body) {
  // have your disk writing logic here.
}

function doAllMyWork() {

  var api1 = callToAPI(someAPIurl1);
  var api2 = callToAPI(someAPIurl2);
  var res1, res2;
  [ res1, res 2] = await Promise.all([ api1, api2 ]);

  var dsk1 = writeToDisk(res1);
  var dsk2 = writeToDisk(res2);
  var dskRes1, dskRes2;

  [ dskRes1, dskRes2 ] = await Promise.all([ dsk1, dsk2 ]);

  console.log(dskRes1);
  console.log(dskRes2);

}

What's wrong with this simplistic model? Well, you wait for all your api calls before you start your disk calls. You could drop the api->disk flow into a single promise and parallelize those promises, as a simplification, and it would add another notch of performance boost, especially if some api calls are longer, slower, or have significantly bigger payloads than others.

Still, your overall performance is going to be limited by (A) io saturation, and (B) the time it takes for the longest operation to happen.