Downloading JSON file in expressjs and reading it

Question

I have a task where I am given a URL such as https://xyz.json. This URL prompts the downloading of the JSON file into the local. I am now required to read the use this JSON data for further processing. Since I am new to NodeJS and express, I find myself confused about how to achieve this in ExpressJS.

This is what I've tried :

const https = require("https"); 
const fs = require("fs");

const file = fs.createWriteStream("outputFile.json");
const request = https.get(
  "https://xyz.json",
  function (response) {
    response.pipe(file);

    // after download completed close filestream
    file.on("finish", () => {
      file.close();
      console.log("Download Completed");
    });
  }
);

Here, in the outputFile.json, no data is present

Qn2) Can I periodically download using setTimeOut(). Would it be efficient or is there any better way of caching data to make the application faster?

Thanks in advance!

Answer 1

Here's a sample app that downloads a json triggered when you hit an API route hosted as ExpressJS sever.

const express = require('express');
const cors = require('cors');
const morgan = require('morgan');
const bodyParser = require('body-parser');
const axios = require('axios');
const fs = require('fs');

const app = express();

app.use(cors());
app.use(morgan(':method :url :status :user-agent - :response-time ms'));
app.use(bodyParser.json());

app.get('/', async (req, res) => {
    try {
        const { status, data } = await axios.get('http://52.87.135.24/json-files/events.json'); // Can be replaced by your json url
        if (status === 200) {
            fs.writeFileSync('data.json', JSON.stringify(data));
            res.status(200).json({
                success: 'Downloaded file.',
                data: data  // Comment it if you don't want to send the data back
            })
        } else {
            res.status(404).json({ 'Failed': 'File not found.' })
        }
    } catch (err) {
        console.log(err);
        res.status(500).json({ 'Error': 'Internal Server Error' });
    }
});

app.listen(process.env.PORT || 3000, function () {
    console.log('Express app running on port ' + (process.env.PORT || 3000))
});

And as I mentioned that this download gets triggered every time you make a request on http://localhost:3000 in this case, you can create a client script that acts like a cron job in which you can use the setTimeout or actually, setInterval to download your file periodically.

const axios = require('axios');

setInterval(async () => {
    await axios.get('http://localhost:3000/');
}, 5000);

Here's such a script along! :)