Firebase cloud function uploading data to the cloud firebase deletes the data without any reason

Question

Hello so a little bit of background. I created a function in JS that fetches some data(multiple objects that can change in number every time I fetch the data ad example : fetch1: I get 3 obj | fetch2: I get 1 obj, I CAN NOT GET 0 obj ) from a website every minute, processes that data, and then sends it to the cloud firestore. The problem is that after the upload of the data, the doc and the collection get deleted without any reason at all and I don't understand what is happening.

I tried to search online but I could not find a fix.

Can anyone give me a hint of what is happening ?

This is the function in JS

import functions = require("firebase-functions");
import admin = require("firebase-admin");
admin.initializeApp();
const database = admin.firestore();
import https = require("https");
let finalData: { [x: string]: any; }[] = [];
const page = 1;
const fiat = "";
const tradeType = "";
const asset = "";
const payTypes = [""];
const baseObj = {
  page,
  rows: ,
  publisherType: ,
  asset,
  tradeType,
  fiat,
  payTypes,
};
const stringData = JSON.stringify(baseObj);
const options = {
  hostname: "",
  port: ,
  path: "",
  method: "",
  headers: {
    "Content-Type": "application/json",
    "Content-Length": stringData.length,
  },
};


const req = https.request(options, (res: any) => {
  finalData = [];
  let output = "";
  res.on("data", (d: string) => {
    output += d;
  });
  res.on("end", () => {
    try {
      const jsonOuput = JSON.parse(output);
      const allData = jsonOuput["data"];
      for (let i = 0; i < allData.length; i++) {
        let payTypesz = "";
        for (let y = 0; y < allData[i]["adv"]["tradeMethods"].length; y++) {
          payTypesz += allData[i]["adv"]["tradeMethods"][y]["payType"];
          if (y < allData[i]["adv"]["tradeMethods"].length - 1) {
            payTypesz += ", ";
          }
        }
        const obj = {
          tradeType: allData[i]["adv"]["tradeType"],
          asset: allData[i]["adv"]["asset"],
          fiatUnit: allData[i]["adv"]["fiatUnit"],
          price: allData[i]["adv"]["price"],
          surplusAmount: allData[i]["adv"]["surplusAmount"],
          maxSingleTransAmount: allData[i]["adv"]["maxSingleTransAmount"],
          minSingleTransAmount: allData[i]["adv"]["minSingleTransAmount"],
          nickName: allData[i]["advertiser"]["nickName"],
          monthOrderCount: allData[i]["advertiser"]["monthOrderCount"],
          monthFinishRate: allData[i]["advertiser"]["monthFinishRate"],
          advConfirmTime: allData[i]["advertiser"]["advConfirmTime"],
          payTypes: payTypesz,
          position: 0,
        };
        finalData.push(obj);
      }
      console.log(finalData);
    } catch (e) {
      console.log(e);
    }
  });
});
exports.scheduledFunction = functions.pubsub
    .schedule("* * * * *")
    .onRun((context: any) => {
      req.write(stringData);
      req.end();
      for (let i = 0; i < finalData.length; i++) {
        database.doc("/$i")
            .set({
              "tradeType": finalData[i]["tradeType"],
              "asset": finalData[i]["asset"],
              "fiatUnit": finalData[i]["fiatUnit"],
              "price": finalData[i]["price"],
              "surplusAmount": finalData[i]["surplusAmount"],
              "maxSingleTransAmount": finalData[i]["maxSingleTransAmount"],
              "minSingleTransAmount": finalData[i]["minSingleTransAmount"],
              "nickName": finalData[i]["nickName"],
              "monthOrderCount": finalData[i]["monthOrderCount"],
              "monthFinishRate": finalData[i]["monthFinishRate"],
              "advConfirmTime": finalData[i]["advConfirmTime"],
              "payTypes": finalData[i]["payTypes"],
              "position": finalData[i]["position"],
            });
      }
      return console.log("Succes Upload of the data ");
    });
//  # sourceMappingURL=index.js.map

This is the setup of the DB.

Answer 1

By default https for Node.js does not return Promises, therefore it can be cumbersome to correctly manage the life cycle of your Cloud Function.

I would suggest you use the axios library and refactor your code as follows, using a batched write to write to Firestore:

exports.scheduledFunction = functions.pubsub
    .schedule("* * * * *")
    .onRun(async (context: any) => {   // <=== See async keyword here
        
        try {
            const httpCallResponse = await axios.get(...); // I let you adapt the code, including the URL to call, according to the axios doc
            const finalData = ... // Again, it's up to you to work out the value based on httpCallResponse
            
            const batch = database.batch();
    
            for (let i = 0; i < finalData.length; i++) {
                batch.set(database.doc(i.toString(10)),
                    {
                        "tradeType": finalData[i]["tradeType"],
                        "asset": finalData[i]["asset"],
                        "fiatUnit": finalData[i]["fiatUnit"],
                        "price": finalData[i]["price"],
                        "surplusAmount": finalData[i]["surplusAmount"],
                        "maxSingleTransAmount": finalData[i]["maxSingleTransAmount"],
                        "minSingleTransAmount": finalData[i]["minSingleTransAmount"],
                        "nickName": finalData[i]["nickName"],
                        "monthOrderCount": finalData[i]["monthOrderCount"],
                        "monthFinishRate": finalData[i]["monthFinishRate"],
                        "advConfirmTime": finalData[i]["advConfirmTime"],
                        "payTypes": finalData[i]["payTypes"],
                        "position": finalData[i]["position"],
                    });
            }
            
            await batch.commit();
            console.log("Succes Upload of the data ");
            
            return null;
           
        } catch (error) {
            console.log(error);
            return true;
        }

    });

Extra Note: Note that with your code you create monotonically increasing IDs and that could be a problem.