fetching data from remote couchdb database

Question

I would like to fetch all the data provided by this api with is a couchdb, https://skimdb.npmjs.com/registry/_all_docs its total size is around 1.9 Million row, I need all this data to do some offline local processing , since the size of the data is huge I would like to fetch it by batch , how I could I do it ? I tried fetching all the data from node app by it take to much time and it returned empty.

Thanks in advance

Answer 1

I found solution to my question : There is many ways to do it , one efficient solution is to specify the start_key and the limit which represents the batch s

import fs from 'fs';


const registeryUrl = "https://skimdb.npmjs.com/registry/_all_docs?"


// In my case the start key id is '-'
const startkey="-"
function fetchdata(startkey){
  console.log(startkey)
  // batch size
  const limit = 1000
  fetch(registeryUrl +"startkey_docid="+startkey+"&limit=" + limit )
    .then((response) => response.json())
    .then((data) => {
        // check if there is data still to fetch from api 
      var isStill = ((data["total_rows"]-data["offset"]) >limit) ? true : false;
      // the last element will be the next startkey 
      var last_element=data["rows"].pop()["id"]
      /** process you data as you want */
      
      return [last_element,isStill]
    })
    .then((array) => {
      const isStill=array[1]
      const last_element=array[0]
      // while there is more data be fetched  
      if (isStill){
        fetchdata(last_element)
      }
      
      else{
        console.log("finished fetching")
      }

    })
}

fetchdata(startkey)