how to get data with nodejs in webscraping

Question

I am trying to scrape this link but I get the data from main tab and I don't know how to solve this problem I tried this with python , nodejs but both have the same results ... https://www.gate.io/marketlist?tab=loan I tried this link and get data from this https://www.gate.io/marketlist?tab=usdt I appreciate if someone help me about this matter. this is the code I wrote.

const PORT = 8000
const axios = require('axios')
const cheerio = require('cheerio')
const { response } = require('express')
const express = require('express')

const app = express ()

const url = 'https://www.gate.io/marketlist?tab=loan'
axios(url)
    .then(response => {
        const html = response.data 
        console.log(html)
    })

app.listen(PORT , () => console.log('server running on PORT ${PORT}'))

Answer 1

I think all you have to do is wrap axios with aync await , then you will get your desired result,

const PORT = 8000
const axios = require('axios')
const cheerio = require('cheerio')
const { response } = require('express')
const express = require('express')

const app = express ()

const url = 'https://www.gate.io/marketlist?tab=loan'

const sendGetRequest = async () => {
    try {
        const response = await axios.get(url);
        console.log(response.data);
        } catch (err) {
            console.error(err);
        }
    };
    
sendGetRequest();
app.listen(PORT , () => console.log('server running on PORT ${PORT}'))

Answer 2

The page you need is rendered using javascript, so cheerio won't help you. Instead, you need to use something like the Puppeteer.

Here is a small example:

const puppeteer = require("puppeteer");

const URL = "https://www.gate.io/marketlist?tab=loan";

async function scrape() {
  const browser = await puppeteer.launch({
    headless: false,
  });
  const page = await browser.newPage();

  await page.goto(URL);
  await page.waitForSelector("your_selector"); //or await page.waitForTimeout("your_timeout");

  // here you do what you need

  browser.close();
}

scrape();

See Puppeteer documentation for more usage information.