Vastly different response times with express server and wrong output order with async/await

Question

I have an express server where I call an API route and loop through an object that gets returned from the fetch method. Within the loop, I'm calling another API and push the contents to an array. When the loop is finished, I return the array in a response object. See my code (the urls were changed for this question and are not actually callable)

app.get('/', async (req, res) => {
  const userId = '123'

  const userData = await fetch(
    `https://www.exampleTourApi.com/users/${userId}/tours?type=tour_recorded&sort_field=date&sort_direction=asc&status=public`
  )

  let tours = null

  try {
    const data = await userData.json()
    tours = data.tours
  } catch (error) {
    console.log('Invalid User ID!')
    return
  }

  let tourContents = []

  await Promise.all(
    tours.map(async tour => {
      try {
        const url = `https://www.exampleTourApi.com/tours/${tour.id}.gpx`

        const gpxFile = await fetch(url)

        if (gpxFile.status !== 200) {
          console.log(gpxFile.status)
          return
        }

        console.log(`Get content for ${url}`)

        const gpxFileContent = await gpxFile.text()

        console.log(`Content length: ${gpxFileContent.length}`)

        tourContents.push(gpxFileContent)

        console.log('Pushed to array')
      } catch (error) {
        console.log(`Unexpected error ${error}`)
        return
      }
    })
  )

  console.log('All finished')

  const response = {
    code: 200,
    tours: tourContents,
  }

  return res.json(response)
})

However, I face two problems:

1. The response times differ heavily when calling the server route via fetch in the frontend. It can be just 4 seconds or a whole minute, even though I'm always calling the same endpoints. I know there can be some variation in the JS execution of a few milliseconds normally, but this is a big gap.

2. I used console.log's to debug the behaviour, but even though I have an async function with await, the console out is in wrong order. Only the 'All finished' console.log fires correctly when all requests are done. Here's an example:

   Get content for https://www.exampleTourApi.com/tours/47742549.gpx
   Get content for https://www.exampleTourApi.com/tours/254832437.gpx
   Content length: 214545
   Pushed to array
   Get content for https://www.exampleTourApi.com/tours/233629471.gpx
   Content length: 214868
   Pushed to array
   Get content for https://www.exampleTourApi.com/tours/231785145.gpx
   Content length: 35955

Can you point me in the right direction? Many thanks!

Answer 1

About the output order. You are executing all fetch(url) in parallel, therefore the completion order depends on which request responds first. That's why you should not push the result into an array manually. Instead you should return the result. Promise.all() ensures all results are placed in the same order as the array you pass to it:

tourContents.push(gpxFileContent) // delete this line

Change your code to this:

    const url = `https://www.exampleTourApi.com/tours/${tour.id}.gpx`

    const gpxFile = await fetch(url)

    if (gpxFile.status !== 200) {
      console.log(gpxFile.status)
      return
    }

    const gpxFileContent = await gpxFile.text()

    return gpxFileContent; // THIS IS IMPORTANT

Then in your Promise.all() do this:

let tourContents =  await Promise.all(
  tours.map(async tour => {
    // ...

If on the other hand you want to fetch sequentially (that is, fetch only one thing at a time) you need to replace Array.prototype.map() with a for loop:

  let tourContents = []

  for (tour of tours) {
    try {
      const url = `https://www.exampleTourApi.com/tours/${tour.id}.gpx`

      const gpxFile = await fetch(url)

      if (gpxFile.status !== 200) {
        console.log(gpxFile.status)
        return
      }

      console.log(`Get content for ${url}`)

      const gpxFileContent = await gpxFile.text()

      console.log(`Content length: ${gpxFileContent.length}`)

      tourContents.push(gpxFileContent)

      console.log('Pushed to array')
    } catch (error) {
      console.log(`Unexpected error ${error}`)
      return
    }
  }