Push data from API to existing page (combining data sources) in Eleventy

Question

I am trying to take an external URL added into a page's front matter --> send URL to scraper --> have scraper take metadata attributes --> push metadata attributes back to page's front matter.

So far the scraper is working, but how would I be able to push this data to the page? Getting the error:

TemplateContentRenderError was thrown
[eleventy:dev] > (./src/index.njk)
[eleventy:dev]   TypeError: Cannot read property 'slice' of undefined

Here is the current code that I have:

config.addCollection('products', collection => {

// This is typical Collection by Tag call
const products = collection.getFilteredByTag('products');

// Map over all the posts
const postsWithUpdates = products.map(item => {

    // get URL from each page and pass it into scraper
    const itemMeta = getMetaData(item.data.refUrl).then((data) => {

        // map each page data with data from the scraper
        item.description = data.description;
        item.Image = data.icon;
        // item.date = item.data.post ? new Date(item.data.post.date) : item.date
        console.log(item.description);

        return item;
    });

    console.log(itemMeta);
    return itemMeta;
});


return postsWithUpdates;
}); 

When I console.log itemMeta or postsWithUpdates it returns as Promise { <pending> }

Any and all insight is very much appreciated, thanks

Answer 1

The issue here is the mixing of synchronous code and asynchronous code with promises. If you're not familiar with promises, MDN has a great guide on promises.

Essentially, a Promise is a way to run some code later after a task has been completed. This is helpful when you're waiting for another process, such as a network request, to finish.

Here, getMetaData() returns a Promise, where you can handle the data in the then handler. However, the code after that (the console.log and return) do not wait for the task to complete, which is why you see Promise { <pending> } logged out.

To fix this, you'll want to switch to async/await syntax. Luckily for us, Eleventy supports async functions in addCollection.

// We need to add the `async` keyword to enable `await` syntax.
config.addCollection('products', async (collection) => {

const products = collection.getFilteredByTag('products');

// When mapping over the posts, we also need to
// add the `async` keyword to the callback function.
// This will return an array of Promises.
const postsWithUpdates = products.map(async (item) => {

    // Now we can use the `await` keyword! This will wait for the
    // task to complete and place the result in `itemMeta`.
    const itemMeta = await getMetaData(item.data.refUrl)

    // map each page data with data from the scraper
    item.description = itemMeta.description;
    item.Image = itemMeta.icon;
    // item.date = item.data.post ? new Date(item.data.post.date) : item.date
    console.log(item.description);    

    console.log(item);
    return item;
});

// Since we now have an array of Promises, we need to resolve each
// one to their final values. We can do this with `Promise.all`.
// Don't forget to `await` the call!
return await Promise.all(postsWithUpdates);

});

Learn more about Promise.all or async functions.