Popup form visible, but html code missing in Puppeteer

Question

I'm currently trying to get some informations from a website (https://www.bauhaus.info/) and fail at the cookie popup form.

This is my code till now:

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://www.bauhaus.info');
    await sleep(5000);
    const html = await page.content();
    fs.writeFileSync("./page.html", html, "UTF-8");
    page.pdf({
        path: './bauhaus.pdf', 
        format: 'a4'
    });
});

function sleep(ms) {
    return new Promise((resolve) => {
        setTimeout(resolve, ms);
    });
}

Till this everything works fine. But I can't accept the cookie banner, because I don't see the html from this banner in puppeteer. But in the pdf I can see the form.

My browser

Puppeteer

Why can I not see this popup in the html code? Bonus quest: Is there any way to replace the sleep method with any page.await without knowing which js method triggers the cookie form to appear?

Answer 1

This element is in a shadow root. Please visit my answer in Puppeteer not giving accurate HTML code for page with shadow roots for additional information about the shadow DOM.

This code dips into the shadow root, waits for the button to appear, then clicks it:

const puppeteer = require("puppeteer"); // ^13.5.1

let browser;
(async () => {
  browser = await puppeteer.launch({headless: false});
  const [page] = await browser.pages();
  const url = "https://www.bauhaus.info/";
  await page.goto(url, {waitUntil: "domcontentloaded"});
  const el = await page.waitForSelector("#usercentrics-root");
  await page.waitForFunction(el =>
    el.shadowRoot.querySelector(".sc-gsDKAQ.dejeIh"), {}, el
  );
  await el.evaluate(el =>
    el.shadowRoot.querySelector(".sc-gsDKAQ.dejeIh").click()
  );
  await page.waitForTimeout(100000); // pause to show that it worked
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;