Evaluation failed: TypeError: Cannot read properties of null (reading 'firstElementChild')

Question

I'm trying to trigger click event but in Ubuntu server it is not working.

....
  await page.evaluate(() => {
    document.querySelector(".phone").firstElementChild.click();
  });
 .....

Answer 1

Evaluation fails if an element is not present in the DOM (Document Object Model). TypeError: Cannot read properties of null (reading '...') is the general error in such cases, I can recommend this StackOverflow question/answer on this topic.

(1) As your script works only locally you need to find the reason what's the difference between the scraped page opened locally vs opened on the server:

• why the element with .phone class is not rendered to the DOM?
• are you recognized as a scraper?
• do you get a captcha immediately?
• does the scraped page looks different on the Ubuntu server vs the local computer (rare, but pages can have different content based on User-Agent)?

For this purpose, you can:

1. log the HTML source of the page (console.log(await page.content()))
2. save a screenshot of the page in Base64:

await page.screenshot({ path: __dirname + '/screen.png' })
const screenBase64 = fs.readFileSync(__dirname + '/screen.png', 'base64')
console.log('data:image/png;base64, ' + screenBase64)

You need to find the answer for this yourself and the solution should address this.

---

(2) If your script would throw the same error even locally it would tell that you try to evaluate .phone too early before it would be rendered to DOM. This could be handled with awaiting the element:

await page.waitForSelector('.phone')
await page.evaluate(() => {
  document.querySelector('.phone').firstElementChild.click()
})