'Scrapy 2.6.1 - Empty HTML response
I am scraping this website (https://www.nst.com.my/news/crime-courts) for the content of (div.article-teaser) using Scrapy 2.6.1 Python.
New Straits Times website image But when I preview the content using scrapy shell with command (response.css('div.article-teaser')), it returns nothing.
Extracted HTML file image The extracted HTML file also missing the 'div.article-teaser' section where it supposed to be appeared after line 532 and 533 in the picture.
I am suspecting of JavaScript issue, may I know if there's any solution to scrape the content in this case?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|