'BeautifulSoup not finding all items in Amazon
I wrote the following source code to scrape titles/authors from Amazon books. However, "find all" only returns me information from the first 30 books instead of all 50 books on the page.
I noticed that the first 30 books are the ones that have already been loaded without scrolling the search bar, but I'm not sure if this is the reason.
s = HTMLSession()
url = "https://www.amazon.com/Best-Sellers-Kindle-Store-Arts-Photography/zgbs/digital-text/154607011/ref=zg_bs_nav_digital-text_3_157325011"
r = s.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
books = soup.find_all("div", {"class":"_p13n-zg-list-grid-desktop_truncationStyles_p13n-sc-css-line-clamp-1__1Fn1y"})
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|