'Get Amazon Items Rank using Selenium

I want to get the rank of all Amazon items on the page using the following Chrome Extension: DS Amazon Quick View

To do so, I use selenium to log into my chrome profile (where the extension is already installed) and try to scrape the rank html info. However, "find_all" return an empty object:

from selenium import webdriver
from bs4 import BeautifulSoup    

options = webdriver.ChromeOptions()
options.add_argument(r"--user-data-dir=C:\\Users\\Edo\\AppData\\Local\\Google\\Chrome\\User Data") # get your owm chrome local directory
options.add_argument(r'--profile-directory=Default') 
driver = webdriver.Chrome(executable_path=r"C:\Program Files (x86)\chromedriver.exe", options=options) #get your own exe directory
driver.get("https://www.amazon.com/Best-Sellers-Kindle-Store/zgbs/digital-text/ref=zg_bs_unv_digital-text_1_154606011_1")
soup = BeautifulSoup(driver.page_source.encode('utf-8').strip(), 'html.parser')
print(soup.find_all("div", {"class":"xtaqv-result"})))

>>> 0


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source