Selenium won't load all the content

Question

I've been lookin at this webpage, https://www.tractorsupply.com/tsc/product/welded-wire-48-in-x-100-ft to try and get the price using Beautiful Soup and Selenium with python. However, when I use selenium to access the page with this code:

    url = 'https://www.tractorsupply.com/tsc/product/welded-wire-48-in-x-100-ft'
    headers = 'Chrome/101.0.4951.41 (Windows NT 10.0; Win64; x64)'
    opts = Options()
    opts.add_argument(f"user-agent={headers}")

    driver = webdriver.Chrome(chrome_options=opts)

    driver.get(url)

    time.sleep(10)

I get this resulting page:

When I should be getting this page:

I've done a little research and it seems they use JavaScript to populate those fields based on my zip code, but i'm not entirely sure becuase I don't know much about JS.

What I really need is text value of that price and i'm not sure how to pursue that.

Thanks

Answer 1

I was able to figure out a solution, the code is as follows:

headers = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36'
opts = Options()
opts.add_argument(f"user-agent={headers}")

opts.add_argument("--disable-blink-features=AutomationControlled") # must be included

opts.add_argument("--enable-javascript") # must be included

opts.headless = True

driver = webdriver.Chrome(options=opts)

driver.get(url)

time.sleep(5)

Answer 2

You are trying to access an instance variable without ever having created an instance of the class.

If you intend to create an instance of it on MainActivity side then you can do something like:

// Create Instance
val dataManager = DataManager()
// Access instance varaible
val users = dataManager.users
// Do whatever you want with users