beautiful soup returning empty list with aria-hidden = "True"

Question

I am trying to make amazon product cost tracker, but i am not able to get past getting the information required and is just showing an empty list.

import requests
import bs4
res = requests.get("https://www.amazon.com/dp/B087R79S92/ref=twister_B08LD4TDF3?_encoding=UTF8&th=1")
soup = bs4.BeautifulSoup(res.text,"lxml")
x = soup.select("span.a-offscreen")
print(x)

Answer 1

Note: First at all - ALWAYS take a deeper look into the soup you are cooking

You will find the following information:

To discuss automated access to Amazon data please contact [email protected]. For information about migrating to our APIs refer to our Marketplace APIs at https://developer.amazonservices.com/ref=rm_5_sv, or our Product Advertising API at https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html/ref=rm_5_ac for advertising use cases.

Yes, there are ways to get the content:

• Follow instructions and check the API
• Set some headers to your request to show your "not a robot", wont work forever
• Use Selenium
• ...