'Beautifulsoup requests.get() redirecting from mentioned url
I use the mentioned code to scrape a specific page
from bs4 import BeautifulSoup
import requests
url = "https://www.mychoize.com/self-drive-car-rentals-pune/cars"
page = requests.get(url=)
print(page.history)
for resp in page.history:
print(resp.status_code, resp.url)
soup = BeautifulSoup(page.content, 'html.parser')
lists = soup.find_all('div', class_ = "product-box")
for list in lists:
title = list.find('h3', class_ = "margin-o ng-binding")
#print(title)
But it keeps scraping the homepage('https://www.mychoize.com'). In order to stop it from redirecting to homepage I tried the following code to explore the response history
from bs4 import BeautifulSoup
import requests
url = "https://www.mychoize.com/self-drive-car-rentals-pune/cars"
page = requests.get(url ,allow_redirects=True)
print(page.history)
for resp in page.history:
print(resp.status_code, resp.url)
soup = BeautifulSoup(page.content, 'html.parser')
lists = soup.find_all('div', class_ = "product-box")
for list in lists:
title = list.find('h3', class_ = "margin-o ng-binding")
#print(title)
I obtained the following output
[<Response [302]>, <Response [301]>]
302 https://www.mychoize.com/self-drive-car-rentals-pune/cars
301 http://www.mychoize.com/
How do I prevent it from redirecting?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
