'I'm using Beautiful Soup to web scrape a website with a password, but it's acting funky
Like I said in the title, I'm trying to web scrape a website that's behind a password. This is the code I have so far:
import requests
from bs4 import BeautifulSoup as bs
url = ('https://www.dummyurl.com')
login = ('/website-extension/')
headers = {'User-Agent': 'User agent information', 'origin': url, 'referer': url + login}
s = requests.session()
login_payload = {
'clientcode': 'uhia8', #This is not the actual info for obvious reasons
'username': 'jferguson',
'password': 'qwerty123'
}
login_req = s.post(url + login, headers=headers, data=login_payload)
print(login_req)
I got <Response [200]>, which from my understanding means that I logged in successfully, but when I, in the session, tried to print the html for the page that I logged into like in the code below, it prints the login page HTML:
soup = bs(s.get(url + login).text, 'html.parser')
print(soup.prettify)
Any idea on what I'm doing wrong? I'm relatively new to coding, and I'm new to stackoverflow, so please be nice :)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
