Log into a website using Requests module in Python

Question

I am new in Python and web scraping, but I keep learning. I have managed to get some exciting results using BeautifulSoup and Requests libraries and my next goal is to log into a website that allows remote access to my heating system to do some web scraping and maybe extend its capabilities further.

Unfortunately, I got stuck. I have used Mozilla's Web Dev Tools to see the url that the form posts to, and the name attributes of the username and password fields. The webpage url is https://emodul.pl/login and the Request payload looks as follows:

{"username":"my_username","password":"my_password","rememberMe":false,"languageId":"en","remote":false}

I am using requests.Session() instance to make a post request to the login url and using the above-mentioned payload:

import requests

url = 'https://emodul.pl/login'
payload = {'username':'my_username','password':'my_password','rememberMe':False,'languageId':'en','remote':False}

with requests.Session() as s:
    p = s.post(url, data=payload)
    print(p.text)

Apparently I'm doing something wrong because I'm getting the "error":"Sorry, something went wrong. Try again." response.

Any advice will be much appreciated.

Answer 1

Services often look at HTTP headers.

• Typical behavior is to restrict access based on user agent; try using an UA value that is similar to your regular browser.

• Some services also check other headers in order to discard requests that they think are not coming from a human.

You can use developer tools to see what headers are being sent, and send all or most of them as part of your requests (as simple as post(url, data=payload, headers={...}).