Web Scraping Python - Immoscout24 - 415 Error

Question

I'm trying to get data from this website:

https://www.immobilienscout24.de/expose/132715777?#/

I keep on getting the same 415 error, despite sending headers. When I put a .json() at the end of the request, it even throws an error.

My Code:

def get_json_data_object():
    headers = {"User-Agent": "Mozilla/5.0 (Linux; U; Android 4.2.2; he-il; NEO-X5-116A Build/JDQ39) AppleWebKit/534.30 ("
                         "KHTML, like Gecko) Version/4.0 Safari/534.30 , 'Accept-Language': 'en-US,en;q=0.5'"}
    api_url = "https://www.immobilienscout24.de/expose/132715777?#/"
    return requests.post(api_url, headers=headers)

Answer 1

HTML Response Code [415][1] indicates that your payload is bad, and cannot be processed by the server. In this case, you're using POST and not GET in order to fetch the page. On a web server these methods are usually received by different handlers, and in the case of POST they are built to expect you will be sending something along - a JSON request, an image, something.

You're not sending anything, which is why it's rejecting the request. However, you're not going ever be able to realistically create a payload that will be accepted (however, the site does publish an external API). Even if you did, it will not get you the response you want - it'll just confirm or deny the item you sent, maybe with some additional data. But it won't get you the page you asked for - that is retrieved with GET.

You mention in comments you can't use GET because it rejects you as a robot. This is probably by design, to prevent you from scraping the site. It may be possible to set up the correct headers and other information to bypass such filters; that would be a different question, at the least, and probably ethically challenging. I'd look in their developer section for what they do or do not allow when using their site for more information. It may be as simple as passing a token along with your request.