'Python: can get json from a website with error msg 'Missing X-LVS-HSToken'
def downloadWebPageJSON(url):
headers = {'Accept': 'application/json'}
r = requests.get(url, headers=headers)
When navigating a on specific website, I can see that part of the data (information) is not available in the html code. I found out that they are loaded from a JSON file instead.
I managed to get the address to reach the JSON, and would like to download the content. However when I requests.get the JSON I get an error message 'Missing X-LVS-HSToken'
First of all, even thought the website has an authentification to get into our account, the webpage I'm talking about can be accessible by anyone without account (ie: I loaded the page with private browser mode without login - and it still displays the content information I want)
So my question is do you have any idea what can be the cause and how to solve. I guess it is all about having a auth code or something to get token.
No idea if it can help:
Header when I want to reach the html page:
{'Date': 'Thu, 24 Mar 2022 16:05:03 GMT', 'Content-Type': 'text/html; charset=utf-8', 'X-Powered-By': 'Express', 'Cache-Control': 'public, max-age=300', 'ETag': 'W/"43725-aI4pOr4UXwc5yoFiJHmtU4ruXZU"', 'Content-Encoding': 'gzip', 'Content-Security-Policy': "frame-ancestors 'self' https://*.fdj.fr;", 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload;', 'Transfer-Encoding': 'chunked', 'Connection': 'Keep-alive', 'Keep-Alive': 'timeout=15, max=100'}status of the html >>> 200
header when I want to reach the JSON:
{'Date': 'Thu, 24 Mar 2022 16:05:03 GMT', 'Content-Type': 'application/json; charset=utf-8', 'X-Request-Id': 'e435f855-a5c9-4e23-af47-a7d017988752', 'ETag': 'W/"23-xWDQjX9a9skCFe+6I7MkqH0I+1c"', 'Set-Cookie': 'abp-pselw=abpwfr11004_1443; Path=/lvs-api', 'Content-Length': '35', 'Connection': 'Keep-alive', 'Keep-Alive': 'timeout=15, max=100'}status >>> 401
On the browser, I can't access to json directly - I get the same error message. However if I load the html page first it seems that the browser get the token and then when I refresh the Json page then it load all the content.
But I could not do the same within my python script. I tried to requests get the html first then the Json but sill the error message remain.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
