'scraping udemy page with python but can't get access
I want to scrape udemy courses reviews but I can't get access to the page web. when I want to read the page using python I get this error:
.
urllib.error.HTTPError: HTTP Error 403: Unauthorized
Solution 1:[1]
403 means that you got forbidden from accessing the Udemy website because your scrapper checks these links to get more info: 403 status code https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403
robot.txt https://moz.com/learn/seo/robotstxt
this is Udemy robot.txt link: https://www.udemy.com/robots.txt
Solution 2:[2]
You can run it anywhere where you host usual dotnet core api.
You can publish the bot like an dotnet core api from visual studio or devops pipeline and deploy the code build to iis or server.
NOTE: you can open the same solution you develop in Bot Framework Composer in an Visual Studio like any dotnet core project.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | |
| Solution 2 | Sanjeevi Subramani |