'Resolving "urllib.error.HTTPError: HTTP Error 403: Forbidden" for Python web scraper
import urllib.request, urllib.parse, urllib.error
import twurl
import ssl
# https://apps.twitter.com/
# Create App and get the four strings, put them in hidden.py
TWITTER_URL = 'https://api.twitter.com/1.1/statuses/user_timeline.json'
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
while True:
print('')
acct = input('Enter Twitter Account: ')
if (len(acct) < 1): break
url = twurl.augment(TWITTER_URL, {'screen_name': acct, 'count':'2'})
print('Retrieving', url)
req = urllib.request.Request(url, headers={'User-Agent': 'XYZ/3.0'})
# connection = urllib.request.urlopen(url, context=ctx)
connection = urllib.request.urlopen(req, timeout=10, context=ctx)
data = connection.read().decode()
print(data[:250])
headers = dict(connection.getheaders())
# Print headers
print('Remaining', headers['x-rate-limit-remaining'])
I have been teaching myself python using the textbook and resources provided by University of Michigan Python for Everyone online course(s).
I already used Twitter developer page to get the proper API keys and Token keys. After quick google searches I learned about the Request function with the headers parameter and 'User- Agent' and even set a timeout value for urlopen. I still get the same Forbidden message.
Any assistance will be greatly appreciated.
EDIT:
Above mentions a hidden.py file. Here's the code for it as well as what twurl.augment is doing:
# Keep this file separate
# https://apps.twitter.com/
# Create new App and get the four strings
def oauth():
return {"consumer_key": "",
"consumer_secret": "",
"token_key": "",
"token_secret": ""}
I left out the actual keys for obvious reasons.
But twurl.augment is in a separate file as:
import urllib.request, urllib.parse, urllib.error
import oauth
import hidden
# https://apps.twitter.com/
# Create App and get the four strings, put them in hidden.py
def augment(url, parameters):
secrets = hidden.oauth()
consumer = oauth.OAuthConsumer(secrets['consumer_key'], secrets['consumer_secret'])
token = oauth.OAuthToken(secrets['token_key'], secrets['token_secret'])
oauth_request = oauth.OAuthRequest.from_consumer_and_token(consumer,
token=token, http_method='GET', http_url=url,
parameters=parameters)
oauth_request.sign_request(oauth.OAuthSignatureMethod_HMAC_SHA1(), consumer, token)
return oauth_request.to_url()
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
