'Resolving "urllib.error.HTTPError: HTTP Error 403: Forbidden" for Python web scraper

import urllib.request, urllib.parse, urllib.error
import twurl
import ssl

# https://apps.twitter.com/
# Create App and get the four strings, put them in hidden.py

TWITTER_URL = 'https://api.twitter.com/1.1/statuses/user_timeline.json'

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

while True:
    print('')
    acct = input('Enter Twitter Account: ')
    if (len(acct) < 1): break
    url = twurl.augment(TWITTER_URL, {'screen_name': acct, 'count':'2'})
    
    print('Retrieving', url)
    req = urllib.request.Request(url, headers={'User-Agent': 'XYZ/3.0'})
    # connection = urllib.request.urlopen(url, context=ctx)
    connection = urllib.request.urlopen(req, timeout=10, context=ctx)
    data = connection.read().decode()
    print(data[:250])
    headers = dict(connection.getheaders())
    # Print headers
    print('Remaining', headers['x-rate-limit-remaining'])
    

I have been teaching myself python using the textbook and resources provided by University of Michigan Python for Everyone online course(s).

I already used Twitter developer page to get the proper API keys and Token keys. After quick google searches I learned about the Request function with the headers parameter and 'User- Agent' and even set a timeout value for urlopen. I still get the same Forbidden message.

Any assistance will be greatly appreciated.

EDIT:

Above mentions a hidden.py file. Here's the code for it as well as what twurl.augment is doing:

# Keep this file separate

# https://apps.twitter.com/
# Create new App and get the four strings

def oauth():
    return {"consumer_key": "",
            "consumer_secret": "",
            "token_key": "",
            "token_secret": ""}

I left out the actual keys for obvious reasons.

But twurl.augment is in a separate file as:

import urllib.request, urllib.parse, urllib.error
import oauth
import hidden

# https://apps.twitter.com/
# Create App and get the four strings, put them in hidden.py

def augment(url, parameters):
    secrets = hidden.oauth()
    consumer = oauth.OAuthConsumer(secrets['consumer_key'], secrets['consumer_secret'])
    token = oauth.OAuthToken(secrets['token_key'], secrets['token_secret'])

    oauth_request = oauth.OAuthRequest.from_consumer_and_token(consumer,
                    token=token, http_method='GET', http_url=url,
                    parameters=parameters)
    oauth_request.sign_request(oauth.OAuthSignatureMethod_HMAC_SHA1(), consumer, token)
    return oauth_request.to_url()


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source