'How to set up a Scrapy proxy with authorization?

My middlewares settings:

from w3lib.http import basic_auth_header 
class CustomProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = "111.11.11.111:1111"
        request.headers['Proxy - Authorization'] = basic_auth_header('login', 'password')

My settings:

DOWNLOADER_MIDDLEWARES = {
    'my_project.middlewares.CustomProxyMiddleware': 350,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400,
}

After launching, I get an error:

scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy 217.29.53.106:51725 [{'status': 407, 'reason': b'Proxy Authentication Required'}]

What is the reason, how to fix it? (I use valid https proxies)

scrapy


Solution 1:[1]

Try changing the header name to be Proxy-Authorization

request.headers['Proxy-Authorization'] = basic_auth_header('login', 'password')

Solution 2:[2]

proxy = [
  'http': 'http://{user}:{password}@{host}:{port}',
  'https': 'https://{user}:{password}@{host}:{port}',
]

yield scrapy.request(url=url, proxy=proxy}

doesn't this work?

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Pavel Bely
Solution 2 insearchof

Related Questions