'Why does match_hostname in python 3.6's SSL module give hostname of local server when issuing remote request?

I'm running into an infrequent bug when issuing a remote request using python's requests module. The stack trace indicates the exception is being thrown down in the ssl module of our python 3.6.6 install. For context, this is a RHEL 6.10 server OpenSSL version is OpenSSL 1.0.1e-fips 11 Feb 2013.

While I don't think it's relevant, the request is being issued from a web server that is sitting behind a WAF that is handling SSL offloading for inbound requests. Again, this is an outbound request that's failing, but I thought I'd mention the above just in case.

The issue is that sometimes, these outbound requests to different domains, such as api.powerbi.com, or api.adp.com are throwing a CertificateError indicating the hostname of whatever certificate was resolved does not match the requested hostname. The strange thing is, the reported hostname of the certificate is our own website.

That was was little confusing to word, but basically it's like outbound requests from our web server are sometimes attempting to load our own SSL certificate, instead of the one for the external URL we're trying to reach.

SSLError at /reports/power-bi/shipped-invoice-list/
HTTPSConnectionPool(host='api.powerbi.com', port=443): Max retries exceeded with url: /v1.0/myorg/groups/12387aasd0-xxxx-1234-82d7-e73929720a25/reports/ (Caused by SSLError(CertificateError("hostname 'api.powerbi.com' doesn't match either of '*.ourdomain.com', 'ourdomain.com'",),))

Traceback (most recent call last):
  File "/u/home/deploy/.virtualenvs/ourdomain.com/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/u/home/deploy/.virtualenvs/ourdomain.com/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/u/home/deploy/.virtualenvs/ourdomain.com/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/u/home/deploy/.virtualenvs/ourdomain.com/lib/python3.6/site-packages/urllib3/connection.py", line 386, in connect
    _match_hostname(cert, self.assert_hostname or server_hostname)
  File "/u/home/deploy/.virtualenvs/ourdomain.com/lib/python3.6/site-packages/urllib3/connection.py", line 396, in _match_hostname
    match_hostname(cert, asserted_hostname)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 327, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))

I am wondering if anyone has seen anything like this before? Whenever I search for these types of errors, I'm mainly getting results about invalid SSL certificates, but nothing about a local SSL certificate being used to valid a request to an external resource.

We are not doing anything special with the requests module, no sessions, etc. It's simply....

import requests

response = requests.post('http://api.powerbi.com...', etc

I should add that we do also issue outbound requests to ourdomain.com from this same application, so I was almost wondering if something in HTTPSConnectionPool is reusing an existing connection that was previously used against ourdomain.com, but is not being used for api.powerbi.com and the SSL isn't being reset? I am not even sure if that is how the connection pooling works, but it was a thought.

Barring that, it would seem like something in our WAF might be occasionally routing outbound requests intended for an external domain, back into our own local hostname somehow.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source