Getting error: "certificate verify failed: unable to get local issuer certificate" trying to webscrape pdfs in PyCharm

Question

Here is the code that I have been trying to fix:

import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup
url = "https://www.pbs.gov.pk/node/3391/?name=001"

#If there is no such folder, the script will create one automatically
folder_location = r'infoFA'
if not os.path.exists(folder_location):os.mkdir(folder_location)
response = requests.get(url)
soup= BeautifulSoup(response.text, "html.parser")     
for link in soup.select("a[href$='.pdf']"):
    #Name the pdf files using the last portion of each link which are unique in this case
    filename = os.path.join(folder_location,link['href'].split('/')[-1])
    with open(filename, 'wb') as f:
        f.write(requests.get(urljoin(url,link['href'])).content)

From this, I get the following error in PyCharm:

Traceback (most recent call last):
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.pbs.gov.pk', port=443): Max retries exceeded with url: /node/3391/?name=001 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/justinlesko/PycharmProjects/pythonProject8/main.py", line 10, in <module>
    response = requests.get(url)
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/justinlesko/PycharmProjects/pythonProject8/venv/lib/python3.9/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.pbs.gov.pk', port=443): Max retries exceeded with url: /node/3391/?name=001 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

Process finished with exit code 1

I have tried the following to fix this ssl error: -setting verify to "False" in requests.get(url) -installing and reinstalling certificates from the python 3.9 folder I have -setting unknown certificates to "trust" in preferences

Answer 1

Rather than turn it off you could add the proper cert to your cacert.pem file. You can append it to the end of the file.

For pycharm it will be beneath the PycharmProjects file when using a python SDK you downloaded with PyCharm. For example:

./PycharmProjects/pythonProject/venv/lib/python3.9/site-packages/certifi/cacert.pem