AWS IOT Data: CERTIFICATE_VERIFY_FAILED

Question

I run test scripts for AWS IOT in a bitbucket pipeline using python + boto3

It worked fine until recently, now i get the following error:

Traceback (most recent call last):
  File "/localDebugRepo/tests/aws/test_iot_api.py", line 119, in test_set_get_owner
    self.iot_util.set_owner(owner, self.test_thing)
  File "/localDebugRepo/aws/iot_api.py", line 176, in set_owner
    self.iot_data.update_thing_shadow(thingName=thing, payload=payload)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 663, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 137, in _send_request
    success_response, exception):
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 256, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/usr/local/lib/python3.6/site-packages/botocore/httpsession.py", line 281, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://data.iot.eu-central-1.amazonaws.com/things/thing-unittest/shadow [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

While I cannot reproduce this on my local system, reproducing the error with the default python:3.6.4 docker image is successful indicating that there might be an invalid certificate.

Intrestingly, running the following command in pipeline is succesfull: openssl s_client -connect data.iot.eu-central-1.amazonaws.com:443

root@f30a34330be5:/localDebugRepo# openssl s_client -connect data.iot.eu-central-1.amazonaws.com:443
CONNECTED(00000003)
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 Secure Server CA - G4
verify return:1
depth=0 C = US, ST = Washington, L = Seattle, O = "Amazon.com, Inc.", CN = *.iot.eu-central-1.amazonaws.com
verify return:1
140686038922896:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=*.iot.eu-central-1.amazonaws.com
   i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
 1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
---

any advice on how can i debug this further would be greatly apreciated

Answer 1

It would appear that AWS has bad certs for the last several hours. I do not subscribe to a support tier, so I don't know how to tell them. I am getting the same problem; boto3 reports that bad cert (which you can verify in a browser).

All of my IoT functions are affected, though if I run it locally (not as a lambda), it seems to work.

Perhaps someone has a way to tell Amazon their little problem?

Edit:

See: https://forums.aws.amazon.com/thread.jspa?messageID=967311&#967311 and https://github.com/boto/boto3/issues/2686 for the fix. You shouldn't use the defaults for creating your dataplane client, because certifi (python) has been fixed to ignore the Symantec CA for the URL, and Amazon isn't going to fix it.

Answer 2

The solution pointed out by Eric Lyons did not worked for me directly. The problem was with the endpoint provided by:

iot_client = boto3.client("iot", region_name=os.getenv("IOT_REGION"))
iot_client.describe_endpoint(endpointType="iot:Data-ATS").get("endpointAddress")

It fails during authentication:

I fixed it by getting the endpoint directly from the IOT-Core settings page:

client('iot-data',
       aws_access_key_id     = '<MY ACCESS KEY>',
       aws_secret_access_key = '<MY ACCESS SECRET KEY>',
       endpoint_url          = '<MY ENDPOINT>');