How to store and use SSL/TLS certificates for authentication in GitHub actions?

Question

The short version of question is: How to store and use SSL/TLS third-party certificates correctly for authentication with third-party services in GitHub actions?

The long story and the setup is as follows:

1. I have a Kafka broker as a cloud service with authentication using SSL/TLS. The server gives me:

• CA certificate: ca.pem
• Access certificate: service.cert
• Access key: service.key

2. Python project with Consumer and Producer with virtual environment pipenv All these are fed to Kafka Producer/Consumer and authorization using these files from my PC running Ubuntu if works just fine.

However, when I try to do a similar trick on GitHub actions, it doesn't giving me SSL error. Here's what I do:

1. Copy content of all files into respective github secrets. (I've tried this in many ways to avoid special symbols issues but result is the same always)

2. When Consumer or Producer is started it requires path to all these 3 files, not their content. So I create certificate files from keys like this:

      pipenv run mkdir ${{ github.workspace }}/certs
      pipenv run printf "%s" $(echo ${{ secrets.CA }}) > ${{ github.workspace }}/certs/ca.pem
   
      pipenv run printf "%s" $(echo ${{ secrets.SERVICE_CERT }}) > ${{ github.workspace }}/certs/service.cert
   
      pipenv run printf "%s" $(echo ${{ secrets.SERVICE_KEY }}) > ${{ github.workspace }}/certs/service.key
   

(github actions hide the content in logs. Anyway, this is not an issue in this case)

3. When during tests Kafka Consumer tries to connect it gives me SSL error. Other tests, which do not require Kafka Consumer including those using cloud DB, run fine. So, this should be a problem with certificates.

The question is: what am I doing wrong and how to solve this? I've searched a lot, but haven't found any meaningful guide on how to correctly store and use SSL/TLS certificate/keys to access third party services. There're some close topics on similar cases, but none of them use an approach with fits my case. like this: https://github.com/Apple-Actions/import-codesign-certs or this: https://github.community/t/secret-ability-to-store-certificates/16930

I also know that this is possible to bypass this issue by enabling SASL authentication on a broker or using a GitHub container solution for Kafka. However, I'd like to get an understanding how to do this. Because it looks like a very common case but there's literally no info on the Internet about it.

The error log looks like this:

 ../../../.local/share/virtualenvs/<my_proj_name>-2S-aWGK9/lib/python3.9/site-packages/kafka/client_async.py:909: in check_version
    version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter']))
../../../.local/share/virtualenvs/<my_proj_name>-2S-aWGK9/lib/python3.9/site-packages/kafka/conn.py:1238: in check_version
    if not self.connect_blocking(timeout_at - time.time()):
../../../.local/share/virtualenvs/<my_proj_name>-2S-aWGK9/lib/python3.9/site-packages/kafka/conn.py:340: in connect_blocking
    self.connect()
../../../.local/share/virtualenvs/<my_proj_name>-2S-aWGK9/lib/python3.9/site-packages/kafka/conn.py:401: in connect
    self._wrap_ssl()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kafka.conn.BrokerConnection object at 0x7f5dda56f7c0>

    def _wrap_ssl(self):
        assert self.config['security_protocol'] in ('SSL', 'SASL_SSL')
        if self._ssl_context is None:
            log.debug('%s: configuring default SSL Context', self)
            self._ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)  # pylint: disable=no-member
            self._ssl_context.options |= ssl.OP_NO_SSLv2  # pylint: disable=no-member
            self._ssl_context.options |= ssl.OP_NO_SSLv3  # pylint: disable=no-member
            self._ssl_context.verify_mode = ssl.CERT_OPTIONAL
            if self.config['ssl_check_hostname']:
                self._ssl_context.check_hostname = True
            if self.config['ssl_cafile']:
                log.info('%s: Loading SSL CA from %s', self, self.config['ssl_cafile'])
>               self._ssl_context.load_verify_locations(self.config['ssl_cafile'])
E               ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)

../../../.local/share/virtualenvs/<my_proj_name>-2S-aWGK9/lib/python3.9/site-packages/kafka/conn.py:473: SSLError

Answer 1

Based on the suggestions in the comments and github community issue, here are the steps you need to do:

• Create base64 encoded string of the certificate file

base64 <certificate_file>

• Store the base64 string as github secret like below

• Add below lines in github

echo "${{ secrets.DC_DEV_SDMS_CERTIFICATE }}" | base64 --decode > <certificate_file_name>