Google Cloud Platform API for Python and AWS Lambda Incompatibility: Cannot import name 'cygrpc'

Question

I am trying to use Google Cloud Platform (specifically, the Vision API) for Python with AWS Lambda. Thus, I have to create a deployment package for my dependencies. However, when I try to create this deployment package, I get several compilation errors, regardless of the version of Python (3.6 or 2.7). Considering the version 3.6, I get the issue "Cannot import name 'cygrpc'". For 2.7, I get some unknown error with the .path file. I am following the AWS Lambda Deployment Package instructions here. They recommend two options, and both do not work / result in the same issue. Is GCP just not compatible with AWS Lambda for some reason? What's the deal?

Neither Python 3.6 nor 2.7 work for me.

NOTE: I am posting this question here to answer it myself because it took me quite a while to find a solution, and I would like to share my solution.

Answer 1

Building off the answer from @Josh Wolff (thanks a lot, btw!), this can be streamlined a bit by using a Docker image for Lambdas that Amazon makes available.

You can either bundle the libraries with your project source or, as I did below in a Makefile script, upload it as an AWS layer.

layer:
    set -e ;\
    docker run -v "$(PWD)/src":/var/task "lambci/lambda:build-python3.6" /bin/sh -c "rm -R python; pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit" ;\
    pushd src ;\
    zip -r my_lambda_layer.zip python > /dev/null ;\
    rm -R python ;\
    aws lambda publish-layer-version --layer-name my_lambda_layer --description "Lambda layer" --zip-file fileb://my_lambda_layer.zip --compatible-runtimes "python3.6" ;\
    rm my_lambda_layer.zip ;\
    popd ;

The above script will:

1. Pull the Docker image if you don't have it yet (above uses Python 3.6)
2. Delete the python directory (only useful for running a second time)
3. Install all requirements to the python directory, created in your projects /src directory
4. ZIP the python directory
5. Upload the AWS layer
6. Delete the python directory and zip file

Make sure your requirements.txt file includes the modules listed above by Josh: google-cloud-vision, protobuf, google-api-python-client, httplib2, uritemplate, google-auth-httplib2

Answer 2

There's a fast solution that doesn't require much coding.

Cloud9 uses AMI so using pip on their virtual environment should make it work.

I created a Lambda from the Cloud9 UI and from the console activated the venv for the EC2 machine. I proceeded to install google-cloud-speech with pip.That was enough to fix the issue.

Answer 3

I was facing same error using goolge-ads API.

{ "errorMessage": "Unable to import module 'lambda_function': cannot import name'cygrpc' from 'grpc._cython' (/var/task/grpc/_cython/init.py)","errorType": "Runtime.ImportModuleError","stackTrace": []}

My Lambda runtime was Python 3.9 and architecture x86_64.

If somebody encounter similar ImportModuleError then see my answer here : Cannot import name 'cygrpc' from 'grpc._cython' - Google Ads API