How to use Tensorflow Lite on AWS Lambda

Question

I'm trying to host a small model I have compiled down to a .tflite on AWS Lambda. Using either the python 3.6 or python 3.7 tflite wheel files available on the tensorflow website I zip up my packages/code, and upload to S3 and link to lambda with space to spare. However, when I test my function it crashes when trying to load tflite. Initially, it couldn't load shared object files. This was the error

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_predict': No module named '_interpreter_wrapper')

I found this shared object file and moved it up into the local directory, and then got another error

Unable to import module 'lambda_predict': /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/_interpreter_wrapper.so)

My base system is Ubuntu (Bionic Beaver) Both these errors come from importing tflite

Answer 1

Okay I solved this today.

Complete solution and compiled dependencies for amazonlinux/aws lambda on my github: https://github.com/tpaul1611/python_tflite_for_amazonlinux

So the problem is that aws lambda runs on amazonlinux which apparently needs a different compilation of the tflite _interpreter_wrapper than the one tensorflow is currently providing on their website. https://www.tensorflow.org/lite/guide/python

My solution was to compile it natively on amazonlinux using docker and the script that tensorflow is providing in their git repo. https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/pip_package

I created a Dockerfile:

FROM amazonlinux

WORKDIR /tflite

RUN yum groupinstall -y development
RUN yum install -y python3.7
RUN yum install -y python3-devel
RUN pip3 install numpy wheel
RUN git clone --branch v2.2.0-rc0 https://github.com/tensorflow/tensorflow.git
RUN sh ./tensorflow/tensorflow/lite/tools/pip_package/build_pip_package.sh
RUN pip3 install tensorflow/tensorflow/lite/tools/pip_package/gen/tflite_pip/python3/dist/tflite_runtime-2.2.0rc0-cp37-cp37m-linux_x86_64.whl

CMD tail -f /dev/null

and then ran the following commands:

docker build -t tflite_amazonlinux .
docker run -d --name=tflite_amazonlinux tflite_amazonlinux
docker cp tflite_amazonlinux:/usr/local/lib64/python3.7/site-packages .
docker stop tflite_amazonlinux

These commands output a folder called site-packages which contain the corretly compiled tflite python dependencies for amazonlinux and therefore also aws lambda.

Answer 2

If you currently are trying to zip it up locally and upload the zip, your binary files may not run on the same OS that lambda runs on.

You may want to try SAM as @Yann suggests as it can build your deployment package for you; however, I am not sure if this can get the proper binary you need.

What I normally do to get the right binary for lambda is launch an ec2 instance with the same ami as what lambda uses under the hood and download it there. You can then export it to s3, download it locally, and then package it.

Info on using binaries on lambda: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package/

Info on what ami to use for ec2: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Answer 3

I know I am late but this is for reference to other people.

We can directly run tensorflow lite on the public.ecr.aws/lambda/python:3.8 image. With AWS SAM we can package the lambda function as a container image. Just place tflite-runtime in the requirement.txt and create the docker

FROM public.ecr.aws/lambda/python:3.8

COPY requirements.txt ./
RUN python3.8 -m pip install -r requirements.txt -t .

COPY app.py ./

CMD ["app.lambda_handler"]

For more information visit this github link https://github.com/kevin3010/sam-tf-lite-image-classifier

Also I have written a detailed blog post on this https://spaceatom.hashnode.dev/deep-learning-on-aws-lambda-with-tensorflow-lite

I hope this helps