'How to run great expectations on AWS lambda

I am trying to use great-expectations, i.e., run expectations suites within an AWS Lambda function.

When I am trying to install the packages in the requirements.txt, I get an error re jupyter lab:

aws-sam\\build\\ValidationFunction\\.\\jupyterlab_widgets-1.1.0.data\\data\\share\\jupyter\\labextension
s\\@jupyter-widgets\\jupyterlab-manager\\schemas\\@jupyter-widgets\\jupyterlab-manager\\package.json.orig'

I am using SAM CLI, version 1.42.0 and am trying to build the function inside a container. Python version 3.9.

Requirements.txt:

    botocore
    boto3
    awslambdaric
    awswrangler
    pandas_profiling
    importlib-metadata==2.0
    great-expectations==0.13.19
    s3fs==2021.6.0
    python-dateutil==2.8.1
    aiobotocore==1.3.0
    requests==2.25.1
    decorator==4.4.2
    pyarrow==2

I read several posts on the internet using Lambda functions to run Great Expectations. However, there are none reporting any issues.

Specifically, the question is does anyone have a solution for running Python code on Lambda functions when the dependencies are a large set of Python packages?

python aws-lambda great-expectations

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'How to run great expectations on AWS lambda

Sources

Related Questions