Why is AWS Lambda asking for dependencies of a pickled model imported from S3?

Question

I a trying to build an AWS Lambda function that receives a json object with numeric inputs and returns a prediction from a custom model. The model is saved as a .pkl file in an S3 bucket. With an AWS notebook I am able to simply import the model and pass it inputs without any imports aside from pandas, pickle, and boto3:

import json
import pandas as pd
import pickle
import boto3

#import model from S3
bucket='bucket_name'
data_key='model.pkl' 

s3 = boto3.resource('s3')
my_model = pickle.loads(s3.Bucket(bucket).Object(data_key).get()['Body'].read())
my_model.predict(pd.DataFrame([[31,10]]))

However, in the lambda function where I implement:

import json
import pandas as pd
import pickle
import boto3

print('Loading function')

#import model from S3
bucket='bucket'
data_key='model.pkl' 

s3 = boto3.resource('s3')
my_model = pickle.loads(s3.Bucket(bucket).Object(data_key).get()['Body'].read())


def lambda_handler(event, context):
    return my_model.predict(event['key'])

I receive the following error when I run my test:

{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'CustomEstimator'",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

CustomEstimator is the module I built to train model.pkl. Do I really have to create a deployment package that contains the module of the custom estimator I created? I thought that was the whole point of pickling the model and saving it in an S3 bucket so that I can just pull the model artifact from that single file. What am I missing??

*Insert here apology that I am obviously new to this. Thanks!

Answer 1

No, you misunderstood how pickle works. Pickle still requires that you are able to import the original module and class definitions, this is very explicit in its documentation:

pickle can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was stored.