ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2._psycopg' zappa

Question

I am using zappa to deploy backend to the AWS Lambda. It worked well, until I decided to use PostgreSQL. I added it in the settings like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': config('DATABASE_NAME'),
        'USER': config('DATABASE_USER'),
        'PASSWORD': config('DATABASE_PASSWORD'),
        'HOST': config('DATABASE_HOST'),
        'PORT': '5432'
    }
}

I am using AWS RDS. I installed psycopg2-binary and also psycopg2 (versions 2.8.6), but the issue remains. The python version is 3.8.

The full error log:

   [1621168086542] [ERROR] ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2._psycopg'
Traceback (most recent call last):
  File "/var/task/handler.py", line 609, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 240, in lambda_handler
    handler = cls()
  File "/var/task/handler.py", line 146, in __init__
    wsgi_app_function = get_django_wsgi(self.settings.DJANGO_SETTINGS)
  File "/var/task/zappa/ext/django_zappa.py", line 20, in get_django_wsgi
    return get_wsgi_application()
  File "/var/task/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/var/task/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/var/task/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/var/task/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/var/task/django/contrib/auth/base_user.py", line 48, in <module>
    class AbstractBaseUser(models.Model):
  File "/var/task/django/db/models/base.py", line 122, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/var/task/django/db/models/base.py", line 326, in add_to_class
    value.contribute_to_class(cls, name)
  File "/var/task/django/db/models/options.py", line 206, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/var/task/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/var/task/django/db/utils.py", line 214, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/var/task/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/var/task/django/db/backends/postgresql/base.py", line 29, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)

Answer 1

You can use the custom compiled psycopg2 library for AWS Lambda aws-psycopg2.

Answer 2

I built my lambda package on macOS Big Sur with zappa==0.53.0 and django==3.2.5. I started getting this error after I upgraded to psycopg2-binary==2.9.1. Rolling back to psycopg2-binary==2.8.6 fixed the issue for me.

Answer 3

I recently had the same issue popping out despite using psycopg2-binary successfully for a long time. I found the following workaround using lambda layers which seems to go in the good future proof direction:

In your zappa config, add the following line:

"layers": ["arn:aws:lambda:<region>:<account_id>:layer:<layer_name>:<layer_version>"]

Referencing one of the layers provided in this repo (it depends on your AWS region and Python version) and remove psycopg2-binary from your environment.

One of the added benefit is that it should reduce your package size as well.

Answer 4

I installed aws-psycopg2 (1.2.1) to resolve the 500 error that I was getting from AWS every time I executed a post request.

Note: I kept psycopg2 (2.9.1) in order so that my local app could still function.

I have a flask app (python 3.8) deployed to AWS lambda via Zappa. Local and hosted versions of my app are both hooked up to an AWS RDS PostgreSQL db. Trying to avoid using a docker container to keep my stack as simple as possible.

Answer 5

Disclosure: I'm the creator.

The recommended to build and deploy Zappa projects is from within a Docker container similar to the AWS Lambda.
The reason your deploy is not working is because Zappa builds all dependencies before it pushes it to AWS Lambda. However its building it for your Computer which runs differently than AWS Lambda.
There is a simple python package that will launch your Terminal in a Docker container similar to the one AWS Lambda uses.
Checkout https://pypi.org/project/zappadock/ .