How to Import Python Numpy in AWS Lambda

Importing a Python library in AWS Lambda is painful. Python version, library version, platform version… All of them should be compatible with each other. In this post, we’ll learn how to import any Python library to your lambda function without a lambda layer. Let’s get started!

AWS Lambda comes with boto3 library by default. To use other libraries you need to include them in Lambda layer. Another option is adding dependencies to your project folder which we’ll use today.

Let’s check some error messages related to library platform compatibility:

ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy c-extensions failed.

Try uninstalling and reinstalling numpy.
If you have already done that, then:
Check that you expected to use Python2.7 from "/usr/bin/python",
and that you have no directories in your PATH or PYTHONPATH that can
interfere with the Python and numpy version "1.17.2" you're trying to use.
If (1) looks fine, you can open a new issue at
https://github.com/numpy/numpy/issues. Please include details on:
how you installed Python
how you installed numpy
your operating system
whether or not you have multiple versions of Python installed
if you built from source, your compiler versions and ideally a build log
If you're working with a numpy git repository, try git clean -xdf
(removes all files not under version control) and rebuild numpy.
Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.
Original error was: No module named _multiarray_umath

or

cannot import name '_ccallback_c'

Also, you may get an import module error like this:

Unable to import module 'lambda_function': No module named 'pandas'

Installing Dependencies

I’m using pandas, NumPy and scipy for my project so in my case installation command is:

$ pip install -t . pandas numpy scipy

This command won’t fetch the right library packages for AWS Lambda because it fetches the libraries by checking our local computer environments like:

• Platform info (linux, mac, windows)
• Platform architecture(arm, x86)
• Python version
• Dependency list

Gathering Info

We are preparing the project package for the AWS Lambda instance. So we should check AWS lambda python runtime version, architecture and platform info. To check the runtime version click the “Code” tab and scroll to the bottom.

Now, we know everything we need.

• Python runtime: 3.7
• Architecture: x86_64
• Platform: Linux (it’s default for Lambda)

Selecting the right packages

We’ll download packages from PyPI. Click on the links below.

https://pypi.org/project/pandas/#files

https://pypi.org/project/numpy/#files

https://pypi.org/project/scipy/#files

You’ll see a lot of installation packages. Confusing right? Naming conversion for the packages is:

{distribution}-{version}-{python tag}-{abi tag}-{platform tag}.whl.

For NumPy, I should download
NumPy-1.21.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
cp37 mean it’s compatible with python 3.7
manylinux match with AWS lambda platform
x86 architecture also matches with AWS Lambda

Note: Check the compatibility with other libraries too. Eg. Pandas 1.41 Dependencies:

Download files to your project directory. Remove previous pandas, NumPy and scipy folders(if there is any). And unzip downloaded whl files.

You can remove dist-info and pycache folders to save space. Fıle permission for folders inside the deployment package should be 755 or 777. If it’s not you can change it using chmod command.

/Upload your zip file to S3. Copy the S3 URI and in the Code tab of AWS Lamda click upload from –> S3 location –> paster

1. Upload zip.zip file to S3. Copy S3 URI
2. Go to AWS Lambda Code tab
3. Click Upload from -> S3 location
4. Paste and Save the changes.

Congratz! Now you can use the new python libraries in your project. Let me know any questions in the comment box below.