Author: Gokul P.
Introduction:
Ever since AWS has become the most popular choice for cloud services, there has been a steady increase in the number of organizations completely deploying their setup in AWS. Consequently, the features offered by AWS have also been explored to a great extent and have opened up a wide range of possibilities for the end-users to further strengthen their cloud setup. AWS Lambda is one of those services that enables users to deploy their code in the cloud and get batch processes done. Combined with AWS S3, the Lambda function can be a game-changer in your cloud journey with AWS.
Having said all this, one of the biggest challenges one would face while deploying the python code in Lambda is that it doesn't allow you to pip install any of the libraries that you need to use in your code. This leaves us with only one option-if your code makes use of external python libraries, then you need to add them as layers to the Lambda function.
What are layers? From the documentation, A Lambda layer is an archive containing additional code, such as libraries, dependencies, or even custom runtimes. When you include a layer in a lambda function, the contents are extracted to the /opt directory in the execution environment.
Now you know what a lambda layer is, the next question would be how to create one.
Creating a lambda layer is a multi-step process. Of the many ways to create a lambda layer from scratch, I'm explaining the two simplest methods here.
First I will explain the ways by which the zip file that has to be uploaded to the AWS Lambda function is made, for each of these methods. The final step of adding these files to the lambda function is the same for both.
METHOD-1:Using Wheel (.whl) files:
This method makes use of the .whl file, which is the compiled files for the packages. This could be downloaded for all the popular packages from https://pypi.org/. Search for any of the packages you want to use in the code. Here, for example, I’m doing it for SciPy, a popular library for scientific functions in python.
Since AWS Lambda works on Linux, make sure you download the file for the Linux environment.
This particular file will work for Python 3.8 on AWS Lambda.
Note:- With some of the libraries like Pandas, you would also need an additional library named Pytz, which allows accurate and cross-platform timezone calculations. This can be downloaded from here:- https://pypi.org/project/pytz/#files
Next, create a folder named ‘Python’ (The name is important). Now extract the downloaded .whl files to the folder. (.whl files could be extracted using any zip tool. If the file has not been recognized by the tool automatically, try using the ‘Open with’ option and it should work)
Once the unzip is completed, zip the ‘Python’ folder into a new file named ‘Python.zip’. Again, the name is important here.
Now we have the file that is ready to be uploaded to AWS to create a layer.
We will quickly go through the next method of creating the zip file, and then move to the procedure of adding the zip file to AWS and creating the layer.
Method 2: Using Command Prompt:
Using this method, we are directly installing the required library files to a folder in our local machine(Windows OS) and then zip it to be uploaded to the Lambda Layer.
Open command prompt (run as administrator). First, create a folder named ‘layer' inside the folder.
Next, navigate to the newly created folder.
Create a folder named python and navigate to that folder.
Now, install the required library using pip, into the folder.
Once this is finished, you should see something like this:
Now, go to the path where you have installed the library in explorer. You should see the installed libraries there.
Now, zip these files into a file named ‘python.zip’ (Again, the name is important).
With this, we have the file that has to be uploaded to the Lambda layer ready!
Final Step-Creating AWS Lambda layer:
Now we have the final part where we upload the zip file into Lambda to create layers. As mentioned earlier, this process is the same for both methods mentioned above.
Login to your AWS account, go to S3, upload the created zip file to any of the buckets, and copy the URI.
Navigate to AWS Lambda and in the side pane, select layers.
Click on create a layer.
Specify the name, S3 URI and Runtime, and click on create.
Once created, go to the function in which you want to use the library, and click on layers.
Click on the add a layer option.
Choose custom layers and then choose the layer and version. (Here it shows version 2 since I already had a version created. If you are creating it for the first time, it will be version 1). Click on Add.
Once the layer has been added, the function will be updated.
Now, deploy the code and test. (You might have to create a test event for the first time which is nothing but entering any name and saving the default settings)
After this, the function will be executed, and as you can see the execution was a success.
With this, we have successfully created a layer for the library which you could use in your python code that runs on the AWS Lambda function.
Conclusion:
We have seen two easy methods with which we could create a custom layer on the AWS Lambda for your code. It is recommended to try the first method before you go for the second one since during our tests, some of the libraries were not working properly when done using the second method. There might be situations where both of these methods don’t work for the library you want to use in your code. Then the final resort would be to move the code to the AWS EC2 instance and there it should definitely work. But keep in mind that it might cost a lot more than what AWS Lambda does.