When I upload a (zipped) deployment package as a lambda function on AWS I get "no module named..." errors for both bs4 and google.
I created a virtual environment using venv and I installed the required dependencies
The app works fine when running from within the virtual environment. But, when I zip it up and upload it as a lambda function on AWS, I get "no module named..." errors for both "bs4" and (if I hash out the import of bs4 for debug reasons) also for "google". I checked the site-packages folder in the zip file and they seem to be there.
Why is AWS saying there is no module when there is?!
I am using python3.6 on Ubuntu.
Lambda needs ZIP with all the libraries and your main python code file in the same folder.
Here is what i do:
Create a new package folder with the following hierarchy
mkdir -p ./package/tmp/lib
Copy Project folder into the temp folder
cp -a ./$(PROJECT)/. ./package/tmp/
Copy python site-packages from virtual env to temp folder in package
cp -a $(VIRTUAL_ENV)/lib/python2.7/site-packages/. ./package/tmp/
Remove any unused libraries (that are not required for this particular lambda to run) from temp folder
rm -rf ./package/tmp/wheel*
Zip the temp package directory
cd ./package/tmp && zip -r ../$(PROJECT).zip .
This final zip so created is ready for upload on Lambda.
Related
I am experimenting with Google Cloud Platform buildpacks, specifically for Python. I started with the Sample Functions Framework Python example app, and got that running locally, with commands:
pack build --builder=gcr.io/buildpacks/builder sample-functions-framework-python
docker run -it -ePORT=8080 -p8080:8080 sample-functions-framework-python
Great, let's see if I can apply this concept on a legacy project (Python 3.7 if that matters).
The legacy project has a structure similar to:
.gitignore
source/
main.py
lib
helper.py
requirements.txt
tests/
<test files here>
The Dockerfile that came with this project packaged the source directory contents without the "source" directory, like this:
COPY lib/ /app/lib
COPY main.py /app
WORKDIR /app
... rest of Dockerfile here ...
Is there a way to package just the contents of the source directory using the buildpack?
I tried to add this config to the project.toml file:
[[build.env]]
name = "GOOGLE_FUNCTION_SOURCE"
value = "./source/main.py"
But the Python modules/imports aren't set up correctly for that, as I get this error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
Putting both main.py and /lib into the project root dir would make this work, but I'm wondering if there is a better way.
Related question, is there a way to see what project files are being copied into the image by the buildpack? I tried using verbose logging but didn't see anything useful.
Update:
The python module error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
was happening because I moved the lib dir into source in my test project, and when I did this, Intellij updated the import statement in main.py without me catching it. I fixed the import, then applied the solution listed below and it worked.
I had been searching the buildpack and Google cloud function documentation, but I discovered the option I need on the pack build documentation page: option --path.
This command only captures the source directory contents:
pack build --builder=gcr.io/buildpacks/builder --path source sample-functions-framework-python
If changing the path, the project.toml descriptor needs to be in that directory too (or specify with --descriptor on command line).
I have an Angular-Flask application that I'm trying to Dockerize.
One of the Python files used is a file called trial.py. This file uses a module called _trial.pyd. When I run the project on local everything works fine. However, when I dockerize using the following code and run, it gives me error "Module _trial not found".
FROM node:latest as node
COPY . /APP
COPY package.json /APP/package.json
WORKDIR /APP
RUN npm install
RUN npm install -g #angular/cli#7.3.9
CMD ng build --base-href /static/
FROM python:3.6
WORKDIR /root/
COPY --from=0 /APP/ .
RUN pip install -r requirements.txt
EXPOSE 5000
ENTRYPOINT ["python"]
CMD ["app.py"]
Where am I going wrong? Do I need to change some path in my dockerfile?
EDIT:
Directory structure:
APP [folder with src Angular files]
static [folder with js files]
templates [folder with index.html]
_trial.pyd
app.py [Starting point of execution]
Dockerfile
package.json
requirements.txt
trial.py
app.py calls trial.py (successfully).
trial.py needs _trial.pyd. I try to call it using import _trial.
It works fine locally, but when I dockerize it, i get the following error message:
Traceback (most recent call last):
File "app.py", line 7, in
from trial import *
File "/root/trial.py", line 15, in
import _trial
ModuleNotFoundError: No module named '_trial'
The commands run are:
docker image build -t prj .
docker container run --publish 5000:5000 --name prj prj
UPDATE
It is able to recognize other .py files but not .pyd module.
I think this could be due to the following reasons
Path
Make sure the required file is in the pythonpath. It sounds like you have done that so probably not this one.
Debug Mode
If you are working with a debug mode you will need to rename this module "_trail.pyd" to "_trail_d.pyd"
Missing DLL
The .pyd file may require a dll or other dependency that is not available and cant be imported due to that. there are tools such as depends.exe or this that allow you to find what is required
Name Space Issue
If there was a file already called "_trail.py" in the python path that could create unwanted behavior.
Solved it!
The issue was that I was using .pyd files in a Linux container. However, it seems that Linux doesn't support .pyd files. This is why it was not able to detect the _trial.pyd module. I had to generate a shared object (.so) file (i.e. _trial.so) and it worked fine.
EDIT: The .so file was generated on a Linux system by me. Generating .so on Windows and then using it in a Linux container gives "invalid ELF header" error.
I have a file with this as the contents.
def print_hello_world():
print ('Hello World')
It's zipped up in a folder with a __init__.py file.
I add this as the layer and set the correct runtime of python3.6.
How do import it into my lambda function via lambda code?
Edit: After researching I found that lambda mounts the layers at /opt and if you add /opt to your syspath via sys.path.insert(0, '/opt') then you can import your layers.
You can see your layers at: print(os.listdir("/opt"))
There's gotta be a more elegant way to do this!
So I've recently ran into this issue, and I believe I found a cleaner way to import your layers.
First for the structure of the zip file which you upload:
You do not need an __init__.py file
Put all the scripts which you want to import into a folder name python
Zip up that python folder (choose any name you want) and upload it to your layer
Once uploaded, and the layer has been configured in your lambda function, you can simply use it with import {filename}
So if your script in the python folder was called something like custom_helper.py, import it in your lambda with import custom_helper.
I am not sure if this is the clean way to do it, but it seems simple enough to start.
Your zip file should have the following structure:
python/lib/python3.7/site-packages
That is, it needs a folder named Python, and within that a folder named lib, and within that a folder named python3.7, and within that a folder named site-packages. Anything inside that folder will be available for import.
(If you're using another version of Python, that version should be in the path instead of 3.7)
You'll have to:
Build you lambda layer using Docker:
sudo docker run -v "$PWD":/var/task "lambci/lambda:build-${penv}" /bin/sh -c "pip install -r requirements.txt -t python/lib/${penv}/site-packages/; exit"
Upload layer to AWS
Add layer to your lambda
Modity sys.path in your Lambda before import
import sys;
sys.path.append('/opt/python/lib/python3.7/site-packages');
import apsw
print(apsw.__file__)
/opt/python/lib/python3.7/site-packages/apsw.cpython-37m-x86_64-linux-gnu.so
Update:
It looks like you need to place build under /opt/python so you can use it without
sys.path.append('/opt/python/lib/python3.7/site-packages');
New build steps:
mkdir -vp my-layer/python && cd my-layer/python
python3 -m pip install click -t ./
cd ..
zip -r9 my-layer python
now add layer to lambda function
and you can import:
import click
SOLUTION
You must to allocate all the packages you install with pip install ... into a zip file that ends with the folllowing file structure:
[your zip file].zip
|_python/
|_all.../
|_your.../
|_packages, modules.../
Another boring way is:
[your zip file].zip
|_python/
|_lib/
|_python3.x/
|_site-packages/
|_all.../
|_your.../
|_packages, modules.../
with this structure you lambda code get the modules you have.
I hope this make done your issue.
I have a Django project that needs to be built using TravisCI. Originally, I put the travis.yml file in the root of the project (where the virtual environment would be) and then built it but for some reason, Travis is using default settings since it can't find the yml file.
I then moved the file into the src directory and rerun, but the build still wasn't finding the file. Where does the file need to be placed in order for Travis to find it?
Travis.yml:
language: python
python:
- "2.7"
# setup environment
env:
- DJANGO_VERSION=1.11.2
- DJANGO_SETTINGS_MODULE='di_photouploader.settings.production'
# install dependencies
install:
- pip install -r requirements.txt
# run test scripts
script:
- python manage.py test
It should be named
.travis.yml
And not
Travis.yml
Place the file at the project root.
my goal is to dockerize python 3.6 web application.
During my work I developed private python package ('VmwareProviderBL') that my web application is using.
Build my docker image working perfectly, but when I am trying to run it, I get an error saying my private Python package is not found.
My Dockerfile
# Use an official Python runtime as a parent image
FROM python:3.6-slim
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
ADD . /app
# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt
# Make port 80 available to the world outside this container
EXPOSE 80
# Define environment variable
ENV NAME World
# Run WebController.py when the container launches
CMD ["python", "WebApi/WebController.py"]
My Error when trying to run the image
Traceback (most recent call last):
File "WebApi/WebController.py", line 3, in <module>
from VmwareProviderBL import SnapshotBL, MachineBL, Utils
ModuleNotFoundError: No module named 'VmwareProviderBL'
My package hierarchy
--VmwareVirtualMachineProvider
--WebApi
--VmwareProviderBL
--requirements
--Dockerfile
Any ideas anyone?
I know I need somehow to add this package to my Dockerfile, did not find any example online.
When you import a module, Python looks in (a) the built-in path (sys.path), (b) locations in the PYTHONPATH variable, and (c) in the directory containing the script you're running.
You're installing the requirements for your module (pip install -r requirements.txt), but you're never installing the VmwareProviderBL module itself. This means it's not going to be available in (a) above. The script you're running (WebApi/WebController.py) isn't located in the same directory as the VmwareProviderBL module, so that rules out (c).
The best way of solving this problem would be to include a setup.py file in your project so that you could simply pip install . to install both the requirements and the module itself.
You could also modify the PYTHONPATH environment variable to include the directory that contains the VmwareProviderBL module. For example, add to your Dockerfile:
ENV PYTHONPATH=/app