I have a custom Python package (call it MyProject) on my filesystem with a setup.py and a requirements.txt. This package needs to be used by a Flask server (which will be deployed on AWS/EC2/EB).
In my Flask project directory, I create a virtualenv and run pip install -e ../path/to/myProject.
But for some reason, MyProject's upstream git repo shows up in pip freeze:
...
llvmlite==0.19.0
-e git+https://github.com/USERNAME/MYPROJECT.git#{some-git-hash}
python-socketio==1.8.0
...
The reference to git is a problem, because the repository is private and the deployment server does not (and should not, and will never) have credentials to access it. The deployment server also doesn't even have git installed (and it seems extremely problematic that pip assumes without my permission that it does). There is nothing in MyProject's requirements.txt or setup.py that alludes to git, so I am not sure where the hell this is coming from.
I can dupe the project to a subdirectory of the Flask project, and then put the following in MyFlaskProject's requirements.txt:
...
llvmlite==0.19.0
./MyProject
python-socketio==1.8.0
...
But this doesn't work, because the path is taken as relative to the working directory of the pip process when it is run, not to requirements.txt. Indeed, it seems pip is broken in this respect. In my case, EC2 runs its install scripts from some other directory (with a full path to requirements.txt specified), and as expected, this fails.
What is the proper way to deploy a custom python package as a dependency of another project?
To install your own python package from a git repo you might want to check this post
To sort out the credential issue, why not having git installed on the EC2? You could simply create an ssh key and share it with MyProject repository.
I am using this solution on ECS instances deployed by Jenkins (with Habitus to hide Jenkin's ssh keys while building the image) and it works fine for me!
Related
I am completely new to Azure. I recently deployed my Python Script on Azure Functions (HTTP). It worked completely fine for me. The problem I faced is when my Python script needed some packages to be installed like (pandas, psycopy2). Although I put them in requirements.txt file. And after deployment requirements.txt is stored in root directory (same as of host.json) but I am getting import error.
I don't really know how to install these packages in azure function.
Any help would be really really appreciated.
I tried deploying python script using multiple techniques but none of them worked for me, I just have one python script and I need to install requirement.txt file in azure function.
Please help me with this problem.
I've installed a package ('requests') and ran http trigger function locally by creating a new function app with python3.9. I was able to deploy it to Azure and triggered successfully without any error.
Note: Make sure that while adding any package in requirements.txt file, install package in the project directory folder itself by giving:
pip install --target="<local project directory path>/.python_packages/lib/site-packages" -r requirements.txt
Here, I've taken requests == 2.19.1 latest version package and imported in init.py
requirements.txt:
azure-functions
requests==2.19.1
While executing locally, I received the desired outcome:-
Deployed to Azure:
requirements.txt file after deploying it to Azure:
Received "200 Ok" response after test/run:
You can check here Refer MSDoc
I am trying to use Google Cloud Platform (specifically, the Vision API) for Python with AWS Lambda. Thus, I have to create a deployment package for my dependencies. However, when I try to create this deployment package, I get several compilation errors, regardless of the version of Python (3.6 or 2.7). Considering the version 3.6, I get the issue "Cannot import name 'cygrpc'". For 2.7, I get some unknown error with the .path file. I am following the AWS Lambda Deployment Package instructions here. They recommend two options, and both do not work / result in the same issue. Is GCP just not compatible with AWS Lambda for some reason? What's the deal?
Neither Python 3.6 nor 2.7 work for me.
NOTE: I am posting this question here to answer it myself because it took me quite a while to find a solution, and I would like to share my solution.
TL;DR: You cannot compile the deployment package on your Mac or whatever pc you use. You have to do it using a specific OS/"setup", the same one that AWS Lambda uses to run your code. To do this, you have to use EC2.
I will provide here an answer on how to get Google Cloud Vision working on AWS Lambda for Python 2.7. This answer is potentially extendable for other other APIs and other programming languages on AWS Lambda.
So the my journey to a solution began with this initial posting on Github with others who have the same issue. One solution someone posted was
I had the same issue " cannot import name 'cygrpc' " while running
the lambda. Solved it with pip install google-cloud-vision in the AMI
amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2 instance and exported the
lib/python3.6/site-packages to aws lambda Thank you #tseaver
This is partially correct, unless I read it wrong, but regardless it led me on the right path. You will have to use EC2. Here are the steps I took:
Set up an EC2 instance by going to EC2 on Amazon. Do a quick read about AWS EC2 if you have not already. Set one up for amzn-ami-hvm-2018.03.0.20180811-x86_64-gp2 or something along those lines (i.e. the most updated one).
Get your EC2 .pem file. Go to your Terminal. cd into your folder where your .pem file is. ssh into your instance using
ssh -i "your-file-name-here.pem" ec2-user#ec2-ip-address-here.compute-1.amazonaws.com
Create the following folders on your instance using mkdir: google-cloud-vision, protobuf, google-api-python-client, httplib2, uritemplate, google-auth-httplib2.
On your EC2 instance, cd into google-cloud-vision. Run the command:
pip install google-cloud-vision -t .
Note If you get "bash: pip: command not found", then enter "sudo easy_install pip" source.
Repeat step 4 with the following packages, while cd'ing into the respective folder: protobuf, google-api-python-client, httplib2, uritemplate, google-auth-httplib2.
Copy each folder on your computer. You can do this using the scp command. Again, in your Terminal, not your EC2 instance and not the Terminal window you used to access your EC2 instance, run the command (below is an example for your "google-cloud-vision" folder, but repeat this with every folder):
sudo scp -r -i your-pem-file-name.pem ec2-user#ec2-ip-address-here.compute-1.amazonaws.com:~/google-cloud-vision ~/Documents/your-local-directory/
Stop your EC2 instance from the AWS console so you don't get overcharged.
For your deployment package, you will need a single folder containing all your modules and your Python scripts. To begin combining all of the modules, create an empty folder titled "modules." Copy and paste all of the contents of the "google-cloud-vision" folder into the "modules" folder. Now place only the folder titled "protobuf" from the "protobuf" (sic) main folder in the "Google" folder of the "modules" folder. Also from the "protobuf" main folder, paste the Protobuf .pth file and the -info folder in the Google folder.
For each module after protobuf, copy and paste in the "modules" folder the folder titled with the module name, the .pth file, and the "-info" folder.
You now have all of your modules properly combined (almost). To finish combination, remove these two files from your "modules" folder: googleapis_common_protos-1.5.3-nspkg.pth and google_cloud_vision-0.34.0-py3.6-nspkg.pth. Copy and paste everything in the "modules" folder into your deployment package folder. Also, if you're using GCP, paste in your .json file for your credentials as well.
Finally, put your Python scripts in this folder, zip the contents (not the folder), upload to S3, and paste the link in your AWS Lambda function and get going!
If something here doesn't work as described, please forgive me and either message me or feel free to edit my answer. Hope this helps.
Building off the answer from #Josh Wolff (thanks a lot, btw!), this can be streamlined a bit by using a Docker image for Lambdas that Amazon makes available.
You can either bundle the libraries with your project source or, as I did below in a Makefile script, upload it as an AWS layer.
layer:
set -e ;\
docker run -v "$(PWD)/src":/var/task "lambci/lambda:build-python3.6" /bin/sh -c "rm -R python; pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit" ;\
pushd src ;\
zip -r my_lambda_layer.zip python > /dev/null ;\
rm -R python ;\
aws lambda publish-layer-version --layer-name my_lambda_layer --description "Lambda layer" --zip-file fileb://my_lambda_layer.zip --compatible-runtimes "python3.6" ;\
rm my_lambda_layer.zip ;\
popd ;
The above script will:
Pull the Docker image if you don't have it yet (above uses Python 3.6)
Delete the python directory (only useful for running a second
time)
Install all requirements to the python directory, created in your projects /src directory
ZIP the python directory
Upload the AWS layer
Delete the python directory and zip file
Make sure your requirements.txt file includes the modules listed above by Josh: google-cloud-vision, protobuf, google-api-python-client, httplib2, uritemplate, google-auth-httplib2
There's a fast solution that doesn't require much coding.
Cloud9 uses AMI so using pip on their virtual environment should make it work.
I created a Lambda from the Cloud9 UI and from the console activated the venv for the EC2 machine. I proceeded to install google-cloud-speech with pip.That was enough to fix the issue.
I was facing same error using goolge-ads API.
{
"errorMessage": "Unable to import module 'lambda_function': cannot import name'cygrpc' from 'grpc._cython' (/var/task/grpc/_cython/init.py)","errorType": "Runtime.ImportModuleError","stackTrace": []}
My Lambda runtime was Python 3.9 and architecture x86_64.
If somebody encounter similar ImportModuleError then see my answer here : Cannot import name 'cygrpc' from 'grpc._cython' - Google Ads API
I'm trying to develop a Python library which will eventually be put on PyPI.
It's a library I use in another project which pulls it from PyPI.
I have unit-tests for the library in its own project repository. But I mainly test the library in use through the main application.
I was previously "publishing" the library locally, using
pip install -e
so that the main project in another repository can pull it from local packages and I can test it in context.
But now I'm moving to pipenv. And I want to be able to do the same. But if I put the dependency in the Pipenv file, it seems to try to pull from the real PyPI, not my local store.
How do I set up this workflow with Pipenv?
Pipenv can install packages from various sources, not only from PyPI. The CLI usage is very similar to pip, which is a feature of pipenv. You can pass a local path or a URL with CVS prefix to pipenv install. Pipenv will add the package to Pipfile accordingly.
CLI Usage
First go to the project folder (which contains the Pipfile) of your main application. Then run
$ pipenv install --dev -e "/path/to/your/local/library"
If the library is version controlled by Git or SVN, you can also use an URL like this:
$ pipenv install --dev -e git+https://github.com/your_user_id/libraryname#develop
If the Git repository for your library is stored locally, use file:// instead of https://github.com. Other protocols like FTP and SSH are also supported.
The above command will pull the package from the source, install it and modify the Pipfile in the current folder to include the package.
Pipfile usage
Usually you do not need to modify the Pipfile directly. For advanced settings in the pipfile, please see the Pipfile's specs. Below are some example entries to pipfile
[dev-packages]
mylibrary = { git = 'https://github.com/xxx/mylibrary.git', ref = '0.0.1', editable = true }
"e1839a8" = {path = "/path/to/your/local/library2", editable = true}
"e51a27" = {file = "/path/to/your/local/library1/build/0.0.1.zip"}
Setup a private PyPI index
Although it would be overkill, just to be complete, setting up a private PyPI server can also work.
I'm using Ubuntu 15.04 with Python 2.7, pip 1.5.6 and virtualenv 1.11.6.
I will create a Django project inside /var/www/html/project_name (and work in that directory) for use it with Apache2.
I created a virtual environment named venv inside the project_name folder for syncing purposes.
With the virtual environment activated, I just cant't run pip install django because I get 'Permission denied' message. So I try the command sudo pip install django, but that will install Django globally.
So, running which pip, I get /var/www/html/project_name/venv/bin/pip.
But running sudo which pip, I get /usr/bin/pip.
Does someone get any idea about how it is possible to install Django (or any other package) inside the virtual environment?
PS: I know it's possible to run sudo venv/bin/pip install django, but it doesn't seem very useful.
Your trouble is presumably that you don't have write access to this /var/www/html/project_name directory. Your choices are:
Change the permissions, recursively, so that you do have permissions to write to that directory.
Run the following commands:
.
$ sudo su
# . venv/bin/activate
# pip install
Just to add what everyone's been saying and I sort of skimmed over: NEVER EVER put sensitive things (that includes your django project) under the document root. You can store it under something like /var/www/scripts or something, but don't put it under the document root.
Another way of deploying is using something like gunicorn as the "main" webserver and then just having whatever world visible webserver (like apache) reverse proxy to gunicorn. I've done this with nginx and it's fairly easy to setup, the only down side is then you have to setup something extra in your system's init scripts to start up gunicorn.
I have just started learning Django, and I'm facing some problems regarding the 'copy' of the projects.
I have two computers and I would like to use both computers for my development. When I was learning PHP (at that time I didnt even know how to use Github), all I had to do was set up a webserver on both computers, and upload the whole files through Google Drive (from one computer) and then download it from the other computer.
However, it seems to me that Django is somewhat different since it is a framework and has a lot of setting ups before starting a project (including virtual environment; I am following a Youtube tutorial and it says that I would be better off if I used virtualenv). I thought it wouldn't work just by downloading the whole project folder to the other computer.
Currently, I have uploaded the whole virtual environment folder on Github.
So, to list my questions,
When downloading it on the other computer, should I setup the virtual environment on that computer and then download the folder?...
Is there any way that I can only sync or commit the files that has been changed in the project automatically?
(That is, i have to change many files in django projects(views, urls, settings... etc) but it would be hard to remember all the files that i have changed and then seperately commit those ones)
Any help would be appreciated.
Edit: Consider using pipenv
I suggest that you also install virtualenvwrapper (here). virtualenvwrapper keeps all files except your project at another location so your project directory contains only your files and you can safely use git add --all.
After its installed, do:
$ mkdir my-project; cd my-project
$ mkvirtualenv my-env-name
$ pip install django <more-good-stuff>
$ pip freeze > requirements.txt
$ git init; git add --all; git commit -m "Initial Commit"
... push to github ...
Now go to other machine, and install virtualenv and virtualenvwrapper
$ git clone <url> my-project; cd my-project
$ mkvirtualenv my-env-name
$ pip install -r requirements.txt
... continue your work, commit and push push and win at life :D
you usually don't want to commit everything blindly. It's better if you use git status to see all the files that you changed and then git add those that you want to commit. Once you've verified that you really want to commit all files, you can simply git add --all (more here). And then you git commit.
And, yes, virtualenv is the way to go. You want to create a virtualenv for your project and have all your dependencies in a requirements.txt file. This will allow to have only your source code and no libraries in your git repo, making it much cleaner. This also can allow you to have a set of verified libraries in production, and if you want to try out a new library you can just install it in your local virtualenv. Or even have two virtualenvs and switch, or whatever, and it does not mess your code repo or other machines' installations.