I'm trying to set up the development environment for modifying a Python library. Currently, I have a fork of the library, I cloned it from remote and installed it with
pip install -e git+file:///work/projects/dev/git_project#branch#egg=git_project
However, it seems that instead of creating a symbolic link with pip install -e to the directory where I cloned my package, pip would copy the package to src/git_project in my virtual environment, making it difficult to modify it from there and push changes to my fork at the same time. Am I missing out on something or pip install -e doesn't actually make a symlink when installing from VCS?
I know that I can also do pip install -e git+git:// to install from my remote, but it makes it difficult to see real-time changes I make without pushing my code to this fork all the time.
Is there a way I can clone a fork to my local development environment, pip install a specific branch from this cloned repo, and create a symlink link to the actual git_project folder so that I can modify the package there, push changes to my remote, and at the same time import the library anywhere in my environment to see real-time changes I make on my branch without committing anything yet?
Thanks for any help!
pip install -e git+URL means "clone the repository from the URL locally and install". If you already have the repository cloned locally and want to simply install from it: just install without Git:
cd /work/projects/dev/git_project
git checkout branch
pip install -e .
Related
I've been spending a few days trying to figure out how best to build a Python Lambda bundle when using Poetry. I found a few blogs that that outline the same technique but those didn't work in my situation. The solution provided in the blogs is to use pip install to install the needed dependencies into a specific directory and zip it up.
poetry run pip install -t dist/lambda .
cd dist/lambda
zip -r ../lambda.zip .
However, this doesn't work if you use path dependencies with Poetry. You get an error from pip stating pip._vendor.pkg_resources.RequirementParseError: Invalid URL: for any local dependency.
I did run into the Poetry Bundle Plugin and it looked promising. Using it did work in that it installed the needed dependencies and the project itself into the chosen target directory.
poetry self add poetry-plugin-bundle
poetry bundle venv .venv-lambda
cd .venv-lambda/lib/python*/site-packages/
zip -r ../../../../dist/lambda.zip .
The problem with this approach is that it installs more than just the mainline dependencies, but also the dev and test dependencies. There is no option to specify which dependency group to include or exclude. There is an open issue with a PR that is waiting to be merged to resolve this. Once that is resolved, this is likely the ideal solution.
Until then, I need something different/better.
Ultimately I found this documentation from AWS for how to create a lambda archive from a Python virtual environment. Using Poetry's install command, I was able to install just the main runtime dependencies into the Poetry projects created virtual environment, including any local path based dependencies. However, this doesn't install the project itself so the source code needs to be copied in before being archived. In my case, I use a dedicated source directory/module for my code.
poetry install --only main --sync
mkdir -p dist/lambda-package
cp --recursive .venv/lib/python*/site-packages/* dist/lambda-package/
cp --recursive my_project_source_directory dist/lambda-package/
cd dist/lambda-package
zip -r ../dist/lambda.zip .
The above commands are what I use on my CI build. The local .venv directory is used because the following Poetry setting virtualenvs.in-project is set to true.
The other thing necessary for this to work is to not use editable path based dependencies, or at least just do that locally during development. Marking them as editable will not install the dependency into the virtual environment, but will rather just create a link to the project source code. This will not get picked up when creating the zip file.
This isn't perfect as there is likely more that gets bundled than necessary but it does remove any dev and test dependencies from the Poetry plugin solution. Also, because on my CI build server, I cache the installed dependencies in the virtual environment, this means at the end of my build, none of the dev or test dependencies are present to be cached and get installed on every run.
I hope this helps someone else in a similar situation.
Here's how I've been doing it:
rm -rf dist package
poetry install --only main --sync
poetry build
poetry run pip install --upgrade -t package dist/*.whl
cd package; mkdir -p out; zip -r -q out/mylambda.zip . -x '*.pyc'
Based on this answer, I can fully understand the use of:
pip install -e /path/to/locations/repo
However, I am yet to see the use of:
pip install -e .
I can understand it from the perspective of doing pip install -e /path/to/locations/repo, but from the working directory of the project dependency. But that's the only use case I can see.
In what use case would I want to install locally the same package I am now working on?
pip install -e
will just create a projekt_name.egg-info file in the venv\Lib\site-packages folder with a link to the repo location. Nothing is copied.
You can continue developing and you can access your project packages as if the repo was properly installed. No dirty sys.path.append-hacks needed.
I'm following the Prepare to Use Extensions document and I'm having issues installing CKAN into a virtual environment.
sudo apt-get install virtualenv python-pip mercurial
virtualenv /home/ubuntu/pyenv
. /home/ubuntu/pyenv/bin/activate
At first this failed, but then I found that virtualenv should be python-virtualenv.
Now I'm having issues with:
pip install -e hg+http://bitbucket.org/okfn/ckan#egg=ckan
I'm getting an error code 255, and when I visit the URL, it looks like the source has been deleted and moved to Github. I'm a beginner to Ubuntu, Python and CKAN so I'm not sure how to properly change this command to point to the new location.
I tried to use the following, but it didn't work for me:
pip install -e hg+https://github.com/ckan/ckan#egg=ckan
How should I continue to install CKAN in the virtual environment?
If you go to that URL in your browser, you will see it has a pointer to github now:
This repository has been deleted
Our apologies, but the repository "ckan" has been deleted.
It now lives at https://github.com/okfn/ckan.
So, instead you do:
pip install -e 'git+git://github.com/ckan/ckan#egg=Package'
or
pip install -e 'git+https://github.com/ckan/ckan#egg=Package'
I quoted the URL as a good practice (because # is comment character in the shell), but in this context it isn't interpreted that way, so it isn't strictly necessary.
I've got a virtualenv set up for a django app. So far I've installed all my packages via pip when the virtualenv is activated, but I now need to clone one from bitbucket. Is there a special way to do this or do I just need to open a terminal, goto venv/lib/python2.7/site-packages and run the clone command?
Here's the repository i'm trying to clone https://bitbucket.org/basti/python-amazon-product-api/src
Use the -e flag and specify a git repo:
pip install -e git://github.com/manojlds/mylib.git#egg=mylib
The url above can be bitbucket, github etc.
-e, --editable <VCS+REPOS_URL[#REV]#EGG=PACKAGE>
Install a package directly from a checkout. Source will be checked
out into src/PACKAGE (lower-case) and installed in-place (using
setup.py develop). You can run this on an existing directory/checkout
(like pip install -e src/mycheckout). This option may be provided
multiple times. Possible values for VCS are: svn, git, hg and bzr.
clone repository,
if your app is has setup.py, then run python setup.py install
when virtual env is actived.
else copy this app inside you django project and add name of it your INSTALLED_APPS in settings.py
or you can use pip install -e <repo_addr>, see doc.
I am attempting to setup a development environment on my new dev machine at home. I have just installed Ubuntu and now I am attempting to clone a remote repo from our web-server and install its dependencies so I can begin work.
So far I have manually installed virtualenv and virtualenvwrapper from pypi and edited my bash.rc appropriately to source my virtualenvs when i start my terminal. I then cloned my repo to ~/projects/project-name/websitename.com. Then I used virtualenvwrapper to mkvirtualenv env-name from ~/projects/project-name/websitename.com. This reflects exactly the file-structure/setup of the web-server I am cloning from. So far so good.
I logged into the dev server and activate the virtualenv there and use pip freeze -l > req.txt to render a dependencies list and scp to my local machine. I activate the virtualenv on my local machine, navigate to the ~/projects/project-name/websitename.com and execute pip install -r path-to-req.txt and it runs through all of the dependencies as if nothing is wrong. However, when i attempt to manage.py syncdb i get an error about not finding core django packages. What the hell? So i figure somehow Django failed to install, i run pip install Django==1.5.1 and it completes successfully. I got to setup my site again and get another error about no module named django_extensions. Okay, what the hell with it, i just installed all of these packages with pip?!
So i pip freeze -l > test.txt and cat test.txt, what does it list? Django==1.5.1, the one package I just manually installed. Why isn't pip installing my dependencies from my specified list into my virtualenv? What am I messing up here?
-EDIT-------------
Which pip gives me the path to pip in my virtualenv
I have only 1 virtualenv and it is activated
My usual workflow is to
pip freeze > someFile.txt
and then install with
pip install -r someFile.txt
So I'm certain that this should work just fine. Unfortunately I can't really tell you anything besides make sure to check that
You really are in the virtualenv that you think you are in. Make sure to run
workon yourVirtualEnvName
to activate it just in case that matters.
Make sure to check that pip is within your virtualenv.
which pip
gives me
/path/to/home/.virtualenvs/myVirtEnv/bin/pip
Sorry I can't give you a more concrete answer. I have to do this semi-regularly and I've never had a problem with it skipping dependencies. Best of luck!
Struggled with some variation of this issue not long ago; it ended up being my cluttered .bash_profile file.
Make sure you don't have anything that might mess up your virtualenv inside your .bash_profile/.bashrc, such as $VIRTUAL_ENV or $PYTHONHOME or $PYTHONPATH environment variables.
I know this is an old post, but I just encountered a similar problem. In my case the cause was that I was running the pip install command using sudo. This made the command run globally and the packages install in the global python path.
Hope that helps somebody.