I have a repository with an inflated requirements.txt file that I'd like to clean up. Using pipreqs I've set my requirements.txt to be a minimal set of packages need for my repository. To test this, I setup a virtualenv to install the packages and then run all my unit tests to make sure they're satisfactory.
virtualenv temp_venv --no-site-packages
source temp_venv/bin/activate
pip install -r requirements.txt
Which runs fine, but I see that a whole bunch of extra packages are collected and installed. Why? Are these identified as needed by required packages, and thus installed? If so, should I then include them in the requirements.txt?
Yes. The packages are dependencies of your dependencies​.
But no, you should not specify them directly. Automatic tools know to download dependencies recursively and it would significantly add to maintenance overhead.
This might have been because of the dependencies of your written libs in requirements.txt. For ex: if you have written scipy as requirement numpy will also be installed because scipy is dependent on numpy.
Well, for me the above answers were not the case. Pip install was installing extra packages not in requirements.txt . The solution was:
Run conda create -n venv_name and conda activate venv_name, where venv_name is the name of your virtual environment.
Run conda install pip. This will install pip to your venv directory.
Then run pip install -r requeriments.txt
The above answer was adapted from here: Using Pip to install packages to Anaconda Environment
Related
im using this guide to generate a python-environment.
pip freeze > requirements.txt
and then
python -m pip install -r requirements.txt
There are some windows-specific packages that I use but do not want to be in the requirements.txt. Is there a way to acive this?
pywin32==228
pywin32-ctypes==0.2.0
if not, can i install and ignore those modules, since the installation process will always stop at pywin and will not install the packages that come afterwards.
You can use Environment Markers in your requirements.txt
pywin32==228; platform_system=="Windows"
I am wondering if there is a way to uninstall packages in PIP including those that are not listed in the requirements.txt but which were installed as dependencies of those that are.
For example, suppose I have Django==2.1 line in requirements.txt. When running pip install -r requirements.txt, the above will instruct PIP to install many extra packages on which Django depends.
However, if I then execute pip uninstall -r requirements.txt, the Django package will be uninstalled, but PIP will retain many of its now unused dependencies.
My question is how to go about cleaning those up nicely. Is there a way to make PIP preserve and consider history explicitly? If the thing which forced PIP to install a package is being uninstalled, it appears that we should also be able to flag it to wipe its now defunct dependencies.
Take a look at pipdeptree Python package and pipdeptree --reverse some_package command in particular.
The easiest option is to use pip-autoremove. After installing it via pip, you can simply call the following from the command line:
pip-autoremove Django
Which uninstalls Django and its unused dependencies including those not listed in requirements.txt.
Trying to install django with different version that in system, it shows me:
Installing collected packages: Django
Found existing installation: Django 1.7.11
Not uninstalling django at /home/user/lib/python2.7, outside environment /home/user/webapps/v2_dev/venv
Successfully installed Django-1.8.19
But in fact there is old version
tried with different commands:
./venv/bin/pip install Django==1.8.11
pip install Django==1.8.11
UPDATED:
When I install my packages it shows:
The required version of setuptools (>=16.0) is not available,
and can't be installed while this script is running. Please
install a more recent version first, using
'easy_install -U setuptools'.
(Currently using setuptools 3.1 (/home/user/lib/python2.7/setuptools-3.1-py2.7.egg))
When I do the upgrade:
venv/bin/pip install --upgrade setuptools
Requirement already up-to-date: setuptools in ./venv/lib/python2.7/site-packages (40.5.0)
I arrived at this post while looking for how to force install something in a virtualenv despite it being already installed in the global python. This happens when the virtual env was created with --system-site-packages.
In this situation, for certain packages it may be important to have a local version within the virtualenv, even if for many other packages we can share the global versions. This is the case of pytest, for example. However, pip will refuse to install a package in the virtualenv if it can already find the most recent version in the system site.
The solution is to use pip install --ignore-installed mypackage.
Instead of installing setuptools and Django like ./venv/bin/pip install ..., try to activate your virtual environment first and install the stuff you need afterwards.
Activating virtual environment:
Go to the folder where your virtual environment is located (typically the root folder of your project) and type one of the two:
source venv/bin/activate (Unix-based systems)
venv\Scripts\activate (Windows)
This will ensure that you are not mixing packages installed in different environments.
Forcing reinstall of the packages:
Simple upgrade can be done by adding: --upgrade or -U
Forcing reinstall of the packages can be done by adding: --force-reinstall
In your case (once the environment is activated):
python -m pip install -U --force-reinstall setuptools Django
Step by step:
Deactivate and delete the old virtual environment
Create new environment using python -m virtualenv venv (python 2) or python -m venv venv (python 3)
python above is the interpreter which you want to use in your project. That's the only point where you might want to use for example python3 or some absolute path instead. Later use the code as is.
source venv/bin/activate
Activating the virtual environment
python -m pip install -U pip
If you have issue with ImportError: No module named _internal than probably you are using an old version of pip. Issue is described here
python -m pip install -U --force-reinstall -r requirements.txt
-U --force-reinstall is a bit of an overkill in case of fresh environment, but it will do no harm
Go to the place where your manage.py is located and start the server using python manage.py runserver
The problem was in Webfaction VPS
Need an empty file named sitecustomize.py in the /home/username/webapps/appName/env/lib/python2.
That empty file overrides their python customizations, one of which is to include any packages in the ~/lib/python2.7 directory.
You might need to deactivate your virtual env and activate it again for changes to take effect.
workaround but it works!
in your virtualenv directory change the properties of the pyvenv.cfg file
include-system-site-packages = True
this will cause the packages installed on the main to be used
Is there a way to install packages with pip to avoid the need to repeatedly delete files like:
pip can't proceed with requirement 'Flask-Restless==0.13.1 (from -r requirements.txt (line 2))' due to a pre-existing build directory.
location: /private/var/folders/0k/t9lwmd2j1212pxydpr6l596h0000gq/T/pip_build_jacob/Flask-Restless
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
I'm on round 4 of doing this and have no idea how long it may take to get through.
Looking at pip --help isn't helpful and man pip returns nothing.
As it has already been mentioned it's better to use virtualenv in order to avoid python package chaos on your system and install the python packages only for particular projects.
However, in your particular case you can try the following in the terminal:
pip uninstall flask-restless
Then try to run the installation again:
pip install -r requirements.txt
The options to consider during installation:
--force-reinstall
--ignore-installed
--no-deps
Add these options to the end of pip install -r requirements.txt to play with them and see if they can help.
Using
--force-reinstall
may solve your issue.
I would also recommend considering using a virtualenv for each project you are working on.
https://virtualenv.pypa.io/en/stable/
You can then activate the virtual environment for that project and pip
pip install -r requrements.txt
will install dependencies for that project in the virtual environment instead of globally. This will reduce the odds of having weird conflicts like you are having and if you do have an issue you can blow away the virtualenv and reinstall just the dependencies for that project without borking your global packages.
Quick question.
Is there a way to ensure that pip freeze > requirements.txt keeps the order in which the packages were installed? This is an issue for me because I continuously get something like this in requirements.txt:
matplotlib==1.1.1
numpy==1.6.2
So an error occurs when I try to install using pip install -r requirements.txt because numpy is a dependency of matplotlib, so I have to install manually numpy first and then rerun pip install -r requirements.txt
Is there any fix on that?
UPDATE: In response to mechmind, I installed matplotlib and numpy in Ubuntu 12.04 using pip with virtualenv --distribute myenv. After installation, I got this freeze file:
argparse==1.2.1
distribute==0.6.28
matplotlib==1.1.1
numpy==1.6.2
wsgiref==0.1.2
Then when I try to reinstall in another virtual environment I get the following error:
REQUIRED DEPENDENCIES
numpy: no
* You must install numpy 1.4 or later to build
* matplotlib.
So maybe it's dependent on the system.
Thanks!
Just tried pip with numpy and matplotlib and pip correctly resolved dependency checks - numpy built first.
Tried on old stock pip from ubuntu 10.10.
EDIT: After playing with pip and virtualenv, i realized that dependency check actually works only when that dependencies was discovered, i.e. when package was installed, removed and installed again.
So actual solution will involve reordering of packages in requrements file (for simple case when there are only two packages with wrong order, you can just reverse requirements file: sort -r | xargs pip install