I am installing packages from requirements.txt
pip install -r requirements.txt
The requirements.txt file reads:
Pillow
lxml
cssselect
jieba
beautifulsoup
nltk
lxml is the only package failing to install and this leads to everything failing (expected results as pointed out by larsks in the comments). However, after lxml fails pip still runs through and downloads the rest of the packages.
From what I understand the pip install -r requirements.txt command will fail if any of the packages listed in the requirements.txt fail to install.
Is there any argument I can pass when running pip install -r requirements.txt to tell it to install what it can and skip the packages that it cannot, or to exit as soon as it sees something fail?
Running each line with pip install may be a workaround.
cat requirements.txt | xargs -n 1 pip install
Note: -a parameter is not available under MacOS, so old cat is more portable.
This solution handles empty lines, whitespace lines, # comment lines, whitespace-then-# comment lines in your requirements.txt.
cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 pip install
Hat tip to this answer for the sed magic.
For windows users, you can use this:
FOR /F %k in (requirements.txt) DO ( if NOT # == %k ( pip install %k ) )
Logic: for every dependency in file(requirements.txt), install them and ignore those start with "#".
For Windows:
pip version >=18
import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
pip version <18
import sys
import pip
def install(package):
pip.main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
The xargs solution works but can have portability issues (BSD/GNU) and/or be cumbersome if you have comments or blank lines in your requirements file.
As for the usecase where such a behavior would be required, I use for instance two separate requirement files, one which is only listing core dependencies that need to be always installed and another file with non-core dependencies that are in 90% of the cases not needed for most usecases. That would be an equivalent of the Recommends section of a debian package.
I use the following shell script (requires sed) to install optional dependencies:
#!/bin/sh
while read dependency; do
dependency_stripped="$(echo "${dependency}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
# Skip comments
if [[ $dependency_stripped == \#* ]]; then
continue
# Skip blank lines
elif [ -z "$dependency_stripped" ]; then
continue
else
if pip install "$dependency_stripped"; then
echo "$dependency_stripped is installed"
else
echo "Could not install $dependency_stripped, skipping"
fi
fi
done < recommends.txt
Building on the answer by #MZD, here's a solution to filter out all text starting with a comment sign #
cat requirements.txt | grep -Eo '(^[^#]+)' | xargs -n 1 pip install
For Windows using PowerShell:
foreach($line in Get-Content requirements.txt) {
if(!($line.StartsWith('#'))){
pip install $line
}
}
One line PowerShell:
Get-Content .\requirements.txt | ForEach-Object {pip install $_}
If you need to ignore certain lines then:
Get-Content .\requirements.txt | ForEach-Object {if (!$_.startswith("#")){pip install $_}}
OR
Get-Content .\requirements.txt | ForEach-Object {if ($_ -notmatch "#"){pip install $_}}
Thanks, Etienne Prothon for windows cases.
But, after upgrading to pip 18, pip package don't expose main to public. So you may need to change code like this.
# This code install line by line a list of pip package
import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
Another option is to use pip install --dry-run to get a list of packages that you need to install and then keep trying it and remove the ones that don't work.
A very general solution
The following code installs all requirements for:
multiple requirement files (requirements1.txt, requirements2.txt)
ignores lines with comments #
skips packages, which are not instalable
runs pip install each line (not each word as in some other answers)
$ (cat requirements1.txt; echo ""; cat requirements2.txt) | grep "^[^#]" | xargs -L 1 pip install
For Windows:
import os
from pip.__main__ import _main as main
error_log = open('error_log.txt', 'w')
def install(package):
try:
main(['install'] + [str(package)])
except Exception as e:
error_log.write(str(e))
if __name__ == '__main__':
f = open('requirements1.txt', 'r')
for line in f:
install(line)
f.close()
error_log.close()
Create a local directory, and put your requirements.txt file in it.
Copy the code above and save it as a python file in the same directory. Remember to use .py extension, for instance, install_packages.py
Run this file using a cmd: python install_packages.py
All the packages mentioned will be installed in one go without stopping at all. :)
You can add other parameters in install function. Like:
main(['install'] + [str(package)] + ['--update'])
Related
This question already has answers here:
How to find "import name" of any package in Python?
(5 answers)
Closed 4 years ago.
I have installed a python library called PyRadiomics by doing:
pip install pyradiomics
When I do pip freeze | grep pyra in the command prompt, I see pyradiomics==2.0.0 clearly showing that the library was installed.
When I start a python interpreter and do import pyradiomics, it does not work. I realized that I need to do import radiomics.
I just happened to figure this out by luck.
How is someone supposed to know how to import a library into their python script after installing it using pip. It seems that currently, you could install a library with pip install some_name and have to do import some_other_name in your Python code. I have been using Python for a while but never came across this before which lead me to assume that some_name is always the same as some_other_name but I found out that this is not the case.
How is someone supposed to know what that other name is after installing a library called some_name
pip can list installed files for any package installed by it:
$ pip show -f packagename
Doing some simple output filtering/transforming with bash, you can easily come up with a custom command that will list all Python packages/modules in a package, for example:
$ pip show -f packagename | grep "\.py$" | sed 's/\.py//g; s,/__init__,,g; s,/,.,g'
Example output with the wheel package:
$ pip install wheel
...
$ pip show -f wheel | grep "\.py$" | sed 's/\.py//g; s,/__init__,,g; s,/,.,g'
wheel
wheel.__main__
wheel.archive
wheel.bdist_wheel
wheel.egg2wheel
wheel.install
wheel.metadata
wheel.paths
wheel.pep425tags
wheel.pkginfo
wheel.signatures
wheel.signatures.djbec
wheel.signatures.ed2551
wheel.signatures.keys
wheel.tool
wheel.util
wheel.wininst2wheel
You can even alias it to a custom shell command, for example pip-list-modules. In your ~/.bashrc (Linux) / ~/.bash_profile (MacOS):
function pip-list-modules() {
pip show -f "$#" | grep "\.py$" | sed 's/\.py//g; s,/__init__,,g; s,/,.,g'
}
Test it:
$ pip-list-modules setuptools wheel pip # multiple packages can passed at once
I want to use setuptools to create a package consisting of two files: foo.py (script) and foo.conf.
Then I want to publish the package on my devpi-server and then install the package using pip.
Suppose I that initially I have my current working directory clean
$ ls -l
total 0
Then I issue pip install (or download?) command
$ pip install -i http://mydevpi.server foo
And get a dir with my two files created
$ tree
.
|
foo
|
|\_ foo.py
|
\_ foo.conf
So questions are:
what setuptools configuration should I use?
what exact pip command should I use to install the package the way I want? Will pip install -i http://mydevpi.server --target=. do the trick?
First write somethings as setup.py in foo directory like:
import setuptools
setuptools.setup(
name='foo_pip',
version='1',
packages=[''],
url='1',
license='1',
author='1',
author_email='1',
description='1'
)
(You can use distutils or setuptools)
Then python setup.py bdist_wheel -d TARGET and there will be a whl file in target directory, copy the path.
You can now install using pip install the_wheel_file_path --prefix="the_path_to_install"
Something like this
Processing .../TARGET/foo_pip-1-py2-none-any.whl
Installing collected packages: foo-pip
Successfully installed foo-pip-1
Then use it by import foo
I'm on MacOS Sierra and have python3 and python installed via brew. Using the command python3 -m venv my_venv, I created a vitual environment for python3. I then tried to activate it with . bin/activate.fish from within my_venv. However I get the exception
$(...) is not supported. In fish, please use '(automate_stuff)'.
bin/activate.fish (line 58): if test -n "$(automate_stuff) "
^
from sourcing file bin/activate.fish
called on line 175 of file /usr/local/Cellar/fish/HEAD/share/fish/config.fish
in function '.'
called on standard input
source: Error while reading file 'bin/activate.fish'
Also I tried to install flake8 for the my_venv with the command (from within my_venv) . bin/pip -m pip install flake8. That also failed with
Missing end to balance this if statement
bin/pip (line 9): if __name__ == '__main__':
^
from sourcing file bin/pip
called on line 175 of file /usr/local/Cellar/fish/HEAD/share/fish/config.fish
in function '.'
called on standard input
source: Error while reading file 'bin/pip'
What is going on and how do I fix it? Just to repeat I run Fish Shell as my default shell.
. bin/pip -m pip install flake8.
Here you are sourcing (the . command is an alias for source) the pip script with fish. Since pip is a python script, you'll get syntax errors.
You want ./bin/pip.
$(...) is not supported. In fish, please use '(automate_stuff)'.
This is a bug in virtualenv - $() isn't valid fish syntax.
I would like to see a list of packages that depend on a certain package with PIP. That is, given django, I would like to see django-cms, django-filer, because I have these packages installed and they all have django as dependency.
Update (2021):
Since pip version 10 you can do:
pkg=httplib2
pip show $pkg | grep ^Required-by
or for bash
pkg=httplib2
grep ^Required-by <(pip show $pkg)
so you could create an alias like:
alias pyreq='pip show $pkg | grep ^Required-by'
and querying by:
pkg=httplib2 pyreq
which should give (for ubuntu):
Required-by: lazr.restfulclient, launchpadlib
Original:
Quite straightforward:
pip show <insert_package_name_here>| grep ^Requires
Or the other way around: (sorry i got it wrong!)
for NAME in $(pip freeze | cut -d= -f1); do REQ=$(pip show $NAME| grep Requires); if [[ "$REQ" =~ "$REQUIRES" ]]; then echo $REQ;echo "Package: $NAME"; echo "---" ; fi; done
before that set your search-string with:
REQUIRES=django
essentially you have to go through the whole list and query for every single one. That may take some time.
Edit:
Also it does only work on installed packages, I don't see pip providing dependencies on not installed packages.
I know there's already an accepted answer here, but really, it seems to me that what you want is to use pipdeptree:
pip install pipdeptree
pipdeptree --help
pipdeptree -r -p django
Since version 10, pip show also includes a "Required-by" entry. So just
pip show <package_name>
is enough nowadays. Or possibly
pip show <package_name> | grep ^Required-by
if you want to get just that single line for a script or whatever.
This one, for pip older than 1.3.1 will list all packages and it's dependencies, you can parse its output with any scripting language, for Requires ... django inclusions:
pip freeze | cut -f 1 -d'=' | xargs -L1 pip show
For example, following snippet:
import os
import re
package = 'numpy'
regex = re.compile('.*{}($|,).*'.format(package))
def chunks(l, n): return [l[i:i+n] for i in range(0, len(l), n)]
cmd = "pip freeze | cut -f 1 -d'=' | xargs -L1 pip show"
packages = os.popen(cmd).read()
pkg_infos = chunks(packages.splitlines(), 5)
print '\n'.join(x[1][6:] for x in filter(lambda x: regex.match(x[-1]), pkg_infos))
outputs pandas on my system.
One liner based on requirements.txt. In this example I'm looking for funcsigs reverse dependency, and found mock. Just change funcsigs by something else.
cat requirements.txt | grep -v git | sed 's/==.*//' | xargs -I % echo 'pip show % 2>/dev/null | grep Requires | grep -q funcsigs && echo %' | sh
When I attempt to run pyramid
[~/env/MyStore]# ../bin/pserve development.ini
It will show the following error
File "/home/vretinfo/env/lib/python3.2/site-packages/Paste-1.7.5.1-py3.2.egg/paste/fileapp.py", line 14, in <module>
from paste.httpheaders import *
File "/home/vretinfo/env/lib/python3.2/site-packages/Paste-1.7.5.1-py3.2.egg/paste/httpheaders.py", line 140, in <module>
from rfc822 import formatdate, parsedate_tz, mktime_tz
ImportError: No module named rfc822
How should I resolve this?
This is what I did to install
$ mkdir opt
$ cd opt
$ wget http://python.org/ftp/python/3.2.3/Python-3.2.3.tgz
$ tar -xzf Python-3.2.3.tgz
$ cd Python-3.2.3
./configure --prefix=$HOME/opt/Python-3.2.3
$ make;
$ make install
$ cd ~
$ wget http://python-distribute.org/distribute_setup.py
$ pico distribute_setup.py
* change first line to opt/Python-3.2.3/python
$ opt/Python-3.2.3/bin/python3.2 distribute_setup.py
$ opt/Python-3.2.3/bin/easy_install virtualenv
$ opt/Python-3.2.3/bin/virtualenv --no-site-packages env
$ cd env
$ ./bin/pip install passlib
$ ./bin/pip install pyramid_beaker
$ ./bin/pip install pyramid_mailer
$ ./bin/pip install pyramid_mongodb
$ ./bin/pip install pyramid_jinja2
$ ./bin/pip install Werkzeug
$ ./bin/pip install pyramid
$ ./bin/pcreate -s pyramid_mongodb MyShop
$ cd MyShop
$ ../bin/python setup.py develop
$ ../bin/python setup.py test -q
Ok, I've done some searching around on pyramid docs ( http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/paste.html ).
It states on the 3rd paragraph
"However, all Pyramid scaffolds render PasteDeploy configuration files, to provide new developers with a standardized way of setting deployment values, and to provide new users with a standardized way of starting, stopping, and debugging an application."
So I made changes to development.ini and replaced
[server:main]
use = egg:waitress#main
and in setup.py, I added 'waitress' into the requires array
Next step, I did to totally remove all things related to Paste, in /home/vretinfo/env/ECommerce/,
$ rm -rf Paste*;rm -rf paste*
After this, I tried running test -q again, this is the stack trace:
[~/env/ECommerce]# ../bin/python setup.py test -q
/home/vretinfo/opt/Python-3.2.3/lib/python3.2/distutils/dist.py:257: UserWarning: Unknown distribution option: 'paster_plugins'
warnings.warn(msg)
running test
Checking .pth file support in .
/home/vretinfo/env/ECommerce/../bin/python -E -c pass
Searching for Paste>=1.7.1
Reading http://pypi.python.org/simple/Paste/
Reading http://pythonpaste.org
Best match: Paste 1.7.5.1
Downloading http://pypi.python.org/packages/source/P/Paste/Paste-1.7.5.1.tar.gz#md5=7ea5fabed7dca48eb46dc613c4b6c4ed
Processing Paste-1.7.5.1.tar.gz
Writing /tmp/easy_install-q5h5rn/Paste-1.7.5.1/setup.cfg
Running Paste-1.7.5.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-q5h5rn/Paste-1.7.5.1/egg-dist-tmp-e3nvmj
warning: no previously-included files matching '*' found under directory 'docs/_build/_sources'
It seems like paste is needed for pyramid1.4 for some reason. Perhaps someone have some insights on this.
I have managed to solve this issue through the folks in IRC#pyramid. I'm posting the solution in here in case someone encounters in future. I've tested this out in Python3.2, works ok now.
After running ./bin/pcreate -s <...>
in the folder of the project, development.ini
Change the following:
1. in the 1st line, rename [app:<Project>] to [app:main]
2. [server:main]
If it is egg:Paste#http, change it to
use = egg:waitress#main
3. remove [pipeline:main] and its section
in the same folder, make the necessary changes to setup.py:
1. requires = [....], add in waitress into the array, remove WebError from the array
2. remove paster_plugins=['pyramid']
Then finally, run
$ ../bin/python setup.py develop
Paste will not be installed or checked if it exists.