I have a file that I suspect was installed by pip.
How can I find which package installed that file?
In other words, I'm looking for a command similar to pacman -Qo filename or dpkg -S filename, but for pip. Does it exist? Or should I use some combination of pip and grep?
In that case, I don't know how to list all the file installed.
You could try with
pip list | tail -n +3 | cut -d" " -f1 | xargs pip show -f | grep "filename"
Then search through the results looking for that file.
You can use a python script like this:
#!/usr/bin/env python
import sys
try:
from pip.utils import get_installed_distributions
except ModuleNotFoundError:
from pip._internal.utils.misc import get_installed_distributions
MYPATH=sys.argv[1]
for dist in get_installed_distributions():
# RECORDs should be part of .dist-info metadatas
if dist.has_metadata('RECORD'):
lines = dist.get_metadata_lines('RECORD')
paths = [l.split(',')[0] for l in lines]
# Otherwise use pip's log for .egg-info's
elif dist.has_metadata('installed-files.txt'):
paths = dist.get_metadata_lines('installed-files.txt')
else:
paths = []
if MYPATH in paths:
print(dist.project_name)
Usage looks like this:
$ python lookup_file.py requests/__init__.py
requests
I wrote a more complete version here, with absolute paths:
https://github.com/nbeaver/pip_file_lookup
Try this!
find_pkg_by_filename(){ for pkg in $(pip list | cut -d" " -f1) ; do if pip show -f "$pkg" | grep "$1" ; then echo "=== Above files found in package $pkg ===" ; fi ; done ; }
find_pkg_by_filename somefilename
Note that if you add -q to the grep, it will exit as soon as there's a match, and then pip will complain about broken pipes.
Related
I have a new m1 laptop and I transferred all my data from my old machine onto. Including old python installations. I can't work out how to find and delete all of them so I can clean install with brew in the new version.
Brew on M1 also looks in a different directory for installs now so it can't find the old versions (inside local/bin) to uninstall them.
If brew list pythonfails, try “the brute force” method, i.e. search your entire drive for executables that are called python-×.×.
This command will not delete anything. It is up to you to manually delete the files you don’t want anymore.
find / -perm +111 -type f -or -type l 2> /dev/null | grep '/python[-0123456789\.]*$'
Explanation
find / \ # search starting at the topmost level of your drive
-perm +111 \ # executables files only
-type f -or -type l \ # only files or symlinks
2> /dev/null \ # discard errors
| grep '/python[-0123456789\.]*$' # regexp to filter names
I'm wondering if its possible to determine the ownership of the contents in <VIRTUAL_ENV_ROOT>/lib/python3.8/site-packages.
In some cases it is obvious: The package/subfolder numpy is (most likely) owned by the package numpy on PyPI.
But what if there is a weirdly_named package/subfolder where it isn't obvious, i.e., which doesn't correspond to a package on PyPI with the same name? Is there a way to determine which installed package has created a particular sub path?
Basically I'm looking for the pip equivalent of dpkg -S which can e.g. tell me that the file /etc/X11/Xsession is owned by the apt package x11-common.
For the time being I'm going for a simple zsh/bash alias:
alias pip_inspect_ownership='pip show --files $(pip freeze | grep "==" | cut -d "=" -f 1 | tr "\n" " ") | less'
The idea is:
pip show --files <package> <package> ... lists all the files owned by each package in the list, separated by a package information header.
The sub-expression $(pip freeze | grep "==" | cut -d "=" -f 1 | tr "\n" " ") filters out packages installed in editable mode, and collects the remaining into a list of all packages.
I pipe the result through less so that I can quickly search for ownership of multiple files.
I'm being asked to make a python program that parses tokens.
The usage is as follows:
$ cat input.txt | program "some text %{0} and %{1S3}" > output.txt
but the best I can manage is:
$ cat input.txt | py program.py "some text %{0} and %{1S3}" > output.txt
or if I make the script executable, remove the file extension, and I'm in the current directory
$ cat input.txt | ./program "some text %{0} and %{1S3}" > output.txt
Is there a way for me to use the first example's style of execution with a python script? Ideally I'd also be able to use it from anywhere, not necessary when pointing at the directory containing the program.
Edit: I've tried this:
Here's what I tried --
$ cd projects/token_parser/
$ ln -s grap /usr/local/bin/grap
ln: failed to create symbolic link '/usr/local/bin/grap': Permission denied
$ sudo ln -s grap /usr/local/bin/grap
[sudo] password for fosssaintdross:
$ grap
bash: grap: command not found
You need to make sure the location containing program.py is in the PATH environment variable. Alternatively you can link to it from a path that is already in the list, for example:
ln -s /path/to/program.py /usr/local/bin/program
I have a folder that will have a file name based on the python version being used.
PyICU-1.9.5-cp27-cp27m-macosx_10_6_x86_64.whl
PyICU-1.9.5-cp35-cp35m-macosx_10_6_x86_64.whl
I want to use bash to detect the 35 or 27, and then execute a pip install <filename>. I have GitHub Gist that has this hard coded, but instead of Hard coding the filename, I would like to detect the version, pick the filename with that version, and use it in a new command.
I would like to change the pip install command based on the file name. I found some code to detect the python version; what's the next piece?
#from http://stackoverflow.com/questions/6141581/detect-python-version-in-shell-script
python -c 'import sys; print(sys.version_info[:])'
How do I detect the numbers after cp, so that I can construct this:
detected 34, so:
pip installPyICU-1.9.5-cp34-cp34-macosx_10_6_x86_64.whl
#!/bin/bash
#Assuming $ver holds the version string...
ver=$(echo "$ver" | awk -F - '{print $3}' | sed 's/^..//')
For a more detailed explanation;
awk -F - '{print $3}' uses - as the field separator and returns the third field. The third field in this example is returned as cp34.
sed 's/^..//' removes the first two characters, and returns the rest, in this case 34.
Since you don't even need to extract the version from the filename, but just execute the file that is in the folder, then you can do:
cd dist/ &&
filename=$(ls|grep 'PyICU-')
pip install $filename
python -c 'import sys; print(sys.version_info[:])'
This yields e. g. (2, 7, 3, 'final', 0), while what you need for the file name would be just 27, so we have to modify the above code a little:
python -c 'import sys; print("%s%s"%sys.version_info[:2])'
what's the next piece?
Now we can set a variable to the output of the above and use it in the file name:
ver=`python -c 'import sys; print("%s%s"%sys.version_info[:2])'`
pip installPyICU-1.9.5-cp${ver}-cp${ver}-macosx_10_6_x86_64.whl
I don't have much space left, and I am tempted to install python libraries.
I would like to know how big a python library is before installing is (just to know how much extra space to request if I don't have enough).
Is there a way to do that, e.g. using pip?
This is not possible using pip as far as I know. But if the package is hosted on PyPI you can append /json to the URL to view more details, including file size.
For example, visit https://pypi.python.org/pypi/pip/json and have a look at the entries in the releases key. The size of each release tells you how big the download would be.
Note that this tells you the download size, not the size after installing (as a package may request additional dependencies for installation, after decompressing the archives, and so on). So your question may be difficult to answer, prior to installation.
If you have a requirements.txt file use this script:
#!/bin/bash
# by rubo77: https://stackoverflow.com/a/68278901/1069083
mkdir -p /tmp/pip-size
for i in $(pip-sync -n requirements.txt|tail -n +2); do
echo -ne "$i:\t"
wget --quiet -O /tmp/pip-size/$i https://pypi.org/pypi/$i/json
cat /tmp/pip-size/$i | jq 'last(.releases[])'[].size | paste -sd+ | bc > /tmp/pip-size/$i.size
cat /tmp/pip-size/$i.size | awk '{$1=$1/1024/1024; print $1,"MB";}'
done
echo -e "--------\ntotal:"
cat /tmp/pip-size/*.size | paste -sd+ | bc | awk '{$1=$1/1024/1024; print $1,"MB";}';
output example:
ninja: 1.4366 MB
numpy: 120.735 MB
opencv-python: 724.788 MB
pillow: 137.695 MB
scikit-image: 7.46707 MB
torch: 4772.73 MB
torchvision: 153.842 MB
--------
total:
5918.69 MB
This is my script i'm currently using
(based off the excellent package walking code from How to see pip package sizes installed?)
Steps :
create a python script to check all currently installed pip packages
create a shell script to create a brand new python environment and install package to test
run shell script
profit :)
Step 1
create a python script called tool-size.py
#!/usr/bin/env python
import os
import pkg_resources
def calc_container(path):
total_size = 0
for dirpath, dirnames, filenames in os.walk(path):
for f in filenames:
fp = os.path.join(dirpath, f)
total_size += os.path.getsize(fp)
return total_size
def calc_installed_sizes():
dists = [d for d in pkg_resources.working_set]
total_size = 0
print (f"Size of Dependencies")
print("-"*40)
for dist in dists:
# ignore pre-installed pip and setuptools
if dist.project_name in ["pip", "setuptools"]:
continue
try:
path = os.path.join(dist.location, dist.project_name)
size = calc_container(path)
total_size += size
if size/1000 > 1.0:
print (f"{dist}: {size/1000} KB")
print("-"*40)
except OSError:
'{} no longer exists'.format(dist.project_name)
print (f"Total Size (including dependencies): {total_size/1000} KB")
if __name__ == "__main__":
calc_installed_sizes()
Step 2
create a bash script called tool-size.sh
#!/usr/bin/env bash
# uncomment to to debug
# set -x
rm -rf ~/.virtualenvs/tool-size-tester
python -m venv ~/.virtualenvs/tool-size-tester
source ~/.virtualenvs/tool-size-tester/Scripts/activate
pip install -q $1
python tool-size.py
deactivate
Step 3
run script with package you want to get the size of
tool-size.sh xxx
say for truffleHog3
$ ./tool-size.sh truffleHog3
Size of Dependencies
----------------------------------------
truffleHog3 2.0.6: 56.46 KB
----------------------------------------
smmap 4.0.0: 108.808 KB
----------------------------------------
MarkupSafe 2.0.1: 40.911 KB
----------------------------------------
Jinja2 3.0.1: 917.551 KB
----------------------------------------
gitdb 4.0.7: 320.08 KB
----------------------------------------
Total Size (including dependencies): 1443.81 KB