I am maintaining a requirements.txt file for my Python project, including version numbers and hashes for each package. (This is so I can install with pip install --no-deps --require-hashes.)
Some packages in the list are my actual requirements (for example, Sphinx) while others are just dependencies of those requirements (for example, everything else you get when you pip install Sphinx).
I want to maintain this list of requirements separately from those dependencies. How can I do this?
At the moment I have simply split my requirements into two files (see below, simplified for readability). But I am not sure how maintainable this will be (false retentions, handling dependency updates...).
requirements.txt
# these are my actual requirements
-r dependencies.txt
Sphinx
dependencies.txt
# these are the dependencies of my actual requirements
alabaster
Babel
certifi
chardet
docutils
idna
imagesize
Jinja2
MarkupSafe
Pygments
pytz
requests
six
snowballstemmer
sphinxcontrib-websupport
urllib3
Related
Summary of Issue
I am creating a Django project and attempting to setup Auth0 integrations. Per Auth0's documentation, I installed the social-auth-app-django library which then went and also installed social-auth-core. So please note that both these libraries are installed.
Here is my py -m pip list output proving such:
Package Version
---------------------- ---------
asgiref 3.4.1
certifi 2021.10.8
cffi 1.15.0
charset-normalizer 2.0.7
colorama 0.4.4
cryptography 35.0.0
defusedxml 0.7.1
Django 3.2.9
ecdsa 0.17.0
idna 3.3
oauthlib 3.1.1
pip 21.3.1
pyasn1 0.4.8
pycparser 2.21
PyJWT 2.3.0
python-dotenv 0.19.2
python-jose 3.3.0
python-social-auth 0.3.6
python3-openid 3.2.0
pytz 2021.3
requests 2.26.0
requests-oauthlib 1.3.0
rsa 4.7.2
setuptools 57.4.0
six 1.16.0
social-auth-app-django 5.0.0
social-auth-core 4.1.0
sqlparse 0.4.2
urllib3 1.26.7
As you can see, I'm running Django 3.2.9. I am also using Python 3.10.
After following all of Auth0's steps, I went to run py manage.py migrate and receive the following ending error message:
ModuleNotFoundError: No module named 'social_django'
What I have tried
I have tried uninstalling and reinstalling the social-auth-app-django and social-auth-core libraries multiple times. I have also tried installing the libraries individually instead of just installing social-auth-app-django. I have also tried installing social-auth-app[django] (using brackets). Nothing has worked.
Every post I can find online is telling the OP to install the social-auth-app-django library and that should fix their problem. But as demonstrated, I believe I've already done that. So unless there's a different way I need to install the library, I need a different solution.
Any and all help would be greatly appreciated. Thank you!
Huge credit to #furas in the comments helping point me in the right direction.
The Solution: Reinitialize the virtual environment using python 3.10 instead of python 3.8 and reinstall necessary packages.
Explanation: The problem was that I had inadvertently created my venv using python 3.8. So even though I'd run py -m pip list successfully while inside of the 3.8 venv, all of my packages were still installed using python 3.8. So when I'd try and migrate using python 3.10, the packages weren't viewable.
Last week I installed awscli with pip3, and today I decided to uninstall it. The uninstall was successful, but pip3 list gives me the following output:
~
❯ pip3 list
Package Version
----------------- ----------
- scli
-wscli 1.16.137
astroid 2.0.4
botocore 1.12.127
certifi 2018.10.15
colorama 0.3.9
docutils 0.14
isort 4.3.4
jmespath 0.9.4
lazy-object-proxy 1.3.1
mccabe 0.6.1
pip 19.0.3
pyasn1 0.4.5
pylint 2.1.1
python-dateutil 2.8.0
PyYAML 3.13
rsa 3.4.2
s3transfer 0.2.0
setuptools 40.8.0
six 1.11.0
urllib3 1.24.1
virtualenv 16.1.0
virtualenv-clone 0.4.0
wheel 0.33.1
wrapt 1.10.11
The top two entries appear to be related to awscli. Even the version number (1.16.137) is the same as awscli's. Anyone know how to resolve this issue?
EDIT:
Found this:
/usr/local/lib/python3.7/site-packages
❯ ls
__pycache__ mccabe-0.6.1.dist-info virtualenv.py
astroid mccabe.py virtualenv_clone-0.4.0.dist-info
astroid-2.0.4.dist-info pip virtualenv_support
botocore pip-19.0.3-py3.7.egg-info wheel
botocore-1.12.130.dist-info pkg_resources wheel-0.32.2-py3.7.egg-info
certifi pylint wheel-0.33.0-py3.7.egg-info
certifi-2018.10.15.dist-info pylint-2.1.1.dist-info wheel-0.33.1-py3.7.egg-info
clonevirtualenv.py setuptools wrapt
easy_install.py setuptools-40.8.0-py3.7.egg-info wrapt-1.10.11.dist-info
isort sitecustomize.py ~-scli-1.16.137.dist-info
isort-4.3.4.dist-info six-1.11.0.dist-info ~wscli-1.16.137.dist-info
lazy_object_proxy six.py
lazy_object_proxy-1.3.1.dist-info virtualenv-16.1.0.dist-info
Safe to delete the two offending directories?
pip list takes this information from .dist-info entries in your path. You appear to have some extra names there, given your listing. Note the two entries at the end:
~-scli-1.16.137.dist-info
~wscli-1.16.137.dist-info
Simply delete these two directory entries.
Note that awscli did not create these directories, especially because pip would have used the universal wheel file to install awscli, so no setup script needed to be run when it was installed. They remind me of the Windows hidden lock files (which start with ~$), so perhaps they were created when you used another tool I'm not familiar with that may have accidentally left these lying around.
It doesn't really matter if those .dist-info entries are directories, symlinks, or files, all that pip list does is take all names that end in .dist-info then splits out version and name at the first -. You can create any phantom entry just by creating empty files:
$ mkdir demo && cd demo && virtualenv-3.8 .
# ....
$ bin/pip list # new, empty virtualenv
Package Version
---------- -------
pip 19.0.3
setuptools 41.0.0
wheel 0.33.1
$ touch lib/python3.8/site-packages/foobar-version.dist-info
$ bin/pip list # surprise package listed
Package Version
---------- -------
foobar version
pip 19.0.3
setuptools 41.0.0
wheel 0.33.1
I'm having trouble installing the proper version of PyBrain. This is a problem.
(rnine) ronaldo#ronaldo-laptop:~$ pip install https://github.com/pybrain/pybrain/archive/0.3.3.zip
Collecting https://github.com/pybrain/pybrain/archive/0.3.3.zip
Downloading https://github.com/pybrain/pybrain/archive/0.3.3.zip
\ 1.5MB 1.4MB/s
Requirement already satisfied: scipy in ./.conda/envs/rnine/lib/python3.5/site-packages (from PyBrain==0.3.1)
Installing collected packages: PyBrain
Running setup.py install for PyBrain ... done
Successfully installed PyBrain-0.3.1
As you can see, it keeps installing PyBrain 0.3.1 even though I'm pulling in the 0.3.3 folder. Why is this happening and how can I fix this.
Download this package and see its setup.py, you will see version="0.3.1", so this is a bug made by author.
You have nothing to do with it except for modifying source code by yourself.
When installing the Fuel machine learning library, I got stuck with some dependencies issues:
alvas#ubi:~$ pip install --upgrade git+git://github.com/mila-udem/fuel.gitYou are using pip version 7.1.0, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting git+git://github.com/mila-udem/fuel.git
Cloning git://github.com/mila-udem/fuel.git to /tmp/pip-xUlqCT-build
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Collecting numpy (from fuel==0.0.1)
Requirement already up-to-date: six in /usr/local/lib/python2.7/dist-packages (from fuel==0.0.1)
Collecting picklable-itertools (from fuel==0.0.1)
Downloading picklable-itertools-0.1.1.tar.gz
Collecting pyyaml (from fuel==0.0.1)
Downloading PyYAML-3.11.tar.gz (248kB)
100% |████████████████████████████████| 249kB 612kB/s
Collecting h5py (from fuel==0.0.1)
Downloading h5py-2.5.0.tar.gz (684kB)
100% |████████████████████████████████| 688kB 398kB/s
Collecting tables (from fuel==0.0.1)
Downloading tables-3.2.2.tar.gz (7.0MB)
100% |████████████████████████████████| 7.0MB 73kB/s
Complete output from command python setup.py egg_info:
/usr/bin/ld: cannot find -lhdf5
collect2: error: ld returned 1 exit status
* Using Python 2.7.6 (default, Jun 22 2015, 17:58:13)
* USE_PKGCONFIG: True
.. ERROR:: Could not find a local HDF5 installation.
You may need to explicitly state where your local HDF5 headers and
library can be found by setting the ``HDF5_DIR`` environment
variable or by using the ``--hdf5`` command-line option.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-wMS1d3/tables
Then after I have done (Installing h5py on an Ubuntu server):
sudo apt-get install libhdf5-dev
sudo HDF5_DIR=/usr/lib/x86_64-linux-gnu/hdf5/serial/ pip install h5py
And then I had to also update my cython with:
sudo pip install cython
My question is not about how to fix the installation issues but what does this command mean?
sudo HDF5_DIR=/usr/lib/x86_64-linux-gnu/hdf5/serial/ pip install h5py
What does specifying the HDF5_DIR do?
Why didn't the fuel dependencies automatically install from the:
https://github.com/mila-udem/fuel/blob/master/requirements.txt
https://github.com/mila-udem/fuel/blob/master/setup.py
What should I do to update the setup.py from fuel so that it can automatically pip install the dependencies?
You do not need to make any modifications to the setup.py from fuel. Just make sure HDF5_DIR is set correctly before updating the lib.
Explanations:
If you look at your error log, you can see that it fails at installing the h5py python lib that is a dependency of fuel. It also tells you why it failed at the end, basically it is because h5py use the C library hdf5 and it needs the headers of this lib to use it.
So the sudo apt-get install libhdf5-dev that you executed is to install the development version of this C library (you can gess that by the -dev). The dev versions install the headers of the lib and not just the compiled lib.
Then, the HDF5_DIR env variable is nedded to tell h5py setup where to find those headers.
So if you whant to update the fuel lib next time, make sure that the HDF5_DIR is set correctly and then it will update its dependencies (including h5py).
I'm trying to use apns-client 0.2.1 but I only get an error message.
22:56:01 $ pip install apnsclient==0.2.1
Collecting apnsclient==0.2.1
Could not find a version that satisfies the requirement apnsclient==0.2.1 (from versions: 0.1)
No matching distribution found for apnsclient==0.2.1
How can I install 0.2.1 version and then save it in requirements.txt? I need it in this way because my hosting use this file too.
The package is called apns-client, notice the dash.
pip install apns-client