I have upraded my Linux distro recently. Python 3.5 was replaced by Python 3.6.
All site packages I have installed with pip3 are still in the /usr/lib/python3.5/site-packages directory and Python does not find them there now, because it looks in .../python3.6/site-packages obviously.
I see the directory contents and I could manually install them again, but that does not look to me like the right way to do it. I could move the contents to the new directory, but again, this seems to me incorrect either.
How am I supposed to handle it properly?
Should I have prepared a pip3 freeze list before the upgrade?
I tried to search, but the keywords are probably too general and got many unrelated answers.
Python 3.5 was replaced by Python 3.6. But you still have the backup option of using python 3.5.
If you want to use python 3.6 you will have to reinstall all pip packages again for python 3.6. And it makes sense.
Say you were changing from 2.7 to 3.5. You would want to preserve both the environments separately. Hence 3.6 environment is different from 3.5.
A quick way to do this would be to pip freeze for 3.5 and then install those dependencies for 3.6.
pip freeze > reqs.txt
upgrade
pip install -r reqs.txt
Since you don't have this option anymore, first try and list all packages in your python3.5
for that you can install pip3.5 as answered by #kabanus.
sudo apt-get install python3=3.5.1*
sudo python3.5 easy_install.py pip
Also it's advised to use virtual environment per project so you can maintain separate environments for each of them.
I just hit the same problem upgrading from Python 3.6 to Python 3.7, I forgot to run pip freeze before I upgraded to Python 3.7. The solution that worked is to specify the --path option as the old site-packages/ directory (which was not deleted):
pip3 freeze --path /usr/local/lib/python3.6/site-packages/ > python3.6_requirements.txt
pip3 install -r python3.6_requirements.txt
This would have made things simpler for you to reinstall. Checkout the description. Using freeze you could have done something like:
$ env1/bin/pip3 freeze > requirements.txt
$ env2/bin/pip3 install -r requirements.txt
Generally the recommended method is you use a virtualenv for site packages, so you don't litter your installation areas, but TBH it never broke something for me. Another option is to check if the linux distribution has the package available for proper retrieval, as in:
sudo apt-get install python3-<somemodule>
This is what I prefer - and could have been upgraded with the distro. As for what to do now, If you really don't want to re-install everything properly you could try to cp /usr/lib/python3.5/site-packages/* /usr/lib/python3.6/site-packages. The differences between versions are not so great such that I believe most packages would work off the bat. You may have to sed to replace python3.5 with python3.6 in all files there though. Forgot delete all pyc files if you do this.
Python modules are self contained enough that if something is broken it can be handled per package, and the site packages are self contained completely, so you could always just remove everything and re-install.
A final note - you can try and install pyton3.5/pip3.5 for your linux, and then do the freeze thing. If there is no package you could install manually (whl or such) or compile a stand alone and configure the site path properly. If you want to keep things on a global site package directory or migrate to virtualenv this may be the safest option.
Related
For some context, my project uses python version 3.8 and has some packages in a requirements.txt file. I'm trying to upgrade the python version to 3.10 and also all the packages in my requirements.txt file to the latest possible version such that there are no conflicts between dependencies. Is there a way to do this?
It is a bit hard to say what will work best for you since you've given no info on your OS, and that thing you need is done differently on macOS and Windows.
If you're on macOS (this one could also work for Linux I guess), the best way to manage python versions is with pyenv and to update python packages with pip-review, both of which you can install via brew.
So first, you want to create a list of all your packages installed. To do that, you type pip3 freeze > requirements.txt in the terminal. The command creates a txt file in your home folder. In the file, you can see the names of all your packages installed, and it will look like this:
astroid==2.11.6
async-generator==1.10
attrs==21.4.0
autopep8==1.6.0
beautifulsoup4==4.11.1
Secondly, to update your python version, you'll need to type pyen install -l in the terminal to get a list of all currently available versions of python. To install the one you need, let's say it's the most recent at the moment 3.10.5, type pyenv install 3.10.5. To set this version as a default one, type pyenv global 3.10.5. If you don't need your previous version of python, just type pyenv uninstall and the number of the version you want to delete after a space (it's really just like installing one).
Thirdly, to move all your previous packages in the newly installed python version, just type pip3 install -r requirements.txt in the terminal. The command will get the packages of the same versions you had before.
Lastly, to upgrade the packages to the most recent version, type pip-review --auto. It will fetch the latest versions available and install them with no more commands from you.
Now here's why I think pip-review works better than anything else I've tried so far: in case some dependencies are broken (i. e. a package you are using needs another package of an older version than you've just upgraded to), it will point that out in the terminal, and you could get the right one.
yes, you can do that just you need to update your requirements file.
and run the requirements file. it will automatically update your all files.
You can update to Python3.10 and drop the previous requirements.txt into your project root directory. From there you can run pip install -r requirements.txt to install all the packages.
You can also run pip install -U <package-name> to update any package to the latest version.
Also, to re-generate a new requirements.txt file, run pip freeze > requirements.txt.
Pretty sure there is no such thing as automatically updating dependencies in such way that there are no conflicts. Some dependencies may have changes that are not backward compatible and a complete test of all your project's features will be necessary whenever you change the version of anything.
There is also an open-source project called Renovate that can help you maintain your packages and update them to recent versions
I have installed some modules using pip and whenever I try to import them in Python I am told that no module exists. I think there is something wrong with my paths. This is some terminal output, does anyone know how I can fix this?
Nicks-MacBook-Pro:~ nickporter$ which python
/usr/bin/python
Nicks-MacBook-Pro:~ nickporter$ which pip
/usr/local/bin/pip
Nicks-MacBook-Pro:~ nickporter$
You can use pip freeze to find the packages installed.
I do not know whether you use a virtualenv. If you use it, you have to source it, to activate it.
It looks like you probably have multiple installations of Python or pip (or both). First, determine if you have multiple Python binaries installed and decide on the one you want to use by default.
You clearly have /usr/bin/python...check if there is another one in /usr/local/bin/python. If you do have one in /usr/local/bin, I'm guessing you installed it with Homebrew. Pick the Python you want to use (if there are multiple) by playing with your PATH. If you want to use /usr/local/bin/python (assuming it exists) ensure /usr/local/bin comes before /usr/bin in your PATH.
Once you do that, I would remove your current pip. After you remove it, look for it again with which pip, just to make sure you don't have any old one laying around. Once you've totally removed it, reinstall.
If you're installing with Homebrew, I'd stick to using the Homebrew python and pip. If you're not using Homebrew I'd install pip using: https://bootstrap.pypa.io/get-pip.py
curl -q https://bootstrap.pypa.io/get-pip.py | sudo python
You have homebrew version of python shadowed by the system python, but because system python doesn't have pip, a pip you see is actually a brew one.
I found, that the reason for this is that brew by default names python2.7 as python2 in /usr/local/bin (probably for preventing possible compatibility issues with some system services (?)) , i.e. you may try to run python2 or try to ln /usr/local/bin/python2 /usr/local/bin/python and restart the term session.
Make sure that /usr/local/bin goes before /usr/bin and /bin in your $PATH.
BTW. brew info {package} often contains recommendations for proper configuration of packages.
I need to work with Both Python 2.7.12 and python 3.5.2 simultaneously on my Ubuntu 16.04.1 LTS. Python 3 came pre-installed so I've no idea where it sits, in terms of path to the directory, while python 2 sits in /usr/local/lib/python2.7/.
I found lots of questions on SO and on askubuntu about how to install but nothing about how to use them separately, installing different libraries, and what should I avoid or be careful of, if I maintain this dual python thing for the long term? For example, I usually run pip install to install a library and I can check that its installed in my python2 directory but how do I install the same package for my python3 without conflicts? Something like: python3 pip install <package> ?? Where is the default python3 installed? And how do I call python3 for paths where python is not part of the command for example: pip freeze, sudo-apt get, etc.?
PS: I've not officially worked with Virtualenv but I've been informed that is usually good for isolating projects within a python language version, rather than isolating two different language versions from each other.
Please let me know.
Thanks
This is absolutely no problem, as Python does that for you. You don't need a virtualenv at all.
If you use Ubuntu packages, make sure you use the python3- versions for Python 3, and the normal python- versions for Python 2.
For example, python3-numpy and python-numpy.
If you use pip to install extra packages, you an either use the pip script with the version number appended: pip2.7 or pip3.5, or, my preferred method, call pip as a module for the respective Python executable:
python2.7 -m pip install <whatever>
and
python3.5 -m pip install <whatever>
Other than that, there should not be any issue: Python stores the packages in completely separate directories, and each Python executable only uses its respective directive.
Do not fiddle around with PYTHONPATH, unless you really know what you're doing. This has the danger of setting your PYTHONPATH to a directory with Python 2.7 modules and then using Python 3.5 to run things.
If you start from scratch, you may need to install pip first.
For the system Python(s), use the relevant package:
sudo apt install python-pip
sudo apt install python3-pip
For your locally installed Python(s), use the built-in bootstrapper module:
pythonx.y -m ensurepip
Note on the OS-installed Python executables:
Python 3.5 lives at /usr/bin/python3.5, Python 2.7 (the OS one) at /usr/bin/python2.7.
You could even use the OS 2.7 one next to your locally installed /usr/local/bin/python2.7 (and confuse yourself when a package can't be found because you used the wrong one).
Or install Python 3.6 next to Python 3.5 (provided you've used make altinstall, so python3 doesn't get overwritten).
This is also why you don't really want to run pip (or even pip2.7) as is: pip2.7 may get you the system one, instead of the one in /usr/local/bin/pip2.7, depending on your PATH.
(The same goes for the python2.7 executable, so if you need to specify the full path /usr/local/bin/python2.7 to run that one (or have an alias), the same holds for pip2.7. If, on the other hand, /usr/local/bin is first on your PATH, you should in principle never run into the same pip and python executables.)
I am using tox to manage some testing environments. I have a dependency (backports.ssl-match-hostname) that I cannot download using the latest version of pip, so I need to revert back to pip 8.0.3 to allow the install to work.
I have included the 8.0.3 version of pip inside my tox.ini file for dependencies.
deps=
pip==8.0.3
However, when I run
source .tox/py27/bin/activate
and enter the virtual testing environment, and then run
pip --version
I end up with
8.1.2
However, outside of my tox environment, when I run the same command, I get
8.0.3
Is there anything special that tox does when grabbing pip? Why am I not able to specify the version of pip that I want to use as a dependency?
EDIT : to add to this, it seems as though I am able to grab the dependency pip==8.0.3, but for the other dependencies, they are still running from the command launched with pip==8.1.2
So, I need to be able to grab pip==8.0.3 first, and then once installed, grab everything else. Still unsure why tox is starting with pip==8.1.2
This was apparently the result of the "virtualenvs" python package containing a pre-selected group of python packages that it refers to, one of which was the latest and greatest pip.
I don't know if this is the preferred way of doing this, but I found success by running
pip uninstall virtualenv
And then reinstalling with the version that worked
pip install virtualenv==15.0.1
With the "correct" version of virtualenv in place, I was able to run my tox command
source .tox/py27/bin/activate
and see the desired version of pip
pip --version
pip 8.0.3
A workaround for this is here: https://github.com/pypa/pip/issues/3666
Although to make it work I had to write "pip install pip==8.1.1" in my script. So to recap:
Add a pip.sh script to your project:
#!/bin/bash
pip install pip==8.1.1
pip install "$#"
Add to your tox.ini:
install_command = {toxinidir}/pip.sh {opts} {packages}
I've recently hit this problem. I've had it for a while but it just didn't register because I had such occasional failures with Python 2/3 code. Another way that this can happen is, if like me, you change the virtualenv between different Python versions and don't clean up.
Check /bin or /Scripts to see whether python2 points to python. If the virtualenv is Python 3 then this will mean that python2 actually calls Python 3. Vice versa, of course, if you the virtualenv is Python 2 and you want to test Python 3 code.
New versions of virtualenv reach out to download the latest pip, setuptools, and wheel -- you can disable this behavior when running through tox with the tox-virtualenv-no-download package See: https://github.com/asottile/tox-virtualenv-no-download#wait-why
I recently installed python 2.7.2 on my Mac running OSX 10.6.8. Previously, I had version 2.6. I set my path in .bash_profile as follows:
export PATH=/usr/local/bin:$PATH
export PATH=/usr/local/share/python:$PATH
so that when I run python it will refer to my new installation. It does.
I would also like to use pip with my new installation, but the problem is that I already have the current version of pip installed at
/usr/local/bin/pip.
I tried to re-install pip with:
easy_install pip
But, of course this does not put pip in the desired new directory
/usr/local/share/python/pip
but simply refers to the existing version in /usr/local/bin/pip.
Can someone tell me how to fix this?
I would like to then use pip to install NumPy and SciPy in the correct directory (I was having trouble getting the SciPy installation to work with my old version of python, hence the new install).
If you'd like, you can visit the website where I found instructions for installing python 2.7, creating/updating my .bash_profile, installing pip, and NumPy and SciPy. Might provide some insight, or I'm happy to give more details if needed. Thanks!
http://www.thisisthegreenroom.com/2011/installing-python-numpy-scipy-matplotlib-and-ipython-on-lion/#python
Install distribute as per the instructions at http://pypi.python.org/pypi/distribute .
Make sure you specify the full path to the python executable (/usr/local/share/python/python or smth in your case).
$ curl -O https://svn.apache.org/repos/asf/oodt/tools/oodtsite.publisher/trunk/distribute_setup.py
$ /usr/local/share/python/python distribute_setup.py
Then you should have /usr/local/share/python/easy_install.
After that, run:
$ /usr/local/share/python/easy_install pip
Then you should have /usr/local/share/python/pip.
Depending on the ordering of things in your PATH, either your old, or the newly installed pip is executed when you execute the pip command, so you either might have to adapt your PATH, or specify the full path to /usr/local/share/python/pip when installing eggs.
(shameless plug:
In any case, you might consider using virtualenv for installing packages into a "project" specific isolated environment, as opposed to installing them globally.)
I needed to uninstall brew's python.
Then, I was left with python v2.7.6
Next to install, pip I ran
sudo easy_install pip
installed fine and working
I had a similar issue, try this:
$ python -m pip install --upgrade --force-reinstall pip
This will force reinstall pip with whatever version of python you use including installing the binary.
A few days ago I had a friend who was starting Python Programming and needed help with the same issue: installing pip. There are debates over which one to choose between easy_install and pip and it seems everybody is heading the pip direction. Either way, installing either of them can be frustrating.
You can use this simple tutorial : installing pip package manager the easy way
Here are what you should keep in mind as you follow the above guide:
If you already have an older version installed, uninstall it or totally remove the python installation
Once that is cleared, download an install Python.
After that, download ez_setup.py file and save it to your desktop - easily accessible from the command line
Now run it from the command line and it will install easy_install for you after which,
You can use it to install pip.
Once again, you can do this or use the above link to find a simple step-by-step guide on how to get it installed on your computer.
Good luck.
Just so that people knew, ATM we can install PIP by downloading get-pip.py from the page with docs and run it like this:
c:\python27\python.exe get-pip.py
BTW, Python 3.4 comes with PIP pre-installed.
One of the command line options lets you choose where to install to.
--install-dir (-d) install package to DIR
So something like - # easy_install pip -d /usr/local/share/python
(Please correct me if I'm wrong.)
Just wanted to say that I found a way to get around my problem. I don't know that I can explain it perfectly, since I am not very good at understanding what I am doing with this stuff just yet! But, the problem seems to have been with my PATH. I removed the PATH that I posted in my original question, and then used easy_install pip. It went straight to python 2.7.2 (my new version) with no problem. I then successfully used pip to install NumPy and SciPy in the correct location, and they both work. Thanks to ErikAllik and FakeRainBrigand for taking the time to look into it!