We Keep Coding

Upgrading python and its packages to a newer version for my project

For some context, my project uses python version 3.8 and has some packages in a requirements.txt file. I'm trying to upgrade the python version to 3.10 and also all the packages in my requirements.txt file to the latest possible version such that there are no conflicts between dependencies. Is there a way to do this?

It is a bit hard to say what will work best for you since you've given no info on your OS, and that thing you need is done differently on macOS and Windows.
If you're on macOS (this one could also work for Linux I guess), the best way to manage python versions is with pyenv and to update python packages with pip-review, both of which you can install via brew.
So first, you want to create a list of all your packages installed. To do that, you type pip3 freeze > requirements.txt in the terminal. The command creates a txt file in your home folder. In the file, you can see the names of all your packages installed, and it will look like this:
astroid==2.11.6
async-generator==1.10
attrs==21.4.0
autopep8==1.6.0
beautifulsoup4==4.11.1
Secondly, to update your python version, you'll need to type pyen install -l in the terminal to get a list of all currently available versions of python. To install the one you need, let's say it's the most recent at the moment 3.10.5, type pyenv install 3.10.5. To set this version as a default one, type pyenv global 3.10.5. If you don't need your previous version of python, just type pyenv uninstall and the number of the version you want to delete after a space (it's really just like installing one).
Thirdly, to move all your previous packages in the newly installed python version, just type pip3 install -r requirements.txt in the terminal. The command will get the packages of the same versions you had before.
Lastly, to upgrade the packages to the most recent version, type pip-review --auto. It will fetch the latest versions available and install them with no more commands from you.
Now here's why I think pip-review works better than anything else I've tried so far: in case some dependencies are broken (i. e. a package you are using needs another package of an older version than you've just upgraded to), it will point that out in the terminal, and you could get the right one.

yes, you can do that just you need to update your requirements file.
and run the requirements file. it will automatically update your all files.

You can update to Python3.10 and drop the previous requirements.txt into your project root directory. From there you can run pip install -r requirements.txt to install all the packages.
You can also run pip install -U <package-name> to update any package to the latest version.
Also, to re-generate a new requirements.txt file, run pip freeze > requirements.txt.

Pretty sure there is no such thing as automatically updating dependencies in such way that there are no conflicts. Some dependencies may have changes that are not backward compatible and a complete test of all your project's features will be necessary whenever you change the version of anything.

There is also an open-source project called Renovate that can help you maintain your packages and update them to recent versions

Should I use pip or pip3? [duplicate]

This question already has answers here:
pip or pip3 to install packages for Python 3?
(10 answers)
Closed 2 years ago.
Eventually, every single time I install a new Linux distribution I do sudo apt-get install python3.
However, once installed I always get confused. python is Python 2.7 and python3 is Python 3.x. But also it appears that pip is for Python 2 and pip3 for Python 3. That said most tutorials I see on Internet always use the traditional pip install even though it is about Python 3.
How should I deal with this? Should I simply continue to put this annoying 3 every time I use Python (pip3, ipython3, python3...)? In most of my lectures I read that creating a symlink python->python3 is a bad practice. Is that correct?

Use python3 -m pip or python -m pip. That will use the correct pip for the python version you want. This method is mentioned in the pip documentation:
python -m pip executes pip using the Python interpreter you specified as python. So /usr/bin/python3.7 -m pip means you are executing pip for your interpreter located at /usr/bin/python3.7.
Symlinking python->python3 is a bad idea because some programs might rely on python being python 2. Though, I have seen some Dockerfiles symlink python->python3, like TensorFlow's CPU dockerfile (it's less of an issue in a Docker image). Coincidentally, that same Dockerfile uses the python3 -m pip install syntax that I recommend.

creating a symlink python->python3 is a bad practice. Is that correct?
Sometimes. Some OSs (looking at you, macOS) deeply rely on python pointing to a Python 2 interpreter for internal tools and tasks. Deleting the shipped Python 2 interpreter (or aliasing python to a Python 3 interpreter) will break stuff. How to uninstall Python 2.7 on a Mac OS X 10.6.4?

Whether the correct command for Python 3 is pip or pip3 or (say) gaschplutzga depends on a number of factors.
If you only have Python 3, and you have a command named pip, that's probably safe to use. Going forward, this will be the simple, obvious, safe answer in more and more places.
If you have both, and there is a command called pip3 installed on your system, probably that's the correct one to use.
More generally, you can go through your PATH and look for commands with suitable names. On Unix-like systems with a POSIX-compatible shell, try the commands command -V pip3 and command -V pip. (On Windows systems, maybe try where pip3 and where pip, or pray to whatever dark deity informed your choice of operating system.)
If you receive output like
/opt/random/nonstandard/whoa/pip
/usr/local/bin/pip
/usr/bin/pip
you can try each of these in turn with the full path and adding the --version option to have them identify themselves. When you specify the full path, you are bypassing the system's PATH mechanism entirely. For example,
/opt/random/nonstandard/whoa/pip --version
might identify itself as belonging to Python version 3.2.1. If that's the one you want, and it's at the top of your PATH, you can simply rely on the PATH to give you this version when you type just pip. If not, perhaps you can shuffle your PATH (but understand that this changes the resolution order for all commands in the directory whose position you change) or create a simple alias or wrapper which bypasses the PATH for this particular command in your personal account. On Unix-like systems with a POSIX-compatible shell, this might look like
alias pip=/opt/random/nonstandard/whoa/pip
(to persist this across sessions, you'd add this to your .profile or similar - for Bash, try .bash_profile if it exists; for Zsh, try .zshrc. The full scoop for each shell is more complicated than I can squeeze into these narrow parentheses); on Windows, you might be able to control this by setting the environment variable PY_PYTHON, but there's a huge can of worms behind that "might".
Some sites and OSes / distros have additional wrappers or conventions which introduce additional options; if you use a specific package manager, perhaps also study its documentation. (One common example is Anaconda, though I don't believe it affects the naming or location of pip specifically.)

Use virtual environments, then pip would be associated with the python used to create that virtual environment. Whether you use pip or pip3, it will be equivalent to python3 -m pip as mentioned in jakub's answer. Also, given that Python 2.7 is already EOL (which means you will most likely work with Python 3) and that pip install-ing things onto the system packages should be avoided, then a virtual environment would be helpful here.
For example, using pipenv:
$ pipenv --python=/usr/local/opt/python#3.8/bin/python3
$ pipenv shell
Launching subshell in virtual environment...
(TEMP) $ pip --version
pip 20.2.3 from /Users/me/.venvs/temp2-SbXvZiFd/lib/python3.8/site-packages/pip (python 3.8)
(TEMP) $ pip3 --version
pip 20.2.3 from /Users/me/.venvs/temp2-SbXvZiFd/lib/python3.8/site-packages/pip (python 3.8)
For example, using venv:
$ python3.8 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip --version
pip 20.2.3 from /Users/me/temp2/.venv/lib/python3.8/site-packages/pip (python 3.8)
(.venv) $ pip3 --version
pip 20.2.3 from /Users/me/temp2/.venv/lib/python3.8/site-packages/pip (python 3.8)
The virtual environment takes care of making sure pip or pip3 in this env refers to the pip from the correct Python version. You can then happily follow tutorials that still use pip install something (unless of course that tutorial refers to a Python 2.7 or a system-wide installation).

You can install pip through pip3 and this should resolve this issue.
$ pip --version
pip 19.0.3 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
Notice that pip here is of Python 2.7 (in this example).
You can then force pip3 of Python 3.X to install pip under itself.
$ sudo pip3 install pip --upgrade
Installing collected packages: pip
Found existing installation: pip 8.1.1
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed pip-19.0.3
Once you check this again, it should reference Python 3.X so you don't have to deal with
what is what.
$ pip --version
pip 19.0.3 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5)
I doubt you'll want to use Python 2 after this, but if you do happen to work with Python 2 code, you can create a virtual environment to access those commands again. Otherwise, you won't have to worry about the pip or pip3 distinction after this.
Not really a duplicate of this question, but this helped me suggest this answer: Can pip (python2) and pip3 (python3) coexist?

Pip is for python version less than 3. and pip3 is used when you want to install packages for python version 3 or higher.

pip/python: normal site-packages is not writeable

I have a new Macbook - a user installed it, and then I installed a new user (mine), granted admin privileges and deleted the old one. I am on OS Catalina.
Since the installation I've been having several permission problems.
VSCode can't find Jupyter Notebook, pip installs packages at ~/Library/Python/3.7/site-packages.
When I do which python3 I get usr/bin/python3.
When I do pip3 install <package> I get: Defaulting to user installation because normal site-packages is not writeable And then it says it has already been installed, even though I can't access it when I do import <package>.
It's seems clear that this is a permission problem, pip can't install to the "base" python, and them python can't find what I've installed into ~/Library/Python/3.7/site-packages.
I've tried reinstalling the OS, but since I haven't done a clean install, it didn't change anything.
What am I missing?
How exactly can I fix permissions? Where do I want packages to be installed (venv sure, but some packages I want global (like jupyter).

As #TomdeGeus mentioned in the comments, this command works for me:
Python 3:
python3 -m pip install [package_name]
Python 2:
python -m pip install [package_name]

It's best to not use the system-provided Python directly. Leave that one alone since the OS can change it in undesired ways, as you experienced.
The best practice is to configure your own Python version(s) and manage them on a per-project basis using virtualenv (for Python 2) or venv, possibly via poetry, (for Python 3). This eliminates all dependency on the system-provided Python version, and also isolates each project from other projects on the machine.
Each project can have a different Python point version if needed, and gets its own site_packages directory so pip-installed libraries can also have different versions by project. This approach is a major problem-avoider.

python3.7 -m pip install [package_name]
(you should use the version that you have, of course)
solved it for me.
The most voted answer python3 -m pip install [package_name] does not help me here.
In my case, this was caused by a conflict with the dominating 3.6 version that was also installed as a default. You might ask yourself why you have 3.6 on your system, you will most probably not use that version now. The reason is that 3.6 is used as an independent default python version for many package installers. Those installers do not want to check which individual version you use and whether that fits, they just use 3.6 as a default, if you like it or not.
Here is a proof by example --upgrade pip:
pip3 install --upgrade pip
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in
/home/USERNAME/.local/lib/python3.6/site-packages (20.3.1)
python3 -m pip install --upgrade pip
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in
/home/USERNAME/.local/lib/python3.6/site-packages (20.3.1)
python3.7 -m pip install --upgrade pip
Collecting pip
Cache entry deserialization failed, entry ignored
Using cached https://files.pythonhosted.org/packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl
Installing collected packages: pip Successfully installed pip-20.3.1

I'm using Anaconda on Ubuntu and had the same problem.I fixed it by the following steps:
deactivating current environment
conda deactivate
Then, the base environment activates. I deactivated the base conda environment too. To do so, I used conda deactivate again.
Finally, I activate my project environment directly (instead of activating from the base environment) by the following command. Afterward, I installed the intended package successfully and worked perfectly.
conda activate myenv
pip install somepackage

sudo pip install
Worked for me. But pip install is not recommended to be run with sudo. The issue I was facing on BIGSUR was, it was using system python. Once I Installed python 3.9 using
brew install python#3.9
Then pip worked fine

For readers who thought themselves accidentally update system pip:
If you saw this info in your terminal output:
Defaulting to user installation because normal site-packages is not writeable
then you will be fine. Use the pip3 you just updated to run:
pyenv global system # since I use pyenv
pip3 uninstall pip # this one does the trick
Then you can check again pip3 --version will point to the original old (XCode/System-)pip. E.g. (2022/2/28):
pip 20.2.3 from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip (python 3.8)

It occurs with me when I the virtual enviroment folder name was : venv.
in this case, It gives errors like :
No module pip
Default folder is unwritable
renaming the folder solve the proplem.

Check on the command line "which python" to see if it is the value you expect.
If you have a virtual environment activated, check /venv/bin/activate to see the value of VIRTUAL_ENV= and make sure it is the correct path . The path may be wrong if you renamed or moved the project. If the path is wrong, you can delete the venv and make a new one.

For me, none of the suggestions worked so I had to delete the current virtual environment folder venv and recreate it using one of the following commands:
python -m venv venv
python3 -m venv venv
Check the source of pip on Ubuntu 20.04
which pip
returns the correct path
/home/myname/fullstack/person_api/venv/bin/pip
UPDATE
I presume that some might encounter this problem because they set python path as environmental variable like this in ~/.bashrc:
python=/path/to/python
which you should not be doing! Instead we could do:
py=python
PATH=/path/to/python:$PATH
I bumped into this issue specifically because of this!

Had this same issue on a fresh install of Debian 9.12.
Rebooting my server solved the issue.

In my case on Linux, the ownership of the conda env directory had changed to another Linux user (long story), and so the the normal site-packages was not writeable due to a permissions issue.
The solution was to change ownership back to the user doing pip install.

I met exactly the same issue.
I just type sudo python3.8 -m pip install .... and the error disappeared.
If I remove the sudo, issue remains.

For those running on a Pi, that accidentally installed pip as root. Just chown the lib folder to the pi user:
sudo chown -R pi:pi /usr/local/lib/python3.9/

in my case python3 -m pip install [package_name] did not solve that.
in my case, it was a problem related to other processes occupying the directory.
I restart Pycharm and close any other program that might occupy this folder, and reinstalled the package in site-packages directory successfully.

When this problem occurred to me I have tried all the mentioned approaches but they don't seem to work.
Instead, restarting Python language server in my VSCode did the job - my SimPy package is now found. On Mac it is Cmd+Shift+P and select "Python: Restart Language Server".

Had similar issue on Ubuntu 20.04.4 LTS in VirtualBox, but none of the suggestions here worked for me.
I was trying to install open3d in a venv and every time I was getting
"Defaulting to user installation because normal site-packages is not writeable"
which at first I didn't even noticed. open3d was always being installed in /usr/bin/python3 environment. I've restarted the VM but without luck, so I guess the problem was not just missing write access.
So in VS Code, which was using the venv, importing open3d was not possible. But testing from terminal from the activated venv with python3 -c "import open3d as o3d; print(o3d.__version__)" was working fine and that confused me totally. I even broke my system pip installation using sudo, see further below if you want to know how to fix it.
Anyhow, the solution to my problem was to explicitly point to the python3 file in the venv where I wanted to install the package:
venv/bin/python3 -m pip install open3d
So I was testing out everything and eventually installed with sudo: sudo pip3 install open3d. This of course didn't solved the problem and open3d was still missing in the venv. Even worse, I got the message:
"WARNING: You are using pip version 21.3.1; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command."
So I did it but with sudo, updating the system pip and then found out here that this is not good:
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Following an advice here, I tried to revert to original version, only then pip3 broke:
sudo pip3 uninstall pip
sudo pip3 --version
sudo: pip3: command not found
The apt package was still there:
sudo apt install python3-pip python3-pip is already the newest version
(20.0.2-5ubuntu1.6).
So I had to reinstalled to fix the problem:
sudo apt-get remove python3-pip
sudo apt install python3-pip

Maybe you have python, python3, pip or pip3 aliased. In that case pip might not work well anymore, as the alias isn't always available and so pip/pip3 might resolve python/python3 differently compared to in your terminal.
That could give rise to pip/pip3 trying to install in the system python, and that could give rise to your error.

I tried ever single recommendation described here. In every instance, I get the exact same result: SyntaxError: invalid syntax (<stdin>, line 1)
I'm not sure who designed the system like this, but it seems basically useless, based on my experience so far. Either create a system that works, or don't create anything at all.

How to install a Python package inside a virtual environment with Pip (OS X)

Edit:
I'm going to close this question as the reason its happening is different from my original assumption, and it's clearer to ask the question anew:
Pip installs packages in the wrong directory with virtualenv
The accepted answer doesn't directly answer the original question, but is a very useful overview.
Based on discussion below the issue is that even after
$ source ~/PycharmProjects/Practice/venv/bin/activate
$ pip install numpy
numpy is installed in /usr/local/lib/python2.7/site-packages
What could the reason for this be?
Original:
Using Python on OS X via Homebrew:
I've been trying much of the day to sort this out but either I get a must supply either home or prefix/exec-prefix -- not both error, or the package I try to install goes into totally the wrong place:
$ pip3 --version
pip 18.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
$ cd venv
$ pip3 install numpy
..... [snip with following error:]
"must supply either home or prefix/exec-prefix -- not both")
Using this hint
$ pip3 install numpy -t .
Then I get a new error,
`Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/.../pip-install-0fvveq3v/package/'
Searching around SO gives various possibilities involving pip install setuptools. but pip install throws the above error or installs in the wrong place. i.e. the solution involves something that's causing the error in the first place.
I tried to use the Python.org installer but it didn't install pip at all. (The custom installer showed the option checked but with size zero).

An introductory overview is available in this nice tutorial. Here is a good summary with more detail. But, if you renamed or moved the virtual env dir after its creation, it could break it. Create a new one from scratch: $ cd ~/PycharmProjects; python3 -mvenv newenv ; Activate: $ source newenv/bin/activate ; Install something: $ pip install colorama (same as pip3 install only if venv activated); Check: ls ~/PycharmProjects/newenv/lib/python3*/site-packages ; Deactivate: $ deactivate
Then you could try this solution for Pycharm: how to associate a virtual environment with a python project in pycharm. PyCharm indeed comes bundled with virtualenv which could have been customized, please look into Pycharm-specific resources: creating virtual environments and installing packages in Pycharm.
If you have installed PyPI's mainstream virtualenv, by default it will create new environments with the python interpreter that virtualenv was installed with. But it's possible to specify an alternate Python Interpreter upon a new env creation: $ virtualenv -p python3.7 newenvname
Regarding the error DistutilsOptionError: must supply either home or prefix - please check this and this for solutions. Homebrew'ed mappings between python and pip are described here. The normal pip install --user is disabled in homebrewed Python, but there are workarounds. MacOS system Python doesn't provide pip, but it can installed, reinstalled or upgraded for any specific python version manually. Original non-brewed installers are also available for all Python versions: https://www.python.org/downloads/mac-osx/
By default there's no pip.conf, but it can be created by hand to customize things. All possible pip.conf locations (per-user, per-venv, and global/system-wide, and how they override each other) are listed here. If someone faces an issue, they could use pip config list command to see their active configuration, or locate pip.conf and find it.
Finally, you may want to ensure you aren't using pip against macOS' system python. Shell commands such as $ brew info python, which pip, which pip3, pip3 -V, which python3 can help you see what you actually use. Since the macOS default $PATH used to be /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin, stock macOS binaries (including python) may take precedence over some homebrew'ed installations (including python). If so, a custom PATH could be exported via the ~/.bashrc if required.

Upgraded Python; Do I have to reinstall all site-packages manually?

I have upraded my Linux distro recently. Python 3.5 was replaced by Python 3.6.
All site packages I have installed with pip3 are still in the /usr/lib/python3.5/site-packages directory and Python does not find them there now, because it looks in .../python3.6/site-packages obviously.
I see the directory contents and I could manually install them again, but that does not look to me like the right way to do it. I could move the contents to the new directory, but again, this seems to me incorrect either.
How am I supposed to handle it properly?
Should I have prepared a pip3 freeze list before the upgrade?
I tried to search, but the keywords are probably too general and got many unrelated answers.

Python 3.5 was replaced by Python 3.6. But you still have the backup option of using python 3.5.
If you want to use python 3.6 you will have to reinstall all pip packages again for python 3.6. And it makes sense.
Say you were changing from 2.7 to 3.5. You would want to preserve both the environments separately. Hence 3.6 environment is different from 3.5.
A quick way to do this would be to pip freeze for 3.5 and then install those dependencies for 3.6.
pip freeze > reqs.txt
upgrade
pip install -r reqs.txt
Since you don't have this option anymore, first try and list all packages in your python3.5
for that you can install pip3.5 as answered by #kabanus.
sudo apt-get install python3=3.5.1*
sudo python3.5 easy_install.py pip
Also it's advised to use virtual environment per project so you can maintain separate environments for each of them.

I just hit the same problem upgrading from Python 3.6 to Python 3.7, I forgot to run pip freeze before I upgraded to Python 3.7. The solution that worked is to specify the --path option as the old site-packages/ directory (which was not deleted):
pip3 freeze --path /usr/local/lib/python3.6/site-packages/ > python3.6_requirements.txt
pip3 install -r python3.6_requirements.txt

This would have made things simpler for you to reinstall. Checkout the description. Using freeze you could have done something like:
$ env1/bin/pip3 freeze > requirements.txt
$ env2/bin/pip3 install -r requirements.txt
Generally the recommended method is you use a virtualenv for site packages, so you don't litter your installation areas, but TBH it never broke something for me. Another option is to check if the linux distribution has the package available for proper retrieval, as in:
sudo apt-get install python3-<somemodule>
This is what I prefer - and could have been upgraded with the distro. As for what to do now, If you really don't want to re-install everything properly you could try to cp /usr/lib/python3.5/site-packages/* /usr/lib/python3.6/site-packages. The differences between versions are not so great such that I believe most packages would work off the bat. You may have to sed to replace python3.5 with python3.6 in all files there though. Forgot delete all pyc files if you do this.
Python modules are self contained enough that if something is broken it can be handled per package, and the site packages are self contained completely, so you could always just remove everything and re-install.
A final note - you can try and install pyton3.5/pip3.5 for your linux, and then do the freeze thing. If there is no package you could install manually (whl or such) or compile a stand alone and configure the site path properly. If you want to keep things on a global site package directory or migrate to virtualenv this may be the safest option.