We Keep Coding

Upgrading python and its packages to a newer version for my project

For some context, my project uses python version 3.8 and has some packages in a requirements.txt file. I'm trying to upgrade the python version to 3.10 and also all the packages in my requirements.txt file to the latest possible version such that there are no conflicts between dependencies. Is there a way to do this?

It is a bit hard to say what will work best for you since you've given no info on your OS, and that thing you need is done differently on macOS and Windows.
If you're on macOS (this one could also work for Linux I guess), the best way to manage python versions is with pyenv and to update python packages with pip-review, both of which you can install via brew.
So first, you want to create a list of all your packages installed. To do that, you type pip3 freeze > requirements.txt in the terminal. The command creates a txt file in your home folder. In the file, you can see the names of all your packages installed, and it will look like this:
astroid==2.11.6
async-generator==1.10
attrs==21.4.0
autopep8==1.6.0
beautifulsoup4==4.11.1
Secondly, to update your python version, you'll need to type pyen install -l in the terminal to get a list of all currently available versions of python. To install the one you need, let's say it's the most recent at the moment 3.10.5, type pyenv install 3.10.5. To set this version as a default one, type pyenv global 3.10.5. If you don't need your previous version of python, just type pyenv uninstall and the number of the version you want to delete after a space (it's really just like installing one).
Thirdly, to move all your previous packages in the newly installed python version, just type pip3 install -r requirements.txt in the terminal. The command will get the packages of the same versions you had before.
Lastly, to upgrade the packages to the most recent version, type pip-review --auto. It will fetch the latest versions available and install them with no more commands from you.
Now here's why I think pip-review works better than anything else I've tried so far: in case some dependencies are broken (i. e. a package you are using needs another package of an older version than you've just upgraded to), it will point that out in the terminal, and you could get the right one.

yes, you can do that just you need to update your requirements file.
and run the requirements file. it will automatically update your all files.

You can update to Python3.10 and drop the previous requirements.txt into your project root directory. From there you can run pip install -r requirements.txt to install all the packages.
You can also run pip install -U <package-name> to update any package to the latest version.
Also, to re-generate a new requirements.txt file, run pip freeze > requirements.txt.

Pretty sure there is no such thing as automatically updating dependencies in such way that there are no conflicts. Some dependencies may have changes that are not backward compatible and a complete test of all your project's features will be necessary whenever you change the version of anything.

There is also an open-source project called Renovate that can help you maintain your packages and update them to recent versions

How to reinstall all user packages after updating Python version in Windows?

I have a Windows 7 machine running Python 3.8.5 with a very large number of physics/electronics/data analysis/simulation packages. As it turned out, I must have - for some inexplicable reason - installed the 32-bit version of Python instead of the 64-bit one despite having a 64-bit system. And I didn't notice until very recently when I was trying to install some packages that require 64-bit Python. Hence I've now downloaded and installed the latest Python version that is supported by Windows 7, which seems to be 3.8.10.
Question: What is the easiest and also fail-safe way to reinstall all the user packages - that I currently have under 3.8.5 - to 3.8.10?
For some reason, I couldn't find any "canonical" solution for this online. As it seems, Python does not come with any built-in support for updating or system migration and I'm honestly wondering why...
Anyway, my first idea was to get a list of all user (= "local"?) packages currently installed under 3.8.5, but I don't know how. Reason: Doing help('modules') inside the interpreter will list all packages and I don't see a way to "selectively apply" pip to a specific Python version, e.g. something like python-3.8.5 -m pip list --local is not supported.
After getting a list of the user packages, I was thinking to pack it into a batch command pip install package_1 package_2 <...> package_N, thus reinstalling everything to Python 3.8.10. And afterwards uninstalling Python 3.8.5 and removing all environment variables from system PATH.
Is this the proper way to do this?

Anyway, my first idea was to get a list of all user (= "local"?) packages currently installed under 3.8.5, but I don't know how.
Create a list of installed packages with pip freeze > pkglist.txt or pip list --format=freeze. If you already have one, that's great.
Then uninstall 32-bit Python 3.8.5 and clean your path for all Python related variables. Now, install 64-bit Python 3.8.10.
After reinstalling, you can install back all the packages with pip install -r pkglist.txt and it will restore the exact versions of the packages.
If you insist on having both 32-bit and 64-bit versions installed and also have the Python Launcher installed, you could invoke 32 and 64 bit versions separately with py -3.8-64 -m pip and py -3.8-32 -m pip.
I don't see a way to "selectively apply" pip to a specific Python version.
This is possible with the Python Launcher on Windows. But only between major/minor versions and not the patch versions according to its help message.
I would also recommend creating a virtual environment this time before installing the packages and leaving the root environment alone. You can create one named venv with just python -m venv venv, activate it with ./venv/Scripts/activate and proceed with the installation of packages.
Nope, doesn't work. After installing the packages with the newer Python version in PATH, e.g. Jupyter won't start.
If the Jupyter error persists, you could try pinning packages to their most recent patch/minor versions to update them and yet not break your code.
As a last resort, you could try installing Python 3.10 alongside your current Python installation (without uninstall or editing the PATH) and then installing the absolute latest versions of the packages in a 3.10 virtual environment to see if it works for you. You would invoke the two versions with Py Launcher, e.g. py -3.10 and py -3.8.

If I understood correctly, you have multiple packages like NumPy, pandas etc. installed on your machine, and you want to reinstall them "automatically" on a fresh installation of python.
The method (I use) to perform such an operation is by creating a file named setup.py which includes a list of all the packages.
Bellow, I am attaching an example of such a file I use in one of my projects:
from setuptools import setup, find_packages
setup(
name='surface_quality_tools',
version='0.1',
install_requires=["matplotlib", "psutil", "numpy", "scipy", "pandas", "trimesh", "pyglet", "networkx", "protobuf",
"numpy-stl", "sklearn", "opencv-python", "seaborn", "scikit-image", "flask", "tqdm", "pytest"],
package_data={'': ['*.json']},
packages=find_packages(include=[])
)
to run the installation you should open a command prompt from inside the project directory and run:
pip install -e .
You can find a nice example in this blog page

One common way of handling packages in Python is via virtual environments. You can use Anaconda (conda), venv or any of several other solutions. For example, see this post:
https://towardsdatascience.com/virtual-environments-104c62d48c54#:~:text=A%20virtual%20environment%20is%20a,a%20system%2Dwide%20Python).
The way this works in by keeping the Python interpreter separate from the virtual environment that contains all the necessary packages.
Probably the main reason Python doesn't feature migration tools (at least as part of standard library) is because pip - the main package tool - doesn't handle conflict resolution all too well. When you update a version of Python it might so happen (especially with niche packages) that some of them won't work any more and pip often won't be able to solve the dependencies. This is why it's a good idea to keep a separate venv for different Python versions and different projects.
The other tool you could use for easy migration is Docker which is a semi-virtual machine working on top of your host OS and containing usually some linux distribution, Python along with the necessary packages necessary for running and development.
It takes a bit of time to set up a container image initially but afterwards setting everythin on a new machine or in the cloud becomes a breeze.
Listing currently installed packages is done via pip freeze command, the output of which you can then pipe into a file to keep a record of project requirements, for example pip freeze > requirements.txt.

Installation in virtualenv is attempting to run the global Python binary and failing

Right now I'm trying to install python (3.10) and all further installations on my new pc (windows 10) and so far everything is set up:
Python installed
Windows paths for "Python" & "Python\Scrips"
I am able to call the python and pip version and also install some packages. But after installing virtualenv and creating one the - at the moment - unfixable error appears: I am unable to install packages into the pip-path of the virtualenviroment itself. Whenever I'm trying to run any pip-command I'm getting the following error:
Unable to create process using 'C:\Users\ExampleUser\AppData\Local\Programs\Python\Python310\python.exe "C:\folder\env\Scripts\pip.exe" '
As you can see, it's always refering to the original python-path, but on the other hand it's refering to the pip-path of the virtualenv!? Don't know if it's helpful, but when typing in where python and where pip the paths inside the venv are the first one listed. I've also watched out for no blank spaces in my path...
Unfortunately no explanation out there could help me until now and I never faced this problem on my old machine - mostly the same, except some older version of python, pip and virtualenv.
Does anyone else has an idea what I am missing?

downloading Python 3 at the official website and installing it via express installation
Copy & Paste the standalone python into the /python folder and overwriting the python version
running python -m pip install --upgrade pip in cmd
Now pip and python 3 are installed in their latest version.
It's work for me

Could you use venv to create your virtual environment, instead of virtualenv (given that venv is the recommended way to create virtual environments for Python 3.3, and newer)?
If using venv is an option, this procedure may give you some idea on how to do it.
I have not done any Python development on Windows, but I think the basics would be:
python3 -m venv your-env-directory
your-env-directory\Scripts\activate.bat
If using venv is not an option, maybe you can try specifying the -v flag when running your virtualenv command to increase verbosity so you can further troubleshoot what's going on.

try upgrade pip version python -m pip install --upgrade pip

[ Sorry if this answer turns out to be more of a comment than an answer. I only have 21 reputation, so I cannot comment ]
When trying to install pip packages and run python files, is the CWD (Current Working Directory) C:\folder\env\Scripts? If so, try chaning your CWD to C:\folder. I had a similar problem and doing this fixed it.

You may need to look into a cygwin environment, and look into a chroot or jail environment to run the application without conflict.

Have you tried to use virtualenv-wrapper-win module.
It helps me a lot to manage virtual envs

Life is much easier using Anaconda 3 (it's definitely bloated compared to normal Python though), or use the minimal Miniconda (barebone install, basically just Python + a package manager). You can download it here: https://docs.conda.io/en/latest/miniconda.html#windows-installers
Then you can make a new virtual environment super easy:
conda create -n myenv pip
conda activate
If you have multiple environments you do: conda activate [environment_name]
Now you're in your new environment with pip installed. And you get drop down menus in the Windows menu to get to your new environment too, so there isn't any searching required. They just appear. Now if you want to link Jupyter Notebook or Spyder to the installation, it takes more steps since you need more packages. I used this guide which basically activates Jupyter first, then Spyder IDE. https://medium.com/#apremgeorge/using-conda-python-environments-with-spyder-ide-and-jupyter-notebooks-in-windows-4e0a905aaac5
Since you created the environment with pip added you can pip install whatever packages you need. I had to do this recently with OpenBLAS backed NumPy and SciPy (the defaults from pip, not from conda). Now Miniconda is the closest thing to basic Python installation, and comes with some nice tools to make your life easier. Hopefully this is helpful.

Unresolved import

I get this error for every single package that I have downloaded from pip whenever I type import [package] or any variation there off. This also means that the packages do not work. This is strange, since it used to work.
Edit: Apparently it works just fine on Jupyter Notebooks, but not in a normal python file.

it seems that you are installing the module in a wrong way, specifically, to improper directory (or python version).
you can alway check if module/package is installed or not by this:
pip list
if you can see the module in list, then it is installed, if not, then you need to install it properly.
sometimes you may have different versions of python installed, you install the package to version X and try to run it on version Z. all you need to do is to clarify your python version, check if the pip is referring to the python version by doing this:
pip -V
it shows the pip version along with the python version it is referring to.
The BEST WAY
always try to create a virtual environment and install packages there.
creating a virtual environment
python -m venv <VENV_NAME>
then activate it:
source <VENV_NAME>/bin/activate # MacOS & Linux
<VENV_NAME>/Scripts/activate # Windows
then, install your packages. there shouldn't be any problems after that.

Keeping Installed Packages When Updating Python

I've got python 3.7 installed on Windows 10. The recommended way to upgrade to 3.8 appears to be to do a new installation, which means I will have both versions installed. I don't need both versions, but I would like to keep all the packages I installed for version 3.7.
How do I achieve this please? Also will new new path variable for 3.8 replace the one for 3.7?
The process for such a common use case seems strangely complex. Am I missing something?

Simple solution would be in CMD to do
pip freeze > packages.txt
This will write all your current packages to the text file 'packages.txt'
Then uninstall Python 3.7 as you would any Windows program then install Python 3.8 and in CMD do
pip install -r packages.txt
This will install all the packages that you had before.
Though I would recommend using conda as that handless Python versions and packages for you, along with environments.

One way to do this is to run:
python3.7 -m pip freeze > installed.txt
Then, after installing the new Python version you can install the packages with:
python3.8 -m pip install -r installed.txt
There is a chance that the packages you installed for your old Python installation are not compatible with the new version. For that reason it is safer to keep both Python installations and then use virtual environments for each of your projects.
You can create a virtualenv for each of your projects, using the Python version you need for that project, and install your dependencies only in the virtualenv for that specific project. This way you can avoid the situation where project A requires an old version of a certain package but project B requires a newer one. If you install all your packages globally you run into problems in this case.
See also What is a virtualenv, and why should I use one?

I would recommend moving over to conda to manage your environments.
https://docs.conda.io/projects/conda/en/latest/user-guide/install/windows.html
The current thinking for most of the development projects that I've worked on involving python is that the version and libraries are specified on a per project basis. Conda allows you to freeze the environment so that it's more portable. You can generate an environment.yml file that allows someone to recreate your environment from scratch, and you can maintain only the packages needed for a given project.
As per your original question, you can set the PYTHONPATH to point to the old and new directories. I can't guarantee that the libraries will work though since there could be version compatibility issues.