Summary: To activate a pyenv environment, pyenv is requiring me to include the path to the environment instead of just the environment name.
I was previously able to run pyenv activate env_name, whereas now I must run pyenv activate 3.9.1/envs/env_name.
I accidentally deleted some files in my home directory, but I still had (at least some of) my .pyenv, and it still contained files in the versions directory with the names of my environments. When I ran pyenv, it did not recognize the command.
So I reinstalled pyenv, first copying the old .pyenv into a separate forlder, and then I copied the contents of versions and shims into the new .pyenv.
Now, if I run pyenv versions or pyenv virtualenvs, it does include the environment I want: 3.9.1/envs/env_name. However, if I type pyenv activate env_name, it says that there is no virtual environment with that name. However, If I type pyenv activate 3.9.1/envs/env_name, it does work. In the past, before accidentally deleting those files, it would recognize my environment without prepending 3.9.1/envs on the environment name.
I'm running linux.
Since your pyenv itself doesn't seem broken (it can create new versions with their alias (symbolic link) just fine), it's probably just the symbolic links that have disappeared.
From a quick test, it appears you can very simply recreate the virtual environment, and everything (packages installed) will still be there, while you get the alias again.
So try
pyenv virtualenv 3.9.1 env_name
It may complain, and ask to continue: just enter y (I think the --force option can also do it, in case you have many virtual environments to repair and want to do that in a loop or so).
Check after doing the first one, if pyenv virtualenvs now lists the full name and the alias, then continue with the others.
If you want to be safe, first make a backup of the first virtual environment you're testing this on, or save the installed packages into a list somewhere (pip freeze > pip.list or so), so it's easy to recover if something goes wrong. But as mentioned, my quick tests shows pyenv virtualenv just recreates the symbolic link, nothing really more, since the other parts of the virtual environment already exists (that is, creating a virtual environment doesn't remove an existing one).
Related
To be honest, up until recently I was hoping to avoid everything related to vurtualenvs. Yep, very naïve.
All of the code I write using python is to be deployed using containers, where "one container - one app" is totally true for now. And also number of CLIs being distributed also to my coworkers as wheels.
What I came up with is two "global" environments living in my homedir:
"default" virtual env with python 3.10 and some general packages installed. Using this venv I develop current versions of my applications and do my regular job. It is sourced automatically in .bashrc.
Virtual env with python 3.9. This is used to support legacy versions of/not yet updated applications in case if something should be done to them.
And also for some of the projects there are dedicated virtual envs living in a project folder needed only for this particular project, python version there may vary from 3.7 to 3.10.
As a consequence, I have to change current environment by manually sourcing the right activate script several times a day, and this isn't funny.
After some research on the topic, I found things like virtualenvwrapper, Pipenv and Poetry, but none of them seem to meet all my needs: "global" env to use across multiple projects and outside projects (because of CLIs); ability to create env with given python version; easy switch between virtual envs.
Is there any solution for this?
UPD: Some illustrations
Next virtual environments were created inside ~/pyenv/:
$ virtualenv -p /usr/local/bin/python3.9 default3.9
$ virtualenv -p /usr/local/bin/python3.10 default3.10
Next, .bashrc was updated to do source ~/pyenv/default3.10/bin/activate.
Now, for the vast majority of tasks it is enough: everything needed for day-to-day job is installed in default3.10 venv.
In case I need to switch to python3.9 the following is required:
$ deactivate
$ source ~/pyenv/default3.9/bin/activate
And if then I need to work on some project with its own venv, inside project folder I have to do:
$ deactivate
$ source env/bin/activate
Too many manual inputs. At minimal, I'd like to do something like $ workon default3.9 no matter of the current directory I'm in, and it should activate default3.9 venv correctly if no venv is active right now or if any venv if already active.
The same should work inside project folder: $ workon <local_venv_name> to switch to the venv living in current project folder.
And something like venv create <enterpreter path/py version> to create new "global" or "local" (in current folder only) venv, and venv list to get the list of available "global" and "local" environments.
to use the global environment inside the project's local virtual environment.
set
include-system-site-packages = true
in pyvenv.cfg file located in .venv (virtual environment installation directory) directory of each project.
I wanted to replace Python 3.8 32-bit with the 64-bit version to install the face_recognition module, so I deleted the previous version and tried to re-route the project to the new Python version by going to File > Settings > Project Interpreter > Show all > Show Paths for Selected Interpreter, and adding all the Python files from the new folder and getting rid of the old ones.
However, it's still showing me this error when I try to install the module:
(Will2.0) C:\Users\solei\PycharmProjects\Will>pip install face_recognition
No Python at 'C:\Users\solei\AppData\Local\Programs\Python\Python38-32\python.exe'
I've also tried going to the Windows System Properties and changing everything that says "Python38-32" there, but it's still not working. It does work when I make a new environment, though, so at least I know that Python installed properly. It's just this one environment that is tripping me up (I'd prefer not to make a new project for this, btw. I've already installed a lot of modules in it.).
Your selected interpreter is not the system interpreter you've replaced with the 64-bit version, but your project's virtual environment interpreter. The virtual environment's files weren't changed in that process and need to be updated before you can use that environment again.
The system interpreter is your Python interpreter installed using the installation executable. In your case it is located in C:\Users\solei\AppData\Local\Programs\Python\Python38\. You can have multiple system interpreters installed, such as having Python 2.7, Python 3.7 and Python 3.8 side-by-side.
The virtual environment interpreter is a copy of another interpreter created using the venv package from the Python standard library. You can have many virtual environments interpreters in the system (one or more for every project, for example)
The base interpreter is the interpreter that was used as a template for the venv package. Every virtual environment interpreter has its base interpreter (usually a system interpreter) that it requires to run. Changing or upgrading the base interpreter requires updating the virtual environment.
If we take a quick look at the documentation, a virtual environment is described as
a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.
That means you can setup an individual environment for every project, which will contain its own packages. The environment is a very efficient way of managing project packages, that's why PyCharm suggests a creation of such environment over the system interpreter by default. In short, it allows you to have two different versions of the same package used by two different projects, without the packages conflicting with each other.
This also explains why your virtual environment files weren't affected by your upgrade.
Now, I am unfortunately no Python expert. I had to spend some time examining how Python handles virtual environments on Windows and Ubuntu. It seems the environment always requires the base system interpreter present in the system. If you remove or change the location of the base interpreter, the environment will fail to function.
As I mentioned before editing this answer, you can in theory simply edit the pyenv.cfg file located in the root folder of the virtual environment. In practice, that will only work in simple cases and it is not the intended way of updating virtual environments.
You need to upgrade your virtual environment's files to work with your new system interpreter. That can mean the 64-bit version over the 32-bit version, or even a newer version of Python - such us upgrading from 3.7 to 3.8.
Close PyCharm
Check if the system interpreter you want to upgrade to is on the system Path
You can quickly check by running
python -c "import platform; print(platform.architecture())"
For you, the output should look like this
('64bit', 'WindowsPE')
If your output is different, you'll need to prefix the absolute path to the Python executable in step 4).
Navigate to the virtual environment's directory
The directory you're looking for contains the Include, Lib and Scripts directories and the pyenv.cfg file. From your screenshots, it seems this directory is your project's root directory, so in your case:
cd C:\Users\solei\PycharmProjects\Will2.0\
Upgrade the virtual environment
python -m venv --upgrade .
... or if Python is not on your path
C:\Users\solei\AppData\Local\Programs\Python\Python38\python.exe -m venv --upgrade .
The . in the commands refers to the current directory.
Open PyCharm and verify your environment is working correctly
... or simply try to run pip directly from the command line. Note you need to first activate the virtual environment by running the Scripts\activate.bat batch file.
If the above-mentioned method doesn't work, you might have to create a new virtual environment. You can create one easily without making a new PyCharm project. See this PyCharm documentation for reference. However, you'll still need to redownload all the required packages again.
For the simplicity, I recommend creating the new virtual environment in a .venv folder located in the project's root.
Disclaimer
I tested only the Python's behavior alone on a fresh Windows installation inside the Windows Sandbox. I was able to install the 32-bit Python, create a virtual environment, replace Python with the 64-bit version and upgrade the virtual environment to have it launch correctly again.
I'm having an issue where for some reason the virtual environments that I'm creating are accessing my system-wide installations of Python and pip when they shouldn't be.
Here's my fairly simple workflow just make sure I'm not missing anything obvious (Windows 10, Python 3.8.2):
python -m venv venv
venv\Scripts\activate.bat
My path is now prepended by (venv), as you'd expect. However,
pip list
lists all of the system-wide pip packages I have instead of just the ones that should be in that venv.
pyvenv.cfg indicates that
include-system-site-packages = false
When I open the interpreter using
python
In the virtual environment,
sys.executable
Returns the path on my C drive and
print(pip.__file__)
Does the same. I suspect they should instead be pointing to the interpreter and pip in the virtual environment but don't know how to make that happen.
Edit: 4/27/20, Still dealing with this issue, I have tried:
uninstalling and reinstalling Python, both from python.org and the MS Store
Installing python in a new location
Clearing my user and system environment variables and then adding in
just the ones for Python 3.8.
I'm really at a loss here, would appreciate any help.
To anyone looking at this later on, I eventually realized that the problem was only occurring in a specific folder. Making virtual environments elsewhere on my computer seems to work fine. Why this is the case? I have no idea. I'm curious, but not enough to spend another second on this problem. So there you go.
This happened to me too. My problem was that I created the virtual environment under the directory including the global executable (I mean C:\Program Files\Python39). Once I removed this virtual environment and created a new one somewhere else (for example C:\Python), running python command took the executable in the virtual environment rather than the global one.
I had the same problem too. It might be that the path to your virtual environment has a character with accent in it. This basically means that you will have to alter the \Scripts\activate.bat file.
In order to see if that's the case, use the following procedure:
Activate the virtual environment by running the file \Scrips\activate.bat
Run the following command:
echo %PATH%
There should appear a bunch of paths, being path\to\your\virtual\environment\Scripts the first of them (this is obligatory). Check if this path is exactly the same as it is supposed to be (meaning it should have no "strange" characters instead of characters with accents).
If there are any discrepancies, I suggest you alter your \Scripts\activate.bat according to https://stackoverflow.com/a/22348546/11923771, reopen your Command Prompt and follow steps 1 and 2 again to see if your problem was solved.
I am using Windows 10 (all commands run as administrator). I created an environment called myenv. Then I used
conda env remove -n myenv
Now, if I try
conda info --envs
I only see the base environment. However, if I try
conda activate myenv
I'm still able to activate it! I think because under the folder envs, there is still a folder with the name myenv there which doesn't get deleted.
How do I delete the environment for good?
Command-line options can only go so far, unless you get very specific; perhaps the simplest approach is to delete things manually:
Locate Anaconda folder; I'll use "D:\Anaconda\"
In envs, delete environment of interest: "D:\Anaconda\envs\myenv"
Are you done? Not quite; even while in myenv, conda will still sometimes install packages to the base environment, in "D:\Anaconda\pkgs\"; thus, to clean traces of myenv,
Delete packages installed to myenv that ended up in "D:\Anaconda\pkgs\"
(If above don't suffice) Anaconda Navigator -> Environments -> myenv -> Remove
(If above don't suffice) Likely corrupted Anaconda; make note of installed packages, completely uninstall Anaconda, reinstall.
Note: step 3 is redundant for the goal of simply removing myenv, but it's recommended to minimize future package conflicts.
In addition to the first command in the question posted, I had to complete one additional step to completely remove the environment. I had to go to the folder where the environment was stored (e.g. C:\Users*username*.conda\envs\ on a windows machine) and remove the folder with the same name as the environment I deleted. After this second step, I was able to reuse the environment name without any errors.
I know that virtualenv, if not passed the --no-site-packages argument when creating a new virtual environment, will link the packages in /usr/local/lib/python2.7/site-packages (for Python 2.7) with a newly-created virtual environment. On Ubuntu 12.04 LTS, I have three locations where Python 2.7 packages can be installed (using the default, Ubuntu-supplied Python 2.7 installation):
/usr/lib/python2.7/dist-packages: this has my global installation of ipython, scipy, numpy, matplotlib – packages that I would find difficult and time-consuming to install individually (and all their dependences) if they were not available via the scipy stack.
/usr/local/lib/python2.7/site-packages: this is empty, and I think it will stay that way on Ubuntu unless I install a package from source.
/usr/local/lib/python2.7/dist-packages: this has very important local packages for astronomy, notably those related to PyRAF, STScI, etc., and they are extremely difficult and time-consuming to install individually.
Note that a global directory such as /usr/lib/python2.7/site-packages does not exist on my system. Note also that my global installation of ipython, scipy, etc. lets me use those packages on-the-fly without having to source/activate a virtual environment every time.
Naturally, I now want to use virtualenv to create one virtual environment in my user home directory which I will source/activate for my future projects. However, I would like this virtual environment, while being created, to link/copy all of my packages in locations (1) and (3) in the list above. The main reason for this is that I don't want to go through the pip install process (if it is even possible) to re-install ipython, scipy, the astro-packages, etc. for this (and maybe other) virtual environments.
Here are my questions:
Is there a way for me to specify to virtualenv that I would like it to link/copy packages in these two dist-packages directories for virtual environments that are created in the future?
When I eventually update my global installation of scipy, ipython, etc. in the two dist-packages directories, will this also update/change the packages that my virtual environment uses (and which it originally got during virtualenv creation)?
If I ever install a package from source on Ubuntu, will it go in /usr/local/lib/python2.7/dist-packages, or /usr/local/lib/python2.7/site-packages?
Thanks in advance for your help!
This might be a legitimate use of PYTHONPATH - an environmental variable that virtualenv doesn't touch, which uses the same syntax as the environmental variable PATH, in bash PYTHONPATH=/usr/lib/python2.7/dist-packages:/usr/local/lib/python2.7/dist-packages in a .bashrc or similar. If you followed this path,
You don't have to tell your virtual environment about this at all, it won't try to change it.
No relinking will be required, and
That will still go wherever it would have gone (pip install always uses /usr/local/lib/python2.7/dist-packages/ for my Ubuntu) if you install them outside of your virtual environment. If you install them from within your virtual environment (while it's activated) then of course it'll be put in the virtualenvironment.
I'm just getting my head around virtualenv, but there seems to be an easier way than mentioned so far.
Since virtualenv 1.7 --no-site-packages has been the default behavior.
Therefore using the --system-site-packages flag to virtualenv is all that is needed to get dist-packages in your path - if you use the tweaked virtualenv shipped by Ubuntu. (This answer and this one give some useful history). I've tested this and it does work.
$ virtualenv --system-site-packages .
I agree with Thomas here - I can't see any action required in virtualenv to see the effect of updates in dist-packages.
Having tested that with python setup.py install, it does (again as Thomas said) still go to dist-packages. You could change that by building your own python, but that's a bit extreme.
PYTHONPATH works for me.
vim ~/.bashrc
add this line below:
export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/dist-packages:/usr/local/lib/python2.7/dist-packages
source ~/.bashrc
In the directory site-packages, create a file dist.pth
In the file dist.path, put the following:
../dist-packages
Now deactivate and activate your virtualenv. You should be set.
What you want to achieve here is essentially add specific folder (dist-packages) to Python search path. You have a number of options for this:
Use path configuration (.pth) file, entries will be appended to the system path.
Modify PYTHONPATH (entries from it go to the beginning of system path).
Modify sys.path directly from your Python script, i.e. append required folders to it.
I think that for this particular case (enable global dist-packages folder) third option is better, because with first option you have to create .pth file for every virtualenv you'll be working in (with some external shell script?). It's easy to forget it when you distribute your package. Second option requires run-time setup (add a envvar), which is, again, easy to miss.
And only third option doesn't require any prerequisites at configure- or run-time and can be distributed without issues (on the same-type system, of course).
You can use function like this:
def enable_global_distpackages():
import sys
sys.path.append('/usr/lib/python2.7/dist-packages')
sys.path.append('/usr/local/lib/python2.7/dist-packages')
And then in __init__.py file of your package:
enable_global_distpackages()