Python venv using system python? - python

Why does python -m venv py38 use the system python still??
This just burned my pretty severely, I had a very customized python 3.8 environment that I thought was isolated from the system as advertised by venv, my system python just upgraded to 3.9 and now I have to re-build everything.
Any suggestions on a way to create actually isolated python environments would be appreciated.

https://docs.python.org/3/library/venv.html
Running this command creates the target directory (creating any parent directories that don’t exist already) and places a pyvenv.cfg file in it with a home key pointing to the Python installation from which the command was run (a common name for the target directory is .venv). It also creates a bin (or Scripts on Windows) subdirectory containing a copy/symlink of the Python binary/binaries (as appropriate for the platform or arguments used at environment creation time).
This makes me think that it's possible to have not a copy of the python installation but a link to it.
Anyway pyenv.cfg should have the path of your python, I suggest to change it.

Related

No module recognised after creating new virtualenv

So, i have python 3.11 installed, i wanted to run a code with flask module, and it was running using original python, but not in vscode, so i followed a suggestion from stack exchange and used command python -m venv env.
This created a new folder called env, and after that i got a prompt that new virtual environment has been created, do you want to use it for the workspace folder, i said yes, and still nothing worked. And now even other modules are not being recognized either by vscode or the original python.exe.
I am a novice in programming and have no idea what to do, please help.
Thanks
Once a venv is created, you must activate it like so: C:\User\venv_folder\Scripts\ activate
If this doesn't work try using C:\User\venv_folder\Scripts\ activate.ps1 or activate.bat
You'll know if it works once you have the (venv_name) before your command line in the terminal
python -m venv env
According to the docs. Running this command creates the target directory (creating any parent directories that don’t exist already) and places a pyvenv.cfg file in it with a home key pointing to the Python installation from which the command was run (a common name for the target directory is .venv). It also creates a bin (or Scripts on Windows) subdirectory containing a copy/symlink of the Python binary/binaries (as appropriate for the platform or arguments used at environment creation time). It also creates an (initially empty) lib/pythonX.Y/site-packages subdirectory (on Windows, this is Lib\site-packages). If an existing directory is specified, it will be re-used.
I think what you need more is to create a conda environment. Use the following command to create it (take python 3.10.4 as an example), and then you can manually select it in python interpreter.
conda create -n env-01 python=3.10.4
You can also read vscode docs for more details.

How do I reroute a project to another Python version in PyCharm? (Error: No Python at 'C:\Users\...\python.exe')

I wanted to replace Python 3.8 32-bit with the 64-bit version to install the face_recognition module, so I deleted the previous version and tried to re-route the project to the new Python version by going to File > Settings > Project Interpreter > Show all > Show Paths for Selected Interpreter, and adding all the Python files from the new folder and getting rid of the old ones.
However, it's still showing me this error when I try to install the module:
(Will2.0) C:\Users\solei\PycharmProjects\Will>pip install face_recognition
No Python at 'C:\Users\solei\AppData\Local\Programs\Python\Python38-32\python.exe'
I've also tried going to the Windows System Properties and changing everything that says "Python38-32" there, but it's still not working. It does work when I make a new environment, though, so at least I know that Python installed properly. It's just this one environment that is tripping me up (I'd prefer not to make a new project for this, btw. I've already installed a lot of modules in it.).
Your selected interpreter is not the system interpreter you've replaced with the 64-bit version, but your project's virtual environment interpreter. The virtual environment's files weren't changed in that process and need to be updated before you can use that environment again.
The system interpreter is your Python interpreter installed using the installation executable. In your case it is located in C:\Users\solei\AppData\Local\Programs\Python\Python38\. You can have multiple system interpreters installed, such as having Python 2.7, Python 3.7 and Python 3.8 side-by-side.
The virtual environment interpreter is a copy of another interpreter created using the venv package from the Python standard library. You can have many virtual environments interpreters in the system (one or more for every project, for example)
The base interpreter is the interpreter that was used as a template for the venv package. Every virtual environment interpreter has its base interpreter (usually a system interpreter) that it requires to run. Changing or upgrading the base interpreter requires updating the virtual environment.
If we take a quick look at the documentation, a virtual environment is described as
a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.
That means you can setup an individual environment for every project, which will contain its own packages. The environment is a very efficient way of managing project packages, that's why PyCharm suggests a creation of such environment over the system interpreter by default. In short, it allows you to have two different versions of the same package used by two different projects, without the packages conflicting with each other.
This also explains why your virtual environment files weren't affected by your upgrade.
Now, I am unfortunately no Python expert. I had to spend some time examining how Python handles virtual environments on Windows and Ubuntu. It seems the environment always requires the base system interpreter present in the system. If you remove or change the location of the base interpreter, the environment will fail to function.
As I mentioned before editing this answer, you can in theory simply edit the pyenv.cfg file located in the root folder of the virtual environment. In practice, that will only work in simple cases and it is not the intended way of updating virtual environments.
You need to upgrade your virtual environment's files to work with your new system interpreter. That can mean the 64-bit version over the 32-bit version, or even a newer version of Python - such us upgrading from 3.7 to 3.8.
Close PyCharm
Check if the system interpreter you want to upgrade to is on the system Path
You can quickly check by running
python -c "import platform; print(platform.architecture())"
For you, the output should look like this
('64bit', 'WindowsPE')
If your output is different, you'll need to prefix the absolute path to the Python executable in step 4).
Navigate to the virtual environment's directory
The directory you're looking for contains the Include, Lib and Scripts directories and the pyenv.cfg file. From your screenshots, it seems this directory is your project's root directory, so in your case:
cd C:\Users\solei\PycharmProjects\Will2.0\
Upgrade the virtual environment
python -m venv --upgrade .
... or if Python is not on your path
C:\Users\solei\AppData\Local\Programs\Python\Python38\python.exe -m venv --upgrade .
The . in the commands refers to the current directory.
Open PyCharm and verify your environment is working correctly
... or simply try to run pip directly from the command line. Note you need to first activate the virtual environment by running the Scripts\activate.bat batch file.
If the above-mentioned method doesn't work, you might have to create a new virtual environment. You can create one easily without making a new PyCharm project. See this PyCharm documentation for reference. However, you'll still need to redownload all the required packages again.
For the simplicity, I recommend creating the new virtual environment in a .venv folder located in the project's root.
Disclaimer
I tested only the Python's behavior alone on a fresh Windows installation inside the Windows Sandbox. I was able to install the 32-bit Python, create a virtual environment, replace Python with the 64-bit version and upgrade the virtual environment to have it launch correctly again.

What makes in Python a virtual environment a virtual environment?

I want to find the Python virtual environments ("venvs" for short) in my file system, and for this I need to know what set of files constitutes a venv.
Some tools - when they want to make sure a directory is a venv - check for the pyvenv.cfg file, but older virtual environments and environments created by Pipenv do not have this file. - Furthermore pyvenv.cfg does not seem to be necessary for activating the venv either.
So what is the minimal set of files to define a working virtual environment? And what is an easy way to check for this in Python?
Clarification:
With "working virtual environment" I mean:
A way to use a specific version of Python executable defined in that environment (and possibly different to the default system Python)
A way to use specific packages (usually installed via pip, bit that's not a requirement).
The Python executable in the venv should be aware of being run inside a venv (see
sys.base_prefix and sys.prefix in the Python documentation.)

Python virtualenv activation working but interpreter doesn't

I've just setup a new environment for my project and uploaded a python repository including bin, lib and project folder. I'm pretty sure I did same previously and it worked without problem. Now when doing the same on an AWS environment I get the error
-bash: /projects/scrapy/bin/python2.7: cannot execute binary file. However when doing source /projects/scrapy/bin/activate it successfully activates the environment.
From what I understand, python should be able to execute without any issue no matter the environment ?
Any help or pointing to the right direction would be much appreciated!
python should be able to execute without any issue no matter the environment ?
No, the Python binary is tied to your specific OS and computer architecture. Python source code can usually be run on different machines (provided you didn't use OS-specific features), but that's only made possible by compiling a Python interpreter for the specific target environment first.
In other words, a Python binary compiled to run on macOS will not work on Linux.
All that source bin/activate achieves is that it configures your terminal setting to use the bin directory as the first directory on the PATH search path. This doesn't make bin/python work in another environment, it just means that both environments have a working shell interpreter that can run that script.
Create a new virtualenv with a Python binary compiled for Linux, and install the same packages there. Use Pipenv or a requirements.txt file to transfer the dependencies from Mac to Linux.
For example, using Pipenv you'd copy over the Pipfile and Pipfile.lock files to the other computer, then run pipenv install in the directory there and re-create the virtualenv and dependencies from those files.
I recommend you read up on Python development best practices in the The Hitchhiker’s Guide to Python; this includes such topics on how to manage an environment for a project.

Virtual Env on Mac with python 3.4

I am trying to get a django project up and running, and I have django running, but I am having trouble with python's virtualenv.
here is my error (on terminal open this shows up)
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named virtualenvwrapper
virtualenvwrapper.sh: There was a problem running the initialization hooks.
If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python and that PATH is
set properly.
Here is what My .bash profile looks like:
# Setting PATH for Python 3.4
# The orginal version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.4/bin:${PATH}"
export PATH
# Setting PATH for Python 2.7
# The orginal version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH}"
export PATH
export WORKON_HOME=$HOME/.virtualenvs
source /Library/Frameworks/Python.framework/Versions/3.4/bin/virtualenvwrapper.sh
export PIP_VIRTUALENV_BASE=$WORKON_HOME
if which rbenv > /dev/null; then eval "$(rbenv init -)"; fi
I am pretty new to command line stuff and virtual environments, so I might not know some "obvious" things I am doing wrong. Thanks.
First some basics of the command-line shell:
PATH is an environment variable that contains a list of filesystem directories. When you type a command such as ls, python or virtualenvwrapper.sh your shell will search each directory starting from the first one listed. To see your current PATH type:
$ echo $PATH
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
To see what a given command is going to resolve to the location of the program that is going to be run, use the which command:
$ which ls
/bin/ls
Now in your example you are first adding a Python 3.4 location to your PATH and then a Python 2.7 location. The latter location is going to be first on your PATH. So all your Python related commands are first going to try and run Python 2.7, if a command isn't found there, it next searches in your Python 3.4 installation.
You appear to have installed virtualenvwrapper for Python 3.4, but when you run virtualenvwrapper.sh from the line:
source /Library/Frameworks/Python.framework/Versions/3.4/bin/virtualenvwrapper.sh
You are explicitly running the virtualenvwrapper.sh installed for Python 3.4. This runs a python command where your Python 2.7 is run, which does not appear to have virtualenvwrapper installed, as shown by this error message:
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named virtualenvwrapper
If you tried removing the Python 2.7 location addition to your PATH and it didn't work, you have to remember that changes to your .bash_profile are not automatically reflected in your terminal. You have to create a new terminal session and that new session will read your new .bash_profile.
If you are trying to get the command virtualenv to resolve to two different Pythons, it's simply not possible. It will always resolve to one specific Python based on the ordering of the locations on your PATH. VirtualEnv does install alternate versions of the command with the version of the Python built into the name. Use the commands virtualenv-3.4 and virtualenv-2.7 to create virtual environments for each different Python.
The key to all of this is remembering that PATH is used to resolve to an absolute path location of a program this is run. Use the echo $PATH and which commands to help you understand how that final path is being resolved.
It may seem tedious, but typing out the full absolute path is always going to side-step the magic of PATH resolution:
$ /Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenv
Try using smaller steps on your way to manipulating the path. Use just virutalenv-3.4 to make a Python 3.4 virtual environment and only bring in virtualenvwrapper as you need it and after your understanding of shell environments is more solid.
Finally, even pro's can get tripped up by playing games manipulating the path. You are ultimately trying to take a bunch of different application locations and flatten them all into one namespace. At some point conflicts can become unresolveable. A Python installation tool such as Buildout has it's own learning curve, but it hard-codes the absolute path to the python of each python script that it installs. Hard-coding absolute paths is the only way to deal with extreme corner cases such as having two builds of Python 3.4 side-by-side where you have two scripts which each need to run on two different builds of the same Python. Hard-coding absolute locations is also desirable in production environments, because then your application is immune to any changes to the bash shell. If you forget about a required ordering of your PATH, or another sysadmin tinkers with the PATH, you won't find your application breaking unexpectedly.

Categories