I'm a bit confused on this and can't find a satisfactory answer searching elsewhere - everything is about using external libraries in some way or another.
In my PyCharm projects, I've got your project root directory, which contains a venv. This venv of course contains the python executable relative to the project, as well as the relevant site-packages and modules I've downloaded via pip or whatever.
But there's always an External Libraries section, that has it's own Python executable, venv, .gitignore, site-packages, etc.
Is this normal? What's the purpose of having two venvs in the same project?
Thanks!
You only have one venv normally in your Project
The other Python Executable should be the global one.
Normally you use a venv to keep track of the needed Packages for this specific project. So that you don't confuse other Packages with other projects or have dependency/Version Issues
If you want you can also use the global Installation of Python which would not create a venv
Related
Let's say I don't want to access this python installation from PATH or py on the console, just call on the python.exe file directly. In that case, can I copy the python installation directory into an app folder for something I'm developing to have essentially a portable python packaged with my app?
Yes you can do that. But there is a concept in Python called virtual environments that you should take a look at. The idea is to seperate a project environment from the one on your operating system. It's the recommended way for developing python applications as you can have multiple different environments depending on your projects requirements.
https://docs.python.org/3/tutorial/venv.html
You can also take a look at Conda. Its a more user friendly way of using Environments (in my opinion). If you are interested in Conda you can also look at that question for more information about the differences of Miniconda and Anaconda: Anaconda vs. miniconda
Installation of Conda: https://docs.conda.io/projects/conda/en/latest/user-guide/install/windows.html
I'm a huge beginner so I'm not very informed about how packages really work. I know that you should create a virtualenv in your project folder to avoid version conflicts etc, and you're not supposed to put your actual project files in the virtual env. So if your project files are in your project directory on the same level as the virtualenv, can your project files "access" the things installed in the virtualenv? Can files outside of your directory access packages in your virtual env?
Yes, it all depends on the context. Your virtualenv can exist anywhere, be it in your project directory, or somewhere else.
When you want to use the virtualenv, you just have to call source command on it. Then whatever python command you execute on whichever file, will have access to the virtualenv. For example, if you store your virtualenv in /home/user/project/virtualenv, then you would do
source /home/user/project/virtualenv/bin/activate
Then whatever you with the python, it would be the version installed in virtualenv.
You can double check if you're using the global python or the virtualenv python by doing which python. It will either point to the global python path which is usually under /usr/bin/python or /home/user/project/virtualenv/bin/python.
So normally, you first do the source command, then you can do pip install on whatever packages you need already. It will be installed in the virtualenv and it will not conflict with other projects.
Why does pip need to use virtual environments to isolate packages per-project, instead of just installing them in a default directory in the project? This seems like added complexity without benefit.
NPM, for example, installs packages in the <project_root>\node_modules by default. No virtual environment necessary, and packages are still installed in a project-independent way.
Edit: To be clear, I'm interested in the practical advantages to pip's use of virtual environments over package management systems like NPM, Nuget, and Webpack, which all use directories in the project. Otherwise, if this is just a limitation of Python's modules system, then I'd be interested to know that too.
Because Python's module system doesn't work that way. If pip were to install, say, requests by just downloading it to a python_modules directory, that wouldn't be enough to for import requests to work; it would have to be import python_modules.requests, but then we'd still have problems whenever requests tried to import one of its dependencies, as that would need python_modules prepended, too, and it'd just be a big mess. The solution that virtual environments use is to modify the PYTHONPATH environment variable to include python_modules, plus some extra stuff to take care of executable scripts and not importing packages from outside the virtualenv.
I think maybe you don't know what a virtual environment actually is.
If you were to put some module in a project-specific directory, like myproj/modules, then you would have to add myproj/modules to the search path that Python uses so that you module can be found. One way to do that is to define or modify the environment variable PYTHONPATH. Any directories listed in that variable will be searched for modules, in addition to some hard-coded set of directories.
$ export PYTHONPATH=./myproj/modules
However, that's really all a virtual environment is. The directory contains the desired version of Python, along with whatever modules you want to use. The activate script you run to "enable" a virtual environment does little more than set the value of PATH and PYTHONPATH so that anytime you run python, both the correct version is used and your project-specific set of modules is used in place of any global library.
I am newbie using VirtualEnv and recently try to create one using PyCharm. During the process, PyCharm ask me to specify the project location, application name and VirtualEnv name and location. My doubt is, after I specify the name and location of the VirtualEnv the location of the Django project files must be inside the VirtualEnv? or it's possible to have the VirtualEnv files in a different location than the Django project files?
Maybe I am not understanding the purpose of the VirtualEnv. Perhaps, VirtualEnv it's just a list of the dependencies of my project, Python version, Django version, Pip version, Jinja2 version and all other required files, but not necessarily the Django application files (the website that is being developed).
Thanks in advance.
Ya, I think you misunderstand what virtualenv does:
https://virtualenv.pypa.io/en/latest/
The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
Your project files don't need to be (and shouldn't be) where the virtualenv files are.
virtualenv installs your app's python dependancies in a folder for the specific virtualenv that is being used.
Let's say you are not using virtualenv, the dependencies would be installed into into the site-packages folder for your system. The dependancies aren't installed in your project directory and your project directory isn't in your system's site-packages directory.
Using virtualenv doesn't change that, it just changes where the dependencies are installed.
virtualenv is not just a list of dependencies! It actually has all the modules under its umbrella. Think of a virtualenv as a space which isolates all the packages used by your project from the rest of the other packages that were installed previously or at a later time.Yes, there is an option to have the virtualenv make use of packages that are "outside" of the environment but that's just an option.
The main purpose of having an virtualenv is to enable the user to use packages versions of his choice and keep them isolated from the rest of the space. Usually, the list of packages belonging to a specific virtualenv are captured into a file, requirements.txt. If you want to run the project on a different machine or share it with someone, having requirements.txt will make it easy to recreate the environment via pip install -r requirement.txt from within virtualenv
I have a Python project which contains three components: main executable scripts, modules which those scripts rely on, and data (sqlite3 databases, flat files, etc.) which those scripts manipulate. The top level has an __init__.py file so that other programs can also borrow from the modules if needed.
The question is, is it more "Pythonic" or "correct" to move my project into the default site-packages directory, or to modify PYTHONPATH to include one directory above my project (so that the project can be imported from)? On the one hand, what I've described is not strictly a "package", but a "project" with data that can be treated as a package. So I'm leaning in the direction of modifying PYTHONPATH (after all, PYTHONPATH must exist for a reason, right?)
Definitely do not add your project to site-packages this is going to spoil your system Python installation and will fire back at the moment some other app would come there or you would need to install something.
There are at last two popular options for installing python apps in isolated manner
Using virtualenv
See virtualenv project. It allows
creation of new isolating python environment - python for this environment is different from system one and has it's own PYTHONPATH setup this allows to keep all installed packages private for it.
activation and deactivation of given virtualenv from command line for command line usage. After activate, you can run pip install etc. and it will affect only given virtualenv install.
calling any Python script by starting by virtualenv Python copy - this will use related virtualenv (note, that there is no need to call any activate)
using zc.buildout
This package provides command buildout. With this, you can use special configuration file and this allows creation of local python environment with all packages and scripts.
Conclusions
virtualenv seems more popular today and I find it much easier to learn and use
zc.buildout might be also working for you, but be prepared for a bit longer learning time
installing into system Python directories shall be always reserved for very special cases (pip, easy_install), better avoid it.
installing into private directories and manipulatig PYTHONPATH is also an option, but you would repeat, what virtualenv already provides