I am starting my first actual python project. I follow "Learn Python the Hard Way" to make an initial Python skeleton and I am using virtualenv too.
Now I want to use git to do the version control. According to some previous questions in SO, I am not suggested to commit any virtualenv files. Instead, I could use pip freeze > requirements.txt, and use .gitignore to ignore the virtualenv directories.
However, both virtualenv and Python project skeleton require a /bin directory, should I commit it as well? (Actually I don't really know what the role /bin is playing in Python project)
Any suggestions are appreciated, if there is something wrong with my process to set up a Python project, please correct me.
yourproject/bin is distinct from yourproject/env/bin, where yourproject/env is the virtual environment's directory (and neither of them is /bin in the root directory). You should ignore everything in env, and indeed, your project should work for someone who isn't using a virtual environment, or is managing it differently. Otherwise, you lose the benefits.
Let's imagine for a second that you finish your project, and I want to use it for a new task. I start a new project with a virtualenv of its own, and install some other components I want to use, and then yours. Oops, now I have an older version of Python than I started out with, and that bug in deactivate from two years ago somehow got resurrected. Imagine debugging that, let alone the annoyance of finding that your project replaced some files of mine.
(As these things go, the bin directory is a pretty static part of the virtual environment; zapping other parts of my private env would be much more destructive. If you committed lib you would have prevented me from installing any other components before yours.)
Related
I'm confused as to where I should put my virtualenvs.
With my first django project, I created the project with the command
django-admin.py startproject djangoproject
I then cd'd into the djangoproject directory and ran the command
virtualenv env
which created the virtual environment directory at the same level as the inner djangoproject directory.
Is this the wrong place in which to create the virtualenv for this particular project?
I'm getting the impression that most people keep all their virtualenvs together in an entirely different directory, e.g. ~/virtualenvs, and then use virtualenvwrapper to switch back and forth between them.
Is there a correct way to do this?
Many people use the virtualenvwrapper tool, which keeps all virtualenvs in the same place (the ~/.virtualenvs directory) and allows shortcuts for creating and keeping them there. For example, you might do:
mkvirtualenv djangoproject
and then later:
workon djangoproject
It's probably a bad idea to keep the virtualenv directory in the project itself, since you don't want to distribute it (it might be specific to your computer or operating system). Instead, keep a requirements.txt file using pip:
pip freeze > requirements.txt
and distribute that. This will allow others using your project to reinstall all the same requirements into their virtualenv with:
pip install -r requirements.txt
Changing the location of the virtualenv directory breaks it
This is one advantage of putting the directory outside of the repository tree, e.g. under ~/.virtualenvs with virutalenvwrapper.
Otherwise, if you keep it in the project tree, moving the project location will break the virtualenv.
See: Renaming a virtualenv folder without breaking it
There is --relocatable but it is known to not be perfect.
Another minor advantage: you don't have to .gitignore it.
The advantages of putting it gitignored in the project tree itself are:
keeps related stuff close together.
you will likely never reuse a given virtualenv across projects, so putting it somewhere else does not give much advantage
This is an annoying design flaw in my opinion. They should implement virutalenv in a way that does not matter where the directory is, as storing in-tree is just simpler and more isolated. Node.js' NPM package manager does it without any problem. And while we are at it: pip should just use local directories by default just like NPM. Having this separate virtualenv layer is wonky. Node.js just have NPM that does it all without extra typing. I can't believe I'm prasing the JavaScript ecosystem on a Python post, but it's true.
The generally accepted place to put them is the same place that the default installation of virtualenvwrapper puts them: ~/.virtualenvs
Related: virtualenvwrapper is an excellent tool that provides shorthands for the common virtualenv commands. http://www.doughellmann.com/projects/virtualenvwrapper/
If you use pyenv install Python, then pyenv-virtualenv will be a best practice. If set .python-version file, it can auto activate or deactivate virtual env when you change work folder. Pyenv-virtualenv also put all virtual env into $HOME/.pyenv/versions folder.
From my personal experience, I would recommend to organize all virtual environments in one single directory. Unless someone has extremely sharp memory and can remember files/folders scattered across file system.
Not a big fan of using other tools just to mange virtual environments. In VSCode if I configure(python.venvPath) directory containing all virtual environments, it can automatically recognize all of them.
For Anaconda installations of Python, the "conda create" command puts it in a directory within the anaconda3 folder by default. Specifically (for Windows):
C:\Users\username\anaconda3\envs
This allows other conda commands to work without specifying the path. One advantage, not noted above, is that putting environments in the project folder allows you to use the same name for all of them (but that is not much of an advantage for me). For more info, see:
https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
In Python 3.5 the recommended way to create virtual environments is with the venv, instead of virtualenv. Still the python packaging tutorial mentions both tools.
However virtualenvwrapper is a recommended wrapper tool to use when using virtualenv.
My questions are then:
Is there a way to use virtualenvwrapper with venv?
Or could one even consider virtualenvwrapper not needed due to venv? (I cannot see how this could be true since it is a wrapper solving
another problem)
Edit: I can see that there is some confusion in the answers to my question. venv is Python's official equivalent of virtualenv, as explained in the links above. Multiple stack overflow questions suggests that venv should be used. As mentioned in the "duplicate" you suggested:
the introduction of venv is in part a response to that mess. If you want to help improve the situation, I suggest you use venv and encourage others to do the same
So it is encouraged to use venv. But as this questions implies, if one is to use venv how does one use a wrapper like virtualenvwrapper
Here is a custom but still clean and clear solution. Append This script to in .bashrc / .bash_profile / .zshrc, it will give you basic management of venv.
In addition, you can extend the script by adding the following line so that it will also display the existing venv lists.
lsvenv(){
ls $VENV_HOME
}
Is there a way to use virtualenvwrapper with venv?
Yes. Simply point WORKON_HOME at your venvs directory. Here's what I do in my ~/.zshrc, and I use a mix of virtualenv (rare now, just for a few legacy py2 needs) and venv (most common). I renamed the default to .venvs to make it clear these are mostly Python 3 venvs and not virtualenvs.
# Python Environment Handling
export WORKON_HOME=$HOME/.venvs # Default name changed from virtualenv to highlight I am using python3 -m venv (aka pyvenv)
export PROJECT_HOME=$HOME/dev
source /usr/local/bin/virtualenvwrapper.sh # symlinked to /Library/Frameworks/Python.framework/Versions/3.7/bin/virtualenvwrapper.sh
Or could one even consider virtualenvwrapper not needed due to venv? (I cannot see how this could be true since it is a wrapper solving another problem)
venv == virtualenv (in short). venv does not supersede virtualenvwrapper for the same reason virtualenv does supersede it. Your hunch on this is correct.
How is this better than just making an alias that sources the activate?
Well, it's up to you to determine what you need, but I find virtualenvwrapper with zsh plugins virtualenv and virtualenvwrapper to be pretty great and better than raw aliases.
workon to list all venvs is very nice, then workon speech_analyzer to jump right in on it.
Other solutions?
You can also set up hooks to activate venvs on directory change, but if that's what you are after and only that, then that's essentially pipenv. Pipenv is great if that's all you want to do. Pipenv also has an interesting and promising lockfile feature, but it is too slow for development and too immature with issues in production to comment on at this time.
But I've never liked the 1:1 environment per project workflow for the same reasons given here: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/. Particularly these needs line up with mine of the multi project single environment: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#nikola
I have six environments on my machine and about 20 projects. Pipenv doesn't extend to that situation. Pipenv insists on 20 environments for 20 projects. It just doesn't work and creates more problems than it solves. If you do have a 1:1 workflow though currently, then pipenv may be the tool you'd like. One word of caution, you can ~only~ do that workflow in pipenv, unfortunately.
[EDIT, 8 Jul 19: Readers will probably find that this answer gives a fuller
description of the different tools for handling virtual environments
in Python. They all have their issues, as does conda, which has a
somewhat more sophisticated concept of "environment".]
As its name suggests, virtualenvwrapper was specifically designed to
wrap virtualenv, on which it depends. I don't believe that anyone
has similarly wrapped venv yet.
venv is intended to do the basic work of creating virtual
environments, but environment management has to be done with
scripts. Although shell scripting is often people's first
resort, the venv module has an extensive API to help you in
those tasks.
Nowadays there are many options for creating Python virtual environments. Besides those you mention, anaconda allows the creation and management of environments and even works pretty well with pip most of the time.
The tools in the virtual environment space have been designed to work together with standard Python distribution tools as far as possible, but the arrival of venv in Python 3.5 did not invalidate either virtualenv or virtualenvwrapper, which should both still work fine.
The venv module is principally a simple toolkit to allow the creation of virtualenvs inside programs, and is by no means intended to replace the convenience of virtualenvwrapper. It simply meets a rather different set of needs.
In this question on S/O:
Can existing virtualenv be upgraded gracefully?
The accepted answer says that you can:
use the Python 2.6 virtualenv to "revirtual" the existing directory
I cannot seem to find any details on how to "revirtual" an existing virtualenv. I know how to manually install python, but I am looking very specifically for the "python / virtualenv" way to upgrade python inside a specific virtualenv.
In my very specific situation, I have become an administrator in someone else' stead. The system python is 2.6.6, however the virtualenvs are all using 2.7.4 within the virtualenv path, which is something like:
/home/user_name/.virtualenvs/application_name/bin/python2.7
While there is no python 2.7.x in the system. I cannot find any evidence of python having been manually installed, and I cannot find any details online about using pip or using apt-get/yum or anything to install different versions of python within the virtualenv.
So, my very specific questions are:
How does one "revirtual" a virtualenv?
How does one upgrade python within a virtualenv the "python" way?
Are there alternative ways to manage python versions within virtualenvs?
Thanks, and please let me know if I can clarify my questions in any way!
-S
Usual caveat: make backup copies first. However, the structure created by virtualenv is not that complex, so it should be possible to either find what you need or create a new one and migrate. The path with ~/.virtualenvs in it means it was probably created with virtualenvwrapper so you could read up on that too.
virtualenv makes copies of the executables in the bin directory, which means they don't have to exist elsewhere. In the lib and includes directories, however, by default there will be symlinks to items from the "source" python (unless someone changed that setting). You could do a ls -l in those directories and maybe you will find where python 2.7 is installed - or maybe some broken symlinks.
Under lib should be one or more python-<version> directories, with some symlinks (probably) to python standard library stuff and a site-packages directory, which is where your installed packages are.
Now, upgrading. If you try to update the virtualenv "in-place" (i.e. you just run virtualenv --python==python<version> <existing-directory>) then you'll probably run into two issues: (1) The bin/python symlink will not be replaced unless you delete/move it, and (2) the directories under lib are per python version, so if you don't use 2.7 then the site-packages directory won't automatically carry over -- you'll have to reinstall packages, or go in and move it, which will probably work unless you have compiled binary extensions, or are migrating to 3.x, or something else weird happens.
An alternate, cleaner approach is to create a new, fresh virtualenv, and then reinstall the necessary packages. Find your own app's package(s) first and figure out if they have a setup.py that works or are just code that's sitting in a directory. If they don't have setup.py that correctly brings in all dependencies, then in your existing virtualenv folder you can run ./bin/pip freeze to get a listing of installed packages. You can save this to a file and use pip install -r <filename> on it later.
Experimenting with using python in a virtualenv on my shared hosting account. Based on this dreamhost tutorial have installed pip and another module or two (echonest, remix), but trying to install numpy the long list of errors starts with non-existing path in 'numpy/distutils': 'site.cfg'
/bin/sh: svnversion: command not found.
The virtualenv instructions I read say, "make sure that your path gives preference to ~/local/bin to /usr/bin so that your "local" copy of Python runs, and that your scripts refer to that location."
Does that suggest to make a link somewhere that points calls to /usr/bin/ to ~/local/bin?
Is the solution to find the install package and edit the paths in it's setup.py file?
this is referring to the linux environment variable $PATH which lists the directories in which to look for executables for commands when you don't specify an absolute path. this will contain a list of comma separated paths eg:
/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
You just need to make sure that the /usr/local... stuff comes first (left) like this:
export PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
Are you sure you're running ~/local/bin/python when running setup.py?
One virtualenv-specific thing you can do is source ~/local/bin/activate, which automatically sets your virtualenv to take preference over everything else in your path. It only works until you log out of your terminal instance or run deactivate
I'm seeing confusion in the comments--- between "local" and "your virtualenv". And there is a parallel problem with $PATH. Is there just one path issue? No... there are two.
True, my own analogous efforts on a different shared-hosting account are ultimately crashing too, but I'm further along and I appear to have some things figured out. The confusion here partly comes about because there are two Dreamhost tutorials that are linked to in the question, but the authors of those two wiki chapters didn't make any effort to integrate the two.
And in the second tutorial on Python the mistake is made of introducing the discussion of the local directory in the virtualenv section though it's only really explained in the next section on building your own Python. And the discussion on building your own isn't complete regarding what to do about $PATH. You don't need to build your own Python or have a local directory just to use virtualenv.
It may even be detrimental. The language at this official virtualenv page should fill you with despair, as it shows no effort at clarity. I think that it means that mod_python and mod_wsgi, either of which you must be using for your Flask experiment, don't use any Python that you might come up with--- they use their own Python interpreter, the system's version. But virtualenv, the program, also puts a Python interpreter inside your virtual environment (the interpreter that you will use to do installs and also ultimately to run flask if you let it). The point of the warning from the virtualenv folks is that you can't run with two different interpreters in the same server process. They show you the workaround.
But I have digressed. The second, Python tutorial instructs about building your own local Python, in the directory /home/yourusername/local. A more complete, yet succinct discussion is to be found at Sugath Mudali's Blog.
Your virtualenv should occupy a second directory, which I'll call yourvirtenv. Alongside ~/local should be OK.
So, per Sugath Mudali's helpful instructions, you need to execute export PATH=/home/yourusername/local/bin:$PATH, which puts /home/yourusername/local/bin at the front of your $PATH, which you can confirm with "echo $PATH". Once you've done that, whenever you use "python" in your shell you'll be using the Python that you just installed, whose path you just put first.
Having done that, if you then cd into to your virtualenv source folder (somewhere in /home/youruserhame) and run "python virtualenv.py /home/youruserhame/yourvirtenv" it will create a directory ~/yourvirtenv, into which it will put a copy of the interpreter of your local Python. Generally it will not only be a distinct copy but it will differ even as to version from that of your system, which is at /usr/bin/ as /usr/bin/python ("usr" here is not you but is the oddly-named directory of the entire shared Linux machine to which no unprivileged user such as you has access, into which program binaries can be installed, by the privileged, for system-wide use).
Note that the Dreamhost Python wiki says "DreamHost has begun upgrading servers to Python 2.6.6 as of February 2012." Version 2.6 is adequate for virtualenv and practically everything else. So I would question the need to build your own Python in your case. (Note that Dreamhost even suggests that you should try to forget about it entirely.)
Anyway, on to the second PATH deal, as limasxgoesto0 noted, once you run virtualenv.py and have thus created myvirtenv, if you're still in the /home/myusername user-root directory you run source myvirtenv/bin/source. That puts /home/myusername/myvirtenv/bin at the front of the path--- until you later simply run "deactivate" (no source in front of it). That is all that activate does.
Now, when you return days later and go back to work consider this: export PATH=/home/yourusername/local/bin:$PATH has evaporated. It's not permanent; goes away when the shell is closed. To make it permanent you have to add that statement to .bash_profile. Here's mine:
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
PATH=$PATH:$HOME/bin
export PATH
export PATH=$HOME/local/bin:$PATH
export LD_LIBRARY_PATH=/home/myusername/local/lib
unset USERNAME
I found it necessary to reassert the LD_LIBRARY_PATH in order to make sqlite3 function, even though I had compiled sqlite3 and done that export prior to building Python.
Note that this .bash_profile invokes .bashrc. Depending on how Dreamhost has arranged things, you may have to put these exports in .bashrc.
Likewise, when you return to the shell myvirtenv will not be found to still be activated even if you never typed-in "deactivate". You only activate it temporarily in the shell to install stuff inside it and to start your server or run some other program that you've installed. Thereafter, the installed stuff knows to use the Python interpreter that has been copied to myvirtenv. To make that not happen, to make the stuff run off of the system Python interpreter you have to do what the aforementioned official virtualenv docs on mod_python, mod_wsgi say.
It looks as though this may be necessary even if you have not built your own Python, because otherwise you would have two Python interpreters running in the same server process. I am however sketchy on this. Flask is, like Django, a Python framework. An A2 Hosting tutorial--- they use Phusion Passerger too--- may be of interest. On their site search for "Django shared" and the tutorial will come up on top. Note that Chris C.'s instructions there tell you to use the --python=/path/to/python switch. In your case that could be /home/myusername/local/bin/python if you have locally installed Python, but in their case it's /usr/bin/python2.6. I think that their point is they don't want to get tech-support tickets from customers on problems caused by upgrades to their system python2.6 that could break your site. So that --python switch (-p directory is the short form; --python=directory is the long form) is to freeze the Python that your app will be using, to avoid such problems. It does so by copying all of the Python into your virtual environment, not just the interpreter.
I don't understand how they are avoiding the problem of mod_wsgi and the customer's apps using two different interpreters, but maybe that's what Phusion Passenger helps out with.
I have a Python project which contains three components: main executable scripts, modules which those scripts rely on, and data (sqlite3 databases, flat files, etc.) which those scripts manipulate. The top level has an __init__.py file so that other programs can also borrow from the modules if needed.
The question is, is it more "Pythonic" or "correct" to move my project into the default site-packages directory, or to modify PYTHONPATH to include one directory above my project (so that the project can be imported from)? On the one hand, what I've described is not strictly a "package", but a "project" with data that can be treated as a package. So I'm leaning in the direction of modifying PYTHONPATH (after all, PYTHONPATH must exist for a reason, right?)
Definitely do not add your project to site-packages this is going to spoil your system Python installation and will fire back at the moment some other app would come there or you would need to install something.
There are at last two popular options for installing python apps in isolated manner
Using virtualenv
See virtualenv project. It allows
creation of new isolating python environment - python for this environment is different from system one and has it's own PYTHONPATH setup this allows to keep all installed packages private for it.
activation and deactivation of given virtualenv from command line for command line usage. After activate, you can run pip install etc. and it will affect only given virtualenv install.
calling any Python script by starting by virtualenv Python copy - this will use related virtualenv (note, that there is no need to call any activate)
using zc.buildout
This package provides command buildout. With this, you can use special configuration file and this allows creation of local python environment with all packages and scripts.
Conclusions
virtualenv seems more popular today and I find it much easier to learn and use
zc.buildout might be also working for you, but be prepared for a bit longer learning time
installing into system Python directories shall be always reserved for very special cases (pip, easy_install), better avoid it.
installing into private directories and manipulatig PYTHONPATH is also an option, but you would repeat, what virtualenv already provides