My project's running fine in virtualenv. Unfortunately, I can't yet run it via my IDE (Eric) because of the import troubles. It stands to reason, as I never told the IDE anything about virtualenv.
I know the drill ($ source project/bin/activate etc.), but lack general understanding. What constitutes "running inside virtualenv"? What IDE options might be relevant?
I think the only required setting to run or debug code is path to python interpreter.
Relevant IDE options could be SDK or Interpreter settings.
Note that you should run not default python (eg. /usr/bin/python) but python binary in your virtual environment (eg /path/to/virtualenv/bin/python)
Also there are some environment variables set by activate, but i think they aren't needed when you point to virtualenv python binary directly.
So, again, what activate does is only environment variables setup: at least, it modifies system $PATH in a way that python and pip commands points to executable files under path/to/virtaulenv/bin directiry.
As far as I know it is possible to run your script/project using your virtualenv simply by calling /path/to/your/venv/python your_script.py. To install new packages in your venv for example, you would run /path/to/your/venv/pip install some_package.
I guess the main advantage of "runnnig inside virtualenv" would be not being concerned about having to inform the location of python packages/executable everytime you want to run some script. But I lack general understanding too.
I usually install a virtualenv with the --no-site-packages option in order to have a "clean" installation of python.
--- EDIT ---
The second answer of this discussion has a nice explanation.
Simply look at the project/bin/activate, everything you need is there to set up the appropriate search.
Usually the most important path is the PYTHONPATH which should point to the site-packages.
Related
I'm having an issue where for some reason the virtual environments that I'm creating are accessing my system-wide installations of Python and pip when they shouldn't be.
Here's my fairly simple workflow just make sure I'm not missing anything obvious (Windows 10, Python 3.8.2):
python -m venv venv
venv\Scripts\activate.bat
My path is now prepended by (venv), as you'd expect. However,
pip list
lists all of the system-wide pip packages I have instead of just the ones that should be in that venv.
pyvenv.cfg indicates that
include-system-site-packages = false
When I open the interpreter using
python
In the virtual environment,
sys.executable
Returns the path on my C drive and
print(pip.__file__)
Does the same. I suspect they should instead be pointing to the interpreter and pip in the virtual environment but don't know how to make that happen.
Edit: 4/27/20, Still dealing with this issue, I have tried:
uninstalling and reinstalling Python, both from python.org and the MS Store
Installing python in a new location
Clearing my user and system environment variables and then adding in
just the ones for Python 3.8.
I'm really at a loss here, would appreciate any help.
To anyone looking at this later on, I eventually realized that the problem was only occurring in a specific folder. Making virtual environments elsewhere on my computer seems to work fine. Why this is the case? I have no idea. I'm curious, but not enough to spend another second on this problem. So there you go.
This happened to me too. My problem was that I created the virtual environment under the directory including the global executable (I mean C:\Program Files\Python39). Once I removed this virtual environment and created a new one somewhere else (for example C:\Python), running python command took the executable in the virtual environment rather than the global one.
I had the same problem too. It might be that the path to your virtual environment has a character with accent in it. This basically means that you will have to alter the \Scripts\activate.bat file.
In order to see if that's the case, use the following procedure:
Activate the virtual environment by running the file \Scrips\activate.bat
Run the following command:
echo %PATH%
There should appear a bunch of paths, being path\to\your\virtual\environment\Scripts the first of them (this is obligatory). Check if this path is exactly the same as it is supposed to be (meaning it should have no "strange" characters instead of characters with accents).
If there are any discrepancies, I suggest you alter your \Scripts\activate.bat according to https://stackoverflow.com/a/22348546/11923771, reopen your Command Prompt and follow steps 1 and 2 again to see if your problem was solved.
I have been trying to run a python script on start-up (on a Pi). I initially did this via an .sh script triggered by cron.
Posting with a problem on StackExchange Pi (https://raspberrypi.stackexchange.com/questions/110868/parts-of-code-not-running-when-autostarting-script-in-crontab) the suggestion is to use systemd.
The person helping me there has suggested not using a virtual environment when executing the Python script (they note their limited familiarity with Python), and using the real environment instead. But other resources strongly suggest the use of a virtual environment (e.g. https://docs.python.org/3/tutorial/venv.html).
In the hope of setting this up correctly could anyone weigh in on the correct approach?
Use the virtual environment. I don't see any reason not to. At some point you might want to have multiple Python applications run at the same time on that system, and these applications might require different versions of the same dependency and then you would be back to square one, so... Use the virtual environment.
When configuring systemd, crontab, or whatever, make sure to use the python binary that is placed inside the virtual environment's bin directory, so that there is no need to activate the virtual environment:
/path/to/venv/bin/python -m my_executable_module
/path/to/venv/bin/python /path/to/my_script.py
/path/to/venv/bin/my_executable_script
systemd is going to try to run your script on startup so your virtual environment will not have been activated yet. You can (maybe) avoid that issue by telling systemd to use the python in the virtualenv's bin, with the appropriate environment variables. Or you can activate as a pre-run step for that script's launch in systemd. Maybe.
But on balance I'd make it easy on systemd and your OS and ignore the virtualenv absolutists. Get the script to work on your dev machine using virtualenv all you want, but then prep systemd to use the global python, with suitable packages installed. You can always use virtualenvs on that pi, for scripts that don't have to work with systemd. Systemd doesn't always have the clearest error messages.
(If you need to import custom modules you could inject directories into sys.path in your script. This could even avoid installing packages, for the global Python, entirely.)
This answer is certainly opinion-based.
So yesterday I had to create a virtualenv in order to be able to install Python modules that wouldn't install thanks to OS X El Capitan's new SIP. I thought I did everything right, but today I'm reaching a different conclusion. I hope I can be clear about it.
my python custom install is at myname/learnp/imdb_module, this is where I created it with virtualenv. Edit: I later moved it to myname/learnp/ayr2/imdb_module.
However, when I try to run the interpreter, it seems to always default to the Python that is in Library or something along these lines. I found out about this because a certain module that I managed to install in this custom python env wouldn't import, when I checked what modules I have, it wasn't the same as what I expected.
Furthermore, it seems that ALL other modules that I wanted to install on the CUSTOM virtualenv were installed on the main python env, and that I wasn't installing those modules on the custom env all along.
Excuse me, but I'm very confused right now.
I know how to create a virtual env
I know how to activate it (it appears to the right on Terminal line)
I don't know how to install modules to my virtual env
I don't know how to make the interpreter run from the virtual env so I can do python operations that are only possible by using custom env modules
Any advice is much appreciated!
Update:
Followed Will Hogan's answer for troubleshooting,and I think something weird is happening, quoting my comment to his answer:
HI, thanks for taking the time to answer. This is basically the way I understood this. However, let me attach a screenshot: http://i.imgur.com/DfpngJq.jpg . Am I right to assume something is wrong here? My prompt is changed with the virtualenv named "imdb_module", but when I type in which python it doesn't list ayr2/imdb_module/bin but rather a folder with the path usr/bin/python, which if I understand correctly is the "default" environment.
And not if this helps in any way, but echo $PATH when (imdv_module) appears to the right of the prompt, gives this (I redacted my name): /Users/REDACTEDNAME/learnp/imdb_module/bin:/Library/Frameworks/Python.framework/Versions/3.4/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
While creating the virtualenv you should see it installing setuptools and pip:
$ virtualenv testvenv
New python executable in testvenv/bin/python2.7
Also creating executable in testvenv/bin/python
Installing setuptools, pip...done.
After ensuring the virtualenv is activated you should see your prompt change:
$ . ./testvenv/bin/activate
(testvenv)$
Now you can confirm the paths to python and pip, which should be in the virtualenv:
(testvenv)$ which python
/private/tmp/testvenv/bin/python
(testvenv)$ which pip
/private/tmp/testvenv/bin/pip
If you aren't seeing the python and pip locations as being under the virtualenv's directory, then the virtualenv has not been activated.
I would also ensure that, if you're executing the .py file directly (and not with "python foo.py"), that your shebang line uses:
#!/usr/bin/env python
Or even the full path to the virtualenv's python, e.g.:
#!/tmp/testvenv/bin/python
As opposed to, say:
#!/usr/bin/python
The first will search in the current environment, which will be set by the virtualenv activation. The second explicitly points to the virtualenv's `python'.
Experimenting with using python in a virtualenv on my shared hosting account. Based on this dreamhost tutorial have installed pip and another module or two (echonest, remix), but trying to install numpy the long list of errors starts with non-existing path in 'numpy/distutils': 'site.cfg'
/bin/sh: svnversion: command not found.
The virtualenv instructions I read say, "make sure that your path gives preference to ~/local/bin to /usr/bin so that your "local" copy of Python runs, and that your scripts refer to that location."
Does that suggest to make a link somewhere that points calls to /usr/bin/ to ~/local/bin?
Is the solution to find the install package and edit the paths in it's setup.py file?
this is referring to the linux environment variable $PATH which lists the directories in which to look for executables for commands when you don't specify an absolute path. this will contain a list of comma separated paths eg:
/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
You just need to make sure that the /usr/local... stuff comes first (left) like this:
export PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
Are you sure you're running ~/local/bin/python when running setup.py?
One virtualenv-specific thing you can do is source ~/local/bin/activate, which automatically sets your virtualenv to take preference over everything else in your path. It only works until you log out of your terminal instance or run deactivate
I'm seeing confusion in the comments--- between "local" and "your virtualenv". And there is a parallel problem with $PATH. Is there just one path issue? No... there are two.
True, my own analogous efforts on a different shared-hosting account are ultimately crashing too, but I'm further along and I appear to have some things figured out. The confusion here partly comes about because there are two Dreamhost tutorials that are linked to in the question, but the authors of those two wiki chapters didn't make any effort to integrate the two.
And in the second tutorial on Python the mistake is made of introducing the discussion of the local directory in the virtualenv section though it's only really explained in the next section on building your own Python. And the discussion on building your own isn't complete regarding what to do about $PATH. You don't need to build your own Python or have a local directory just to use virtualenv.
It may even be detrimental. The language at this official virtualenv page should fill you with despair, as it shows no effort at clarity. I think that it means that mod_python and mod_wsgi, either of which you must be using for your Flask experiment, don't use any Python that you might come up with--- they use their own Python interpreter, the system's version. But virtualenv, the program, also puts a Python interpreter inside your virtual environment (the interpreter that you will use to do installs and also ultimately to run flask if you let it). The point of the warning from the virtualenv folks is that you can't run with two different interpreters in the same server process. They show you the workaround.
But I have digressed. The second, Python tutorial instructs about building your own local Python, in the directory /home/yourusername/local. A more complete, yet succinct discussion is to be found at Sugath Mudali's Blog.
Your virtualenv should occupy a second directory, which I'll call yourvirtenv. Alongside ~/local should be OK.
So, per Sugath Mudali's helpful instructions, you need to execute export PATH=/home/yourusername/local/bin:$PATH, which puts /home/yourusername/local/bin at the front of your $PATH, which you can confirm with "echo $PATH". Once you've done that, whenever you use "python" in your shell you'll be using the Python that you just installed, whose path you just put first.
Having done that, if you then cd into to your virtualenv source folder (somewhere in /home/youruserhame) and run "python virtualenv.py /home/youruserhame/yourvirtenv" it will create a directory ~/yourvirtenv, into which it will put a copy of the interpreter of your local Python. Generally it will not only be a distinct copy but it will differ even as to version from that of your system, which is at /usr/bin/ as /usr/bin/python ("usr" here is not you but is the oddly-named directory of the entire shared Linux machine to which no unprivileged user such as you has access, into which program binaries can be installed, by the privileged, for system-wide use).
Note that the Dreamhost Python wiki says "DreamHost has begun upgrading servers to Python 2.6.6 as of February 2012." Version 2.6 is adequate for virtualenv and practically everything else. So I would question the need to build your own Python in your case. (Note that Dreamhost even suggests that you should try to forget about it entirely.)
Anyway, on to the second PATH deal, as limasxgoesto0 noted, once you run virtualenv.py and have thus created myvirtenv, if you're still in the /home/myusername user-root directory you run source myvirtenv/bin/source. That puts /home/myusername/myvirtenv/bin at the front of the path--- until you later simply run "deactivate" (no source in front of it). That is all that activate does.
Now, when you return days later and go back to work consider this: export PATH=/home/yourusername/local/bin:$PATH has evaporated. It's not permanent; goes away when the shell is closed. To make it permanent you have to add that statement to .bash_profile. Here's mine:
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
PATH=$PATH:$HOME/bin
export PATH
export PATH=$HOME/local/bin:$PATH
export LD_LIBRARY_PATH=/home/myusername/local/lib
unset USERNAME
I found it necessary to reassert the LD_LIBRARY_PATH in order to make sqlite3 function, even though I had compiled sqlite3 and done that export prior to building Python.
Note that this .bash_profile invokes .bashrc. Depending on how Dreamhost has arranged things, you may have to put these exports in .bashrc.
Likewise, when you return to the shell myvirtenv will not be found to still be activated even if you never typed-in "deactivate". You only activate it temporarily in the shell to install stuff inside it and to start your server or run some other program that you've installed. Thereafter, the installed stuff knows to use the Python interpreter that has been copied to myvirtenv. To make that not happen, to make the stuff run off of the system Python interpreter you have to do what the aforementioned official virtualenv docs on mod_python, mod_wsgi say.
It looks as though this may be necessary even if you have not built your own Python, because otherwise you would have two Python interpreters running in the same server process. I am however sketchy on this. Flask is, like Django, a Python framework. An A2 Hosting tutorial--- they use Phusion Passerger too--- may be of interest. On their site search for "Django shared" and the tutorial will come up on top. Note that Chris C.'s instructions there tell you to use the --python=/path/to/python switch. In your case that could be /home/myusername/local/bin/python if you have locally installed Python, but in their case it's /usr/bin/python2.6. I think that their point is they don't want to get tech-support tickets from customers on problems caused by upgrades to their system python2.6 that could break your site. So that --python switch (-p directory is the short form; --python=directory is the long form) is to freeze the Python that your app will be using, to avoid such problems. It does so by copying all of the Python into your virtual environment, not just the interpreter.
I don't understand how they are avoiding the problem of mod_wsgi and the customer's apps using two different interpreters, but maybe that's what Phusion Passenger helps out with.
I know that virtualenv, if not passed the --no-site-packages argument when creating a new virtual environment, will link the packages in /usr/local/lib/python2.7/site-packages (for Python 2.7) with a newly-created virtual environment. On Ubuntu 12.04 LTS, I have three locations where Python 2.7 packages can be installed (using the default, Ubuntu-supplied Python 2.7 installation):
/usr/lib/python2.7/dist-packages: this has my global installation of ipython, scipy, numpy, matplotlib – packages that I would find difficult and time-consuming to install individually (and all their dependences) if they were not available via the scipy stack.
/usr/local/lib/python2.7/site-packages: this is empty, and I think it will stay that way on Ubuntu unless I install a package from source.
/usr/local/lib/python2.7/dist-packages: this has very important local packages for astronomy, notably those related to PyRAF, STScI, etc., and they are extremely difficult and time-consuming to install individually.
Note that a global directory such as /usr/lib/python2.7/site-packages does not exist on my system. Note also that my global installation of ipython, scipy, etc. lets me use those packages on-the-fly without having to source/activate a virtual environment every time.
Naturally, I now want to use virtualenv to create one virtual environment in my user home directory which I will source/activate for my future projects. However, I would like this virtual environment, while being created, to link/copy all of my packages in locations (1) and (3) in the list above. The main reason for this is that I don't want to go through the pip install process (if it is even possible) to re-install ipython, scipy, the astro-packages, etc. for this (and maybe other) virtual environments.
Here are my questions:
Is there a way for me to specify to virtualenv that I would like it to link/copy packages in these two dist-packages directories for virtual environments that are created in the future?
When I eventually update my global installation of scipy, ipython, etc. in the two dist-packages directories, will this also update/change the packages that my virtual environment uses (and which it originally got during virtualenv creation)?
If I ever install a package from source on Ubuntu, will it go in /usr/local/lib/python2.7/dist-packages, or /usr/local/lib/python2.7/site-packages?
Thanks in advance for your help!
This might be a legitimate use of PYTHONPATH - an environmental variable that virtualenv doesn't touch, which uses the same syntax as the environmental variable PATH, in bash PYTHONPATH=/usr/lib/python2.7/dist-packages:/usr/local/lib/python2.7/dist-packages in a .bashrc or similar. If you followed this path,
You don't have to tell your virtual environment about this at all, it won't try to change it.
No relinking will be required, and
That will still go wherever it would have gone (pip install always uses /usr/local/lib/python2.7/dist-packages/ for my Ubuntu) if you install them outside of your virtual environment. If you install them from within your virtual environment (while it's activated) then of course it'll be put in the virtualenvironment.
I'm just getting my head around virtualenv, but there seems to be an easier way than mentioned so far.
Since virtualenv 1.7 --no-site-packages has been the default behavior.
Therefore using the --system-site-packages flag to virtualenv is all that is needed to get dist-packages in your path - if you use the tweaked virtualenv shipped by Ubuntu. (This answer and this one give some useful history). I've tested this and it does work.
$ virtualenv --system-site-packages .
I agree with Thomas here - I can't see any action required in virtualenv to see the effect of updates in dist-packages.
Having tested that with python setup.py install, it does (again as Thomas said) still go to dist-packages. You could change that by building your own python, but that's a bit extreme.
PYTHONPATH works for me.
vim ~/.bashrc
add this line below:
export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/dist-packages:/usr/local/lib/python2.7/dist-packages
source ~/.bashrc
In the directory site-packages, create a file dist.pth
In the file dist.path, put the following:
../dist-packages
Now deactivate and activate your virtualenv. You should be set.
What you want to achieve here is essentially add specific folder (dist-packages) to Python search path. You have a number of options for this:
Use path configuration (.pth) file, entries will be appended to the system path.
Modify PYTHONPATH (entries from it go to the beginning of system path).
Modify sys.path directly from your Python script, i.e. append required folders to it.
I think that for this particular case (enable global dist-packages folder) third option is better, because with first option you have to create .pth file for every virtualenv you'll be working in (with some external shell script?). It's easy to forget it when you distribute your package. Second option requires run-time setup (add a envvar), which is, again, easy to miss.
And only third option doesn't require any prerequisites at configure- or run-time and can be distributed without issues (on the same-type system, of course).
You can use function like this:
def enable_global_distpackages():
import sys
sys.path.append('/usr/lib/python2.7/dist-packages')
sys.path.append('/usr/local/lib/python2.7/dist-packages')
And then in __init__.py file of your package:
enable_global_distpackages()