Experimenting with using python in a virtualenv on my shared hosting account. Based on this dreamhost tutorial have installed pip and another module or two (echonest, remix), but trying to install numpy the long list of errors starts with non-existing path in 'numpy/distutils': 'site.cfg'
/bin/sh: svnversion: command not found.
The virtualenv instructions I read say, "make sure that your path gives preference to ~/local/bin to /usr/bin so that your "local" copy of Python runs, and that your scripts refer to that location."
Does that suggest to make a link somewhere that points calls to /usr/bin/ to ~/local/bin?
Is the solution to find the install package and edit the paths in it's setup.py file?
this is referring to the linux environment variable $PATH which lists the directories in which to look for executables for commands when you don't specify an absolute path. this will contain a list of comma separated paths eg:
/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
You just need to make sure that the /usr/local... stuff comes first (left) like this:
export PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
Are you sure you're running ~/local/bin/python when running setup.py?
One virtualenv-specific thing you can do is source ~/local/bin/activate, which automatically sets your virtualenv to take preference over everything else in your path. It only works until you log out of your terminal instance or run deactivate
I'm seeing confusion in the comments--- between "local" and "your virtualenv". And there is a parallel problem with $PATH. Is there just one path issue? No... there are two.
True, my own analogous efforts on a different shared-hosting account are ultimately crashing too, but I'm further along and I appear to have some things figured out. The confusion here partly comes about because there are two Dreamhost tutorials that are linked to in the question, but the authors of those two wiki chapters didn't make any effort to integrate the two.
And in the second tutorial on Python the mistake is made of introducing the discussion of the local directory in the virtualenv section though it's only really explained in the next section on building your own Python. And the discussion on building your own isn't complete regarding what to do about $PATH. You don't need to build your own Python or have a local directory just to use virtualenv.
It may even be detrimental. The language at this official virtualenv page should fill you with despair, as it shows no effort at clarity. I think that it means that mod_python and mod_wsgi, either of which you must be using for your Flask experiment, don't use any Python that you might come up with--- they use their own Python interpreter, the system's version. But virtualenv, the program, also puts a Python interpreter inside your virtual environment (the interpreter that you will use to do installs and also ultimately to run flask if you let it). The point of the warning from the virtualenv folks is that you can't run with two different interpreters in the same server process. They show you the workaround.
But I have digressed. The second, Python tutorial instructs about building your own local Python, in the directory /home/yourusername/local. A more complete, yet succinct discussion is to be found at Sugath Mudali's Blog.
Your virtualenv should occupy a second directory, which I'll call yourvirtenv. Alongside ~/local should be OK.
So, per Sugath Mudali's helpful instructions, you need to execute export PATH=/home/yourusername/local/bin:$PATH, which puts /home/yourusername/local/bin at the front of your $PATH, which you can confirm with "echo $PATH". Once you've done that, whenever you use "python" in your shell you'll be using the Python that you just installed, whose path you just put first.
Having done that, if you then cd into to your virtualenv source folder (somewhere in /home/youruserhame) and run "python virtualenv.py /home/youruserhame/yourvirtenv" it will create a directory ~/yourvirtenv, into which it will put a copy of the interpreter of your local Python. Generally it will not only be a distinct copy but it will differ even as to version from that of your system, which is at /usr/bin/ as /usr/bin/python ("usr" here is not you but is the oddly-named directory of the entire shared Linux machine to which no unprivileged user such as you has access, into which program binaries can be installed, by the privileged, for system-wide use).
Note that the Dreamhost Python wiki says "DreamHost has begun upgrading servers to Python 2.6.6 as of February 2012." Version 2.6 is adequate for virtualenv and practically everything else. So I would question the need to build your own Python in your case. (Note that Dreamhost even suggests that you should try to forget about it entirely.)
Anyway, on to the second PATH deal, as limasxgoesto0 noted, once you run virtualenv.py and have thus created myvirtenv, if you're still in the /home/myusername user-root directory you run source myvirtenv/bin/source. That puts /home/myusername/myvirtenv/bin at the front of the path--- until you later simply run "deactivate" (no source in front of it). That is all that activate does.
Now, when you return days later and go back to work consider this: export PATH=/home/yourusername/local/bin:$PATH has evaporated. It's not permanent; goes away when the shell is closed. To make it permanent you have to add that statement to .bash_profile. Here's mine:
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
PATH=$PATH:$HOME/bin
export PATH
export PATH=$HOME/local/bin:$PATH
export LD_LIBRARY_PATH=/home/myusername/local/lib
unset USERNAME
I found it necessary to reassert the LD_LIBRARY_PATH in order to make sqlite3 function, even though I had compiled sqlite3 and done that export prior to building Python.
Note that this .bash_profile invokes .bashrc. Depending on how Dreamhost has arranged things, you may have to put these exports in .bashrc.
Likewise, when you return to the shell myvirtenv will not be found to still be activated even if you never typed-in "deactivate". You only activate it temporarily in the shell to install stuff inside it and to start your server or run some other program that you've installed. Thereafter, the installed stuff knows to use the Python interpreter that has been copied to myvirtenv. To make that not happen, to make the stuff run off of the system Python interpreter you have to do what the aforementioned official virtualenv docs on mod_python, mod_wsgi say.
It looks as though this may be necessary even if you have not built your own Python, because otherwise you would have two Python interpreters running in the same server process. I am however sketchy on this. Flask is, like Django, a Python framework. An A2 Hosting tutorial--- they use Phusion Passerger too--- may be of interest. On their site search for "Django shared" and the tutorial will come up on top. Note that Chris C.'s instructions there tell you to use the --python=/path/to/python switch. In your case that could be /home/myusername/local/bin/python if you have locally installed Python, but in their case it's /usr/bin/python2.6. I think that their point is they don't want to get tech-support tickets from customers on problems caused by upgrades to their system python2.6 that could break your site. So that --python switch (-p directory is the short form; --python=directory is the long form) is to freeze the Python that your app will be using, to avoid such problems. It does so by copying all of the Python into your virtual environment, not just the interpreter.
I don't understand how they are avoiding the problem of mod_wsgi and the customer's apps using two different interpreters, but maybe that's what Phusion Passenger helps out with.
Related
TL;DR - Microsoft Store Apps are broken (0 bytes), hence the Python interpreter is unable to create process and run "Python" inside virtualenv, I failed to follow numerous explanations of how to change virtualenv path for Python.
Recently, without any changes to my computer/environment, an issue started occurring when executing (also tried python3, which brings the same):
python manage.py runserver
This brought back the following issue:
Unable to create process using
'"C:\Users\MyUser\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\python.exe"'
As I dug deeper, I realized that all of the apps installed under this folder are 0 bytes, hence they are completed broken, all of a sudden.
Therefore I figured it's not an issue with Django, rather the python itself.
I tried changing the virtualenv path for Python.exe, instead of using WindowsApps version, I installed the original Python from the original website.
I failed to do so, multiple times.
I tried: "https://stackoverflow.com/questions/4757178/how-do-you-set-your-pythonpath-in-an-already-created-virtualenv/47184788#47184788" - Adding the path inside "activate" and "activate.bat", as
set PYTHONPATH="C:\Users\MyUser\AppData\Local\Programs\Python\Python310\python.exe"
And the issue persists.
I tried every solution/article I found in regards to the issue I have. Many of them claim a simple result, whereas the rest claim a complete refactor is required.
Even downloading from Microsoft store is broken, it always fails.
Since I'm unable to remove the broken apps, and I wasn't able to find a way to change the virtualenv Python interpreter, I am here, asking for your assistance.
Thanks in advance,
Since you have installed python from python.org, ensure that it has been properly added to the PATH and you're not inadvertently using the windows store version.
Open command prompt
Execute where python to check which python your system is using.
If you're only seeing the WindowsApps version of python listed in the output of where command, then python installer apparently didn't add the location to PATH, so you'll need to manually add it yourself.
Also, if you're not restricted to use virtualenv, you can give conda a try.
As I began rebuilding, I figured it out.
When building a virtualenv, the python I used was directing to:
"C:\Users\MyUser\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\python.exe"'
Python interpreter is saved as the "PATH" to execute python.
Since my Python interpreter from Microsoft Store went broke, I was unable to execute python inside the virtual environment.
Venv configures a file called:
pyvenv.cfg
This is how the virtual environment knows what Python interpreter to use.
Inside it, the first line states:
home = C:\Users\MyUser\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\
Therefore, we just need to re-configure it to the new path of Python interpreter:
home = C:\Users\MyUser\AppData\Local\Programs\Python\Python310
And it worked for me.
Thanks for everyone's assistant, glad it's done.
we have a bunch of code in python - we have a sort of bootstrap bash script that kicks off various things - and we are trying to just set a python_bin variable that points to python 3. I tried this:
if [[ $( which python 2>/dev/null ) =~ /Python3 ]]; then
python_bin=python
elif which python3 >/dev/null 2>&1; then
python_bin=python3
else
python_bin=python
fi
but on some windows boxes it's not finding the right python because it's behind some runner "py". Is there a definitive way in bash to get to python3 that works on any platform where python and bash are both installed in the path?
Basically we want someone to be able to get clone our repository and run setup.sh - and keep finding environments where the python install is different in some new and interesting way.
Is there a definitive way in bash to get to python3 that works on any platform where python and bash are both installed in the path?
First off, are you sure that's sufficient? There could be any number of Python installations within folders that are on the PATH. For that matter, are you sure that your script will run correctly even under 3.0?
Second, no, you can't ever do this in a completely foolproof way. For one thing, Windows users are under no compulsion to leave python.exe's name alone. After all, nothing in Windows is dependent upon it. Linux and Mac users even have the same freedom for other installations besides the system one. For another, just because a system has Python installed doesn't mean it has any 3.x installations.
(Oh, and you should probably take a moment to make sure you understand how virtual environments work, and do some testing with those. Depending on the platform and the user's preferences, these might either copy the base Python environment or symlink it. Then there are all the alternate implementations: Jython, Pypy, IronPython....)
On the other hand, 2.7 has been EOL for over two years; it is approximately as old as Windows 7. Your users should be expecting any new code to demand 3.x anyway - and a reasonably up-to-date version at that. Even 3.6 is EOL now.
some windows boxes it's not finding the right python because it's behind some runner "py".
Keep in mind that if you are using Bash on Windows then you are effectively reliant on hacks anyway. I would recommend writing a separate .bat version of the bootstrap, if possible.
The Python launcher for Windows exists because Windows does not have a built-in concept of shebang lines. Not everyone on Windows uses it. The entire point of py is that it circumvents the PATH environment variable; py itself is on the path (typically in the Windows install directory, for global installations), and it maintains its own registry. It does this exactly in order to support more complex logic for choosing an executable; using PATH can only ever give you the first valid thing that was found. Systems that use py will generally not put any of their installed Python installations on the PATH, because py will take care of it. But they could, and who knows which one will be found first.
If you really want to stick with this detection logic, try prioritizing a search for py, and use py -3 if py is available. But...
Basically we want someone to be able to get clone our repository and run setup.sh - and keep finding environments where the python install is different in some new and interesting way.
The best you can do is to just write a shebang line, set the executable bit for the script on systems that support it, use the Bash script to run these executable scripts as scripts (rather than trying to find the Python executable), and let users be responsible for their own environment variables. People who set up their environments in weird ways are supposed to be responsible for understanding the weirdness. I appreciate the desire to try to provide a turn-key solution for people who might not be that technically proficient - but if your audience includes the sort of person who installs Bash on Windows, there are at least some reasonable guarantees on technical proficiency.
On ordinary Linux and Mac setups, a shebang like #!/usr/bin/env python3 should give you the default 3.x Python in the current context. Such a shebang is also recognized by py, which will parse it in a completely different way: the .py file extension is registered to be run by py; py sees the shebang and applies some regexes; it sees that the command includes python3, and runs whichever Python it has registered as the default 3.x installation.
I am starting my first actual python project. I follow "Learn Python the Hard Way" to make an initial Python skeleton and I am using virtualenv too.
Now I want to use git to do the version control. According to some previous questions in SO, I am not suggested to commit any virtualenv files. Instead, I could use pip freeze > requirements.txt, and use .gitignore to ignore the virtualenv directories.
However, both virtualenv and Python project skeleton require a /bin directory, should I commit it as well? (Actually I don't really know what the role /bin is playing in Python project)
Any suggestions are appreciated, if there is something wrong with my process to set up a Python project, please correct me.
yourproject/bin is distinct from yourproject/env/bin, where yourproject/env is the virtual environment's directory (and neither of them is /bin in the root directory). You should ignore everything in env, and indeed, your project should work for someone who isn't using a virtual environment, or is managing it differently. Otherwise, you lose the benefits.
Let's imagine for a second that you finish your project, and I want to use it for a new task. I start a new project with a virtualenv of its own, and install some other components I want to use, and then yours. Oops, now I have an older version of Python than I started out with, and that bug in deactivate from two years ago somehow got resurrected. Imagine debugging that, let alone the annoyance of finding that your project replaced some files of mine.
(As these things go, the bin directory is a pretty static part of the virtual environment; zapping other parts of my private env would be much more destructive. If you committed lib you would have prevented me from installing any other components before yours.)
My project's running fine in virtualenv. Unfortunately, I can't yet run it via my IDE (Eric) because of the import troubles. It stands to reason, as I never told the IDE anything about virtualenv.
I know the drill ($ source project/bin/activate etc.), but lack general understanding. What constitutes "running inside virtualenv"? What IDE options might be relevant?
I think the only required setting to run or debug code is path to python interpreter.
Relevant IDE options could be SDK or Interpreter settings.
Note that you should run not default python (eg. /usr/bin/python) but python binary in your virtual environment (eg /path/to/virtualenv/bin/python)
Also there are some environment variables set by activate, but i think they aren't needed when you point to virtualenv python binary directly.
So, again, what activate does is only environment variables setup: at least, it modifies system $PATH in a way that python and pip commands points to executable files under path/to/virtaulenv/bin directiry.
As far as I know it is possible to run your script/project using your virtualenv simply by calling /path/to/your/venv/python your_script.py. To install new packages in your venv for example, you would run /path/to/your/venv/pip install some_package.
I guess the main advantage of "runnnig inside virtualenv" would be not being concerned about having to inform the location of python packages/executable everytime you want to run some script. But I lack general understanding too.
I usually install a virtualenv with the --no-site-packages option in order to have a "clean" installation of python.
--- EDIT ---
The second answer of this discussion has a nice explanation.
Simply look at the project/bin/activate, everything you need is there to set up the appropriate search.
Usually the most important path is the PYTHONPATH which should point to the site-packages.
I have a Python project which contains three components: main executable scripts, modules which those scripts rely on, and data (sqlite3 databases, flat files, etc.) which those scripts manipulate. The top level has an __init__.py file so that other programs can also borrow from the modules if needed.
The question is, is it more "Pythonic" or "correct" to move my project into the default site-packages directory, or to modify PYTHONPATH to include one directory above my project (so that the project can be imported from)? On the one hand, what I've described is not strictly a "package", but a "project" with data that can be treated as a package. So I'm leaning in the direction of modifying PYTHONPATH (after all, PYTHONPATH must exist for a reason, right?)
Definitely do not add your project to site-packages this is going to spoil your system Python installation and will fire back at the moment some other app would come there or you would need to install something.
There are at last two popular options for installing python apps in isolated manner
Using virtualenv
See virtualenv project. It allows
creation of new isolating python environment - python for this environment is different from system one and has it's own PYTHONPATH setup this allows to keep all installed packages private for it.
activation and deactivation of given virtualenv from command line for command line usage. After activate, you can run pip install etc. and it will affect only given virtualenv install.
calling any Python script by starting by virtualenv Python copy - this will use related virtualenv (note, that there is no need to call any activate)
using zc.buildout
This package provides command buildout. With this, you can use special configuration file and this allows creation of local python environment with all packages and scripts.
Conclusions
virtualenv seems more popular today and I find it much easier to learn and use
zc.buildout might be also working for you, but be prepared for a bit longer learning time
installing into system Python directories shall be always reserved for very special cases (pip, easy_install), better avoid it.
installing into private directories and manipulatig PYTHONPATH is also an option, but you would repeat, what virtualenv already provides