we have a bunch of code in python - we have a sort of bootstrap bash script that kicks off various things - and we are trying to just set a python_bin variable that points to python 3. I tried this:
if [[ $( which python 2>/dev/null ) =~ /Python3 ]]; then
python_bin=python
elif which python3 >/dev/null 2>&1; then
python_bin=python3
else
python_bin=python
fi
but on some windows boxes it's not finding the right python because it's behind some runner "py". Is there a definitive way in bash to get to python3 that works on any platform where python and bash are both installed in the path?
Basically we want someone to be able to get clone our repository and run setup.sh - and keep finding environments where the python install is different in some new and interesting way.
Is there a definitive way in bash to get to python3 that works on any platform where python and bash are both installed in the path?
First off, are you sure that's sufficient? There could be any number of Python installations within folders that are on the PATH. For that matter, are you sure that your script will run correctly even under 3.0?
Second, no, you can't ever do this in a completely foolproof way. For one thing, Windows users are under no compulsion to leave python.exe's name alone. After all, nothing in Windows is dependent upon it. Linux and Mac users even have the same freedom for other installations besides the system one. For another, just because a system has Python installed doesn't mean it has any 3.x installations.
(Oh, and you should probably take a moment to make sure you understand how virtual environments work, and do some testing with those. Depending on the platform and the user's preferences, these might either copy the base Python environment or symlink it. Then there are all the alternate implementations: Jython, Pypy, IronPython....)
On the other hand, 2.7 has been EOL for over two years; it is approximately as old as Windows 7. Your users should be expecting any new code to demand 3.x anyway - and a reasonably up-to-date version at that. Even 3.6 is EOL now.
some windows boxes it's not finding the right python because it's behind some runner "py".
Keep in mind that if you are using Bash on Windows then you are effectively reliant on hacks anyway. I would recommend writing a separate .bat version of the bootstrap, if possible.
The Python launcher for Windows exists because Windows does not have a built-in concept of shebang lines. Not everyone on Windows uses it. The entire point of py is that it circumvents the PATH environment variable; py itself is on the path (typically in the Windows install directory, for global installations), and it maintains its own registry. It does this exactly in order to support more complex logic for choosing an executable; using PATH can only ever give you the first valid thing that was found. Systems that use py will generally not put any of their installed Python installations on the PATH, because py will take care of it. But they could, and who knows which one will be found first.
If you really want to stick with this detection logic, try prioritizing a search for py, and use py -3 if py is available. But...
Basically we want someone to be able to get clone our repository and run setup.sh - and keep finding environments where the python install is different in some new and interesting way.
The best you can do is to just write a shebang line, set the executable bit for the script on systems that support it, use the Bash script to run these executable scripts as scripts (rather than trying to find the Python executable), and let users be responsible for their own environment variables. People who set up their environments in weird ways are supposed to be responsible for understanding the weirdness. I appreciate the desire to try to provide a turn-key solution for people who might not be that technically proficient - but if your audience includes the sort of person who installs Bash on Windows, there are at least some reasonable guarantees on technical proficiency.
On ordinary Linux and Mac setups, a shebang like #!/usr/bin/env python3 should give you the default 3.x Python in the current context. Such a shebang is also recognized by py, which will parse it in a completely different way: the .py file extension is registered to be run by py; py sees the shebang and applies some regexes; it sees that the command includes python3, and runs whichever Python it has registered as the default 3.x installation.
Related
I've installed a python library (https://github.com/rsagroup/pyrsa) on my Mac via the terminal. This package is not part of Anaconda. I would like to work with it in Spyder now, which I just installed via the Anaconda distribution. I have scoured the internet but not been able to figure out how to do this. Would appreciate any tips!
Thank you.
Normally python checks certain locations for modules/packages:
current directory
the sub-directory called 'site-packages'
the path given by the environment variable PYTHONPATH
So therefore, as long as the module' directory is in 1 of the 3 path descriptions given above, and contains a file (empty or not) called '_init_.py', python can find it and you can import it.
Note that Anaconda is nothing more than a distribution of python. Which is more or less like a bunch of python packages, the (i)python interpreters and an IDE (spyder/IDLE) bundled together.
More or less the same applies to using Spyder: this is a shell around a python interpeter (actually, I think it is a ipython interpreter, but I'm not sure, since I don't use Spyder). Therefore whether you use Spyder, PyCharm, IDLE or whatever should not impact the directories that python checks for modules/packages.
Summarizing: the package locations any python interpreter checks are always the same. This is not linked to whether you use python shipped via Anaconda or the python interpreter preset in your linux/windows operating system.
In your case it might be the best choice between adding the directory in which you've stored the package to the PYTHONPATH environment variable, but opinions may as always differ on the matter.
Situation: I've installed a bunch of packages with pip. I've now written code using these packages.
I have a myscript.py
My friend is on windows.
has python installed.
He has no packages.
He cannot get any packages.
He has pip
He will never be able to use the internet to get more packages everything must be hand delivered.
In fact about 10 minutes after he runs whatever I give him, he formats his machine and it's gone.
How do I take myscript.py and give it to him on a USB stick so that he can copy the file myscript.py onto his computer and run it?
I thought Pipenv would do it but it looks like it just creates a LIST of packages to download from the internet. (a very well defined list... but a list not the actual files needed to run something. Do I understand it correctly?
Right now I'm giving him .exe made with py2exe. This isn't very elegant considering he has python already.
tl;dr how do I give a python script .py to an end user that doesn't have the internet?
You can package everything up in a virtual environment and give him the complete environment necessary to run the script. You can read about this at https://developer.mozilla.org/en-US/docs/Python/Virtualenv and an IDE like PyCharm will help you create such an environment easily.
It's actually a good thing to learn about and do anyway.
I'm planning to install Anaconda3 for Python 3.4. Since by default, Mac OS X uses Python2, will install Anaconda3 change the default Python version for the system? I don't want that to happen since Python3 can break backwards compatibility. If it does change the default Python version, how can I avoid that?
Apple has a built-in system for managing multiple versions of software, and switching between them. But you don't even need to worry about that, because Anaconda installations are self-contained: Everything lives under the top Anaconda installation directory (probably /Applications/anaconda). The only effect outside this directory is that during installation, Anaconda will offer to modify the PATH variable in your .bashrc. If you agree, it will add one line at the end of your .bashrc, something like this:
PATH="/Applications/anaconda/bin:$PATH"
As you can see, Anaconda puts itself first in the system path. This means that typing python at the shell prompt will launch python 3, which may not be what you want. I run Anaconda 3.4 like this and have had absolutely no problems with my system, but I did need to modify my own executable python2 scripts that launched python like this:
#!/usr/bin/env python
This is a nice way to find python wherever it is, but in this case it will find python 3-- oops! Changing the above to #!/usr/bin/python or to #!/usr/bin/env python2 ensures that they continue to work correctly. In my experience this was not necessary with any of the system's own scripts; everything is already set up to find the right python.
Alternative 1: You could decline the PATH modification, and use Anaconda via the launcher. In that case there is no change to the rest of your execution environment. The launcher will start a special bash prompt with the anaconda environment activated, but execution in normal shells is completely unaffected. If you will continue to program a lot in python 2, this may be for you.
Alternative 2: A minimal-impact alternative is to put the anaconda directory last in your path:
PATH="$PATH:/Applications/anaconda/bin"
This ensures that non-anaconda binaries take precedence over anaconda, so python will start good old /usr/bin/python (that is, python 2). You can start anaconda's variant by typing python3, idle3, etc. I did not have IPython before I installed anaconda, so typing ipython finds the anaconda version.
No it won't, you can have multiple python installs, once you don't remove your system python or manually change the default you will be fine.
Experimenting with using python in a virtualenv on my shared hosting account. Based on this dreamhost tutorial have installed pip and another module or two (echonest, remix), but trying to install numpy the long list of errors starts with non-existing path in 'numpy/distutils': 'site.cfg'
/bin/sh: svnversion: command not found.
The virtualenv instructions I read say, "make sure that your path gives preference to ~/local/bin to /usr/bin so that your "local" copy of Python runs, and that your scripts refer to that location."
Does that suggest to make a link somewhere that points calls to /usr/bin/ to ~/local/bin?
Is the solution to find the install package and edit the paths in it's setup.py file?
this is referring to the linux environment variable $PATH which lists the directories in which to look for executables for commands when you don't specify an absolute path. this will contain a list of comma separated paths eg:
/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
You just need to make sure that the /usr/local... stuff comes first (left) like this:
export PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
Are you sure you're running ~/local/bin/python when running setup.py?
One virtualenv-specific thing you can do is source ~/local/bin/activate, which automatically sets your virtualenv to take preference over everything else in your path. It only works until you log out of your terminal instance or run deactivate
I'm seeing confusion in the comments--- between "local" and "your virtualenv". And there is a parallel problem with $PATH. Is there just one path issue? No... there are two.
True, my own analogous efforts on a different shared-hosting account are ultimately crashing too, but I'm further along and I appear to have some things figured out. The confusion here partly comes about because there are two Dreamhost tutorials that are linked to in the question, but the authors of those two wiki chapters didn't make any effort to integrate the two.
And in the second tutorial on Python the mistake is made of introducing the discussion of the local directory in the virtualenv section though it's only really explained in the next section on building your own Python. And the discussion on building your own isn't complete regarding what to do about $PATH. You don't need to build your own Python or have a local directory just to use virtualenv.
It may even be detrimental. The language at this official virtualenv page should fill you with despair, as it shows no effort at clarity. I think that it means that mod_python and mod_wsgi, either of which you must be using for your Flask experiment, don't use any Python that you might come up with--- they use their own Python interpreter, the system's version. But virtualenv, the program, also puts a Python interpreter inside your virtual environment (the interpreter that you will use to do installs and also ultimately to run flask if you let it). The point of the warning from the virtualenv folks is that you can't run with two different interpreters in the same server process. They show you the workaround.
But I have digressed. The second, Python tutorial instructs about building your own local Python, in the directory /home/yourusername/local. A more complete, yet succinct discussion is to be found at Sugath Mudali's Blog.
Your virtualenv should occupy a second directory, which I'll call yourvirtenv. Alongside ~/local should be OK.
So, per Sugath Mudali's helpful instructions, you need to execute export PATH=/home/yourusername/local/bin:$PATH, which puts /home/yourusername/local/bin at the front of your $PATH, which you can confirm with "echo $PATH". Once you've done that, whenever you use "python" in your shell you'll be using the Python that you just installed, whose path you just put first.
Having done that, if you then cd into to your virtualenv source folder (somewhere in /home/youruserhame) and run "python virtualenv.py /home/youruserhame/yourvirtenv" it will create a directory ~/yourvirtenv, into which it will put a copy of the interpreter of your local Python. Generally it will not only be a distinct copy but it will differ even as to version from that of your system, which is at /usr/bin/ as /usr/bin/python ("usr" here is not you but is the oddly-named directory of the entire shared Linux machine to which no unprivileged user such as you has access, into which program binaries can be installed, by the privileged, for system-wide use).
Note that the Dreamhost Python wiki says "DreamHost has begun upgrading servers to Python 2.6.6 as of February 2012." Version 2.6 is adequate for virtualenv and practically everything else. So I would question the need to build your own Python in your case. (Note that Dreamhost even suggests that you should try to forget about it entirely.)
Anyway, on to the second PATH deal, as limasxgoesto0 noted, once you run virtualenv.py and have thus created myvirtenv, if you're still in the /home/myusername user-root directory you run source myvirtenv/bin/source. That puts /home/myusername/myvirtenv/bin at the front of the path--- until you later simply run "deactivate" (no source in front of it). That is all that activate does.
Now, when you return days later and go back to work consider this: export PATH=/home/yourusername/local/bin:$PATH has evaporated. It's not permanent; goes away when the shell is closed. To make it permanent you have to add that statement to .bash_profile. Here's mine:
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
PATH=$PATH:$HOME/bin
export PATH
export PATH=$HOME/local/bin:$PATH
export LD_LIBRARY_PATH=/home/myusername/local/lib
unset USERNAME
I found it necessary to reassert the LD_LIBRARY_PATH in order to make sqlite3 function, even though I had compiled sqlite3 and done that export prior to building Python.
Note that this .bash_profile invokes .bashrc. Depending on how Dreamhost has arranged things, you may have to put these exports in .bashrc.
Likewise, when you return to the shell myvirtenv will not be found to still be activated even if you never typed-in "deactivate". You only activate it temporarily in the shell to install stuff inside it and to start your server or run some other program that you've installed. Thereafter, the installed stuff knows to use the Python interpreter that has been copied to myvirtenv. To make that not happen, to make the stuff run off of the system Python interpreter you have to do what the aforementioned official virtualenv docs on mod_python, mod_wsgi say.
It looks as though this may be necessary even if you have not built your own Python, because otherwise you would have two Python interpreters running in the same server process. I am however sketchy on this. Flask is, like Django, a Python framework. An A2 Hosting tutorial--- they use Phusion Passerger too--- may be of interest. On their site search for "Django shared" and the tutorial will come up on top. Note that Chris C.'s instructions there tell you to use the --python=/path/to/python switch. In your case that could be /home/myusername/local/bin/python if you have locally installed Python, but in their case it's /usr/bin/python2.6. I think that their point is they don't want to get tech-support tickets from customers on problems caused by upgrades to their system python2.6 that could break your site. So that --python switch (-p directory is the short form; --python=directory is the long form) is to freeze the Python that your app will be using, to avoid such problems. It does so by copying all of the Python into your virtual environment, not just the interpreter.
I don't understand how they are avoiding the problem of mod_wsgi and the customer's apps using two different interpreters, but maybe that's what Phusion Passenger helps out with.
I have 3 versions of Python (2.5, 2.7, 3.2) installed on a Windows machine, 2.5 being the default one (first in PATH and default for open action).
Now the weird thing appear when I run a python script with filename.py (without specifying the interpreter) or by clicking the file in Explorer: Python 2.5 is running the script (expected) BUT Python 2.7 PATH being inserted before the original system PATH.
Still if I look at the command line, it seems that Python 2.5 was executed.
Where is the first record in PATH (C:\Ptyhon27\) comming from, I can assure you this is added when the scripts runs, but by whom?
As you already observed I do have several versions of Python in PATH, this is not something anormal, because they can also have versioned executables instead and because Windows always picks the first one it the PATH.
Looking closely, it seems you have various versions of Python in your PATH environment variable.
The standard Python installer for Windows doesn't add itself to the PATH; I always do this manually for the version of Python I want to use by default. If you're using a different Python installer (such as ActiveState or Enthought) that may be the cause, but I haven't tried those.
In any case, you can edit your PATH environment variable manually and clean it up, leaving only the path to the version of Python you wish to be the default, as described here (scroll down a bit to get to the relevant section).
You might be interested in the Python Launcher for Windows project, a.k.a. PEP 397. Install it, and remove all Python dirs from PATH, leaving only the launcher one, and use py/pyw instead of python/pythonw.
Ahh, I think I have it! You didn't mention exactly how you're running Python...
Python adds the directory from which the command is run to the PATH! If you run it from the command line, this will be the directory in which the Python file you execute is located, or the directory you ran Python from if you just opened an interpreter. However, various tools, interpreters and development environments start up differently, and some of them will use a certain Python interpreter (depends on their configuration) and add its location to PATH.
If you want more help, please give a detailed description of how you're running Python when this happens.