First: I'm running Macports. No problems with that, except:
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin
which is the value of sys.exec_prefix, for my macports python even though:
/opt/local/lib/python2.6/site-packages/
seems to be quite a logical place to put things, /opt/local being the macports --prefix, as it were. Why does easy_install put things in this odd Frameworks/Python.framework thing?
More importantly, can i use the methods here, to ensure that all my systemwide python, particularly the scripts which I really want in /opt/local/bin, things I use all over the place like (i|b)python for example are accessible?
So first I'd make the observation that the directory
/Library/Frameworks/Python.framework/Versions/2.6/...
is where you can find the OS X provided version of Python so I'd guess that what the MacPorts developers want to do is replicate the OS X Python directory structure but keep it as far away from the OS X version of Python as possible. Also I think replicating the structure allows you to install third-party extensions outside of the MacPorts modules. I've done that before so I know it's possible, its just you still need to make sure:
When installing the modules, the version that the installer encounters when doing a PATH search is the Python installation you want to install it to (ie. a module meant for MacPorts Python doesn't end up in OS X Python). The page you cited earlier spells out the procedure an installer goes through when trying to find a Python installation to install to so you have some hints there about what to look into if you have problems installing third party modules.
Once you have installed your modules you'll have to keep track of them yourself because MacPorts won't do that for you. On the bright side everything should be in the site-extensions folder of the Python installation, I think.
This question from the MacPorts FAQ kind of lays the groundwork for their philosophy of directory organization I think. Also this question too.
Related
The core question is: How can I find those pythons which were installed by MacPorts?
A bit more more information:
I installed a port from MacPorts which had both, python37 and python38, as dependencies.
I am now trying to find out where these distributions are installed, since I need to call respective scripts with those pythons. They wouldn't run if I use a different one which has already been on my system.
I found /opt/local/bin/python3.8, which I think was not there before. I am not sure though. This folder however does not contain a python3.7 as well.
My system has other distributions on it as well. Those which came with MacOS, and the one which came with Anaconda. I know where I can find the anaconda distribution, and I think the MacOS ones are in /usr/bin/, but I am also not 100% sure if all of those located there came with the OS, or if they may also contain the ones which were ported from MacPorts.
Thanks for help!
I would post a comment if I could (low reputation)...
Since I'm going through a similar situation at the moment, I thought about a few things that could help you (which can be obvious, but then again, posting them as a comment could be more appropriate.)
Yes, everything MacPorts-related is installed at /opt/local, usually. You can search there for any Python version. Some other points:
Perhaps since you already had Python 3.7 installed somewhere MacPorts didn't install it anew
Take a look at your $PATH environment variable. It will give you an idea of what's being searched for first.
You can always run something like sudo find / -type f -name "python*" to try and find anything Python-related (you don't strictly need sudo but it helps avoiding some annoying error messages). This way you will know which Python versions are installed where and by which distribution.
Oh, perhaps this can help too: port installed would show you everything MacPorts has installed. From here: See what has been installed via MacPorts
I finally managed to install numpy, but it seems to only work in python2.6 . I don't know how to install it in the 2.7 folder (been trying for hours, but I'm just a beginner developer in my first months). Anyway, if I use Python 2.7 and append the absolute path to sys.path, could there be problems?
Any suggestions?
Thank you.
It could partially work but this is a bad idea. Just don't do it. Even if it seems to work, it may not. And if it really does, then it will fail randomly in the future.
These are the potential problems that come into my mind:
Extensions (those written in C, C++ etc.) are specific to a particular Python version; and numpy has a few extensions, AFAICS. It will work only if you don't use any of them (i.e. use pure Python modules);
Python compiles modules into bytecode. The bytecode is specific to a particular Python version. If you use modules from python2.6 directory in python2.7, the compiled files will collide. I doubt this will cause major problems except for the fact that they will be recompiled every time Python version is switched;
Python code can be version-specific. It's unlikely for minor versions (but for example Python 2/3 could have serious differences) but still can happen. In other words, the modules installed for Python2.6 can be actually a bit different than those for Python2.7;
If you change the loading order, Python2.7 may start loading some standard modules from Python2.6. it could work, it could cause random breakages;
It will make all modules installed for Python2.6 visible. It can cause a few random switches somewhere with unpredictable result. I doubt there's something specific for that version but some modules may actually decide to use some kind of deprecated interface finding it visible.
There could be more. You may actually try but be prepared that you may waste a lot of time trying to find out why something does not work as expected later.
And unless I'm missing something, I think installing numpy for Python2.7 involves mostly running the setup using Python 2.7; like:
python2.7 setup.py clean
python2.7 setup.py build
python2.7 setup.py install
Depending on your particular install/system, it may be preferable to use the package manager, binary bundle or a tool like pip instead. If you'd like more details on how to enforce Python2.7 with those, you'd have to tell us which one is of your choice.
I am having trouble installing matplotlib on mac os 10.6, so I used macports and installed all dependencies it needed, which is great, but on top of it a new python version. Now I have two python versions and that bothers me. The matplotlib is working fine on the macport python, and the rest of my stuff is with the default python.
What is the best solution for integrating both into one, and which one to use as the default python?
I tried copying all packages from my default python /defaultPython/../site-packages into the /opt/../site-packages. Same thing would be to add /defaultPython/../site-packages on the PYTHONPATH of macports python.
Some did work fine, but not all.
For example on
import scipy.sparse
I got this error
import _csr
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/sparsetools/_csr.so, 2): no suitable image found. Did find:
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/sparsetools/_csr.so: no matching architecture in universal wrapper
Any thoughts or dirty fix for this?
Thanks a lot!
You should never remove or alter the system Python that Apple supplies with Mac OS X -- that's the specific build they've tested their OS with, and you really don't want to break that.
If you want to use handy macports-installed extensions, you need the macports version of Python for that purpose, so you can't remove that either.
There is no real reason to worry because you have more than one Python version on your system (I typically have at least half a dozen on mine;-).
Just put /opt/... (wherever macports keeps its bin directory) at the start of your PATH environment variable (e.g. in .bashrc) and make sure you install with the macports Python any extension you need there.
On Tiger, I used a custom python installation to evaluate newer versions and I did not have any problems with that*. Now Snow Leopard is a little more up-to-date and by default ships with
$ ls /System/Library/Frameworks/Python.framework/Versions/
2.3 2.5 2.6 #Current
What could be considered best practice? Using the python shipped with Mac OS X or a custom compiled version in, say $HOME.
Are there any advantages/disadvantages using the one option over the other?
My setup was fairly simple so far and looked like this: Custom compiled Python in $HOME and a $PATH that would look into $HOME/bin first, and subsequently would use my private Python version. Also $PYTHONPATH pointed to this local installation. This way, I did not need to sudo–install packages - virtualenv took care of the rest. Note: I liked this setup, so I'm just curious and thought I inquire the hive mind.
I guess it depends on your needs. Personally, I use the latest version for any series (2.5, 2.6, etc.) from MacPorts.
I compile it myself because this gives me the newest 64-bit version. The official OS X builds seem to be 32-bit only. I dropped MacPorts several months ago because its dependency system and its often outdated packages were too annoying.
The problem with using the Python version that's shipped with your OS is that it may contain bugs, or be limited in other ways. If you install Python from Fink or MacPorts, you have the liberty of updating it.
Another, important advantage of managing your own version of Python with a package manager (Fink or MacPorts) is that they help a lot with the compilation of module dependencies (for instance when you use a module that depends on compiled C code). Thus, installing Python modules is certainly easier if you do not use the Python shipped with OS X. This is an important point to consider, before making your choice.
What you do on your on machine is totally up to you. If you're going to deploy your code to other people's, I'd say it's clearly preferable to use the shipped version unless you really need the newer version.
I'm digging up this old topic.
There are no real answers to create your own python distro/framework from source and bundle and package it in a correct way. I tried to compile it from source, linking it against my C programmed application which uses python 3 and works on my computer. But when I move it around the filesystem (e.g to /tmp) it has hardcoded paths in the python compile. And I have no clue what script/wrapper to make.
My point to ship our own python distro is to make sure there is no oddness in interpeter change involved when shipping the application and rely on the OS python install.
It is also nowhere documented on the python docs.
I already found this post 4206511
short version: how can I get rid of the multiple-versions-of-python nightmare ?
long version: over the years, I've used several versions of python, and what is worse, several extensions to python (e.g. pygame, pylab, wxPython...). Each time it was on a different setup, with different OSes, sometimes different architectures (like my old PowerPC mac).
Nowadays I'm using a mac (OSX 10.6 on x86-64) and it's a dependency nightmare each time I want to revive script older than a few months. Python itself already comes in three different flavours in /usr/bin (2.5, 2.6, 3.1), but I had to install 2.4 from macports for pygame, something else (cannot remember what) forced me to install all three others from macports as well, so at the end of the day I'm the happy owner of seven (!) instances of python on my system.
But that's not the problem, the problem is, none of them has the right (i.e. same set of) libraries installed, some of them are 32bits, some 64bits, and now I'm pretty much lost.
For example right now I'm trying to run a three-year-old script (not written by me) which used to use matplotlib/numpy to draw a real-time plot within a rectangle of a wxwidgets window. But I'm failing miserably: py26-wxpython from macports won't install, stock python has wxwidgets included but also has some conflict between 32 bits and 64 bits, and it doesn't have numpy... what a mess !
Obviously, I'm doing things the wrong way. How do you usally cope with all that chaos ?
I solve this using virtualenv. I sympathise with wanting to avoid further layers of nightmare abstraction, but virtualenv is actually amazingly clean and simple to use. You literally do this (command line, Linux):
virtualenv my_env
This creates a new python binary and library location, and symlinks to your existing system libraries by default. Then, to switch paths to use the new environment, you do this:
source my_env/bin/activate
That's it. Now if you install modules (e.g. with easy_install), they get installed to the lib directory of the my_env directory. They don't interfere with existing libraries, you don't get weird conflicts, stuff doesn't stop working in your old environment. They're completely isolated.
To exit the environment, just do
deactivate
If you decide you made a mistake with an installation, or you don't want that environment anymore, just delete the directory:
rm -rf my_env
And you're done. It's really that simple.
virtualenv is great. ;)
Some tips:
on Mac OS X, use only the python installation in /Library/Frameworks/Python.framework.
whenever you use numpy/scipy/matplotlib, install the enthought python distribution
use virtualenv and virtualenvwrapper to keep those "system" installations pristine; ideally use one virtual environment per project, so each project's dependencies are fulfilled. And, yes, that means potentially a lot of code will be replicated in the various virtual envs.
That seems like a bigger mess indeed, but at least things work that way. Basically, if one of the projects works in a virtualenv, it will keep working no matter what upgrades you perform, since you never change the "system" installs.
Take a look at virtualenv.
What I usually do is trying to (progressively) keep up with the Python versions as they come along (and once all of the external dependencies have correct versions available).
Most of the time the Python code itself can be transferred as-is with only minor needed modifications.
My biggest Python project # work (15.000+ LOC) is now on Python 2.6 a few months (upgrading everything from Python 2.5 did take most of a day due to installing / checking 10+ dependencies...)
In general I think this is the best strategy with most of the interdependent components in the free software stack (think the dependencies in the linux software repositories): keep your versions (semi)-current (or at least: progressing at the same pace).
install the python versions you need, better if from sources
when you write a script, include the full python version into it (such as #!/usr/local/bin/python2.6)
I can't see what could go wrong.
If something does, it's probably macports fault anyway, not yours (one of the reasons I don't use macports anymore).
I know I'm probably missing something and this will get downvoted, but please leave at least a little comment in that case, thanks :)
I use the MacPorts version for everything, but as you note a lot of the default versions are bizarrely old. For example vim omnicomplete in Snow Leopard has python25 as a dependency. A lot of python related ports have old dependencies but you can usually flag the newer version at build time, for example port install vim +python26 instead of port install vim +python. Do a dry run before installing anything to see if you are pulling, for example, the whole of python24 when it isn't necessary. Check portfiles often because the naming convention as Darwin ports was getting off the ground left something to be desired. In practice I just leave everything in the default /opt... folders of MacPorts, including a copy of the entire framework with duplicates of PyObjC, etc., and just stick with one version at a time, retaining the option to return to the system default if things break unexpectedly. Which is all perhaps a bit too much work to avoid using virtualenv, which I've been meaning to get around to using.
I've had good luck using Buildout. You set up a list of which eggs and which versions you want. Buildout then downloads and installs private versions of each for you. It makes a private "python" binary with all the eggs already installed. A local "nosetests" makes things easy to debug. You can extend the build with your own functions.
On the down side, Buildout can be quite mysterious. Do "buildout -vvvv" for a while to see exactly what it's doing and why.
http://www.buildout.org/docs/tutorial.html
At least under Linux, multiple pythons can co-exist fairly happily. I use Python 2.6 on a CentOS system that needs Python 2.4 to be the default for various system things. I simply compiled and installed python 2.6 into a separate directory tree (and added the appropriate bin directory to my path) which was fairly painless. It's then invoked by typing "python2.6".
Once you have separate pythons up and running, installing libraries for a specific version is straightforward. If you invoke the setup.py script with the python you want, it will be installed in directories appropriate to that python, and scripts will be installed in the same directory as the python executable itself and will automagically use the correct python when invoked.
I also try to avoid using too many libraries. When I only need one or two functions from a library (eg scipy), I'll often see if I can just copy them to my own project.