So, id like to start serious python development, and its proven to be a big pain. Im not worried at all about the language itself; I like it well enough and I will have no problems picking it up. But the ecosystem is driving me crazy.
First I tried to get up and running under windows. I gave up on that after a few days, as 90% of packages dont include windows support / install instructions. So I switched to macosx, which people said was good for mac development.
More frustration ensues. Id like to use python as a matlab replacement and tool development platform, so spyderlib seems like an excellent tool. But now ive been busy trying to build pyqt on my mac for two days, to no avail, and im starting to question the wisdom of it all. Obviously, following several guides literally invariantly ends in cryptic errors. For which platform was this dependency built? What arcane compiler flags need to be set? I dont know and I dont care; why doesnt the installer figure it out? Oh wait, there isnt any... I want to USE these tools, not first completely reverse engineer them to find out how to build them.
There is a vast amount of implied knowledge in all the documentation I can find on these matters, both with regard to unix and pythonic quirks. Is there any way to scale this mountain, in a place with a managable learning curve? Right now I have no idea what im doing. Or should I go back to windows and try to coerce the unix packages I need into cooperation?
On Mac OS X, you can get spyder with macports. This should build everything needed.
If you prefer Windows, take a look at Python(x,y). It has a bunch of scientific tools pre-built, including spyder.
Finally, the Enthought Python distribution is worth considering for scientific work.
Have you tried ActivePython?
Why battle with compiling the modules yourself when you can get the pre-built packages from PyPM?
pypm install pyqt4 matplotlib scipy numpy
From my experience the best platform for kind of project you're describing is Linux. There you just install the libs you need from package manager and that's it. Binary packages, so compiling is not required.
If you want to stick with MacOS X, you should install either MacPorts or Fink. It's usually easy to use. Problem is, that things like Qt take forever to compile. But you won't be doing that very often.
As for installing Python modules, the best is PIP, which is very nice replacement for easy_install did does much more. Especially useful if you want virtualenv setup.
This is nearly the exact opposite of my experience with Python on Windows. Python itself installs with a binary installer, most add-on packages support easy_install, others provide binary installers of their own. The IDE I use is SciTE, which uses the old DOS install model - copy the files to a directory and run the SciTE.exe file. If you get a source distribution of a Python package, go to the directory containing setup.py and run python setup.py install. Maybe that's the implied knowledge you're talking about.
You can also find many unofficial Windows binaries at http://www.lfd.uci.edu/~gohlke/pythonlibs/.
I switched to Mac a few years ago and found that it took me quite a while of googling to properly install all the packages I needed for Python development. While I installed everything I made a list of the steps required to setup a functional system that may be appropriate for you as well. I usually use NetCDF4, HDF5, Numpy, Matplotlib, f2py, and Fortran in combination with Python. I published my list of 22 setup-steps for installing from source on my website. Installing from source is somewhat more time-consuming than using macports and fink, but enables you to have a working environment that is optimized for your system.
Related
I have read a bunch of posts here and on Google, but my question is far more basic than the answers: If Python(2.7) came pre-installed on my MacBook Pro (High Sierra), can I just do sudo easy_install pip (as suggested) from the command line--withOUT causing issues? I have a vague understanding of global/local installations, and my understanding is certain Python installations aren't compatible with local/global kernel installations. I hope I am getting the terminology right, but I saw several warnings about installing pip for "a homebrew based python installation", but I am not sure whether Python on my laptop is installed via homebrew (nor how to find out).
My question came about because I wanted to install the Hydrogen package to use in Atom, the text editor (to help me learn Python). I finally succeeded in installing Hydrogen, but got stumped by the missing kernels (not sure which ones I need, so I am willing to install them all). But I can't seem to install the kernels without pip. So here I am.
My apologies for asking such a basic question--and thanks!
The rule of thumb is: If your operating system has a package manager, use it.
Unfortunately, MacOS is the only UNIX-like operating system that does not come with a decent package managment system.
(There is the app store, but that is useless for a lot open source software for several different reasons. It's also a walled garden.)
You have several choices (in descreasing suitability):
Use one of the package managers available for MacOS. Which one is the best choice for you depends on all the packages you need being available.
Use a Python distribution. I've used Anaconda on ms-windows, and that has saved me a lot of hassle. A good choice if you are only looking for Python and related libraries.
Build everything yourself. This can be very time-consuming and is a duplication of effort. You will learn a lot though.
I would second Piinthesky's comment that you install Python 3.6. Python 2.7 is now a legacy version.
Well although I am no Mac expert I've given it a shot anyway:
Yes you could but do you really want to risk it (or even do it)?
Mac-OS must rely on Python to fulfill something in the OS otherwise it would not come inbuilt. This means two things:
The Python installation will be minimal. By that I mean it will have things missing (any large library for a start). They will do this mainly to cut down on the OS size. Therefore you will not have the full Python library and in the long term you may end up missing out.
Second if anything went wrong (IE you broke your installation or even modified it -yep I've done this in Linux and have ended up factory resetting) then you may cause something to stop working and may need to factory reset or perform some other drastic action on your OS. A separate installation would prevent your from risking this. This is very useful because there comes a time when you may decide to update certain modules with pip and find it can't or it updates something that you shouldn't be messing with.
Yes it's possible you may run into compatibility problems but I think it's most widely accepted that you do not use the inbuilt one as it needs to remain unchanged if the OS is to use it correctly. Messing with it increases the chances of it breaking.
Conclusion: So even though installing modules with pip (and getting pip) can be done with the inbuilt Python it comes down whether you want to risk harming your OS. I strongly suggest you get a separate installation and leave the inbuilt one as it is. Second as you mentioned you will find that the inbuilt versions are never up-to-date or are built were they are not really compatible with standard libraries (expect things like the missing runtime libraries all the time) , just another reason to stay clear of them.
This is how I solved this problem-for those newbies who just want Hydrogen to work:
Installed Python 3 (instead of messing around with Python 2.7 and pip).
Followed instructions here (https://packaging.python.org/tutorials/installing-packages/#ensure-you-can-run-pip-from-the-command-line) for 'get-pip.py'.
In Atom, cmd+shift+p to bring up the packages menu, clicked on 'Hydrogen Run', which gave me the errors again.
Copied the code from the warnings and installed the kernels needed (via the command line).
Hydrogen is now working.
Thanks for all the tips!
I am a beginner and I want to learn computer programming. So, for now, I have started learning Python by myself with some knowledge about programming in C and Fortran.
Now, I have installed Python version 3.6.0 and I have struggled finding a suitable text for learning Python in this version. Even the online lecture series ask for versions 2.7 and 2.5 .
Now that I have got a book which, however, makes codes in version 2 and tries to make it as close as possible in version 3 (according to the author); the author recommends "downloading Anaconda for Windows" for installing Python.
So, my question is: What is this 'Anaconda'? I saw that it was some open data science platform. What does it mean? Is it some editor or something like Pycharm, IDLE or something?
Also, I downloaded my Python (the one that I am using right now) for Windows from Python.org and I didn't need to install any "open data science platform".
So what is this happening?
Please explain in easy language. I don't have too much knowledge about these.
Anaconda is a commercial python and R distribution. It aims to provide everything you need (Python-wise) for data science "out of the box".
It includes:
The core Python language
100+ Python "packages" (libraries)
Spyder (IDE/editor - like PyCharm) and Jupyter
conda, Anaconda's own package manager, used for updating Anaconda and packages
Your course may have recommended it as it comes with these extras but if you don't need them and are getting on fine with vanilla Python that's OK too.
Learn more: https://www.anaconda.com/distribution/
Anaconda is a Python distribution that makes it easy to install Python plus a number of its most often used 3rd party libraries in a flexible way on a Windows or Linux machine.
My experiences with it are very positive, both on Windows and Linux. It is quite complete and avoids problems in building libraries that you need from source code, that frequently plague one by one installations of those libraries by tools like pip.
By the way: It's very wise to start with 3.5 or 3.6 since 2.7 is approaching the end of its lifecycle, though many applications still depend on it.
As for tutorials: Pythons own docs are quite suitable for learning the language.
https://docs.python.org/3/tutorial/
Anaconda is a Python-based data processing and scientific computing platform. It has built in many very useful third-party libraries. Installing Anaconda is equivalent to automatically installing Python and some commonly used libraries such as Numpy, Pandas, Scrip, and Matplotlib, so it makes the installation so much easier than regular Python installation. If you don't install Anaconda, but instead only install Python from python.org, you also need to use pip to install various libraries one by one. It is painful and you need to consider compatibility, thus it is highly recommended to directly install Anaconda.
I'm at a bit of despair.
I've spent the last two days trying to set-up a proper Python (2.7x) Windows enviornment.
I tried a bunch of different "bundles", and it boils down to this:
It's either I can't install the packages I want;
or I can't use the IDE I want.
I tried a simple install of "python" with no additions.
Very early, I come to a stop while using pip/easy_install, or exe files to install additional packages.
I then tried "Portable Python" which weathered much better.
It has a nice (though limited) bunch of packages.
I can use it with any IDE I want (usually PyDev/Eclipse).
Here too, It's practically impossible (for me, following what I read) to install most packages.
I then went with Enthought Canopy (formerly EPD).
This has a superb tool for package managing. I did not find any problem, with package handling.
However, it comes with a very limited (almost insulting) IDE.
All my actions to point eclipse interpreter to the Canopy directories, went awry.
Lastly, I tried WinPython.
This is somewhere in-between "Portable" and "Canopy".
It has more packages pre-bundled, than "portable" has, but less than canopy.
It also has a package management GUI, but unlike canopy's, this one fails.
Please, tell me how you set-up a Win/Python extendable enviornment.
And please, do not flag this as a duplicate - I have read dozens of related questions here on SO, and elsewhere. Non actually helped me.
I would not be wasting your times, unless I tried all paths before (obviously using SO and Google to solve my problems before going through another lengthy install).
[EDIT (SOLVED) ]:
I uninstalled everything, and then installed "plain" python.
I also manually removed all references to orphaned "pythons" of past from the PATH and PYTHONHOME env vars.
That seems to work.
I have read through various posts here on the pros/cons of a wide array of Python environments. That being said, I am even more confused than prior to engaging in this due diligence.
My application is algorithmic trading, backtesting and analytical tool development and so I know I want the following functionality:
-numpy
-scipy
-Matplotlib
-spyder IDE
It seems like the installation of spyder on macos is non-trivial based on comments posted by users here http://code.google.com/p/spyderlib/wiki/Installation and so before I go down a rabbit hole I was wondering if someone could better guide me on the process and pre-requisites.
I came across the following post on stackoverflow, which seemed to indicate that I could just install the EPD-7.1 distribution and then install spyder separately but I'm not sure if the EPD distribution comes with all of the pre-requisites necessary for spyder.
Python distributions and environments for scientific computing
The following seems to be a guide on how to install spyder on macos but I'm not sure if this is current and whether installation of EPD makes some of the steps unnecessary.
http://works13.com/blog/mac/howto-install-spyder-1-x-on-mac-os-x-with-64bit-python.htm
Lastly, if I were to install EPD - is it better to go with 32-bit or 64-bit.
Thank you very much for your guidance.
Anaconda seems to be exactly what you're looking for: spyder, numpy, scipy, and matplotlib all work right out of the box.
The shortest path to a working analytic environment is the Enthought Python Distribution. The PythonXY build is also nice (easy to install, etc) but has fewer preinstalled features (a lighter footprint).
Spyderlib works cleanly with PythonXY. With EPD, there a couple of open issues:
http://code.google.com/p/spyderlib/issues/detail?id=627
http://code.google.com/p/spyderlib/issues/detail?id=723
I can't think of any reason to prefer a 32-bit build over a 64-bit build.
It depends on what kind of algo trading you intend to do but for anything that is time sensitive (such as actual trading) Python will likely be too slow. That said, it can be a good tool for backtesting and strategy development, although things like Streambase that can do everything might be a better choice.
I was in the boat as well. Too many options and too many opinions. I was also very much lost and to be honest, still feeling a bit soo! I am new to python and I am also not a proficient user of the terminal on Mac OS. Still, I managed to do the following to get spyder3 running on my new Mac running on High Sierra 10.13.4
Install Python3 using terminal Mac OS command:
$ brew doctor
$ brew install python3
(please note that my Mac OS High Sierra already had Python 2.x installed in it and I did not touch it. )
Then I installed Spyder using the recommended approach described here
https://github.com/spyder-ide/spyder/releases
Hope it helps!
short version: how can I get rid of the multiple-versions-of-python nightmare ?
long version: over the years, I've used several versions of python, and what is worse, several extensions to python (e.g. pygame, pylab, wxPython...). Each time it was on a different setup, with different OSes, sometimes different architectures (like my old PowerPC mac).
Nowadays I'm using a mac (OSX 10.6 on x86-64) and it's a dependency nightmare each time I want to revive script older than a few months. Python itself already comes in three different flavours in /usr/bin (2.5, 2.6, 3.1), but I had to install 2.4 from macports for pygame, something else (cannot remember what) forced me to install all three others from macports as well, so at the end of the day I'm the happy owner of seven (!) instances of python on my system.
But that's not the problem, the problem is, none of them has the right (i.e. same set of) libraries installed, some of them are 32bits, some 64bits, and now I'm pretty much lost.
For example right now I'm trying to run a three-year-old script (not written by me) which used to use matplotlib/numpy to draw a real-time plot within a rectangle of a wxwidgets window. But I'm failing miserably: py26-wxpython from macports won't install, stock python has wxwidgets included but also has some conflict between 32 bits and 64 bits, and it doesn't have numpy... what a mess !
Obviously, I'm doing things the wrong way. How do you usally cope with all that chaos ?
I solve this using virtualenv. I sympathise with wanting to avoid further layers of nightmare abstraction, but virtualenv is actually amazingly clean and simple to use. You literally do this (command line, Linux):
virtualenv my_env
This creates a new python binary and library location, and symlinks to your existing system libraries by default. Then, to switch paths to use the new environment, you do this:
source my_env/bin/activate
That's it. Now if you install modules (e.g. with easy_install), they get installed to the lib directory of the my_env directory. They don't interfere with existing libraries, you don't get weird conflicts, stuff doesn't stop working in your old environment. They're completely isolated.
To exit the environment, just do
deactivate
If you decide you made a mistake with an installation, or you don't want that environment anymore, just delete the directory:
rm -rf my_env
And you're done. It's really that simple.
virtualenv is great. ;)
Some tips:
on Mac OS X, use only the python installation in /Library/Frameworks/Python.framework.
whenever you use numpy/scipy/matplotlib, install the enthought python distribution
use virtualenv and virtualenvwrapper to keep those "system" installations pristine; ideally use one virtual environment per project, so each project's dependencies are fulfilled. And, yes, that means potentially a lot of code will be replicated in the various virtual envs.
That seems like a bigger mess indeed, but at least things work that way. Basically, if one of the projects works in a virtualenv, it will keep working no matter what upgrades you perform, since you never change the "system" installs.
Take a look at virtualenv.
What I usually do is trying to (progressively) keep up with the Python versions as they come along (and once all of the external dependencies have correct versions available).
Most of the time the Python code itself can be transferred as-is with only minor needed modifications.
My biggest Python project # work (15.000+ LOC) is now on Python 2.6 a few months (upgrading everything from Python 2.5 did take most of a day due to installing / checking 10+ dependencies...)
In general I think this is the best strategy with most of the interdependent components in the free software stack (think the dependencies in the linux software repositories): keep your versions (semi)-current (or at least: progressing at the same pace).
install the python versions you need, better if from sources
when you write a script, include the full python version into it (such as #!/usr/local/bin/python2.6)
I can't see what could go wrong.
If something does, it's probably macports fault anyway, not yours (one of the reasons I don't use macports anymore).
I know I'm probably missing something and this will get downvoted, but please leave at least a little comment in that case, thanks :)
I use the MacPorts version for everything, but as you note a lot of the default versions are bizarrely old. For example vim omnicomplete in Snow Leopard has python25 as a dependency. A lot of python related ports have old dependencies but you can usually flag the newer version at build time, for example port install vim +python26 instead of port install vim +python. Do a dry run before installing anything to see if you are pulling, for example, the whole of python24 when it isn't necessary. Check portfiles often because the naming convention as Darwin ports was getting off the ground left something to be desired. In practice I just leave everything in the default /opt... folders of MacPorts, including a copy of the entire framework with duplicates of PyObjC, etc., and just stick with one version at a time, retaining the option to return to the system default if things break unexpectedly. Which is all perhaps a bit too much work to avoid using virtualenv, which I've been meaning to get around to using.
I've had good luck using Buildout. You set up a list of which eggs and which versions you want. Buildout then downloads and installs private versions of each for you. It makes a private "python" binary with all the eggs already installed. A local "nosetests" makes things easy to debug. You can extend the build with your own functions.
On the down side, Buildout can be quite mysterious. Do "buildout -vvvv" for a while to see exactly what it's doing and why.
http://www.buildout.org/docs/tutorial.html
At least under Linux, multiple pythons can co-exist fairly happily. I use Python 2.6 on a CentOS system that needs Python 2.4 to be the default for various system things. I simply compiled and installed python 2.6 into a separate directory tree (and added the appropriate bin directory to my path) which was fairly painless. It's then invoked by typing "python2.6".
Once you have separate pythons up and running, installing libraries for a specific version is straightforward. If you invoke the setup.py script with the python you want, it will be installed in directories appropriate to that python, and scripts will be installed in the same directory as the python executable itself and will automagically use the correct python when invoked.
I also try to avoid using too many libraries. When I only need one or two functions from a library (eg scipy), I'll often see if I can just copy them to my own project.