I write scientific software in Numpy/Scipy/Matplotlib. Having developed applications on my home computer, I am now interested in writing simple web applications. Example: user uploads image or audio file, my program processes it using Numpy/Scipy, and output is displayed on the browser using Matplotlib, or perhaps the user can download a processed file.
I already pay for hosting that does have Python 2.4.3 installed, but no Numpy/Scipy. I don't have shell access via command line, either. Just drag-and-drop FTP. Pretty limited, but I can get simple Python/CGI scripts working.
Surprisingly, a web search revealed few suitable options for web hosting with these capabilities already built in. (Please guide me if I am wrong.) I am learning about the Google App Engine, but I still don't have a full understanding about its tools and limitations. What the web did tell me is that others have similar concerns.
Hoping for solutions, I thought I would ask these simple questions to the awesome SO community:
Is there a simple way of installing numpy (or any third-party package/library) onto my already hosted space? I know the Python path on my hosted space, and I know the relevant Python/Numpy directories on my home computer. Can I simply copy files over and have it work? Both local and remote systems run Ubuntu.
What hosting sites exist (either free or paid) which have Numpy/Matplotlib installed or, if not installed, the possibility of installing it? Are there any documented sites that you can reference with working applications, no matter how simple?
Can Google App Engine help me in any way? Or is it totally for something else? Have you or others used it to write scientific applications in Python/Numpy? If so, could you reference them?
Thank you for your help.
EDIT: After the useful answers below, I bought the $20 plan at Slicehost, and I love it so far! (I first tried Amazon EC2. I must be stupid, but I just couldn't get it to work.) Setting up the Ubuntu server with Apache took mere hours (and I'm an Apache novice). It allows me to do exactly what I wanted with Python plus much more. I now have my own remote repository for version control, too. Thanks again!
EDIT 2: Nearly two years later, I tried Linode and EC2 (again). Linode is great. EC2 seemed easier this time around -- maybe it's just added experience, or maybe it's the improvements that Amazon made to the AWS management console. For those interested in Numpy/Scipy/Matplotlib/Audiolab, here is my Ubuntu cheat sheet whenever I launch an EC2 instance:
ec2:~$ sudo aptitude install build-essential python-scipy ipython
python-matplotlib python-dev python-setuptools libsndfile-dev
libasound2-dev mysql-server python-mysqldb
Upload scikits.audiolab-0.11.0
ec2:~/scikits.audiolab-0.11.0$ sudo python setup.py install
ec2:~$ sudo rm -rf scikits.audiolab-0.11.0
ec2:~$ nano .ipython/ipy_user_conf.py
ip.ex('import matplotlib; matplotlib.use("Agg"); import scipy, pylab,
scipy.signal as sig, scipy.linalg as lin, scipy.sparse as spar,
os, sys, MySQLdb, boto; from scikits import audiolab')
import ipy_greedycompleter
import ipy_autoreload
1: Installing third party packages to hosted spaces
You can indeed install third party packages to your hosted space. If it's a pure python package, all that's needed is to unpack it to a directory and then add that directory to your PYTHONPATH environment variable or sys.path.
This can be tiring to do often, and won't work easily for compiled modules. If you have shell access to your python host, the excellent virtualenv package allows you to do set up a private python environment with its own libraries.
To set up your virtualenv, you'll do something like this at the shell:
$ virtualenv $HOME/my_python
$ $HOME/my_python/bin/easy_install numpy
You can keep running easy_install for anything else you want to install in your personal python environment.
Now, when you write your python scripts, you will want to use your private python interpreter, if that is possible:
#!/home/myuser/my_python/bin/python
import numpy
# script here
If your python env cannot be specified (such as if run by mod_wsgi), you will need to add it to the import path:
import sys
sys.path.insert(0, '/home/myuser/my_python/lib/python2.5/site-packages')
import numpy
2: Hosting sites with numpy
I can't think of any hosting sites offhand which offer numpy pre-installed. However, Dreamhost/Bluehost for sharedhosts provide SSH access, and with shell access you can install numpy using the methods I described above. Any Virtual Private Server such as Linode/Slicehost will allow you to install whatever you desire, as well.
3: AppEngine
As mentioned above, AppEngine will not allow you to install C extensions (but pure python ones do work) so it's unlikely numpy will work for you on there, since I suspect some of its features use C speedups.
App Engine does not support any of numpy, scipy, or matplotlib, alas.
If you know exactly what OS and CPU your host is using, you could make an identical installation for yourself, download and install the same version of Python they're using, download the sources of packages you require and build them into .so (or .pyd, depending on the platform) files, and upload those -- sounds like a real tour de force.
Any of the many, many sites that offer normal virtual hosting (a virtual machine, typically Linux, with modest HW resources, but root privileges for you, ssh shell access, and a gcc you can use in particular) will be much easier to work with -- essentially, you'll download and install the software you need just about the same way you'd do on your own Linux workstation!
2 What hosting sites exist (either free or paid) which have Numpy/Matplotlib installed
PythonAnywhere offers web hosting and a simple in-browser IDE; lots of Python packages (including NumPy and Matplotlib) are pre-installed. There's a free plan that you can use to play around, and "Premium" and "Hosting" accounts with more features are $5 and $10/month respectively.
Full disclosure: I work there...
I do not have privledges yet to comment but I can provide an "answer".
3: AppEngine
Numpy is now available on Google App Engine:
https://code.google.com/p/googleappengine/issues/detail?id=190
However, matplotlib is still waiting:
http://code.google.com/p/googleappengine/issues/detail?id=482
Perhaps more people starring this issue will make it happen.
I will note that svgfig is an option as it is pure python:
http://code.google.com/p/svgfig/
Update:
Turns out matplotlib is now available:
https://developers.google.com/appengine/docs/python/tools/libraries27#matplotlib
Related
I have a project with multiple dependencies installed using virtualenv and pip. I want to run my project on a server which does not have pip installed. Unfortunately, installing pip is not an option.
Is there a way to export my required packages and bundle them with my project? What is the common approach in this situation?
Twitter uses pex files to bundle Python code with its dependencies. This will produce a single file. Another relevant tool is platter which also aims to reduce the complexity of deploying Python code to a server.
Another alternative is to write a tool yourself which creates a zip file with the Python and dependencies and unzips it in the right location on the server.
In Python 3.5 the module zipapp was introduced to improve support for this way of deploying / using code. This allows you to manage the creation of zip files containing Python code and run them directly using the Python interpreter.
#Simeon Visser's answer is a good way to deal with that. Mine is to build my python project with buildout.
This may be outside of scope of the question, but if your need is deploying applications to servers with their dependencies, have a look at virtualization and linux containers.
It is by far the most used solution to this problem, and will work with any type of application (python or not), and it is lightweight (the performance hit of LXC is not noticeable in most cases, and isolating apps is a GREAT feature).
Docker containers, besides being trendy right now, are a very convenient way to deploy applications without caring about dependencies, etc...
The same goes for development envs with vagrant.
First let me explain the current situation:
We do have several python applications which depend on custom (not public released ones) as well as general known packages. These depedencies are all installed on the system python installation. Distribution of the application is done via git by source. All these computers are hidden inside a corporate network and don't have internet access.
This approach is bit pain in the ass since it has the following downsides:
Libs have to be installed manually on each computer :(
How to better deploy an application? I recently saw virtualenv which seems to be the solution but I don't see it yet.
virtualenv creates a clean python instance for my application. How exactly should I deploy this so that usesrs of the software can easily start it?
Should there be a startup script inside the application which creates the virtualenv during start?
The next problem is that the computers don't have internet access. I know that I can specify a custom location for packages (network share?) but is that the right approach? Or should I deploy the zipped packages too?
Would another approach would be to ship the whole python instance? So the user doesn't have to startup the virutalenv? In this python instance all necessary packages would be pre-installed.
Since our apps are fast growing we have a fast release cycle (2 weeks). Deploying via git was very easy. Users could pull from a stable branch via an update script to get the last release - would that be still possible or are there better approaches?
I know that there are a lot questions. Hopefully someone can answer me r give me some advice.
You can use pip to install directly from git:
pip install -e git+http://192.168.1.1/git/packagename#egg=packagename
This applies whether you use virtualenv (which you should) or not.
You can also create a requirements.txt file containing all the stuff you want installed:
-e git+http://192.168.1.1/git/packagename#egg=packagename
-e git+http://192.168.1.1/git/packagename2#egg=packagename2
And then you just do this:
pip install -r requirements.txt
So the deployment procedure would consist in getting the requirements.txt file and then executing the above command. Adding virtualenv would make it cleaner, not easier; without virtualenv you would pollute the systemwide Python installation. virtualenv is meant to provide a solution for running many apps each in its own distinct virtual Python environment; it doesn't have much to do with how to actually install stuff in that environment.
Finally making a legitimate mac installer for my product. I've made a successful Windows installer with Inno installer. I'm not sure how to do this in Mac.
This must happen:
-Python is installed
-Wx is Installed
-Py Serial is installed
-Program is copied
-Shortcut is made.
I was doing this with Bash scripts before, but my customers having been complaining about those. Perhaps X-code package maker is the solution? I know the recommended method is "just copy files" but these libraries must be installed somehow.
Thanks in advance!
Unless I am using Fink for installing packages, I usually just download the .tar.gz file from the source and install it from terminal inside the unzipped folder containing the install.py file. Terminal command:
sudo python ./setup.py install
If you would like, I can show you how to set up and use Fink, which is another easy way to install packages / libraries.
tl;dr: py2app will make a self-contained application bundle out of your Python scripts, making it real easy to employ the 'just copy files' installation process. The libraries you need are bundled into the app bundle itself, so they don't need to be installed systemwide.
Also check out Optimizing for Mac OS X from the wxPython wiki; it gives good tips on using py2app and other useful information on building a Mac-friendly Python application.
On OS X, programs are generally installed through one of three ways: the Mac App Store, a package installer (.pkg/.mpkg), or a copyable application bundle on a disk image (.app in a .dmg). Each has its strengths and weaknesses.
The Mac App Store requires that you subscribe to Apple's restrictions and requirements, and may be a good choice for apps expecting wider distribution (though, nowadays, it can be a good way to reach that wider audience easily). Copyable application bundles are by far the simplest installation method pre-App Store, and still remain one of the more popular ways to install programs. Finally, an Installer package is a user-friendly way to install more complex programs requiring more than a simple application bundle (e.g. system services, files in particular directories, system-dependent components, advanced installation logic, etc.). I should note, though, that do exist complex applications which ship as application bundles and perform the bulk of their 'installation' at first run.
My experience with the Mac App Store is limited, so I won't talk about it. You can find more details at the official website.
Python is quite amenable to being shipped as an application bundle. You can use py2app to automatically create an application bundle for the program, and then drop that bundle into a Mac disk image (.dmg) using Disk Utility to create a complete installation package. This doesn't support making shortcuts, but on OS X it is much more usual for users to just drop the app into /Applications and make the necessary dock shortcut themselves if they want.
The next way is to make a metapackage (.mpkg) which will be installed using the OS X standard Installer utility. This is in line with what users will expect from a Mac application. IIRC, both Mac Python and wxPython ship as .pkg already, which should make it easier to integrate them into a metapackage. bdist_mpkg can help with making packages for pyserial and your own program, which can be added to the metapackage. Finally, using the third-party dockutil script, you can automatically add a dock shortcut. Note, however, that installers generally do not add shortcuts to the dock; it is more usual to have a program installed to the /Applications directory.`
I use Kodingen.com to test python scripts. I would like to install new python packages on Kodingen. I know I can use the 1-Click installation of the App-Store to install new packages from ohloh.net. When I install a package, it create a new directory in httpdocs folder. And then? How to setup the python package to use import in my scripts? Is it possible for a user to do it?
I'm looking for the complete procedure. As example, we can use PIL (Python Imaging Library). How to install it?
For information, Kodingen use a linux server.
I asked a similar question but this one is a more general question about the python packages install.
hey thammas, we are working on kodingen v1.0 with which you will be able to use your own server connected to your account. then you can install anything and use it freely.
but now, kodingen provides shared hosting account that doesn't enable anyone to install anything. same applies to ruby gems. but we will make this possible very soon.
just to reiterate, kodingen is able to run python/perl/php scripts, just not external/custom libraries until we give you your own servers. hope it clarifies.
I have developed a python CGI application which works just fine on my development box. My hosting provider however gives me little control of its server: I use a lot of custom stuff in my python environment (like sqlalchemy and mako templating) and the servers python version is far too old to be used. My question is: how do I set up a isolated, complete, standalone python environment in my home directory and install my required modules to run my app? ...the easiest way ;)
how do I set up a isolated, complete, standalone python environment in my home directory
mkdir /home/me/.local (if it doesn't already exist. You don't have to use .local but it is becoming the normal place to put this)
mkdir /home/me/.local/src (ditto)
cd /home/me/.local/src
wget http://python.org/ftp/python/2.6.4/Python-2.6.4.tgz
gzip -d Python-2.6.4.tgz
tar xf Python-2.6.4.tar
cd Python-2.6.4
./configure --prefix=/home/me/.local
make
make install
Hopefully you can now run Python:
/home/me/.local/bin/python
Install packages you need using the usual setup.py script, but with your version of Python:
/home/me/.local/bin/python setup.py install
Set hashbang on CGI files to use your version of Python:
#!/home/me/.local/bin/python
Consider migrating your application to WSGI if you can. You can of course still deploy WSGI apps through CGI using a wsgiref.handlers.CGIHandler for now, but in the future when you have a less woeful hosting environment you'll be able to deploy using a much less wasteful server interface such as mod_wsgi.
In your shoes, I'd use pyinstaller to bundle Python, my code, and all my dependencies into one installer executable, upload it, and run it. Just be sure to use the SVN trunk of pyinstaller -- the "released" version is WAY obsolete.
Be aware that with SQLAlchemy and everything else, with CGI you may find out you're really slow, since you're paying the full startup price everytime the page gets visited. But if CGI is all you can afford, I guess that's the way I would try to cope!-)
This looks like a job for virtualenv. From the site:
Also, what if you can't install packages into the global site-packages directory? For instance, on a shared host.
This looks to be right up your alley.
I am on Dreamhost's shared plan. Besides CGI, they also offer FastCGI which makes things much faster than CGI. You should check if your hosting provider offers that. Or maybe they provide Passenger for Ruby that you could piggyback your Python with.
If you compile Python yourself, keep in mind the UCS setting if you try to install precompiled packages and experience failures. See the StackOverflow article. Dreamhost's wiki has some advice on how you could build and deploy Python yourself on their servers; you might want to adapt to your needs.