I have a NodeJS web server (Express) locally that uses Python-Shell to fire off Python scripts (mostly scraper stuff using PyCurl). The Python scripts also use various modules that I have installed using pip. This all works locally with no issues, running Node v4.1.1 and Python 2.7.10.
Now, I need a server to run this setup on, and I'm cheap, so I really want to use Openshift, which I have used before for other Node-only projects.
My problem is that I need to work with Python, and while it is installed (v2.6 anyway), I can't do much with it. Pip is not installed at all, and I don't have administrative rights to install it. Easy_install seems to be present, but I get permission issues when trying to execute it (even just the test via "easy_install" with no parameters). I need pip to install several modules for my Python scripts.
I was able to follow the instructions here and get Python 2.7, setuptools, and pip all installed successfully, but they are only available by prefixing the path $OPENSHIFT_DATA_DIR/bin (which resolves to /var/lib/openshift/[my-id]/app-root/data/bin). So...
\> $OPENSHIFT_DATA_DIR/bin/python -V
Python 2.7.3
I can also successfully run my Python scripts if I prefix the python command with that path.
So my question is how can I either get python-shell to use that path for Python, or how can I update Openshift's environment variables so that the standard "python" command is pointed to my manually-built v2.7.3 executable?
I'm stumped, but I know next to nothing about Linux or Openshift's architecture specifically, so I'm hoping you guru's out there can help me out!
If someone ever has a real answer to this, feel free to post it up, but I ended up just abandoning Python altogether and re-wrote the scraper in Node using Cheerio. Much better!
I have a project with multiple dependencies installed using virtualenv and pip. I want to run my project on a server which does not have pip installed. Unfortunately, installing pip is not an option.
Is there a way to export my required packages and bundle them with my project? What is the common approach in this situation?
Twitter uses pex files to bundle Python code with its dependencies. This will produce a single file. Another relevant tool is platter which also aims to reduce the complexity of deploying Python code to a server.
Another alternative is to write a tool yourself which creates a zip file with the Python and dependencies and unzips it in the right location on the server.
In Python 3.5 the module zipapp was introduced to improve support for this way of deploying / using code. This allows you to manage the creation of zip files containing Python code and run them directly using the Python interpreter.
#Simeon Visser's answer is a good way to deal with that. Mine is to build my python project with buildout.
This may be outside of scope of the question, but if your need is deploying applications to servers with their dependencies, have a look at virtualization and linux containers.
It is by far the most used solution to this problem, and will work with any type of application (python or not), and it is lightweight (the performance hit of LXC is not noticeable in most cases, and isolating apps is a GREAT feature).
Docker containers, besides being trendy right now, are a very convenient way to deploy applications without caring about dependencies, etc...
The same goes for development envs with vagrant.
In my free time I am developing a small python project based on django. I use several other python packages to improve user and my development experience.
I have understand, that I run pip install pkgname to install a module named pkgname and then I use from pkgname import somewhat and go one and everything is fine and fluffy. I understand why to use virtualenv and already use serious tools like coverage and travis-ci.
Now, I want to the next step. At the moment, my project is not ready for world wide deployment. But I like to prepare for this as soon as possible. It's called Palco and hosted on GitHub.
Usually, I think, pip is the current default to install python modules. But the application I develop is nothing developers will use in their project but users will. So, nobody will do pip install palco and then write from palco import hellyeah. But they will configure their favourite web server to run django's wsgi-handler.
Currently, I would tell people,
please download this archive.zip, extract it and configure your webserver as mentioned
But this is a very php-way of deployment (just throw your files in a directory).
How make I my end-user python project deployable? Do I need a setup.py? Do I miss something?
I know, trac is installed with pip and then I "deploy" my very on trac-instance with trac-admin.py. Should I have a palco-admin.py with the same magic effect as trac-admin?
First let me explain the current situation:
We do have several python applications which depend on custom (not public released ones) as well as general known packages. These depedencies are all installed on the system python installation. Distribution of the application is done via git by source. All these computers are hidden inside a corporate network and don't have internet access.
This approach is bit pain in the ass since it has the following downsides:
Libs have to be installed manually on each computer :(
How to better deploy an application? I recently saw virtualenv which seems to be the solution but I don't see it yet.
virtualenv creates a clean python instance for my application. How exactly should I deploy this so that usesrs of the software can easily start it?
Should there be a startup script inside the application which creates the virtualenv during start?
The next problem is that the computers don't have internet access. I know that I can specify a custom location for packages (network share?) but is that the right approach? Or should I deploy the zipped packages too?
Would another approach would be to ship the whole python instance? So the user doesn't have to startup the virutalenv? In this python instance all necessary packages would be pre-installed.
Since our apps are fast growing we have a fast release cycle (2 weeks). Deploying via git was very easy. Users could pull from a stable branch via an update script to get the last release - would that be still possible or are there better approaches?
I know that there are a lot questions. Hopefully someone can answer me r give me some advice.
You can use pip to install directly from git:
pip install -e git+http://192.168.1.1/git/packagename#egg=packagename
This applies whether you use virtualenv (which you should) or not.
You can also create a requirements.txt file containing all the stuff you want installed:
-e git+http://192.168.1.1/git/packagename#egg=packagename
-e git+http://192.168.1.1/git/packagename2#egg=packagename2
And then you just do this:
pip install -r requirements.txt
So the deployment procedure would consist in getting the requirements.txt file and then executing the above command. Adding virtualenv would make it cleaner, not easier; without virtualenv you would pollute the systemwide Python installation. virtualenv is meant to provide a solution for running many apps each in its own distinct virtual Python environment; it doesn't have much to do with how to actually install stuff in that environment.
I write scientific software in Numpy/Scipy/Matplotlib. Having developed applications on my home computer, I am now interested in writing simple web applications. Example: user uploads image or audio file, my program processes it using Numpy/Scipy, and output is displayed on the browser using Matplotlib, or perhaps the user can download a processed file.
I already pay for hosting that does have Python 2.4.3 installed, but no Numpy/Scipy. I don't have shell access via command line, either. Just drag-and-drop FTP. Pretty limited, but I can get simple Python/CGI scripts working.
Surprisingly, a web search revealed few suitable options for web hosting with these capabilities already built in. (Please guide me if I am wrong.) I am learning about the Google App Engine, but I still don't have a full understanding about its tools and limitations. What the web did tell me is that others have similar concerns.
Hoping for solutions, I thought I would ask these simple questions to the awesome SO community:
Is there a simple way of installing numpy (or any third-party package/library) onto my already hosted space? I know the Python path on my hosted space, and I know the relevant Python/Numpy directories on my home computer. Can I simply copy files over and have it work? Both local and remote systems run Ubuntu.
What hosting sites exist (either free or paid) which have Numpy/Matplotlib installed or, if not installed, the possibility of installing it? Are there any documented sites that you can reference with working applications, no matter how simple?
Can Google App Engine help me in any way? Or is it totally for something else? Have you or others used it to write scientific applications in Python/Numpy? If so, could you reference them?
Thank you for your help.
EDIT: After the useful answers below, I bought the $20 plan at Slicehost, and I love it so far! (I first tried Amazon EC2. I must be stupid, but I just couldn't get it to work.) Setting up the Ubuntu server with Apache took mere hours (and I'm an Apache novice). It allows me to do exactly what I wanted with Python plus much more. I now have my own remote repository for version control, too. Thanks again!
EDIT 2: Nearly two years later, I tried Linode and EC2 (again). Linode is great. EC2 seemed easier this time around -- maybe it's just added experience, or maybe it's the improvements that Amazon made to the AWS management console. For those interested in Numpy/Scipy/Matplotlib/Audiolab, here is my Ubuntu cheat sheet whenever I launch an EC2 instance:
ec2:~$ sudo aptitude install build-essential python-scipy ipython
python-matplotlib python-dev python-setuptools libsndfile-dev
libasound2-dev mysql-server python-mysqldb
Upload scikits.audiolab-0.11.0
ec2:~/scikits.audiolab-0.11.0$ sudo python setup.py install
ec2:~$ sudo rm -rf scikits.audiolab-0.11.0
ec2:~$ nano .ipython/ipy_user_conf.py
ip.ex('import matplotlib; matplotlib.use("Agg"); import scipy, pylab,
scipy.signal as sig, scipy.linalg as lin, scipy.sparse as spar,
os, sys, MySQLdb, boto; from scikits import audiolab')
import ipy_greedycompleter
import ipy_autoreload
1: Installing third party packages to hosted spaces
You can indeed install third party packages to your hosted space. If it's a pure python package, all that's needed is to unpack it to a directory and then add that directory to your PYTHONPATH environment variable or sys.path.
This can be tiring to do often, and won't work easily for compiled modules. If you have shell access to your python host, the excellent virtualenv package allows you to do set up a private python environment with its own libraries.
To set up your virtualenv, you'll do something like this at the shell:
$ virtualenv $HOME/my_python
$ $HOME/my_python/bin/easy_install numpy
You can keep running easy_install for anything else you want to install in your personal python environment.
Now, when you write your python scripts, you will want to use your private python interpreter, if that is possible:
#!/home/myuser/my_python/bin/python
import numpy
# script here
If your python env cannot be specified (such as if run by mod_wsgi), you will need to add it to the import path:
import sys
sys.path.insert(0, '/home/myuser/my_python/lib/python2.5/site-packages')
import numpy
2: Hosting sites with numpy
I can't think of any hosting sites offhand which offer numpy pre-installed. However, Dreamhost/Bluehost for sharedhosts provide SSH access, and with shell access you can install numpy using the methods I described above. Any Virtual Private Server such as Linode/Slicehost will allow you to install whatever you desire, as well.
3: AppEngine
As mentioned above, AppEngine will not allow you to install C extensions (but pure python ones do work) so it's unlikely numpy will work for you on there, since I suspect some of its features use C speedups.
App Engine does not support any of numpy, scipy, or matplotlib, alas.
If you know exactly what OS and CPU your host is using, you could make an identical installation for yourself, download and install the same version of Python they're using, download the sources of packages you require and build them into .so (or .pyd, depending on the platform) files, and upload those -- sounds like a real tour de force.
Any of the many, many sites that offer normal virtual hosting (a virtual machine, typically Linux, with modest HW resources, but root privileges for you, ssh shell access, and a gcc you can use in particular) will be much easier to work with -- essentially, you'll download and install the software you need just about the same way you'd do on your own Linux workstation!
2 What hosting sites exist (either free or paid) which have Numpy/Matplotlib installed
PythonAnywhere offers web hosting and a simple in-browser IDE; lots of Python packages (including NumPy and Matplotlib) are pre-installed. There's a free plan that you can use to play around, and "Premium" and "Hosting" accounts with more features are $5 and $10/month respectively.
Full disclosure: I work there...
I do not have privledges yet to comment but I can provide an "answer".
3: AppEngine
Numpy is now available on Google App Engine:
https://code.google.com/p/googleappengine/issues/detail?id=190
However, matplotlib is still waiting:
http://code.google.com/p/googleappengine/issues/detail?id=482
Perhaps more people starring this issue will make it happen.
I will note that svgfig is an option as it is pure python:
http://code.google.com/p/svgfig/
Update:
Turns out matplotlib is now available:
https://developers.google.com/appengine/docs/python/tools/libraries27#matplotlib