Related
Ideally I’d like to build a package to deploy to Debian. Ideally the installation process would check the system has the required dependencies installed, as well as configure Cronjobs, set up users etc.
I’ve tried googling around and I understand a .deb is the format I can distribute in - but that is as far as I got since I’m getting confused now with the tooling I need to get up to speed with. The other option is to just git clone on the server and configure the environment manually… but that’s not preferable for obvious reasons.
How can I get started with building a Debian package and is that the right direction for deploying web applications? If anyone could point me in the right direction tools-wise and perhaps a tutorial that would be massively appreciated :) also if you advise to just take the simple route with git, happy to take that advice as well if you explain why. if it makes any difference I’m deploying one nodejs and one python web application
You can for sure package everything as a Linux application; for example using pyinstaller for your python webapp.
Besides that, it depends on your use case.
I will focus on the second part of your question,
How can I get started with building a Debian package and is that the right direction for deploying web applications?
as that seems to be what you are after when considering other alternatives to .dev already in your question.
I want to deploy 1-2 websites on my linux server
In this case, I'd say manually git clone and configure everything. Its totally fine when you know that there won't be much more running on the server and is pretty hassle free.
Why spend time packaging when noone will need the package ever again after you just installed it on your server?
I want to distribute my webapps to others on Debian
Here a .deb would make total sense. For example Plex media server and other applications are shipped like this.
If the official Debian wiki is too abstract, there are also other more hands on guides to get you started quickly. You could also get other .deb Packages and extract them to see what they are made up from. You mentioned one of your websites is using python, so I just suspect it might be flask or Django. If it's Django, there is an example repository you might want to check out.
I want to run a lot of stuff on my server / distribute to other devs and platforms / or scale soon
In this case I would make the webapps into docker containers. They are easy to build, share, and deploy. On top you can easily bundle all dependencies and scripts to make sure everything is setup right. Also they are easy to run and stop. So you have a simple "on/off" switch if your server is running low on resources while you want to run something else. I highly favour this solution, as it also allows you to easily control what is running on what ip when you deploy more and more applications to your server. But, as you pointed out, it runs with a bit of overhead and is not the best solution on weak hardware.
Also, if you know for sure what will be running on the server long term and don't need the flexibility I would probably skip Docker as well.
I have a python program that I would like to present as a simple web application. The program currently uses sqlite for storage. I also need to distribute the whole thing to colleagues so having something standalone and easy to start would be ideal ( no install if possible). This web app is meant to be used locally , not by multiple users over a network.
Is there a suitable python framework that might fit my needs? I looked at Django so far but it seems a bit heavy handed for what I need.
Thanks for any suggestions.
I have never tried it myself, but you could try Bottle:
Bottle is a fast, simple and lightweight WSGI micro web-framework for
Python. It is distributed as a single file module and has no
dependencies other than the Python Standard Library.
try http://docs.python.org/library/simplehttpserver.html
As web frameworks are not part of the standard lib, you will have to install something in every case. I would propse to look at http://flask.pocoo.org/. It has a build in WSGI server.
Lots of choices for Python web frameworks! Another is web2py which is designed to work out of the box and allows, but doesn't require, through-the-web development. It is mature and has a strong community and is well-documented.
Tornado as a framework may be a lot more than what you're looking for. However it will meet the requirement of being a completely python based web server. http://tornadoweb.org
I generally just download the source, drop it in /tornado/ of my project and do includes there from the app.
I don't think that any web framework is specifically oriented for the use case you're talking about; They all assume they are running on a server and there's a browser on a remote machine that is accessing them.
A better approach is to think about the HTTP server you'll be using. It's probably preferable to use a server that's as easy to pack and ship as the rest of the python code you'll be using. Now most frameworks provide a 'development' server that's easy to invoke from the command line, but most of them are intended to be "easy for the developer" which often means they are restricted to a single thread. This is bad for deployment because single threaded servers will always feel a bit sluggish.
CherryPy stands out in contrast, by providing a full featured, embedded server that's easy to configure for many use cases, and is available by default with the rest of the framework. There are probably others, but I haven't used 'em.
I've made a small little "application" utilizing Django as a framework. This is an application that is not ment to be deployed to a server but run locally on a machine. Thus the runserver.py works just nice.
I, as an developer is comfortable with fireing up the terminal, running python manage.py runserver and using it.
But I have some Mac OS X and Windows friends wanting to use my application, and they dont have virtualenv, git or anything else.
Is there a way I can package this to be a standalone product? Of course it would depend on Python being installed on the system, but it is possbile to package the virtualenv — with django and everything, and just copy it to another system and make it work?
And maybe even run the runserver in some kind a deamon mode?
Use setuptools and easy_install.
Here's an introductory article.
Yes, you can package it. Django may not be the easiest to do this with, but the principles are the same for other frameworks. You need to make an installer that installs everything you need. And that installer needs to be different for different platforms. such as Windows, Ubuntu, OS X etc. That also means that the answer is significantly different for each platform, and only half of the answer is dependning on Django. :-(
This kinda sucks, but that's life, currently. There is no nice platform independent way to install software for end users.
I also haven't found the perfect solution for this yet.
My current approach is to provide a docker image because that's really easy to use for everyone. This includes an alpine base image because it's tiny and python + django and the app itself. You can also include a webserver like nginx and an app server like uwsgi or gunicorn and expose a port for it.
So in the end your user would just run the container and access the web app under http://localhost:9000/ or something like this.
This is really handy and also my preferred way of trying out some app I've found.
The "proper" way would be do build a package for every OS and distribution you are targeting and a simple zip bundle so people can also install the app manually.
To build the packages I suggest using fpm. It takes most of the pain of doing the packaging with their native tools away. The packages would then depend on a proper application server like uwsgi or gunicorn.
So in the end you could then install it like apt install your-package and it would depend on python-django, uwsgi etc.
For the location and where to put all the files in the package every distribution has their own way of doing it. I prefer putting everything under /usr/share/webapps/myapp/ and having the config under /etc/myapp/config.py or something like that.
For Windows and macOS there are solutions like PyInstaller. I haven't used it yet for a django app but it should do the job. You should include a app server like uwsgi in there too.
In generel you don't want to run the django dev server in a production environment. So keep that in mind when packaging.
I hope that helps a bit.
There are several ways of doing this. I think you are looking more for build tools (which includes packaging) rather than just a Python solution. Here are a couple that I've used in the past:
zc.buildout: Used to build and deploy Python modules and applications, but is also able to work with other languages with a little massaging. Easy to use (for a build tool).
make: The software build classic. Works with practically all languages but a little archaic and hard to learn for a first timer.
The new snap package manager for Linux should suit the task perfectly. It provides all the solutions that were quite a pain for Python apps so far (dependencies, interpreter etc) and at the same time avoiding the complexity of Docker.
These days Docker is probably a good answer
User needs to install Docker first, but it runs on Windows and OSX as well as Linux.
Your Dockerfile takes care of installing all the dependencies and then runs the devserver (or you could even run a proper webserver in the container)
I currently work with .NET exclusively and would like to have a go at python. To this end I need to set up a python development environment. I guide to this would be handy. I guess I would be doing web development so will need a web server and probably a database. I also need pointers to popular ORM's, an MVC framework, and a testing library.
One of my main criteria with all this is that I want to understand how it works, and I want it to be as isolated as possible. This is important as i am wary of polluting what is a working .NET environment with 3rd party web and database servers. I am perfectly happy using SQLite to start with if this is possible.
If I get on well with this I am also likely to want to set up automated build and ci server (On a virtual machine, probably ubuntu). Any suggestions for these would be useful.
My ultimate aim if i like python is to have similar sorts of tools that i have available with .NET and to really understand the build and deployment of it all. To start with I will settle for a simple development environment that is as isolated as possible and will be easy to remove if I don't like it. I don't want to use IronPython as I want the full experience of developing a python solution using the tools and frameworks that are generally used.
It's not that hard to set up a Python environment, and I've never had it muck up my .NET work. Basically, install Python --- I'd use 2.6 rather than 3.0, which is not yet broadly accepted --- and add it to your PATH, and you're ready to go with the language. I wouldn't recommend using a Ubuntu VM as your development environment; if you're working on Windows, you might as well develop on Windows, and I've had no significant problems doing so. I go back and forth from Windows to Linux with no trouble.
If you have an editor that you're comfortable with that has basic support for Python, I'd stick with it. If not, I've found Geany to be a nice, light, easy-to-use editor with good Python support, though I use Emacs myself because I know it; other people like SCITE, NotePad++, or any of a slew of others. I'd avoid fancy IDEs for Python, because they don't match the character of the language, and I wouldn't bother with IDLE (included with Python), because it's a royal pain to use.
Suggestions for libraries and frameworks:
Django is the standard web framework, but it's big and you have to work django's way; I prefer CherryPy, which is also actively supported, but is light, gives you great freedom, and contains a nice, solid webserver that can be replaced easily with httpd.
Django includes its own ORM, which is nice enough; there's a standalone one for Python, though, which is even nicer: SQL Alchemy
As far as a testing library goes, pyunit seems to me to be the obvious choice
Good luck, and welcome to a really fun language!
EDIT summary: I originally recommended Karrigell, but can't any more: since the 3.0 release, it's been continuously broken, and the community is not large enough to solve the problems. CherryPy is a good substitute if you like a light, simple framework that doesn't get in your way, so I've changed the above to suggest it instead.
Well, if you're thinking of setting up an Ubuntu VM anyway, you might as well make that your development environment. Then you can install Apache and MySQL or Postgres on that VM just via the standard packaging tools (apt-get install), and there's no danger of polluting your Windows environment.
You can either do the actual development on your Windows machine via your favourite IDE, using the VM as a networked drive and saving the code there, or you can just use the VM as a full desktop environment and do everything there, which is what I would recommend.
Install the pre-configured ActivePython release from activestate.
Among other features, it includes the PythonWin IDE (Windows only) which makes it easy to explore Python interactively.
The recommended reference is Dive Into Python, mentioned many times on similar SO discussions.
You should install python 2.4, python 2.5, python 2.6 and python 3.0, and add to your path the one you use more often (Add c:\Pythonxx\ and c:\Pythonxx\Scripts).
For every python 2.x, install easy_install; Download ez_setup.py and then from the cmd:
c:\Python2x\python.exe x:\path\to\ez_setup.py
c:\Python2x\Scripts\easy_install virtualenv
Then each time you start a new project create a new virtual environment to isolate the specific package you needs for your project:
mkdir <project name>
cd <project name>
c:\Python2x\Scripts\virtualenv --no-site-packages .\v
It creates a copy of python and its libraries in .v\Scripts and .\v\Lib. Every third party packages you install in that environment will be put into .\v\Lib\site-packages. The -no-site-packages don't give access to the global site package, so you can be sure all your dependencies are in .\v\Lib\site-packages.
To activate the virtual environment:
.\v\Scripts\activate
For the frameworks, there are many. Django is great and very well documented but you should probably look at Pylons first for its documentions on unicode, packaging, deployment and testing, and for its better WSGI support.
For the IDE, Python comes with IDLE which is enough for learning, however you might want to look at Eclipse+PyDev, Komodo or Wingware Python IDE. Netbean 6.5 has beta support for python that looks promising (See top 5 python IDE).
For the webserver, you don't need any; Python has its own and all web framework come with their own. You might want to install MySql or ProgreSql; it's often better to develop on the same DB you will use for production.
Also, when you have learnt Python, look at Foundations of Agile Python Development or Expert Python Programming.
Using Python on Windows
SO: Python tutorial for total beginners?
Take a look at Pylons, read about WSGI and Paste.
There's nice introductory Google tech talk about them: ReUsable Web Components with Python and Future Python Web Development.
Here's my answer to similar question:
Django vs other Python web frameworks?
NOTE: I included a lot of links to frameworks, projects and what-not, but as a new user I was limited to 1 link per answer. If someone else with enough reputation to edit wants/can edit them into this answer instead of the footnotes, I'd be grateful.
There are some Python IDE's such as Wing IDE[1], I believe some people use Eclipse[2] with a python plugin[3] as well. A lot of people in the #python channel of FreeNode seem to prefer vim, emacs, nano and similar text editors in favor of IDE's. My personal preffered editor is Vim, but if you've mostly done .NET development on windows, presumably with the usual Visual X IDE's, vim and emacs will probably cause you culture shock and you'd be better of using an IDE.
Nearly all python web frameworks* support the WSGI standard[4], most of the large web servers have some sort of plugin to support WSGI, the others support WSGI via fast cgi or plain cgi.
The Zope[5] and Django[6] frameworks have their own ORM's, of other ORM's the two most well known appear to be SQL Alchemy[7] and SQL Object[8]. I only have experience with the former, but both support all possible sane database choices, including SQLite which is installed together with Python and hence perfectly suited to testing and experimenting without polluting your .NET environment with 3rd part web servers and database servers.
The builtin unittest[9] and pyunit[10] frameworks seem to be the preffered solutions for unit testing, but I don't have much experience with these.
bpython[11] and ipython[12] offer enhanced interactive python shells which can greatly help speed up and testing small bits of code and hence worth looking in to.
As for a list of well known and often used web frameworks, look into the following frameworks**:
Twisted[13] is a generic networking framework, which supports almost every single protocol under the sun.
Pylons[14] is light-weight framework aimed at being as flexible as possible and leaving all the choices about what ORM, templating language and what-not to you.
CherryPy[15] tries to provide an interface to expose Python objects to the web.
Django[6] attempts to be an all-in-one solution, builtin template system, ORM, admin pages and internationalization. While the previous frameworks have more DIY wiring together various frameworks work involved with them.
Zope[5] is aimed to be suitable for large enterprise applications, I've heard nothing but good things about it, but consensus seems to be that for smaller you're probably better off with one of the simpler and smaller frameworks.
TurboGears[16] is the framework I know the least about, but it seems to be mostly competition for Django.
This is everything I can think of right now, I'll edit and add stuff if I can think of it. I hope this helps you some in the wonderful world of python.
* - The main exception would be Apache's mod_python, which you should avoid for exactly that reason, use mod_wsgi instead.
** - Word of warning, I have not personally used these frameworks this is just a very short impression I have gotten from talking to other people about each framework, it may be wildly inaccurate. (If anyone has any corrections, do comment and I'll try to edit and fix this answer).
(The http:// is missing since they're recognized as links otherwise)
[1] www.wingware.com/
[2] www.eclipse.org/
[3] pydev.sourceforge.net/
[4] wsgi.org/wsgi/
[5] www.zope.org/
[6] www.djangoproject.com/
[7] www.sqlalchemy.org/
[8] www.sqlobject.org/
[9] docs.python.org/library/unittest.html
[10] pyunit.sourceforge.net/pyunit.html
[11] www.bpython-interpreter.org/
[12] ipython.scipy.org/
[13] twistedmatrix.com/trac/
[14] pylonshq.com/
[15] www.cherrypy.org/
[16] turbogears.org/
Environment?
Here is the simplest solution:
Install Active Python 2.6. Its the Python itself, but comes with some extra handy useful stuff, like DiveintoPython chm.
Use Komodo Edit 5. It is among the good free editor you can use for Python.
Use IDLE. Its the best simplest short snippet editor, with syntax highlighting and auto complete unmatched by most other IDEs. It comes bundled with python.
Use Ipython. Its a shell that does syntax highlighting and auto complete, bash functions, pretty print, logging, history and many such things.
Install easy_install and/or pip for installing various 3rd party apps easily.
Coming from Visual Studio and .Net it will sound a lot different, but its an entirely different world.
For the framework, django works the best. Walk thro the tutorial and you will be impressed enough. The documentation rocks. The community, you have to see for yourself, to know how wonderful it is!!
Python has build in SQL like database and web server, so you wouldn't need to install any third party apps. Remember Python comes with batteries included.
If you've worked with Eclipse before you could give Pydev a try
I'm running Mac OS X Leopard and wanted to know what the easy way to setup a web development environment to use Python, MySQL, Apache on my machine which would allow me to develop on my Mac and then easily move it to a host in the future.
I've been trying to get mod_wsgi installed and configured to work with Django and have a headache now. Are there any web hosts that currently use mod_wsgi besides Google, so I could just develop there?
FWIW, we've found virtualenv [http://pypi.python.org/pypi/virtualenv] to be an invaluable part of our dev setup. We typically work on multiple projects that use different versions of Python libraries etc. It's very difficult to do this on one machine without some way to provide a localized, customized Python environment, as virtualenv does.
Most Python applications are moving away from mod_python. It can vary by framework or provider, but most development effort is going into mod_wsgi.
Using the WSGI standard will make your Python application server agnostic, and allow for other nice additions like WSGI middleware. Other providers may only provide CGI (which won't scale well performance wise), or FastCGI.
I've worked with Django using only the included server in the manager.py script and have not had any trouble moving to a production environment.
If you put your application in a host that does the environment configuration for you (like WebFaction) you should not have problems moving from development to production.
I run a Linux virtual machine on my Mac laptop. This allows me to keep my development environment and production environments perfectly in sync (and make snapshots for easy experimentation / rollback). I've found VMWare Fusion works the best, but there are free open source alternatives such as VirtualBox if you just want to get your feet wet.
I share the source folders from the guest Linux operating system on my Mac and edit them with the Mac source editor of my choosing (I use Eclipse / PyDev because the otherwise excellent TextMate doesn't deal well with Chinese text yet). I've documented the software setup for the guest Linux operating system here; it's optimized for serving multiple Django applications (including geodjango).
For extra added fun, you can edit your Mac's /etc/hosts file to make yourdomainname.com resolve to your guest Linux boxes internal IP address and have a simple way to work on / test multiple web projects online or offline without too much hassle.
What you're looking for is Mod_Python. It's an Apache-based interpreter for Python. Check it out here:
http://www.modpython.org/
Google App Engine has done it for you. Some limitations but it works great, and it gives you a path to hosting free.
Of course Mac OS X, in recent versions, comes with Python and Apache. However you may want to have more flexibility in the versions you use, or you may not like the tweaks Apple has made to the way they are configured. A good way to get a more generic set of tools, including MySQL, is to install them anew. This will help your portability issues. The frameworks can be installed relatively easily with one of these open source package providers.
Fink
MacPorts
MAMP
mod_wsgi is really, really simple.
Pyerweb is a really simple (~90 lines including comments/whitespace) WSGI-compliant routing-framework I wrote. Basically the WSGI API is just a function that gets passed environ, and wsgi_start_response, and it returns a string.
envrion is a dict with the request info, for example environ['PATH_INFO'] is the request URI)
wsgi_start_response which is a callable function which you execute to set the headers,:
wsgi_start_response(output_response, output_headers)
output_response is the string containing the HTTP status you wish to send (200 OK etc), and output_headers is a list-of-tuples containing your headers (for example, [("Content-type", "text/html")] would set the content-type)
Then the function returns a string containing your output.. That's all there is to it!
To run it, using spawning you can just do spawn scriptname.my_wsgi_function_nae and it will start listening on port 8080.
To use it via mod_wsgi, it's documentation is good, http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide and there is a django specific section
The up-side to using mod_wsgi is it's the standard for serving Python web-applications. I recently decided to play with Google App Engine, and was surprised when Pyerweb (which I linked to at the start of this answer) worked perfectly on it, completely unintentionally. I was even more impressed when I noticed Django applications run on it too.. Standardisation is a good thing!
You may want to look into web2py. It includes an administration interface to develop via your browser. All you need in one package, including Python.
Check out WebFaction—although I don't use them (nor am I related to / profit from their business in any way). I've read over and over how great their service is and particularly how Django-friendly they are. There's a specific post in their forums about getting up and running with Django and mod_wsgi.
Like others before me in this thread, I highly recommend using Ian Bicking's virtualenv to isolate your development environment; there's a dedicated page in the mod_wsgi documentation for exactly that sort of setup.
I'd also urge you to check out pip, which is basically a smarter easy_install which knows about virtualenv. Pip does two really nice things for virtualenv-style development:
Knows how to install from source control (SVN, Git, etc...)
Knows how to "freeze" an existing development environement's requirements so that you can create that environment somewhere else—very very nice for multiple developers or deployment.