I have a Python application running in a framework that drives a network protocol to control remote devices. Now I want to add a browser-based monitoring and control and I am looking at the Pyramid framework to build it.
Normally you start a Pyramid application using pserve from a command line, but I can't find any documentation or examples for how to invoke it inside a host application framework. This needs to be done in such a way that the Pyramid code can access objects in the host application.
Is this a practical use case for Pyramid or should I be looking for some other WSGI-based framework to do this?
A WSGI app is basically a function which receives some input and returns a response, you don't really need pserve to serve a WSGI app, it's more like a wrapper which assembles an application from an .ini file.
Have a look at Creating Your First Pyramid Application chapter in Pyramid docs:
from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response
def hello_world(request):
return Response('Hello %(name)s!' % request.matchdict)
if __name__ == '__main__':
config = Configurator()
config.add_route('hello', '/hello/{name}')
config.add_view(hello_world, route_name='hello')
app = config.make_wsgi_app()
server = make_server('0.0.0.0', 8080, app)
server.serve_forever()
the last two lines create a server which listens on port 8080.
Now, the trickier problem is that the serve_forever call is blocking, i.e.the program stops on that line until you hit Ctrl-C and stop the script. This makes it a bit non-trivial to have your program to "drive a network protocol to control remote devices" and to serve web pages at the same time (this is unlike other event-based platforms such as Node.js where it's trivial to have two servers to listen on different ports within the same process).
One possible solution to this problem would be to run the webserver in a separate thread.
Related
I have made a website with django. I want to deploy this application to production.
Now I am confused about several things: at the time of the development I can run my application using the command: python manage.py runserver IP_OF_SERVER:PORT.
Now by this approach I can do everything. My tool (website) will only be used locally.
Is it fine that I deploy my website with this command only? Is is necessary to do django production processes and if it is necessary how to do that? I am new to django. Please help me to understand.
Usually, these kinds of things are deployed in a three tier fashion.
Here, the general approach is like so
[Database] <-(db calls)-> [Django app] <- wsgi -> [gunicorn] <- http -> [nginx] <- http -> public
your application is the "Django app" block over here. You could run it with something like manage.py runserver but that's a very lightweight toy server which can't really handle high levels of traffic. If you have a request that takes 1ms to handle and 100 users try to make the same request, the last client will have to wait 100ms before she can get the response. It's easy to solve this by just running more instances of your application but the dev server can't do that.
A so called "app server" like gunicorn will allow you to run a more powerful web server for your application that can spawn off multiple workers and handle some kinds of mischievous traffic patterns.
Now, even gunicorn can be bested by a high performance server especially for serving static assets like images, css, js etc. This is something like nginx. So, we set up our thing to have nginx facing the world and serving all static assets directly. And request to the actual application will be proxied to gunicorn and that will be served by your actual app.
This is not as complex as it sounds and you should be able to get something running within a day or so. All the technologies I've mentioned have substitutes with different characteristics. This is a good tutorial on how to get things going and what to look out for during deployment.
If you want to setup your Django production server through IIS in Windows server (If users are less, you can even use normal Windows 7 or 10 professional machines as well). This video can help you do it step by step
https://www.youtube.com/watch?v=kXbfHtAvubc
I have brought up couple of production websites this way.
Although the one you are trying to do also works, but the only problem with your approach is that you should take care that console is never closed by anyone or you accidentally. But there are lot more hidden problems, usually for production, this is not recommended.
To avoid accidental closures, you can do below in Windows (running it as a service):
For this approach, you need pywin32 needs to be installed, install it from here: https://sourceforge.net/projects/pywin32/files/pywin32/Build%20217/
import win32service
import win32serviceutil
import win32event
import subprocess
import os
class PySvc(win32serviceutil.ServiceFramework):
# you can NET START/STOP the service by the following name
_svc_name_ = "Name your Service here"
# this text shows up as the service name in the Service
# Control Manager (SCM)
_svc_display_name_ = "External Display Name of your service"
# this text shows up as the description in the SCM
_svc_description_ = "Description what this service does"
def __init__(self, args):
import platform
win32serviceutil.ServiceFramework.__init__(self, args)
# create an event to listen for stop requests on
self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
# core logic of the service
def SvcDoRun(self):
os.chdir('your site root directory')
subprocess.Popen('python manage.py runserver IP:PORT')
# if you need stdout and stderr, open file handlers and keep redirecting to those files with subprocess `stdout` and `stderr`
# called when we're being shut down
def SvcStop(self):
# tell the SCM we're shutting down
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
# fire the stop event
win32event.SetEvent(self.hWaitStop)
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(PySvc)
I have a python script.
Main thread (if name=='main', etc): when the main thread initiates, it runs several threads to listen to data streams, events, and to process them. The main thread then starts running the Flask application (app.run()). Processing and data is sent to the front-end Flask app (no issues here)
The Apache Server and mod_wsgi requires me to directly import the app, meaning that my other threads won't run.
My dilemma. In the examples I've seen, the .wsgi script from someapp imports app as application. This would only run the flask application. If I managed to somehow run the python script instead as main, the flask application would be ran on localhost:5000 by default and is not recommended in production to change or use .run().
First of all, is it possible to get this application on a server in this current structure? How would I get the whole application to work on a server? Would I need to completely restructure it? Is it not possible to specify host: 0.0.0.0 port:80 then run the python script instead of just importing the app? Any help is appreciated, any forwarding to other documentations.
Edit: for the sake of testing, I will be using AWS Ubuntu (any other linux distro can be used/switched to if needed).
Sort and misleading answer is yes, it is possible (make sure there is any other program that uses port 80 such as apache etc):
if __name__ == '__main__':
app.run(host='0.0.0.0', port=80)
However, you should not do that. Not recommended as it states in the documentation:
You can use the builtin server during development, but you should use
a full deployment option for production applications. (Do not use the
builtin development server in production.)
Proxy HTTP traffic through apache2 to Flask is much better.
This way, apache2 can handle all your static files and act as a reverse proxy for your dynamic content, passing those requests to Flask.
To have threads check the documentation of WSGIDaemonProcess.
Example of Apache/mod_wsgi configuration should looks like this:
WSGIDaemonProcess mysite processes=3 threads=2 display-name=mod_wsgi
WSGIProcessGroup mysite
WSGIScriptAlias / /some/path/wsgi.py
I managed to find an answer to this without diverging too far from guides on how to get a Flask application working with Python3 and Apache2.
In short, when you initialise Flask, you most likely do something like this:
from flask import Flask
app = Flask(__name__)`
The proposed solution:
import atexit #for detecting flask exit
import threading
from flask import Flask
shareddata = 0
running = False
def init_app():
global shareddata
global running
running = True
app = Flask(__name__)
# some threading goes here
# e.g.
def jointhread():
running=False
t.join()
def MyThread1():
while(running):
#do something
t1 = threading.Thread(target=MyThread1, args=[])
t1.start()
atexit.register(jointhread)
return app
app = init_app()
Threading might not work, whichever's applicable.
I had a similar issue where there was a thread I wanted to constantly monitor data using an API. I ended up importing the function(s) I wanted threaded to my WSGI file and kicked them off there.
Example
import threading
from main import <threaded_function>
my_thread = threading.Thread(target=<threaded_function>)
my_thread.start()
I have a website (which running in Amazon EC2 Instance) running Python Bottle application with CherryPy as its front end web server.
Now I need to add another website with a different domain name already registered. To reduce the cost, I want to utilize the existing website host to do that.
Obviously, virtual host is the solution.
I know Apache mod_wsgi could play the trick. But I don't want to replace CherryPy.
I've googled a a lot, there are some articles showing how to make virtual hosts on CherryPy, but they all assume Cherrypy as Web Sever + Web application, Not CherrPy as Web server and Bottle as Application.
How to use CherrPy as Web server and Bottle as Application to support multiple virtual hosts?
As you mentioned, use VirtualHost. In the example cherrypy.Application instances are used, but any WSGI callable (e. g. Bottle app) will do.
perhaps you can simply put nginx as reverse proxy and configure it to send the traffic to the two domains to the right upstream (the cherryPy webserver).
Another idea would be to use Nginx (http://wiki.nginx.org/Main) with uWsgi(http://projects.unbit.it/uwsgi/) & (uWsgi-python) plug-in
uWsgi has a module named emperor that you can link vhosts(vassals) in, sort of.
i'm a newbie at this myself, so not necessarily an answer but rather a suggestion to check it out.
just a heads up, uWsgi and Nginx can be a hassle to get it to work, depending on your linux distro. Does work nicely with bottle, tested it myself.
hope it helps
jwalker's answer is pretty clear. In case any CherryPy newbie need whole script for reference, I post one below.
import cherrypy
from bottle import Bottle
import os
app1 = Bottle()
app2 = Bottle()
#app1.route('/')
def homePage():
return "========= home1 ==============="
#app2.route('/')
def homePage_2():
return "========= home2 ==============="
vhost = cherrypy._cpwsgi.VirtualHost(None,
domains={
'www.domain1.com': app1,
'www.domain2.com': app2,
}
)
cherrypy.tree.graft(vhost)
cherrypy.config.update({
'server.socket_host': '192.168.1.4',
'server.socket_port': 80,
})
cherrypy.engine.start()
cherrypy.engine.block()
you could make www.domain1.com and www.domain1.com point to one IP adress of you server, so it servers for 2 domain in one Web Server.
I have a Django web application. I also have a spell server written using twisted running on the same machine having django (running on localhost:8090). The idea being when user does some action, request comes to Django which in turn connects to this twisted server & server sends data back to Django. Finally Django puts this data in some html template & serves it back to the user.
Here's where I am having a problem. In my Django app, when the request comes in I create a simple twisted client to connect to the locally run twisted server.
...
factory = Spell_Factory(query)
reactor.connectTCP(AS_SERVER_HOST, AS_SERVER_PORT, factory)
reactor.run(installSignalHandlers=0)
print factory.results
...
The reactor.run() is causing a problem. Since it's an event loop. The next time this same code is executed by Django, I am unable to connect to the server. How does one handle this?
The above two answers are correct. However, considering that you've already implemented a spelling server then run it as one. You can start by running it on the same machine as a separate process - at localhost:PORT. Right now it seems you have a very simple binary protocol interface already - you can implement an equally simple Python client using the standard lib's socket interface in blocking mode.
However, I suggest playing around with twisted.web and expose a simple web interface. You can use JSON to serialize and deserialize data - which is well supported by Django. Here's a very quick example:
import json
from twisted.web import server, resource
from twisted.python import log
class Root(resource.Resource):
def getChild(self, path, request):
# represents / on your web interface
return self
class WebInterface(resource.Resource):
isLeaf = True
def render_GET(self, request):
log.msg('GOT a GET request.')
# read request.args if you need to process query args
# ... call some internal service and get output ...
return json.dumps(output)
class SpellingSite(server.Site):
def __init__(self, *args, **kwargs):
self.root = Root()
server.Site.__init__(self, self.root, **kwargs)
self.root.putChild('spell', WebInterface())
And to run it you can use the following skeleton .tac file:
from twisted.application import service, internet
site = SpellingSite()
application = service.Application('WebSpell')
# attach the service to its parent application
service_collection = service.IServiceCollection(application)
internet.TCPServer(PORT, site).setServiceParent(service_collection)
Running your service as another first class service allows you to run it on another machine one day if you find the need - exposing a web interface makes it easy to horizontally scale it behind a reverse proxying load balancer too.
reactor.run() should be called only once in your whole program. Don't think of it as "start this one request I have", think of it as "start all of Twisted".
Running the reactor in a background thread is one way to get around this; then your django application can use blockingCallFromThread in your Django application and use a Twisted API as you would any blocking API. You will need a little bit of cooperation from your WSGI container, though, because you will need to make sure that this background Twisted thread is started and stopped at appropriate times (when your interpreter is initialized and torn down, respectively).
You could also use Twisted as your WSGI container, and then you don't need to start or stop anything special; blockingCallFromThread will just work immediately. See the command-line help for twistd web --wsgi.
You should stop reactor after you got results from Twisted server or some error/timeout happening. So on each Django request that requires query your Twisted server you should run reactor and then stop it. But, it's not supported by Twisted library — reactor is not restartable. Possible solutions:
Use separate thread for Twisted reactor, but you will need to deploy your django app with server, which has support for long running threads (I don't now any of these, but you can write your own easily :-)).
Don't use Twisted for implementing client protocol, just use plain stdlib's socket module.
I've created a web.py application, and now that it is ready to be deployed, I want to run in not on web.py's built-in webserver. I want to be able to run it on different webservers, Apache or IIS, without having to change my application code. This is where WSGI is supposed to come in, if I understand it correctly.
However, I don't understand what exacly I have to do to make my application deployable on a WSGI server? Most examples assume you are using Pylons/Django/other-framework, on which you simply run some magic command which fixes everything for you.
From what I understand of the (quite brief) web.py documentation, instead of running web.application(...).run(), I should use web.application(...).wsgifunc(). And then what?
Exactly what you need to do to host it with a specific WSGI hosting mechanism varies with the server.
For the case of Apache/mod_wsgi and Phusion Passenger, you just need to provide a WSGI script file which contains an object called 'application'. For web.py 0.2, this is the result of calling web.wsgifunc() with appropriate arguments. For web.py 0.3, you instead use wsgifunc() member function of object returned by web.application(). For details of these see mod_wsgi documentation:
http://code.google.com/p/modwsgi/wiki/IntegrationWithWebPy
If instead you are having to use FASTCGI, SCGI or AJP adapters for a server such as Lighttpd, nginx or Cherokee, then you need to use 'flup' package to provide a bridge between those language agnostic interfaces and WSGI. This involves calling a flup function with the same WSGI application object above that something like mod_wsgi or Phusion Passenger would use directly without the need for a bridge. For details of this see:
http://trac.saddi.com/flup/wiki/FlupServers
Important thing is to structure your web application so that it is in its own self contained set of modules. To work with a particular server, then create a separate script file as necessary to bridge between what that server requires and your application code. Your application code should always be outside of the web server document directory and only the script file that acts as bridge would be in server document directory if appropriate.
As of July 21 2009, there is a much fuller installation guide at the webpy install site, that discusses flup, fastcgi, apache and more. I haven't yet tried it, but it seems like it's much more detailed.
Here is an example of two hosted apps using cherrypy wsgi server:
#!/usr/bin/python
from web import wsgiserver
import web
# webpy wsgi app
urls = (
'/test.*', 'index'
)
class index:
def GET(self):
web.header("content-type", "text/html")
return "Hello, world1!"
application = web.application(urls, globals(), autoreload=False).wsgifunc()
# generic wsgi app
def my_blog_app(environ, start_response):
status = '200 OK'
response_headers = [('Content-type','text/plain')]
start_response(status, response_headers)
return ['Hello world! - blog\n']
"""
# single hosted app
server = wsgiserver.CherryPyWSGIServer(
('0.0.0.0', 8070), application,
server_name='www.cherrypy.example')
"""
# multiple hosted apps with WSGIPathInfoDispatcher
d = wsgiserver.WSGIPathInfoDispatcher({'/test': application, '/blog': my_blog_app})
server = wsgiserver.CherryPyWSGIServer(('0.0.0.0', 8070), d)
server.start()