I have a python script.
Main thread (if name=='main', etc): when the main thread initiates, it runs several threads to listen to data streams, events, and to process them. The main thread then starts running the Flask application (app.run()). Processing and data is sent to the front-end Flask app (no issues here)
The Apache Server and mod_wsgi requires me to directly import the app, meaning that my other threads won't run.
My dilemma. In the examples I've seen, the .wsgi script from someapp imports app as application. This would only run the flask application. If I managed to somehow run the python script instead as main, the flask application would be ran on localhost:5000 by default and is not recommended in production to change or use .run().
First of all, is it possible to get this application on a server in this current structure? How would I get the whole application to work on a server? Would I need to completely restructure it? Is it not possible to specify host: 0.0.0.0 port:80 then run the python script instead of just importing the app? Any help is appreciated, any forwarding to other documentations.
Edit: for the sake of testing, I will be using AWS Ubuntu (any other linux distro can be used/switched to if needed).
Sort and misleading answer is yes, it is possible (make sure there is any other program that uses port 80 such as apache etc):
if __name__ == '__main__':
app.run(host='0.0.0.0', port=80)
However, you should not do that. Not recommended as it states in the documentation:
You can use the builtin server during development, but you should use
a full deployment option for production applications. (Do not use the
builtin development server in production.)
Proxy HTTP traffic through apache2 to Flask is much better.
This way, apache2 can handle all your static files and act as a reverse proxy for your dynamic content, passing those requests to Flask.
To have threads check the documentation of WSGIDaemonProcess.
Example of Apache/mod_wsgi configuration should looks like this:
WSGIDaemonProcess mysite processes=3 threads=2 display-name=mod_wsgi
WSGIProcessGroup mysite
WSGIScriptAlias / /some/path/wsgi.py
I managed to find an answer to this without diverging too far from guides on how to get a Flask application working with Python3 and Apache2.
In short, when you initialise Flask, you most likely do something like this:
from flask import Flask
app = Flask(__name__)`
The proposed solution:
import atexit #for detecting flask exit
import threading
from flask import Flask
shareddata = 0
running = False
def init_app():
global shareddata
global running
running = True
app = Flask(__name__)
# some threading goes here
# e.g.
def jointhread():
running=False
t.join()
def MyThread1():
while(running):
#do something
t1 = threading.Thread(target=MyThread1, args=[])
t1.start()
atexit.register(jointhread)
return app
app = init_app()
Threading might not work, whichever's applicable.
I had a similar issue where there was a thread I wanted to constantly monitor data using an API. I ended up importing the function(s) I wanted threaded to my WSGI file and kicked them off there.
Example
import threading
from main import <threaded_function>
my_thread = threading.Thread(target=<threaded_function>)
my_thread.start()
Related
The usual flags: I'm new to Python, I'm new to PyTest, I'm new to Flask.
I need to create some server independent tests to test an api which calls a third-party.
I cannot access that api directly, but I can tell it what url to use for each third-party.
So what I want to do is to have a fake api running on the side (localhost) while I'm running my tests, so when the api that I'm testing needs to consume the third-parties, it uses my fake-api instead.
So I created the following app.py:
from flask import Flask
from src.fakeapi.routes import configure_routes
app = Flask(__name__)
configure_routes(app)
def start_fake_api():
app.run(debug=True)
And my_test.py:
from src.fakeapi.app import start_fake_api
#start_fake_api()
def test_slack_call():
send_request_to_api_to_configure_which_url_to_use_to_call_third_party("http://127.0.0.1:5000/")
send_request_to_api_to_populate_table_using_third_party()
Now, this might be an oversimplified example, but that's the idea. My problem obviously is that once I run Flask the process just stays in stand by and doesn't continue with the tests.
I want to avoid having to depend on manually running the server before running the tests, and I want to avoid running my tests in parallel.
What's the best way to do this?
Can I somehow execute app.py when I execute pytest? Maybe by altering pytest.ini somehow?
Can I force a new thread just for the server to run?
Thanks in advance!
I don't see a good reason to run a fake server, when you can instead use mock libraries such as requests-mock or responses to respond.
That said, if you really do need to run a real server, you could set up a session scoped fixture with a cleanup.
Adding autouse will make the tests automagically start the server, but you can leave that out and just invoke the fixture in your test, รก la test_foo(fake_api)
Implementing the TODOed bit can be a little tricky; you'd probably need to set up the Werkzeug server in a way that you can signal it to stop; e.g. by having it wait on a threading.Event you can then raise.
#pytest.mark.fixture(scope="session", autouse=True)
def fake_api():
app = ...
port = random.randint(1025, 65535) # here's hoping no one is on that port
t = threading.Thread(target=lambda: app.run(debug=True, port=port))
t.start()
yield port
# TODO: implement cleanly terminating the thread here :)
I have made a website with django. I want to deploy this application to production.
Now I am confused about several things: at the time of the development I can run my application using the command: python manage.py runserver IP_OF_SERVER:PORT.
Now by this approach I can do everything. My tool (website) will only be used locally.
Is it fine that I deploy my website with this command only? Is is necessary to do django production processes and if it is necessary how to do that? I am new to django. Please help me to understand.
Usually, these kinds of things are deployed in a three tier fashion.
Here, the general approach is like so
[Database] <-(db calls)-> [Django app] <- wsgi -> [gunicorn] <- http -> [nginx] <- http -> public
your application is the "Django app" block over here. You could run it with something like manage.py runserver but that's a very lightweight toy server which can't really handle high levels of traffic. If you have a request that takes 1ms to handle and 100 users try to make the same request, the last client will have to wait 100ms before she can get the response. It's easy to solve this by just running more instances of your application but the dev server can't do that.
A so called "app server" like gunicorn will allow you to run a more powerful web server for your application that can spawn off multiple workers and handle some kinds of mischievous traffic patterns.
Now, even gunicorn can be bested by a high performance server especially for serving static assets like images, css, js etc. This is something like nginx. So, we set up our thing to have nginx facing the world and serving all static assets directly. And request to the actual application will be proxied to gunicorn and that will be served by your actual app.
This is not as complex as it sounds and you should be able to get something running within a day or so. All the technologies I've mentioned have substitutes with different characteristics. This is a good tutorial on how to get things going and what to look out for during deployment.
If you want to setup your Django production server through IIS in Windows server (If users are less, you can even use normal Windows 7 or 10 professional machines as well). This video can help you do it step by step
https://www.youtube.com/watch?v=kXbfHtAvubc
I have brought up couple of production websites this way.
Although the one you are trying to do also works, but the only problem with your approach is that you should take care that console is never closed by anyone or you accidentally. But there are lot more hidden problems, usually for production, this is not recommended.
To avoid accidental closures, you can do below in Windows (running it as a service):
For this approach, you need pywin32 needs to be installed, install it from here: https://sourceforge.net/projects/pywin32/files/pywin32/Build%20217/
import win32service
import win32serviceutil
import win32event
import subprocess
import os
class PySvc(win32serviceutil.ServiceFramework):
# you can NET START/STOP the service by the following name
_svc_name_ = "Name your Service here"
# this text shows up as the service name in the Service
# Control Manager (SCM)
_svc_display_name_ = "External Display Name of your service"
# this text shows up as the description in the SCM
_svc_description_ = "Description what this service does"
def __init__(self, args):
import platform
win32serviceutil.ServiceFramework.__init__(self, args)
# create an event to listen for stop requests on
self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
# core logic of the service
def SvcDoRun(self):
os.chdir('your site root directory')
subprocess.Popen('python manage.py runserver IP:PORT')
# if you need stdout and stderr, open file handlers and keep redirecting to those files with subprocess `stdout` and `stderr`
# called when we're being shut down
def SvcStop(self):
# tell the SCM we're shutting down
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
# fire the stop event
win32event.SetEvent(self.hWaitStop)
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(PySvc)
I am trying for the first time to deploy a django website on a microsoft webserver in service mode (without being logged in) using nssm.
To do so, it seems I can't use the usual
python manage.py runserver 0.0.0.0
So I have tried to add a new tornado.py file in my project and the nssm would point to it:
from tornado.wsgi import WSGIContainer
from tornado.ioloop import IOLoop
from tornado.web import FallbackHandler, RequestHandler, Application, StaticFileHandler
from wsgi import application
from mySite.settings import *
class MainHandler(RequestHandler):
def get(self):
self.write("Hi Tornado")
tr = WSGIContainer(application)
app = Application([
(r"/tornado", MainHandler),
(r"/static/(.*)", StaticFileHandler, {'path': STATIC_ROOT}),
(r"/media/(.*)", StaticFileHandler, {'path': MEDIA_ROOT}),
(r".*", FallbackHandler, dict(tr)),
])
if __name__ == '__main__':
app.listen(8000)
IOLoop.instance().start()
The nssm pointing to this file makes the deployment okay, but unfortunately, I have some long requests calling other APIs in the back-end django and when I call one of these long services, it is impossible to make an other request i.e I have to wait for the first request to be finished.
From what I have seen on other questions related to the same issue on this website, I need to add the tornado decorator below someway:
#gen
I have tried a couple of options without success and can't find an example working with django.
I use django only for the mvc framework, but have no use of the orm or the auth.
I would have to keep django because there is a lot of work in it and microsoft webserver/nssm because it is the best practice in my company, but is tornado able to help me on this? Am I looking in the right direction?
Thanks in advance to all those who would take time to help me on this problem.
Heed the warning on the WSGIContainer docs: Tornado's WSGIContainer has no parallelism and is almost certainly a worse choice than other WSGI servers like gunicorn or uwsgi for WSGI-based applications.
The #gen.coroutine is for native Tornado applications; it is not available in any useful way for foreign applications running inside a WSGIContainer.
Tornado's Windows support is also limited.
I have a Python application running in a framework that drives a network protocol to control remote devices. Now I want to add a browser-based monitoring and control and I am looking at the Pyramid framework to build it.
Normally you start a Pyramid application using pserve from a command line, but I can't find any documentation or examples for how to invoke it inside a host application framework. This needs to be done in such a way that the Pyramid code can access objects in the host application.
Is this a practical use case for Pyramid or should I be looking for some other WSGI-based framework to do this?
A WSGI app is basically a function which receives some input and returns a response, you don't really need pserve to serve a WSGI app, it's more like a wrapper which assembles an application from an .ini file.
Have a look at Creating Your First Pyramid Application chapter in Pyramid docs:
from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response
def hello_world(request):
return Response('Hello %(name)s!' % request.matchdict)
if __name__ == '__main__':
config = Configurator()
config.add_route('hello', '/hello/{name}')
config.add_view(hello_world, route_name='hello')
app = config.make_wsgi_app()
server = make_server('0.0.0.0', 8080, app)
server.serve_forever()
the last two lines create a server which listens on port 8080.
Now, the trickier problem is that the serve_forever call is blocking, i.e.the program stops on that line until you hit Ctrl-C and stop the script. This makes it a bit non-trivial to have your program to "drive a network protocol to control remote devices" and to serve web pages at the same time (this is unlike other event-based platforms such as Node.js where it's trivial to have two servers to listen on different ports within the same process).
One possible solution to this problem would be to run the webserver in a separate thread.
I've created a web.py application, and now that it is ready to be deployed, I want to run in not on web.py's built-in webserver. I want to be able to run it on different webservers, Apache or IIS, without having to change my application code. This is where WSGI is supposed to come in, if I understand it correctly.
However, I don't understand what exacly I have to do to make my application deployable on a WSGI server? Most examples assume you are using Pylons/Django/other-framework, on which you simply run some magic command which fixes everything for you.
From what I understand of the (quite brief) web.py documentation, instead of running web.application(...).run(), I should use web.application(...).wsgifunc(). And then what?
Exactly what you need to do to host it with a specific WSGI hosting mechanism varies with the server.
For the case of Apache/mod_wsgi and Phusion Passenger, you just need to provide a WSGI script file which contains an object called 'application'. For web.py 0.2, this is the result of calling web.wsgifunc() with appropriate arguments. For web.py 0.3, you instead use wsgifunc() member function of object returned by web.application(). For details of these see mod_wsgi documentation:
http://code.google.com/p/modwsgi/wiki/IntegrationWithWebPy
If instead you are having to use FASTCGI, SCGI or AJP adapters for a server such as Lighttpd, nginx or Cherokee, then you need to use 'flup' package to provide a bridge between those language agnostic interfaces and WSGI. This involves calling a flup function with the same WSGI application object above that something like mod_wsgi or Phusion Passenger would use directly without the need for a bridge. For details of this see:
http://trac.saddi.com/flup/wiki/FlupServers
Important thing is to structure your web application so that it is in its own self contained set of modules. To work with a particular server, then create a separate script file as necessary to bridge between what that server requires and your application code. Your application code should always be outside of the web server document directory and only the script file that acts as bridge would be in server document directory if appropriate.
As of July 21 2009, there is a much fuller installation guide at the webpy install site, that discusses flup, fastcgi, apache and more. I haven't yet tried it, but it seems like it's much more detailed.
Here is an example of two hosted apps using cherrypy wsgi server:
#!/usr/bin/python
from web import wsgiserver
import web
# webpy wsgi app
urls = (
'/test.*', 'index'
)
class index:
def GET(self):
web.header("content-type", "text/html")
return "Hello, world1!"
application = web.application(urls, globals(), autoreload=False).wsgifunc()
# generic wsgi app
def my_blog_app(environ, start_response):
status = '200 OK'
response_headers = [('Content-type','text/plain')]
start_response(status, response_headers)
return ['Hello world! - blog\n']
"""
# single hosted app
server = wsgiserver.CherryPyWSGIServer(
('0.0.0.0', 8070), application,
server_name='www.cherrypy.example')
"""
# multiple hosted apps with WSGIPathInfoDispatcher
d = wsgiserver.WSGIPathInfoDispatcher({'/test': application, '/blog': my_blog_app})
server = wsgiserver.CherryPyWSGIServer(('0.0.0.0', 8070), d)
server.start()