We Keep Coding

Logging with command line waitress-serve

Is there a way to log waitress-serve output into a file?
The current command I use is:
waitress-serve --listen=localhost:8080 --threads=1 my_app_api:app
The application we used was not written with waitress in mind earlier, so we choose to serve it with command line to avoid change (for now at least).

TLDR waitress-serve doesn't provide a way to do it. See the 'how do i get it to log' section.
Background
Per the documentation for the command-line usage of waitress-serve, no - there's no way to setup logging. See arguments docs.
waitress-serve is just an executable to make running your server more convenient. It's source-code is here runner.py. If you read it, you can see it actually is basically just calling from waitress import serve; serve(**args) for you. (That code clip is not literally what it's doing, but in spirit yes).
The documentation for waitress says that it doesn't log http traffic. That's not it's job. But it will log it's own errors or stacktraces. logging docs. If you read the waitress source trying to find when it logs stuff, you'll notice it doesn't seem to log http traffic anywhere github log search. It primarily logs stuff to do with the socket layer.
Waitress does say that if you want to log http traffic, then you need another component. In particular, it points you to pastedeploy docs which is some middle-ware that can log http traffic for you.
The documentation from waitress is actually kind of helpful answering you question, though not direct and explicit. It says
The WSGI design is modular.
per the logging doc
I.e. waitress won't log http traffic for you. You'll need another WSGI component to do that, and because WSGI is modular, you can probably choose a few things.
If you want some background on how this works, there's a pretty good post here leftasexercise.com
OK, how do I get it to log?
Use tee
Basically, if you just want to log the same stuff that is output from waitress-serve then you don't need anything special.
waitress-serve --listen=localhost:8080 --threads=1 my_app_api:app | tee -a waitress-serve.log
Python logging
But if you're actually looking for logging coming from python's standard logger (say you app is making logger calls or you want to log http traffic) then, you can set that up in your python application code. E.g. edit your applications soure-code and get it to setup logging to a file
import logging
logging.basicConfig(filename='app.log', encoding='utf-8', level=logging.DEBUG)
PasteDeploy middleware for http logs
Or if your looking for apache type http logging then you can use something like PasteDeploy to do it. Note, PasteDeploy is another python dependency so you'll need to install it. E.g.
pip install PasteDeploy
Then you need to setup a .ini file that tells PasteDeploy how to start your server and then also tell it to use TransLogger to create apache type http logs. This is explained more detail here logging with pastedeploy The ini file is specific to each app, but from your question is sounds like the ini file should look like:
[app:wsgiapp]
use = my_app_api:app
[server:main]
use = egg:waitress#main
host = 127.0.0.1
port = 8080
[filter:translogger]
use = egg:Paste#translogger
setup_console_handler = False
[pipeline:main]
pipeline = translogger
app
You'll still need to edit the source-code of your app to get PasteDeploy to load the app with your configuration file:
from paste.deploy import loadapp
wsgi_app = loadapp('config:/path/to/config.ini')
Webframework-dependent roll-your-own http logging
Even if you want to log http traffic, you don't necessarily need something like PasteDeploy. For example, if you are using flask as the web-framework, you can write your own http logs using after_request decorator:
#app.after_request
def after_request(response):
timestamp = strftime('[%Y-%b-%d %H:%M]')
logger.error('%s %s %s %s %s %s', timestamp, request.remote_addr, request.method, request.scheme, request.full_path, response.status)
return response
See the full gist at https://gist.github.com/alexaleluia12/e40f1dfa4ce598c2e958611f67d28966

Multi-process Flask Application on Apache

I am trying to create a simple web application with Python3/Flask and serve it on Apache. I could not figure out how can I make my application to respond multiple requests.
This is my wsgi file:
import sys
import os
sys.path.insert(0, '/var/www/html/FlaskDeploy')
from apps import app as application
This code excerpt from httpd.conf file:
<VirtualHost *:80>
DocumentRoot /var/www/html/FlaskDeploy
WSGIScriptAlias / /var/www/html/FlaskDeploy/app.wsgi
WSGIDaemonProcess apps threads=1 python-path=/var/www/html/FlaskDeploy/env/bin:/var/www/html/FlaskDeploy/env/lib/python3.6/site-packages
<Directory /var/www/html/FlaskDeploy>
WSGIScriptReloading On
WSGIProcessGroup apps
WSGIApplicationGroup %{GLOBAL}
Order deny,allow
Allow from all
</Directory>
</VirtualHost>
Everything works fine but the application runs the requests one by one. For example, assume that each user performs a heavy database operation that takes 3 minutes. In this case, when 3 users from different locations open the application at the same time, the last one has to wait for 9 minutes (including the others' operations to be completed).
Basically I want to create a web application that is able to handle multiple requests.
I am coming from NodeJS world and I have never encountered with this problem on NodeJS. It runs on a single thread but can handle multiple requests.

It is only capable of only handling one request at a time, because that is what you told mod_wsgi to do in using:
threads=1
Don't set that option and it will instead default to 15 threads in the daemon process group and so that is how many requests it can handle concurrently.
If your requests are I/O bound that should be fine to begin with and you can tune things later. If your requests are more CPU bound than I/O bound, start to introduce additional processes as well and distribute requests across them.
processes=3 threads=5
Even if heavily I/O bound, do not increase threads too far per process, it is better to still spread them across processes as Python doesn't work as well with high number of threads per process.
For more information read the documentation:
http://modwsgi.readthedocs.io/en/develop/user-guides/processes-and-threading.html

Multithreaded Flask application on Apache server

I have a python script.
Main thread (if name=='main', etc): when the main thread initiates, it runs several threads to listen to data streams, events, and to process them. The main thread then starts running the Flask application (app.run()). Processing and data is sent to the front-end Flask app (no issues here)
The Apache Server and mod_wsgi requires me to directly import the app, meaning that my other threads won't run.
My dilemma. In the examples I've seen, the .wsgi script from someapp imports app as application. This would only run the flask application. If I managed to somehow run the python script instead as main, the flask application would be ran on localhost:5000 by default and is not recommended in production to change or use .run().
First of all, is it possible to get this application on a server in this current structure? How would I get the whole application to work on a server? Would I need to completely restructure it? Is it not possible to specify host: 0.0.0.0 port:80 then run the python script instead of just importing the app? Any help is appreciated, any forwarding to other documentations.
Edit: for the sake of testing, I will be using AWS Ubuntu (any other linux distro can be used/switched to if needed).

Sort and misleading answer is yes, it is possible (make sure there is any other program that uses port 80 such as apache etc):
if __name__ == '__main__':
app.run(host='0.0.0.0', port=80)
However, you should not do that. Not recommended as it states in the documentation:
You can use the builtin server during development, but you should use
a full deployment option for production applications. (Do not use the
builtin development server in production.)
Proxy HTTP traffic through apache2 to Flask is much better.
This way, apache2 can handle all your static files and act as a reverse proxy for your dynamic content, passing those requests to Flask.
To have threads check the documentation of WSGIDaemonProcess.
Example of Apache/mod_wsgi configuration should looks like this:
WSGIDaemonProcess mysite processes=3 threads=2 display-name=mod_wsgi
WSGIProcessGroup mysite
WSGIScriptAlias / /some/path/wsgi.py

I managed to find an answer to this without diverging too far from guides on how to get a Flask application working with Python3 and Apache2.
In short, when you initialise Flask, you most likely do something like this:
from flask import Flask
app = Flask(__name__)`
The proposed solution:
import atexit #for detecting flask exit
import threading
from flask import Flask
shareddata = 0
running = False
def init_app():
global shareddata
global running
running = True
app = Flask(__name__)
# some threading goes here
# e.g.
def jointhread():
running=False
t.join()
def MyThread1():
while(running):
#do something
t1 = threading.Thread(target=MyThread1, args=[])
t1.start()
atexit.register(jointhread)
return app
app = init_app()
Threading might not work, whichever's applicable.

I had a similar issue where there was a thread I wanted to constantly monitor data using an API. I ended up importing the function(s) I wanted threaded to my WSGI file and kicked them off there.
Example
import threading
from main import <threaded_function>
my_thread = threading.Thread(target=<threaded_function>)
my_thread.start()

using wsgi script to call exe

I'm running apache, Django and wsgi. I also use this other software called SAS to do statistical analysis. Just to give you some context. My end goal is when a client hits submit on a form written in the django, the appropriate sas script is called (via a python wsgi script) which performs calculations on the server, and then redirects the client to the output page.
I have a basic script called test5.py. It looks like this:
import os
import subprocess
def application(environ, start_response):
status = '200 OK'
output = 'Running External Program!'
f = open("C:\Documents and Settings\eric\Desktop\out.txt", 'a')
f.write('hi')
f.close()
#os.system(r'start "C:\Program Files\SAS92\SASFoundation\9.2\sas.exe"')
#subprocess.call([r'C:\Program Files\SAS92\SASFoundation\9.2\sas.exe'])
#os.startfile(r'C:\Program Files\SAS92\SASFoundation\9.2\sas.exe')
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
#start_response('301 Redirect', [('Location', 'http://myserver/reports'),])
start_response(status, response_headers)
return [output]
So what happens is that the out.txt file does get created and have hi written in the file. That's quite cool. The first 3 commented lines were 3 attempts to have this same script also call sas.exe which lives on the server. I'm just trying to get any .exe to work right now, so calling paint or wordpad would be fine. Those lines however do not seems to execute in the wsgi context. If I just load the Python command line, I can get the .exes to execute just fine. Also the last comment seems to be working properly in the redirecting. I'm not sure if I need to configure apache to add executables. Please forgive me if I'm using terms incorrectly. I am still quite new to all of this.
Thanks
Hi Paulo,
I was trying to look into your last comment. I am a bit confused as to exactly what i am looking for or how to look for it. Here is some information that I have gathered. By the way i am running on windows XP and using Apache 2.2.
My apache is installed for all users as in regedit the variable ServerRoot is under HKEY_LOCAL_MACHINE (http://httpd.apache.org/docs/2.2/platform/windows.html). Also I believe SAS is installed under all users. I tested this by having my coworker sign in using her login and I still had access. I’m not sure if that is a sufficient test though.
The log I get when I run the wsgi is the following. I’m not sure if it matters that the process is empty.
[Mon Aug 20 10:33:17 2012] [info] [client 10.60.8.71] mod_wsgi (pid=5980, process='', application='..com|/test5'): Reloading WSGI script 'C:/Sites/cprm/pyscripts/test5.wsgi'.
Also I tried the .bat trick from the link I posted in the comment i posted earlier to no avail. I made a simple batch file that just echoes 'hi' and placed it in the same directory where my wsgi scripts live. I feel like there should be no access problems there, but I may be mistaken. I also just tried calling a simple python script using subprocess just to test. Also nothing happened.
Also just to show you, my httpd.conf file looks like such:
AllowOverride None
Options None
Order allow,deny
Allow from all
WSGIScriptAlias /test1 "C:/sites/cprm/pyscripts/test1.wsgi"
WSGIScriptAlias /test2 "C:/sites/cprm/pyscripts/test2.py"
WSGIScriptAlias /test3 C:/sites/cprm/pyscripts/test3.py
WSGIScriptAlias /test4 "C:/sites/cprm/pyscripts/test4.py"
WSGIScriptAlias /test5 "C:/sites/cprm/pyscripts/test5.wsgi"
WSGIScriptAlias / "C:/sites/cprm/wsgi.py"
Is this information helpful or not really? Also, am i looking for a specific environ variable or something?
Thanks again

For web applications that perform background calculations or other tasks, IMHO it is best to queue the tasks for processing instead of calling an external process from a Django view and hang everything until the task completes. This leads to better:
user experience (request returns instantly - use ajax to signal task status and present the download link once task completes)
security (background process can run under safer credentials)
scalability (tasks can be distributed among servers)
resilience (by default many webservers will send an 'error 500' if your application fails to answer under 30 seconds or so)
For a background daemon processing all entries in the queue, there are several approaches depending on how big you want to scale:
a cron job
a daemon using supervisor (or your watchdog of choice)
an AMQP module like django-celery
[edit]
The process you start from a WSGI script will run under the same user that is running the webserver. In linux it is generally 'www-data' or 'nobody', in Windows/IIS it is 'IUSR_MachineName' (or authenticated user if using IIS authentication). Check if you can start the program using the same credentials your WSGI is running under.

Processing chunked encoded HTTP POST requests in python (or generic CGI under apache)

I have a j2me client that would post some chunked encoded data to a webserver. I'd like to process the data in python. The script is being run as a CGI one, but apparently apache will refuse a chunked encoded post request to a CGI script. As far as I could see mod_python, WSGI and FastCGI are no go too.
I'd like to know if there is a way to have a python script process this kind of input. I'm open to any suggestion (e.g. a confoguration setting in apache2 that would assemble the chunks, a standalone python server that would do the same, etc.) I did quite a bit of googling and didn't find anything usable, which is quite strange.
I know that resorting to java on the server side would be a solution, but I just can't imagine that this can't be solved with apache + python.

I had the exact same problem a year ago with a J2ME client talking to a Python/Ruby backend. The only solution I found which doesn't require application or infrastructure level changes was to use a relatively unknown feature of mod_proxy.
Mod_proxy has the ability to buffer incoming (chunked) requests, and then rewrite them as a single request with a Content-Length header before passing them on to a proxy backend. The neat trick is that you can create a tiny proxy configuration which passes the request back to the same Apache server. i.e. Take an incoming chunked request on port 80, "dechunk" it, and then pass it on to your non-HTTP 1.1 compliant server on port 81.
I used this configuration in production for a little over a year with no problems. It looks a little something like this:
ProxyRequests Off
<Proxy http://example.com:81>
Order deny,allow
Allow from all
</Proxy>
<VirtualHost *:80>
SetEnv proxy-sendcl 1
ProxyPass / http://example.com:81/
ProxyPassReverse / http://example.com:81/
ProxyPreserveHost On
ProxyVia Full
<Directory proxy:*>
Order deny,allow
Allow from all
</Directory>
</VirtualHost>
Listen 81
<VirtualHost *:81>
ServerName example.com
# Your Python application configuration goes here
</VirtualHost>
I've also got a full writeup of the problem and my solution detailed on my blog.

I'd say use the twisted framework for building your http listener.
Twisted supports chunked encoding.
http://python.net/crew/mwh/apidocs/twisted.web.http._ChunkedTransferEncoding.html
Hope this helps.

Apache 2.2 mod_cgi works fine for me, Apache transparently unchunks the request as it is passed to the CGI application.
WSGI currently disallows chunked requests, and mod_wsgi does indeed block them with a 411 response. It's on the drawing board for WSGI 2.0. But congratulations on finding something that does chunk requests, I've never seen one before!

You can't do what you want with mod_python. You can do it with mod_wsgi if you are using version 3.0. You do however have to step outside of the WSGI 1.0 specification as WSGI effectively prohibits chunked request content.
Search for WSGIChunkedRequest in http://code.google.com/p/modwsgi/wiki/ChangesInVersion0300 for what is required.

Maybe it is a configuration issue? Django can be fronted with Apache by mod_python, WSGI and FastCGI and it can accept file uploads.