Is there a way to log waitress-serve output into a file?
The current command I use is:
waitress-serve --listen=localhost:8080 --threads=1 my_app_api:app
The application we used was not written with waitress in mind earlier, so we choose to serve it with command line to avoid change (for now at least).
TLDR waitress-serve doesn't provide a way to do it. See the 'how do i get it to log' section.
Background
Per the documentation for the command-line usage of waitress-serve, no - there's no way to setup logging. See arguments docs.
waitress-serve is just an executable to make running your server more convenient. It's source-code is here runner.py. If you read it, you can see it actually is basically just calling from waitress import serve; serve(**args) for you. (That code clip is not literally what it's doing, but in spirit yes).
The documentation for waitress says that it doesn't log http traffic. That's not it's job. But it will log it's own errors or stacktraces. logging docs. If you read the waitress source trying to find when it logs stuff, you'll notice it doesn't seem to log http traffic anywhere github log search. It primarily logs stuff to do with the socket layer.
Waitress does say that if you want to log http traffic, then you need another component. In particular, it points you to pastedeploy docs which is some middle-ware that can log http traffic for you.
The documentation from waitress is actually kind of helpful answering you question, though not direct and explicit. It says
The WSGI design is modular.
per the logging doc
I.e. waitress won't log http traffic for you. You'll need another WSGI component to do that, and because WSGI is modular, you can probably choose a few things.
If you want some background on how this works, there's a pretty good post here leftasexercise.com
OK, how do I get it to log?
Use tee
Basically, if you just want to log the same stuff that is output from waitress-serve then you don't need anything special.
waitress-serve --listen=localhost:8080 --threads=1 my_app_api:app | tee -a waitress-serve.log
Python logging
But if you're actually looking for logging coming from python's standard logger (say you app is making logger calls or you want to log http traffic) then, you can set that up in your python application code. E.g. edit your applications soure-code and get it to setup logging to a file
import logging
logging.basicConfig(filename='app.log', encoding='utf-8', level=logging.DEBUG)
PasteDeploy middleware for http logs
Or if your looking for apache type http logging then you can use something like PasteDeploy to do it. Note, PasteDeploy is another python dependency so you'll need to install it. E.g.
pip install PasteDeploy
Then you need to setup a .ini file that tells PasteDeploy how to start your server and then also tell it to use TransLogger to create apache type http logs. This is explained more detail here logging with pastedeploy The ini file is specific to each app, but from your question is sounds like the ini file should look like:
[app:wsgiapp]
use = my_app_api:app
[server:main]
use = egg:waitress#main
host = 127.0.0.1
port = 8080
[filter:translogger]
use = egg:Paste#translogger
setup_console_handler = False
[pipeline:main]
pipeline = translogger
app
You'll still need to edit the source-code of your app to get PasteDeploy to load the app with your configuration file:
from paste.deploy import loadapp
wsgi_app = loadapp('config:/path/to/config.ini')
Webframework-dependent roll-your-own http logging
Even if you want to log http traffic, you don't necessarily need something like PasteDeploy. For example, if you are using flask as the web-framework, you can write your own http logs using after_request decorator:
#app.after_request
def after_request(response):
timestamp = strftime('[%Y-%b-%d %H:%M]')
logger.error('%s %s %s %s %s %s', timestamp, request.remote_addr, request.method, request.scheme, request.full_path, response.status)
return response
See the full gist at https://gist.github.com/alexaleluia12/e40f1dfa4ce598c2e958611f67d28966
Related
I'm building a website using Flask and I'm now in the process of adding some logging to it for which I found these docs. The basic example is as follows:
if not app.debug:
import logging
from themodule import TheHandlerYouWant
file_handler = TheHandlerYouWant(...)
file_handler.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)
after which you can log using app.logger.error('An error occurred'). This works fine, but apart from the fact that I do not see any advantage over the regular python logging module I also see a major downside: if I want to log outside of a request context (when for example running some code with a cron job) I get errors because I'm using app outside of the request context.
So my main question; why would I use the Flask logger at all? What is the reason that it was ever built?
The Flask logger uses the "generic" Python logging, it's a logging.getLogger(name) like any other.
The logger exists so that Flask app and views can log things that happen during execution. For example, it will log tracebacks on 500 errors during debug mode. The configuration example is there to show how to enable these logs, which are still useful in production, when you are not in debug mode.
Having an internal logger is not unique to Flask, it's the standard way to use logging in Python. Every module defines it's own logger, but the configuration of the logging is only handled by the last link in the chain: your project.
You can also use app.logger for your own messages, although it's not required. You could also create a separate logger for your own messages. In the end, it's all Python logging.
I need to setup logging in a custom web app which ideally would match the magic which happens when running a web app in Google app engine
For example, in GAE there is a request_log which can be viewed. This groups all log statements together under each request and each request has the http status code together with the endpoint path of the url. Here is an example (I apologise in advance for the crude editing here)
In a flask application I are deploying to Google Kubernetes Engine I would like to get the same level of logging in place. Trouble is I just do not know where to start.
I have got as far as installing the google-cloud-logging python library and have some rudimentary logging in place like this....
..but this is no where near the level I would like.
So the question is - where do I start?? Any searches / docs I have found so far have come up short.
Structured Logging
In Stackdriver Logging, structured logs refer to log entries that use the jsonPayload field to add structure to their payloads. If you use the Stackdriver Logging API or the command-line utility, gcloud logging, you can control the structure of your payloads. Here's an example of what a jsonPayload would look like:
{
insertId: "1m9mtk4g3mwilhp"
jsonPayload: {
[handler]: "/"
[method]: "GET"
[message]: "200 OK"
}
labels: {
compute.googleapis.com/resource_name: "add-structured-log-resource"
}
logName: "projects/my-sample-project-12345/logs/structured-log"
receiveTimestamp: "2018-03-21T01:53:41.118200931Z"
resource: {
labels: {
instance_id: "5351724540900470204"
project_id: "my-sample-project-12345"
zone: "us-central1-c"
}
type: "gce_instance"
}
timestamp: "2018-03-21T01:53:39.071920609Z"
}
You can set your own customizable jsonPayload with the parameters and values that you would like to obtain and then write this information to Stackdriver Logs Viewer.
Setting Debug mode to True
When setting debug=True, you will be able see your app in debugging mode. You will be able to see the HTTP requests, as they will appear on your console for debugging purposes, which you could then write these requests to Stackdriver Logs Viewer. An example of a Hello world Flask app running in Debug mode.
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
if __name__ == "__main__":
app.run(port='5000', debug=True)
Which you could add a Flask logging handler as follows:
import logging
from logging.handlers import RotatingFileHandler
from flask import Flask
app = Flask(__name__)
#app.route('/')
def foo():
app.logger.warning('A warning occurred (%d apples)', 42)
app.logger.error('An error occurred')
app.logger.info('Info')
return "foo"
if __name__ == '__main__':
handler = RotatingFileHandler('foo.log', maxBytes=10000, backupCount=1)
handler.setLevel(logging.INFO)
app.logger.addHandler(handler)
app.run()
As you can see, there are ways to achieve this, by following the proper log configuration; although, the Stackdriver Logs Viewer UI will not look the same for Kubernetes logs as in App Engine Stackdriver Logs Viewer.
Additionally, you could also take a look into Combining correlated log lines in Google Stackdriver since it will give you a better idea of how to batch your logs by categories or groups in case you need to do so.
Click on "View options" at top right corner in the logs panel > "Modify Custom fields"
https://cloud.google.com/logging/docs/view/overview#custom-fields
I am writing this here letting people know what I have come up with during my investigations.
The information supplied by sllopis got me to to the closest solution - using a mixture of structured logging and refactoring some of the code in the flask-gcp-log-groups library I am able to get requests logged in Stackdriver with log lines correlated underneath
Unfortunately this solution has a few gaping holes making it far from ideal albeit it is the best I can come up with so far based on Stackdrivers rigidness.
Each time I drill into a request there is a "flash" as Stackdriver searches and grabs all the trace entries matching that request. The bigger the collection of entries, the longer the flash takes to complete.
I cannot search for text within the correlated lines when only looking at the "request" log. For example, say a correlated log entry underneath a request has a string with the text "now you see me" - if I search for the string "see" it will not bring up that request in the list of search results.
I may be missing something obvious but I have spent several very frustrating days trying to achieve something which you think should be quite simple.
Ideally I would create a protoPayload per log entry, within I would put an array under the property "line" similar to how Google App Engine does its logging.
However there does not appear to be a way of doing this as protoPayload is reserved for Audit Logs.
Thanks to sllopis for the information supplied - if I don't find a better solution soon I will mark the answer as correct as it is the closest I believe I will get to what I want to achieve.
Given the situation I am very tempted to ditch Stackdriver in favour of a better logging solution - any suggestions welcome!
I'm building a website using Flask and I'm now in the process of adding some logging to it for which I found these docs. The basic example is as follows:
if not app.debug:
import logging
from themodule import TheHandlerYouWant
file_handler = TheHandlerYouWant(...)
file_handler.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)
after which you can log using app.logger.error('An error occurred'). This works fine, but apart from the fact that I do not see any advantage over the regular python logging module I also see a major downside: if I want to log outside of a request context (when for example running some code with a cron job) I get errors because I'm using app outside of the request context.
So my main question; why would I use the Flask logger at all? What is the reason that it was ever built?
The Flask logger uses the "generic" Python logging, it's a logging.getLogger(name) like any other.
The logger exists so that Flask app and views can log things that happen during execution. For example, it will log tracebacks on 500 errors during debug mode. The configuration example is there to show how to enable these logs, which are still useful in production, when you are not in debug mode.
Having an internal logger is not unique to Flask, it's the standard way to use logging in Python. Every module defines it's own logger, but the configuration of the logging is only handled by the last link in the chain: your project.
You can also use app.logger for your own messages, although it's not required. You could also create a separate logger for your own messages. In the end, it's all Python logging.
I'm running my app on the GAE development server, with app-engine-patch to run Django.
One of my views is bugged , so I want to log everything that happens.
I added in myapp.views:
import logging
LOG_FILENAME = '/mylog.txt'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG)
and my function is:
def function(string):
logging.debug('new call')
#do stuff
#logging.debug('log stuff')
My problem is that I can't find the log. When I run my app I get no errors, but the log is not created.
I also tried various paths: /mylog.txt ,mylog.txt , c:\mylog.txt, c:\complete\path\to \my\app\mylog.txt, but it doesn't work.
On the other hand I tried to create and run a separate test:
#test.py
import logging
LOG_FILENAME = '/mylog.txt'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG)
logging.debug('test')
And the log is created without problems: c:\mylog.txt
I'm not familiar with logging so I don't know if there might be some issues with django, or appengine.
Thanks
You can't write to files on App Engine - thus, any attempt to log to text files is also doomed to failure. Log output will appear on the SDK console in development, or in the logs console in production.
I am guessing the problem is that you put your log configuration in a view. A general rule of thumb for django logging is to set the log configuration in your settings.py.
By default, dev_appserver suppresses the debug log. To turn it on, run it with the --debug option. Note that dev_appserver itself uses the same logger, and will spew lots of debugging stuff you might not care about.
Your logging will print to stderr.
paster serve --reload development.ini
..for debug = true
THis is what I do to load a development server for Pylons.
However, when I do:
print "hello world"
THis message doesn't print out in the console. In Django, it does.
In Pylons logging package is the method to perform logging:
import logging
log = logging.getLogger(__name__)
log.debug('hello world')
This will work as long as you have logging setup configured correctly in your development.ini. I think the code above should be sufficient without any modifications to default configuration. In case it isn't you can call log.info, log.warn, log.error or log.critical instead of log.debug for your message to pass through.
I highly recommend reading this chapter of Pylons Book.