Will calls to logger block request processing in a Django views?
For instance, in the example below, will the logger.info() at line 2 block the response on line 3?
def random_view(request):
xyz.do_something()
logger.info("Does this block the next line execution?")
return HttpResponse("Done")
I've looked at the Django logging documentation and the python logging module docs but I can't find a concrete answer to this question.
I understand the the Handler decides where each log message goes, but I am not sure if the Handler itself is non-blocking. Would appreciate if someone could clarify or point to documentation explaining this.
From Django Docs
Django uses Python’s builtin logging module to perform system logging.
The usage of this module is discussed in detail in Python’s own
documentation.
From Python Docs
Sometimes you have to get your logging handlers to do their work
without blocking the thread you’re logging from. This is common in Web
applications, though of course it also occurs in other scenarios.
So, although not explicitly mentioned yet logging does seem to be blocking. For details see Python Docs.
Related
I have written a simple python package which has a set of functions that perform simple operations (data manipulation). I am trying to enhance the package and add more functionality for logging, which leads me to this question.
Should I expect the user of the package to pass in a file descriptor or a file handler of the python logging module into the methods of the package, or should the package itself have its own logging module which the methods within the package employ.
I can see benefits (user controls logging and can maintain a flow of function calls based on the same handler) and cons (users logger is not good enough) in both, however what is / are best practices in this case.
In your module, create a logger object:
import logging
LOGGER = logging.getLogger(__name__)
And then call the appropriate functions on that object:
LOGGER.debug('you probably dont care about this')
LOGGER.info('some info message')
LOGGER.error('whoops, something went wrong')
If the user has configured the logging subsystem correctly, then the messages will automatically go where the user wants them to go (a file, stderr, syslog, etc.)
I am trying to create a python script that well periodically connect to haproxy socket, send a "show info" command and exit. This script will be used along with keepalived in order to determine, through this script, when to switch master/slave servers. Therefore, my goal is that my python script been non-blocking.
Within my script, i have initiated a SysLogHandler to send to my local syslogd various messages (exceptions e.t.c). However, i don't know if SysLogHandler is Blocking or not.
The python documentation on logging is the main point where you should start.
SysLogHandler is a subclass of Handler(There are many more. See the docs).
This is the relevant part that you are looking for I think.
I went a bit further and checked the actual implementation of Handler.
You can see that it uses threading.RLock().
In the app I'm currently working on (2.7 runtime), I'm trying to make sure that exceptions at the API level (i.e. not my code) are handled correctly within my application. However, it appears that Google/AppEngine handle those exceptions internally and doesn't bubble them up. For instance, using Thing which is a previously defined ndb.Model
t = Thing(id=1,name='thingy')
try:
t.put()
except Exception as e:
self.log(e)
self.abort(500)
In the unlikely event that something goes awry with the put() I have no way to catch/log that event -- or is there?
A similar thing happens with storing data to the blobstore where exceptions are, apparently, caught and raised internally and leaving no chance for me to log those.
Perhaps I'm missing a key point? I've looked through the API docs but the exceptions raised by services and how to catch them doesn't seem to be a priority for documentation team.
Actually App Engine logs every single request. Just go to the application's dashboard and click on Logs.
If you want to log something on your own you should use the logging library and you can read more about it in the documentation.
So instead of self.log you should use logging.error.
Using Google App Engine, Python 2.7, threadsafe:true, webapp2.
I would like to include all logging.XXX() messages in my API responses, so I need an efficient way to collect up all the log messages that occur during the scope of a request. I also want to operate in threadsafe:true, so I need to be careful to get only the right log messages.
Currently, my strategy is to add a logging.Handler at the start of my webapp2 dispatch method, and then remove it at the end. To collect logs only for my thread, I instantiate the logging.Handler with the name of the current thread; the handler will simply throw out log records that are from a different thread. I am using thread name and not thread ID because I was getting some unexpected results on dev_appserver when using the ID.
Questions:
Is it efficient to constantly be adding/removing logging.Handler objects in this fashion? I.e., every request will add, then remove, a Handler. Is this "cheap"?
Is this the best way to get only the logging messages for my request? My big assumption is that each request gets its own thread, and that thread name will actually select the right items.
Am I fundamentally misunderstanding Python logging? Perhaps I should only have a single additional Handler added once at the "module-level" statically, and my dispatch should do something lighter.
Any advice is appreciated. I don't have a good understanding of what Python (and specifically App Engine Python) does under the hood with respect to logging. Obviously, this is eminently possible because the App Engine Log Viewer does exactly the same thing: it displays all the log messages for that request. In fact, if I could piggyback on that somehow, that would be even better. It absolutely needs to be super-cheap though - i.e., an RPC call is not going to cut it.
I can add some code if that will help.
I found lots of goodness here:
from google.appengine.api import logservice
entries = logservice.logs_buffer().parse_logs()
I'm working on an app that uses the standard logging module to do logging. We have a setup where we log to a bunch of files based on levels etc. We also use celery to run some jobs out of the main app (maintenance stuff usually that's time consuming).
The celery task does nothing other than call functions (lets say spam) which do the actual work. These functions use the logging module to output status messages. Now, I want to write a decorator that hijacks all the logging calls made by spam and puts them into a StringIO so that I can put them somewhere.
One of the solutions I had was to insert a handler for the root logger while the function is executing that grabs everything. However, this is messing with global shared objects which might be problematic later.
I came across this answer but it's not exactly what I'm looking for.
The thing about the StringIO is, there could be multiple processes running (Celery tasks), hence multiple StringIOs, right?
You can do something like this:
In the processes run under Celery, add to the root logger a handler which sends events to a socket (SocketHandler for TCP or DatagramHandler for UDP).
Create a socket receiver to receive and handle the events, as documented here. This acts like a consolidated StringIO across multiple Celery-run processes.
If you are using multiprocessing, you can also use the approach described here. Though that post talks about Python 3.2, the functionality is also available for Python 2.x using logutils.
Update: If you want to avoid a separate receiver process, you can log to a database directly, using a handler similar to that in this answer. If you want to buffer all the logging till the end of the process, you can use a MemoryHandler in conjunction with a database handler to achieve this.
For the StringIO handler, you could add an extra handler for the root logger that would grab everything, but at the same time add a dummy filter (Logger.addFilter) that filters everything out (so nothing is actually logged to StringIO).
You could then write a decorator for spam that removes the filter (Logger.removeFilter) before the function executes, and adds the dummy filter back after.