I am developing a new library and I am completely new to the concept of logging.
I have added logging using Python's logging module for same. The logging I gave has a specific FileHandler set for debug level and StreamHandler set at Warning level. Python documentation about logging says libraries should have only Null Handlers.
Here is the documentation link https://docs.python.org/3/howto/logging.html#library-config
Will it be a problem if I still have an exclusive file and stream handlers in my library.
I am not able to understand why one should create logs in libraries if they cannot have their own customized handlers.
It would be very helpful if someone could clear my understanding gap about implementing logging in libraries.
A secondary question: How will an application developer who uses my library be able to access/enable the logs that I created in the library if I set Null handler?
to your first question - [from the python docs] -
"The application developer knows their target audience and what
handlers are most appropriate for their application: if you add
handlers ‘under the hood’, you might well interfere with their ability
to carry out unit tests and deliver logs which suit their
requirements." 1
as a user of your library, I may want to show logs from your_pkg.foo.baz,
but not your_pkg.foo module.
adding handlers from within your library may force me to do that (depending on the log level that was set to the loggers and handlers).
to your second question -
adding a Nullhandler allows a user to choose his custom logging needs by configuring new handlers through the logging.get_logger("your_pkg.foo.baz").add_handler(...).
to fully understand the logging mechanism (loggers, handlers, filters, and propagation)-
you could look here -
logging flow
Related
I'm making a project and using a library from the requirements of the project. The library implements logging and automatically logs to a file, when I call its functions.
Now, I'm implementing logging by myself, and only want my own messages to be logged to the file and not the library messages.
One solution I thought of would be switching the logging file each time I call a function from the library and then removing that file, but that seems overly complicated and clutterly. What can I do in order to not log the messages from the library?
P.S.:
I'm using the logging library and I initalize it as:
logging.basicConfig(level = logging.INFO,filename = loggingFile,format = "%(message)s")
, which means, all messages from myself and from the library will get logged in loggingFile
Libraries should not directly output anything to logs - that should be done only by handlers configured by an application. A library that logs output is an anti-pattern - if a library definitely does that, I'd log a bug against that library's issue tracker.
On the other hand, it might be that the library is only outputting stuff because you have configured output via your basicConfig() call. If you need more than basic configuration, don't usebasicConfig() - use the other APIs provided.
In my Superset web application, I am interested in setting the level of logging in Flask-OAuthLib to DEBUG. We can see Flask-OAuthLib access its logger here on line 26
from a Superset web application.
Superset is a web application implemented using Flask-AppBuilder. It
allows for OAuth2 authentication via Flask-OAuthLib.
I want to configure Flask-OAuthLib logging from custom_sso_security_manager.py ... a module described in
the Superset docs on custom OAuth configuration.
You can access the logger in exactly the same way. They are added to a global dictionary that items can be get from with getLogger(key). So all you need is put something like this into your file after you imported the oauth lib:
oauth_logger = logging.getLogger('flask_oauthlib')
oauth_logger.setLevel(logging.DEBUG)
# it is custom for libs to have no handler (except the NullHandler)
# so you may want to add one:
oauth_logger.addHandler(logging.StreamHandler()) # just an example
I'm building a website using Flask and I'm now in the process of adding some logging to it for which I found these docs. The basic example is as follows:
if not app.debug:
import logging
from themodule import TheHandlerYouWant
file_handler = TheHandlerYouWant(...)
file_handler.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)
after which you can log using app.logger.error('An error occurred'). This works fine, but apart from the fact that I do not see any advantage over the regular python logging module I also see a major downside: if I want to log outside of a request context (when for example running some code with a cron job) I get errors because I'm using app outside of the request context.
So my main question; why would I use the Flask logger at all? What is the reason that it was ever built?
The Flask logger uses the "generic" Python logging, it's a logging.getLogger(name) like any other.
The logger exists so that Flask app and views can log things that happen during execution. For example, it will log tracebacks on 500 errors during debug mode. The configuration example is there to show how to enable these logs, which are still useful in production, when you are not in debug mode.
Having an internal logger is not unique to Flask, it's the standard way to use logging in Python. Every module defines it's own logger, but the configuration of the logging is only handled by the last link in the chain: your project.
You can also use app.logger for your own messages, although it's not required. You could also create a separate logger for your own messages. In the end, it's all Python logging.
In a nutshell, I write ETL pipelines. They are usually described in high-level scripts. In them, I use different internal libraries (we manage them) that provide utility functions, tooling or internal data structure.
What are the common best practices about logging when dealing with multiple packages import from different repositories?
My questions are:
1) Should I put logs in libraries? Or only in top-level scripts?
On one hand, It could be useful to display some information in some library functions/classes. On the other hand, it imposes the library client usage of a particular logger.
I checked a few open-source projects and it seems that there are no logs at all.
2) If we indeed put logs in all shared libraries, what is the best practices in Python to pass a unique logger to everything?
I want my logging format and strategy to be consistent in each library call as everything is run as part "as a whole". Should I init my logger in the main script and pass the same logger in every object I create? It seems redundant to me. I saw another pattern where all classes that need logging would inherit from a logging class. It seems to me that it might overkill and complicates the overall architecture.
I read in another stackoverflow that Actually every logger is a child of the parent's package logger. How to apply that when the packages come different repositories?
thanks
Add a logger with no handlers (or with just the null handler) to the library and do all the internal logging with that. Give it a name that is related to the library. When you do that any app that uses the lib can get the logger and add a handler to access the logs as needed.
An example would be the requests library which does something similar to that.
import logging
import requests
r = logging.getLogger('requests')
r.addHandler(logging.StreamHandler())
r.setLevel(logging.DEBUG)
requests.get('http://stackoverflow.com')
will print
Starting new HTTP connection (1): stackoverflow.com
http://stackoverflow.com:80 "GET / HTTP/1.1" 301 143
Starting new HTTPS connection (1): stackoverflow.com
https://stackoverflow.com:443 "GET / HTTP/1.1" 200 23886
I'm building a website using Flask and I'm now in the process of adding some logging to it for which I found these docs. The basic example is as follows:
if not app.debug:
import logging
from themodule import TheHandlerYouWant
file_handler = TheHandlerYouWant(...)
file_handler.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)
after which you can log using app.logger.error('An error occurred'). This works fine, but apart from the fact that I do not see any advantage over the regular python logging module I also see a major downside: if I want to log outside of a request context (when for example running some code with a cron job) I get errors because I'm using app outside of the request context.
So my main question; why would I use the Flask logger at all? What is the reason that it was ever built?
The Flask logger uses the "generic" Python logging, it's a logging.getLogger(name) like any other.
The logger exists so that Flask app and views can log things that happen during execution. For example, it will log tracebacks on 500 errors during debug mode. The configuration example is there to show how to enable these logs, which are still useful in production, when you are not in debug mode.
Having an internal logger is not unique to Flask, it's the standard way to use logging in Python. Every module defines it's own logger, but the configuration of the logging is only handled by the last link in the chain: your project.
You can also use app.logger for your own messages, although it's not required. You could also create a separate logger for your own messages. In the end, it's all Python logging.