I'm making a project and using a library from the requirements of the project. The library implements logging and automatically logs to a file, when I call its functions.
Now, I'm implementing logging by myself, and only want my own messages to be logged to the file and not the library messages.
One solution I thought of would be switching the logging file each time I call a function from the library and then removing that file, but that seems overly complicated and clutterly. What can I do in order to not log the messages from the library?
P.S.:
I'm using the logging library and I initalize it as:
logging.basicConfig(level = logging.INFO,filename = loggingFile,format = "%(message)s")
, which means, all messages from myself and from the library will get logged in loggingFile
Libraries should not directly output anything to logs - that should be done only by handlers configured by an application. A library that logs output is an anti-pattern - if a library definitely does that, I'd log a bug against that library's issue tracker.
On the other hand, it might be that the library is only outputting stuff because you have configured output via your basicConfig() call. If you need more than basic configuration, don't usebasicConfig() - use the other APIs provided.
Related
In a nutshell, I write ETL pipelines. They are usually described in high-level scripts. In them, I use different internal libraries (we manage them) that provide utility functions, tooling or internal data structure.
What are the common best practices about logging when dealing with multiple packages import from different repositories?
My questions are:
1) Should I put logs in libraries? Or only in top-level scripts?
On one hand, It could be useful to display some information in some library functions/classes. On the other hand, it imposes the library client usage of a particular logger.
I checked a few open-source projects and it seems that there are no logs at all.
2) If we indeed put logs in all shared libraries, what is the best practices in Python to pass a unique logger to everything?
I want my logging format and strategy to be consistent in each library call as everything is run as part "as a whole". Should I init my logger in the main script and pass the same logger in every object I create? It seems redundant to me. I saw another pattern where all classes that need logging would inherit from a logging class. It seems to me that it might overkill and complicates the overall architecture.
I read in another stackoverflow that Actually every logger is a child of the parent's package logger. How to apply that when the packages come different repositories?
thanks
Add a logger with no handlers (or with just the null handler) to the library and do all the internal logging with that. Give it a name that is related to the library. When you do that any app that uses the lib can get the logger and add a handler to access the logs as needed.
An example would be the requests library which does something similar to that.
import logging
import requests
r = logging.getLogger('requests')
r.addHandler(logging.StreamHandler())
r.setLevel(logging.DEBUG)
requests.get('http://stackoverflow.com')
will print
Starting new HTTP connection (1): stackoverflow.com
http://stackoverflow.com:80 "GET / HTTP/1.1" 301 143
Starting new HTTPS connection (1): stackoverflow.com
https://stackoverflow.com:443 "GET / HTTP/1.1" 200 23886
I am developing a new library and I am completely new to the concept of logging.
I have added logging using Python's logging module for same. The logging I gave has a specific FileHandler set for debug level and StreamHandler set at Warning level. Python documentation about logging says libraries should have only Null Handlers.
Here is the documentation link https://docs.python.org/3/howto/logging.html#library-config
Will it be a problem if I still have an exclusive file and stream handlers in my library.
I am not able to understand why one should create logs in libraries if they cannot have their own customized handlers.
It would be very helpful if someone could clear my understanding gap about implementing logging in libraries.
A secondary question: How will an application developer who uses my library be able to access/enable the logs that I created in the library if I set Null handler?
to your first question - [from the python docs] -
"The application developer knows their target audience and what
handlers are most appropriate for their application: if you add
handlers ‘under the hood’, you might well interfere with their ability
to carry out unit tests and deliver logs which suit their
requirements." 1
as a user of your library, I may want to show logs from your_pkg.foo.baz,
but not your_pkg.foo module.
adding handlers from within your library may force me to do that (depending on the log level that was set to the loggers and handlers).
to your second question -
adding a Nullhandler allows a user to choose his custom logging needs by configuring new handlers through the logging.get_logger("your_pkg.foo.baz").add_handler(...).
to fully understand the logging mechanism (loggers, handlers, filters, and propagation)-
you could look here -
logging flow
I am a newbie using python and I wanted to ask for your help in showing me how can I print messages from Python into robot framework console.
There are several ways for a python function to send information to the robot logs or to the console. These are all documented in the Robot framework user's guide, in the section titled Logging information.
The cleanest way is to use the logging API, which gives specialized functions for various kinds of logging. For example, to send information to the console you would use logger.console(message).
Using the logging API
Here is a library file that uses this method:
# ExampleKeywords.py
from robot.api import logger
def write_to_console(s):
logger.console(s)
You can use this library in the following manner:
*** Settings ***
| Library | ExampleKeywords.py
*** Test Cases ***
| Use a custom keyword to write to the console
| | Write to console | Hello, world
This will appear in the console only, and will not show up in the logs. If you want the information to show up in the logs you can use logger methods info, warn, debug, or trace. To log an error you would simply throw an exception.
Calling built-in keywords
There are other ways for your custom keywords to send information to the logs. For example, you can get a reference to the BuiltIn library, and directly call the log or log to console keywords like this:
from robot.libraries.BuiltIn import BuiltIn
def write_to_console(s):
BuiltIn().log_to_console("Hello, world")
Using print statements
Finally, you can write information to the logs (but not only to the console) using print statements. You can prefix the string with *<level>* to affect the log level. For example, to print a warning you can do:
print "*WARN* Danger Will Robinson"
Summary
Using the API is arguably the best method to log information from your keywords. However, this is a fairly new API only available since Robot Framework 2.6, so if you are using an older version of Robot you may need to use one of the other techniques.
What it sounds like you're asking for is a way to write a library such that your testcase can print to the console during test execution.
This is pretty easy, you can find it in the RF Manual under Logging Information. The short version is you can log a "WARN" that will appear in both the log and on screen with a print statement, like so:
print "*WARN* This text will show up on the console."
This command can be used to print any message or content on robot console.
from robot.libraries.BuiltIn import BuiltIn
BuiltIn().log_to_console("Message that needs to be printed")
I'm struggling getting something going that should be pretty simple. I'm working on some uniittests that calls code with some very verbose debug logging. How can I tell the logging module "set everything to debug, except this one module, make it info"
disable query logging - Google Appengine SDK Python
check the log filter, which can be used as module based filter.
I developed a statistics system for online web service user behavior research in python, which mostly relies on reading and analyzing logs from production server. Currently I shared log folders internally under SMB protocol for the routine analytics program to read, but for the data accessing method I have 2 questions,
Are there any other way accessing logs other than via SMB? or other strategy?
I guess a lot read may block HD of the production and affect normal log writing, any solution to solve this?
I hoped I could come up with some real number but currently don't have. Any guy can give me some guide on doing this more gracefully?
If you are open to using a third party log aggregation tool, you have a couple of options:
http://graylog2.org/
http://www.logstash.net/
http://www.octopussy.pm/
https://github.com/facebook/scribe
In addition, if you are logging to syslog - many of the commonly used syslog daemons ( eg syslog-ng ) can be configured to forward logs from various applications to one or more of these aggregators. It is trivial to log to syslog from a python application - there is a syslog module in the standard library
Well, if you have a HTTP server in between (IHS, OHS, I guess Apache too...) then you can expose your physical repositories via a URL: each of your files will benefit from a URL too, and via this kind of code you can download them quite easily:
import os
import urllib2
# Open our local file for writing
f = urllib2.urlopen(url)
with open(os.path.basename(url), 'wb') as local_file:
local_file.write(f.read())