I want to print basic error messages from a script. They should have the name of the script (i.e. sys.argv[0]) then the message, and should be printed to stderr. Is there an easy way to do this using the standard library?
I've written the below (inspired by this related answer) and put it in its own module so I can import it in my scripts, but I feel like there's a better way.
from __future__ import print_function
import sys
def print_error(*args, **kwargs):
"""Print script name and anything else to stderr."""
print_stderr(sys.argv[0] + ':', *args, file=sys.stderr, **kwargs)
The easiest way is to use logging. First set its format to '%(pathname)s: %(message)s', then use logging.error() to print. For example:
# Setup
import logging
logging.basicConfig(format='%(pathname)s: %(message)s')
# Print error
logging.error('Test error message')
Usage:
$ python3 test.py
test.py: Test error message
$ python3 ~/test.py
/home/wja/test.py: Test error message
See LogRecord attributes for more info about pathname and other available info.
Related
I would like to use rich.logging.RichHandler from the rich library to handle all captured logs in pytest.
Say I have two files,
# library.py
import logging
logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())
def func():
x = {"value": 5}
logger.info(x)
# test_library.py
from library import func
def test_func():
func()
assert False
Running pytest shows the log message as expected, but I want it formatted by rich so I tried to put the following into conftest.py:
import logging
import pytest
from rich.logging import RichHandler
#pytest.hookimpl
def pytest_configure(config: pytest.Config):
logger = logging.getLogger()
logger.addHandler(RichHandler())
which results in the following output:
Under "captured stdout call" the log message appears as formatted by the RichHandler but below that it appears a second time under "captured log call" which is not what I want. Instead the message below "captured log call" should be formatted by the RichHandler and should not appear twice.
It does not perfectly answer your question however it could help getting a closest result.
Disable live logs: log_cli = 0 in pytest.ini
Disable capture: -s flag equivalent to --capture=no
Disable showing captured logs: --show-capture=no (should be disabled when capture is turned off, but ...)
So by disabling live logs and by running pytest -s --show-capture=no you should get rid of the duplicated logs and get only rich-formatted logs.
I have the impression (but do not find the documentation for it) that unittest sets the logging level to WARNING for all loggers. I would like to:
be able to specify the logging level for all loggers, from the command line (when running the tests) or from the test module itself
avoid unittest messing around with the application logging level: when running the tests I want to have the same logging output (same levels) as when running the application
How can I achieve this?
I don't believe unittest itself does anything to logging, unless you use a _CapturingHandler class which it defines. This simple program demonstrates:
import logging
import unittest
logger = logging.getLogger(__name__)
class MyTestCase(unittest.TestCase):
def test_something(self):
logger.debug('logged from test_something')
if __name__ == '__main__':
# DEBUG for demonstration purposes, but you could set the level from
# cmdline args to whatever you like
logging.basicConfig(level=logging.DEBUG, format='%(name)s %(levelname)s %(message)s')
unittest.main()
When run, it prints
__main__ DEBUG logged from test_something
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
showing that it is logging events at DEBUG level, as expected. So the problem is likely to be related to something else, e.g. the code under test, or some other test runner which changes the logging configuration or redirects sys.stdout and sys.stderr. You will probably need to provide more information about your test environment, or better yet a minimal program that demonstrates the problem (as my example above shows that unittest by itself doesn't cause the problem you're describing).
See below example for logging in Python. Also you can change LOG_LEVEL using 'setLevel' method.
import os
import logging
logging.basicConfig()
logger = logging.getLogger(__name__)
# Change logging level here.
logger.setLevel(os.environ.get('LOG_LEVEL', logging.INFO))
logger.info('For INFO message')
logger.debug('For DEBUG message')
logger.warning('For WARNING message')
logger.error('For ERROR message')
logger.critical('For CRITICAL message')
This is in addition to #Vinay's answer above. It does not answer the original question. I wanted to include command line options for modifying the log level. The intent was to get detailed loggin only when I pass a certain parameter from the command line. This is how I solved it:
import sys
import unittest
import logging
from histogram import Histogram
class TestHistogram(unittest.TestCase):
def test_case2(self):
h = Histogram([2,1,2])
self.assertEqual(h.calculateMaxAreaON(), 3)
if __name__ == '__main__':
argv = len(sys.argv) > 1 and sys.argv[1]
loglevel = logging.INFO if argv == '-v' else logging.WARNING
logging.basicConfig(level=loglevel)
unittest.main()
The intent is to get more verbose logging. I know it does not answer the question, but I'll leave it here in case someone comes looking for a similar requirement such as this.
this worked for me:
logging.basicConfig(level=logging.DEBUG)
And if I wanted a specific format:
logging.basicConfig(
level=logging.DEBUG,
datefmt="%H:%M:%S",
format="%(asctime)s.%(msecs)03d [%(levelname)-5s] %(message)s",
)
Programmatically:
Put this line of code in each test function defined in your class that you want to set the logging level:
logging.getLogger().setLevel(logging.INFO)
Ex. class:
import unittest
import logging
class ExampleTest(unittest.TestCase):
def test_method(self):
logging.getLogger().setLevel(logging.INFO)
...
Command Line:
This example just shows how to do it in a normal script, not specific to unittest example. Capturing the log level via command line, using argparse for arguments:
import logging
import argparse
...
def parse_args():
parser = argparse.ArgumentParser(description='...')
parser.add_argument('-v', '--verbose', help='enable verbose logging', action='store_const', dest="loglevel", const=logging.INFO, default=logging.WARNING)
...
def main():
args = parse_args()
logging.getLogger().setLevel(args.loglevel)
I want to ensure that a certain condition in my code causes a log message to be written to the django log. How would I do this with the Django unit testing framework?
Is there a place where I can check logged messages, similarly to how I can check sent emails? My unit test extends django.test.TestCase.
Using the mock module for mocking the logging module or the logger object. When you've done that, check the arguments with which the logging function is called.
For example, if you code looks like this:
import logging
logger = logging.getLogger('my_logger')
logger.error("Your log message here")
it would look like:
from unittest.mock import patch # For python 2.x use from mock import patch
#patch('this.is.my.module.logger')
def test_check_logging_message(self, mock_logger):
mock_logger.error.assert_called_with("Your log message here")
You can also use assertLogs from django.test.TestCase
When you code is
import logging
logger = logging.getLogger('my_logger')
def code_that_throws_error_log():
logger.error("Your log message here")
This is the test code.
with self.assertLogs(logger='my_logger', level='ERROR') as cm:
code_that_throws_error_log()
self.assertIn(
"ERROR:your.module:Your log message here",
cm.output
)
This lets you avoid patching just for logs.
The common way of mocking out the logger object (see the splendid chap Simeon Visser's answer) is slightly tricky in that it requires the test to mock out the logging in all the places it's done. This is awkward if the logging comes from more than one module, or is in code you don't own. If the module the logging comes from changes name, it will break your tests.
The splendid 'testfixtures' package includes tools to add a logging handler which captures all generated log messages, no matter where they come from. The captured messages can later be interrogated by the test. In its simplest form:
Assuming code-under-test, which logs:
import logging
logger = logging.getLogger()
logger.info('a message')
logger.error('an error')
A test for this would be:
from testfixtures import LogCapture
with LogCapture() as l:
call_code_under_test()
l.check(
('root', 'INFO', 'a message'),
('root', 'ERROR', 'an error'),
)
The word 'root' indicates the logging was sent via a logger created using logging.getLogger() (i.e. with no args.) If you pass an arg to getLogger (__name__ is conventional), that arg will be used in place of 'root'.
The test does not care what module created the logging. It could be a sub-module called by our code-under-test, including 3rd party code.
The test asserts about the actual log message that was generated, as opposed to the technique of mocking, which asserts about the args that were passed. These will differ if the logging.info call uses '%s' format strings with additional arguments that you don't expand yourself (e.g. use logging.info('total=%s', len(items)) instead of logging.info('total=%s' % len(items)), which you should. It's no extra work, and allows hypothetical future logging aggregation services such as 'Sentry' to work properly - they can see that "total=12" and "total=43" are two instances of the same log message. That's the reason why pylint warns about the latter form of logging.info call.)
LogCapture includes facilities for log filtering and the like. Its parent 'testfixtures' package, written by Chris Withers, another splendid chap, includes many other useful testing tools. Documentation is here: http://pythonhosted.org/testfixtures/logging.html
Django has a nice context manager function called patch_logger.
from django.test.utils import patch_logger
then in your test case:
with patch_logger('logger_name', 'error') as cm:
self.assertIn("Error message", cm)
where:
logger_name is the logger name (duh)
error is the log level
cm is the list of all log messages
More details:
https://github.com/django/django/blob/2.1/django/test/utils.py#L638
It should work the same for django < 2.0, independently of python version (as long as it's supported by dj)
If you're using test classes, you can use following solution:
import logger
from django.test import TestCase
class MyTest(TestCase):
#classmethod
def setUpClass(cls):
super(MyTest, cls).setUpClass()
cls.logging_error = logging.error
logging.error = cls._error_log
#classmethod
def tearDownClass(cls):
super(MyTest, cls).tearDownClass()
logging.error = cls.logging_error
#classmethod
def _error_log(cls, msg):
cls.logger = msg
def test_logger(self):
self.assertIn('Message', self.logger)
This method replaces error function of logging module with your custom method only for test purposes and put stdout into cls.logger variable which is available in every test case by calling self.logger. At the end it reverts changes by placing error function from logging module back.
I want to have in my application a common logging module that logs to a file.
For example in my commonlog.py I can have something like this:
# Python logging module
import logging
logging.basicConfig(filename="test.log", level=logging.DEBUG)
From the other modules in the application I want to import this module and be able to use it like if it was the Python logging module but without replicating all its functions, for example from module test.py:
import commonlog
commonlog.debug("debug message")
commonlog.info("info message")
commonlog.ANY_OTHER_METHOD_THAT_BELONGS_TO_LOGGING()
How can I "proxy" in my commonlog all the methods from the logging module ?
Doing:
commonlogging.logging.etc..
is not a valid solution because it's using the logging module directly.
I've never had to "inherit" from a module before so I don't know it's naive to do a from logging import * at the top of commonlogging. Here's code showing that it appears to work:
>>> with open('mylogging.py', 'w') as f:
... f.write('''from logging import *
... my_customization = "it works"''')
...
>>> import mylogging
>>> print mylogging.my_customization
it works
>>> help(mylogging.log)
Help on function log in module logging:
log(level, msg, *args, **kwargs)
Log 'msg % args' with the integer severity 'level' on the root logger.
trying to understand and learn how to write packages... testing with something i've always used, logging...
Can you please help me understand why the 'log' variable is not working... and no logging is working on the screen?
Thanks!
main.py :
#!/opt/local/bin/python
import sys
sys.path.append('CLUSTER')
import clusterlogging.differentlogging
clusterlogging.differentlogging.consolelogging()
log.debug("Successfully logged in")
differentlogging.py
#!/opt/local/bin/python
def consolelogging():
import logging
class NullHandler(logging.Handler):
def emit(self, record):
pass
print "Console Logging loaded"
DEFAULTLOGLEVEL=logging.INFO
log = logging.getLogger(__name__)
log.addHandler(NullHandler())
log.debug("Successfully logged in")
def mysqllogging():
print "mysql logging module here"
def sysloglogging():
print "rsyslog logging module here"
output
Console Logging loaded
Traceback (most recent call last):
File "./svnprod.py", line 10, in <module>
log.debug("Successfully logged in")
NameError: name 'log' is not defined
log is a global variable in the differentlogging module. Thus you can access it as
clusterlogging.differentlogging.log.
You could also do something like from clusterlogging.differentlogging import log and then access it as just log.
Edit: actually, on reviewing your code again I don't know what to make of it. Could you please fix up your code indentation so that it makes sense? Are you defining log inside the consolelogging function? If so, you'll need to either make it global with global log or return it from the function and assign it to a variable log on the line where you call the function.
This will return the log array, and you will be able to use the logging function associated.
main.py:
#!/usr/bin/env python
import sys
sys.path.append('CLUSTER')
import clusterlogging.differentlogging
log=clusterlogging.differentlogging.ttylogging()
log.debug("Logging module loaded")
log.info ("It worked")
differentlogging.py :
#!/usr/bin/env python
def ttylogging():
print "Console Logging loaded"
import sys
import logging
class NullHandler(logging.Handler):
def emit(self, record):
pass
DEFAULTLOGLEVEL=logging.INFO
log = logging.getLogger(__name__)
log.addHandler(NullHandler())
log.setLevel(DEFAULTLOGLEVEL)
logStreamHandler = logging.StreamHandler(sys.stdout)
logStreamHandler.setFormatter(logging.Formatter("%(asctime)s %(levelname)5s %(name)s %(lineno)d: %(message)s"))
log.addHandler(logStreamHandler)
return log
def mysqllogging():
print "mysql logging module here"
def sysloglogging():
print "rsyslog logging module here"
Your main.py doesn't do anything to define the name log in the global namespace. Importing a module can define names in the namespace of that module, but can't put anything in the global namespace.
In your main.py you should add this statement:
from clusterlogging.differentlogging import log
By the way, I that is such a long module name, I would use import as:
import clusterlogging.differentlogging as difflogging
log = difflogging.log
EDIT: I originally recommended this but it won't work:
from difflogging import log # doesn't work
You might even want to use a really short name like dl:
import clusterlogging.differentlogging as dl
dl.log('whatever')
Since dl is really short, maybe you don't need to get log bound in the global namespace.
Also, you could get every name from a module by using import * but this is not recommended.
from clusterlogging.differentlogging import * # not recommended
You usually don't want to clutter the global namespace with all the stuff defined in a module. Import just what you need. This is tidier and helps document what you are actually using.