We Keep Coding

Testing if warnings are sent as logmessages

I have set logging.captureWarnings(True) in an application, and would like to test if warnings are logged correctly. I'm having difficulties understanding some of the behavior I'm seeing where tests are influencing each other in ways that I don't quite get.
Here is an example test suite which reproduces the behavior I'm seeing:
test_warning_logs.py
import warnings
import logging
def test_a(caplog):
logging.captureWarnings(True)
logging.basicConfig()
warnings.warn("foo")
assert "foo" in caplog.text
def test_b(caplog):
logging.captureWarnings(True)
logging.basicConfig()
warnings.warn("foo")
assert "foo" in caplog.text
Both tests are identical. When run in isolation (pytest test_warning_logs.py -k test_a, pytest test_warning_logs.py -k test_b), they each pass. When both of them are executed in the same run (pytest test_warning_logs.py), only the first one will pass:
============== test session starts ========================
platform linux -- Python 3.10.2, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/me
plugins: mock-3.10.0, dependency-0.5.1
collected 2 items
test_warning_logs.py .F [100%]
==================== FAILURES =============================
_____________________ test_b ______________________________
caplog = <_pytest.logging.LogCaptureFixture object at 0x7f8041857c40>
def test_b(caplog):
logging.captureWarnings(True)
logging.basicConfig()
warnings.warn("foo")
> assert "foo" in caplog.text
E AssertionError: assert 'foo' in ''
E + where '' = <_pytest.logging.LogCaptureFixture object at 0x7f8041857c40>.text
[...]
Additional Information
First I thought that the commands logging.captureWarnings and logging.basicConfig aren't idempotent, and running them more than once is the issue. But if you remove them from test_b, it still fails.
My current assumption is that it's a pytest issue, because when the code is executed without it, both warnings are logged:
# add this block to the bottom of test_warning_logs.py
if __name__ == '__main__':
from unittest.mock import MagicMock
test_a(MagicMock(text="foo"))
test_b(MagicMock(text="foo"))
$ python test_warning_logs.py
WARNING:py.warnings:/home/me/test_warning_logs.py:9: UserWarning: foo
warnings.warn("foo")
WARNING:py.warnings:/home/me/test_warning_logs.py:17: UserWarning: foo
warnings.warn("foo")

If it is an option to log the warnings using the logging module instead of warnings, then you won't face this issue.
import logging
def test_a(caplog):
logging.captureWarnings(True)
logging.basicConfig()
logging.warning("foo")
assert "foo" in caplog.text
def test_b(caplog):
logging.captureWarnings(True)
logging.basicConfig()
# This will be deprecated eventually, you can use logging.warning() instead
logging.warn("foo")
assert "foo" in caplog.text
I don't know why it doesn't work using warnings module. According to their documentation:
Repetitions of a particular warning for the same source location are
typically suppressed.
So I assumed this is what is happening here, but even calling warnings.resetwarnings() does not change the behavior.

How to use custom log handler in pytest

I would like to use rich.logging.RichHandler from the rich library to handle all captured logs in pytest.
Say I have two files,
# library.py
import logging
logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())
def func():
x = {"value": 5}
logger.info(x)
# test_library.py
from library import func
def test_func():
func()
assert False
Running pytest shows the log message as expected, but I want it formatted by rich so I tried to put the following into conftest.py:
import logging
import pytest
from rich.logging import RichHandler
#pytest.hookimpl
def pytest_configure(config: pytest.Config):
logger = logging.getLogger()
logger.addHandler(RichHandler())
which results in the following output:
Under "captured stdout call" the log message appears as formatted by the RichHandler but below that it appears a second time under "captured log call" which is not what I want. Instead the message below "captured log call" should be formatted by the RichHandler and should not appear twice.

It does not perfectly answer your question however it could help getting a closest result.
Disable live logs: log_cli = 0 in pytest.ini
Disable capture: -s flag equivalent to --capture=no
Disable showing captured logs: --show-capture=no (should be disabled when capture is turned off, but ...)
So by disabling live logs and by running pytest -s --show-capture=no you should get rid of the duplicated logs and get only rich-formatted logs.

Adding system or utility live logs for pytest

I'm building a pytest fixture that starts some long process before any tests are executed.
I would like to have utility logs reporting about that process while it's taking place after being initiated by pytest.
The fixture looks something like this:
import logging
logger = logging.getLogger(__name__)
#fixture
def some_long_process():
logger.info("Process started")
logger.info("Process ongoing..")
yield
logger.info("Process ended")
The problem is that pytest automatically captures all outputs during test runs and although you can enable live logging I wish I could enable live logging specifically for my logger.
Is there a way to do that? Or alternatively, a way to add system logs to pytest?

Basically you just need to enable live logs.
It can be done directly at the command line by passing the log_cli_level .
pytest --log-cli-level=INFO test_log_feature.py
# ----- live log setup -------------------------------------------
# INFO test_log_feature:test_log_feature.py:9 Process started
# INFO test_log_feature:test_log_feature.py:10 Process ongoing..
# PASSED [100%]
# ----- live log teardown ----------------------------------------
# INFO test_log_feature:test_log_feature.py:12 Process ended
You can do the same in a pytest.ini configuration file.
[pytest]
log_cli=true
log_cli_level=INFO
Update
I want live logging solely for my logger, is that possible?
I think in this case you have to play with loggers and define various loggers according to your needs and enable log_cli_level at the lower level you want to see. Here is an example to illustrate this solution.
import logging
import pytest
fixture_logger = logging.getLogger("fixture")
logger = logging.getLogger(__name__)
# can also be set by configuration
logger.setLevel(logging.WARNING)
#pytest.fixture()
def some_long_process(caplog):
fixture_logger.info("Process started")
fixture_logger.info("Process ongoing..")
yield
fixture_logger.info("Process ended")
def test_long_process(some_long_process, caplog, request):
logger.info("Log I don't want to see")
pass
Gives.
pytest --log-cli-level=INFO test_log_feature.py
# --- live log setup ---
# INFO fixture:test_log_feature.py:10 Process started
# INFO fixture:test_log_feature.py:11 Process ongoing..
# PASSED [100%]
# ---- live log teardown ---
# INFO fixture:test_log_feature.py:13 Process ended

How to suppress third party logs in pytest

We've just switched from nose to pytest and there doesn't seem to be an option to suppress third party logging. In nose config, we had the following line:
logging-filter=-matplotlib,-chardet.charsetprober,-PIL,-fiona.env,-fiona._env
Some of those logs are very chatty, especially matplotlib and we don't want to see the output, just output from our logs.
I can't find an equivalent setting in pytest though. Is it possible? Am I missing something? Thanks.

The way I do it is by creating a list of logger names for which logs have to be disabled in conftest.py.
For example, if I want to disable a logger called app, then I can write a conftest.py as below:
import logging
disable_loggers = ['app']
def pytest_configure():
for logger_name in disable_loggers:
logger = logging.getLogger(logger_name)
logger.disabled = True
And then run my test:
import logging
def test_brake():
logger = logging.getLogger("app")
logger.error("Hello there")
assert True
collecting ... collected 1 item
test_car.py::test_brake PASSED
[100%]
============================== 1 passed in 0.01s ===============================
Then, Hello there is not there because the logger with the name app was disabled in conftest.py.
However, if I change my logger name in the test to app2 and run the test again:
import logging
def test_brake():
logger = logging.getLogger("app2")
logger.error("Hello there")
assert True
collecting ... collected 1 item
test_car.py::test_brake
-------------------------------- live log call --------------------------------- ERROR app2:test_car.py:5 Hello there PASSED
[100%]
============================== 1 passed in 0.01s ===============================
As you can see, Hello there is in because a logger with app2 is not disabled.
Conclusion
Basically, you could do the same, but just add your undesired logger names to conftest.py as below:
import logging
disable_loggers = ['matplotlib', 'chardet.charsetprober', <add more yourself>]
def pytest_configure():
for logger_name in disable_loggers:
logger = logging.getLogger(logger_name)
logger.disabled = True

Apart from the ability to tune logging levels or not show any log output at all which I'm sure you've read in the docs, the only way that comes to mind is to configure your logging in general.
Assuming that all of those packages use the standard library logging facilities, you have various options of configuring what gets logged. Please take a look at the advanced tutorial for a good overview of your options.
If you don't want to configure logging for your application in general but only during testing, you might do so using the pytest_configure or pytest_sessionstart hooks which you might place in a conftest.py at the root of your test file hierarchy.
Then I see three options:
The brute force way is to use the default behaviour of fileConfig or dictConfig to disable all existing loggers. In your conftest.py:
import logging.config
def pytest_sessionstart():
# This is the default so an empty dictionary should work, too.
logging.config.dictConfig({'disable_existing_loggers': True})
The more subtle approach is to change the level of individual loggers or disable them. As an example:
import logging.config
def pytest_sessionstart():
logging.config.dictConfig({
'disable_existing_loggers': False,
'loggers': {
# Add any noisy loggers here with a higher loglevel.
'matplotlib': {'level': 'ERROR'}
}
})
Lastly, you can use the pytest_addoption hook to add a command line option similar to the one you mention. Again, at the root of your test hierarchy put the following in a conftest.py:
def pytest_addoption(parser):
parser.addoption(
"--logging-filter",
help="Provide a comma-separated list of logger names that will be "
"disabled."
)
def pytest_sessionstart(pytestconfig):
for logger_name in pytestconfig.getoption("--logging-filter").split(","):
# Use `logger_name.trim()[1:]` if you want the `-name` CLI syntax.
logger = logging.getLogger(logger_name.trim())
logger.disabled = True
You can then call pytest in the following way:
pytest --logging-filter matplotlib,chardet,...
The default approach by pytest is to hide all logs but provide the caplog fixture to inspect log output in your test cases. This is quite powerful if you are looking for specific log lines. So the question is also why you need to see those logs at all in your test suite?

Adding a log filter to conftest.py looks like it might be useful and I'll come back to that at some point in the future. For now though, we've just gone for silencing the logs in the application. We don't see them at any point when the app is running, not just during testing.
# Hide verbose third-party logs
for log_name in ('matplotlib', 'fiona.env', 'fiona._env', 'PIL', 'chardet.charsetprober'):
other_log = logging.getLogger(log_name)
other_log.setLevel(logging.WARNING)

Change log level in unittest

I have the impression (but do not find the documentation for it) that unittest sets the logging level to WARNING for all loggers. I would like to:
be able to specify the logging level for all loggers, from the command line (when running the tests) or from the test module itself
avoid unittest messing around with the application logging level: when running the tests I want to have the same logging output (same levels) as when running the application
How can I achieve this?

I don't believe unittest itself does anything to logging, unless you use a _CapturingHandler class which it defines. This simple program demonstrates:
import logging
import unittest
logger = logging.getLogger(__name__)
class MyTestCase(unittest.TestCase):
def test_something(self):
logger.debug('logged from test_something')
if __name__ == '__main__':
# DEBUG for demonstration purposes, but you could set the level from
# cmdline args to whatever you like
logging.basicConfig(level=logging.DEBUG, format='%(name)s %(levelname)s %(message)s')
unittest.main()
When run, it prints
__main__ DEBUG logged from test_something
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
showing that it is logging events at DEBUG level, as expected. So the problem is likely to be related to something else, e.g. the code under test, or some other test runner which changes the logging configuration or redirects sys.stdout and sys.stderr. You will probably need to provide more information about your test environment, or better yet a minimal program that demonstrates the problem (as my example above shows that unittest by itself doesn't cause the problem you're describing).

See below example for logging in Python. Also you can change LOG_LEVEL using 'setLevel' method.
import os
import logging
logging.basicConfig()
logger = logging.getLogger(__name__)
# Change logging level here.
logger.setLevel(os.environ.get('LOG_LEVEL', logging.INFO))
logger.info('For INFO message')
logger.debug('For DEBUG message')
logger.warning('For WARNING message')
logger.error('For ERROR message')
logger.critical('For CRITICAL message')

This is in addition to #Vinay's answer above. It does not answer the original question. I wanted to include command line options for modifying the log level. The intent was to get detailed loggin only when I pass a certain parameter from the command line. This is how I solved it:
import sys
import unittest
import logging
from histogram import Histogram
class TestHistogram(unittest.TestCase):
def test_case2(self):
h = Histogram([2,1,2])
self.assertEqual(h.calculateMaxAreaON(), 3)
if __name__ == '__main__':
argv = len(sys.argv) > 1 and sys.argv[1]
loglevel = logging.INFO if argv == '-v' else logging.WARNING
logging.basicConfig(level=loglevel)
unittest.main()
The intent is to get more verbose logging. I know it does not answer the question, but I'll leave it here in case someone comes looking for a similar requirement such as this.

this worked for me:
logging.basicConfig(level=logging.DEBUG)
And if I wanted a specific format:
logging.basicConfig(
level=logging.DEBUG,
datefmt="%H:%M:%S",
format="%(asctime)s.%(msecs)03d [%(levelname)-5s] %(message)s",
)

Programmatically:
Put this line of code in each test function defined in your class that you want to set the logging level:
logging.getLogger().setLevel(logging.INFO)
Ex. class:
import unittest
import logging
class ExampleTest(unittest.TestCase):
def test_method(self):
logging.getLogger().setLevel(logging.INFO)
...
Command Line:
This example just shows how to do it in a normal script, not specific to unittest example. Capturing the log level via command line, using argparse for arguments:
import logging
import argparse
...
def parse_args():
parser = argparse.ArgumentParser(description='...')
parser.add_argument('-v', '--verbose', help='enable verbose logging', action='store_const', dest="loglevel", const=logging.INFO, default=logging.WARNING)
...
def main():
args = parse_args()
logging.getLogger().setLevel(args.loglevel)