I have a script that performs database operations alongside an alembic API call to upgrade a newly created database to head. I am having an issue with a python logger instance where logs are written to a file using a module level logger.
Then the script invokes alembic.config.main(argv=alembic_args) to run a migration. However, every log statement after the alembic call, using the original logger instance, isn't written to the expected log file.
Here is an example script that reproduces the behavior.
#!/usr/bin/env python3
import logging
import os
import alembic.config
from .utilities import get_migration_dir
logging.basicConfig(filename='test.log',
level=logging.DEBUG)
CUR_DIR = os.path.dirname(__file__)
LOG = logging.getLogger('so_log')
LOG.info('Some stuff')
LOG.info('More stuff')
alembic_config = (
'--raiseerr',
'upgrade', 'head'
)
os.chdir(get_migration_dir())
alembic.config.main(argv=alembic_config)
os.chdir(CUR_DIR)
LOG.debug('logging after alembic call.')
LOG.debug('more logging after alembic call.')
print('Code still running after alembic')
Log file output
INFO:so_log:Some stuff
INFO:so_log:More stuff
stdout
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
print statement before alembic
Code still running after alembic
It seems as though the logger instance, LOG, is losing context or being directed elsewhere after calling the alembic API.
Also, I've tried running the alembic call in a separate thread which yielded the same result. What I expect to happen should be that log statements continue to write to the specified file after using alembic for migrations but that is not happening. And further, it's actually breaking the LOG instance for any code that's called afterward; Unless I'm just missing something here.
I just learned that fileConfig takes a keyword argument, disable_existing_loggers, which defaults to True. Simply adding disable_existing_loggers = False to the call to fileConfig in env.py
e.g:
fileConfig(config.config_file_name, disable_existing_loggers=False)
appears to allow both logging configurations to work without interfering with each other (which may be preferred rather than having to choose one over the other, in some cases)
This is because alembic sets up logging using fileConfig from alembic.ini, you can see it in your env.py script:
# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)
This effectively overrides your original logger config.
To avoid this, you can simply remove this line from env.py, however this will result in no logs being produced when running alembic from console.
A more robust option is to run alembic commands via alembic.command instead of alembic.config.main. This way you can override alembic config at runtime:
from alembic.config import Config
import alembic.command
config = Config('alembic.ini')
config.attributes['configure_logger'] = False
alembic.command.upgrade(config, 'head')
Then in env.py:
if config.attributes.get('configure_logger', True):
fileConfig(config.config_file_name)
In case anyone comes across this issue here is what worked for me:
Updating the fileConfig call in env.py:
fileConfig(config.config_file_name, disable_existing_loggers = False)
Updating the level for [logger_root] in alembic.ini to a lower level, since it is set by default to WARN
Related
Ok so in my environment.py file I am able to log stuff by:
logging.basicConfig(level=logging.DEBUG, filename="example.log")
def before_feature(context, feature):
logging.info("test logging")
but when I am inside the steps file I cannot perform logging:
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
#given("we have a step")
def step_impl(context):
logger.debug("Test logging 2")
The logging message inside the step does not show up. I am using the python behave module. Any ideas?
I have tried enabling and disabling logcapture when I run behave but it makes no difference.
By default, behave tends to capture logs during feature execution, and only display them in cases of failure.
To disable this, you can set
log_capture=false
in behave.ini
Or, you can use the --no-logcapture command line option
Further Reading : Behave API Reference, Behave LogCapture
what worked for me:
behave --no-capture --no-capture-stderr --no-logcapture
and add in the environment.py the follwing snipest:
def after_step(context, step):
print("")
Why: I disovered that behave does not log the last print statement of a step. So I just added a empty print after each step with the previous snipest.
Hope it helped
Importing logging from environment.py in steps.py solved problem for me.
from features.environment import logging
I am not sure but I guess the problem is that every time you import logging it rewrites your previous configs because disable_existing_loggers is True by default. (Here is the documentation paragraph explaining this)
I'm sure this is a simple fix, but I'd like to view log messages in the PyCharm console while running a unit test. The modules I'm testing have their own loggers, and normally I'd set a root logger to catch the debugging messages at a certain level, and pipe the other logs to a file. But I can't figure out how this works with unit tests.
I'm using the unittest2 module, and using PyCharm's automatic test discovery (which probably is based on nose, but I don't know).
I've tried fooling with the run configurations, but there doesn't seem to be a straightforward way to do this.
The PyCharm documentation isn't particularly helpful here either, if any of you work there.
In edit: It DOES appear that the console catches critical level log messages. I want to know if there is a way to configure this to catch debug level messages.
This post (Pycharm unit test interactive debug command line doesn't work) suggests adding the -s option to the build configuration, which does not produce the desired result.
The only solution I've found is to do the normal logging setup at the top of the file with tests in it (assuming you're just running one test or test class): logging.basicConfig(level=logging.DEBUG). Make sure you put that before most import statements or the default logging for those modules will have already been set (that was a hard one to figure out!).
I needed to add too the option --nologcapture to nosetests as param in pycharm 2016.3.2
Run>Edit COnfigurations > Defaults > Python Tests > Nosetests : activate the check for Prams option and add --nologcapture
In Edit Configurations:
delete old configurations
go to Defaults / Python tests / NoseTests
add --nologcapture to "additional Arguments"
worked like a "charm" for me in pyCharm 2017.2.3
thanks bott
My problem was similar, I only saw messages with level WARNING or higher while running tests in PyCharm. A colleague suggested to configure the logger in __init__.py which is in the directory of my tests.
# in tests/__init__.py
import logging
import sys
# Reconfiguring the logger here will also affect test running in the PyCharm IDE
log_format = '%(asctime)s %(levelname)s %(filename)s:%(lineno)d %(message)s'
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format=log_format)
Then I can simply log in the test code like:
import logging
logging.info('whatever')
I experienced this problem all of a sudden after updating Pycharm. I had been running my tests with the Unittest runner, and all of a sudden Pycharm decided the default should be the pytest runner. When I changed the default test runner back to Unittest the logs appeared as I expected they would.
You can change the default test runner in Project Settings/Preferences >> Tools >> Python Integrated Tools
I assume adding the -nologcapture flag as noted in other answers probably would have worked in my case as well, but I prefer a solution that's exposed by the IDE rather than a manual override.
BTW choosing Unittest also solves an additional error that I got once when trying to run tests - No module named 'nose'
The -s option mentioned in the question in combination with adding the StreamHandler using stream=sys.stdout did work for me.
import logging
import sys
logger = logging.getLogger('foo')
logger.addHandler(logging.StreamHandler(stream=sys.stdout))
logger.warning('foo')
Pycharm unit test interactive debug command line doesn't work
Add a stream handler, pointing to sys.stdout if doctest or pytest is running:
import logging
import sys
logger = logging.getLogger()
def setup_doctest_logger(log_level:int=logging.DEBUG):
"""
:param log_level:
:return:
>>> logger.info('test') # there is no output in pycharm by default
>>> setup_doctest_logger()
>>> logger.info('test') # now we have the output we want
test
"""
if is_pycharm_running():
logger_add_streamhandler_to_sys_stdout()
logger.setLevel(log_level)
def is_pycharm_running()->bool:
if ('docrunner.py' in sys.argv[0]) or ('pytest_runner.py' in sys.argv[0]):
return True
else:
return False
def logger_add_streamhandler_to_sys_stdout():
stream_handler=logging.StreamHandler(stream=sys.stdout)
logger.addHandler(stream_handler)
I am working on a python app that uses the default python logging system. Part of this system is the ability to define handlers in a logging config file. One of the handlers for this app is the django admin email handler, "django.utils.log.AdminEmailHandler". When the app is initializing the logging system, it makes a call to logging.config.fileconfig. This is done on a background thread and attempts to reload the config file periodically. I believe that is important.
I have traced through the python logging source code down to the method:
def _resolve(name):
"""Resolve a dotted name to a global object."""
name = name.split('.')
used = name.pop(0)
found = __import__(used)
for n in name:
used = used + '.' + n
try:
found = getattr(found, n)
except AttributeError:
__import__(used)
found = getattr(found, n)
return found
in the file python2.7/logging/config.py
When this function is given the paramater "django.utils.log.AdminEmailHandler" in order to create that handler, my app hangs on the command
__import__(used)
where used is "django".
I did a little research and I have seen some mentions of __import__ not being thread safe and to avoid its use in background threads. is this accurate? And knowing that __import__("django") does cause a deadlock, is there anything I could do to prevent it?
I suggest using the default Django LOGGING setting to control logging. For development, starting the server with manage.py runserver will automatically reload Django if any files are changed, including the settings file with the logging configuration. In practice it works quite well!
https://docs.djangoproject.com/en/dev/topics/logging/#examples
I think there was a default log file, but I didn't find it yet.
Sometimes the HTTP request process would throw an exception on the screen, but I suggest it also goes somewhere on the disk or I wouldn't know what was wrong during a long run test.
P.S.: write an exception handler is another topic; first I'd like to know my question's answer.
I found something here:
https://groups.google.com/forum/?fromgroups=#!topic/python-tornado/px4R8Tkfa9c
But it also didn't mention where can I find those log.
It uses standard python logging module by default.
Here is definition:
access_log = logging.getLogger("tornado.access")
app_log = logging.getLogger("tornado.application")
gen_log = logging.getLogger("tornado.general")
It doesn't write to files by default. You can run it using supervisord and define in supervisord config, where log files will be located. It will capture output of tornado and write it to files.
Also you can think this way:
tornado.options.options['log_file_prefix'].set('/opt/logs/my_app.log')
tornado.options.parse_command_line()
But in this case - measure performance. I don't suggest you to write to files directly from tornado application, if it can be delegated.
FYI: parse_command_line just enables pretty console logging.
With newer versions, you may do
args = sys.argv
args.append("--log_file_prefix=/opt/logs/my_app.log")
tornado.options.parse_command_line(args)
or as #ColeMaclean mentioned, providing
--log_file_prefix=PATH
at command line
There's no logfile by default.
You can use the --log_file_prefix=PATH command line option to set one.
Tornado just uses the Python stdlib's logging module, if you're trying to do anything more complicated.
Use RotatingFileHandler:
import logging
from logging.handlers import RotatingFileHandler
log_path = "/path/to/tornado.access.log"
logger_ = logging.getLogger("tornado.access")
logger_.setLevel(logging.INFO)
logger_.propagate = False
handler = RotatingFileHandler(log_path, maxBytes=1024*1024*1024, backupCount=3)
handler.setFormatter(logging.Formatter("[%(name)s][%(asctime)s][%(levelname)s][%(pathname)s:%(lineno)d] > %(message)s"))
logger_.addHandler(handler)
It appears that if you invoke logging.info() BEFORE you run logging.basicConfig, the logging.basicConfig call doesn't have any effect. In fact, no logging occurs.
Where is this behavior documented? I don't really understand.
You can remove the default handlers and reconfigure logging like this:
# if someone tried to log something before basicConfig is called, Python creates a default handler that
# goes to the console and will ignore further basicConfig calls. Remove the handler if there is one.
root = logging.getLogger()
if root.handlers:
for handler in root.handlers:
root.removeHandler(handler)
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
Yes.
You've asked to log something. Logging must, therefore, fabricate a default configuration. Once logging is configured... well... it's configured.
"With the logger object configured,
the following methods create log
messages:"
Further, you can read about creating handlers to prevent spurious logging. But that's more a hack for bad implementation than a useful technique.
There's a trick to this.
No module can do anything except logging.getlogger() requests at a global level.
Only the if __name__ == "__main__": can do a logging configuration.
If you do logging at a global level in a module, then you may force logging to fabricate it's default configuration.
Don't do logging.info globally in any module. If you absolutely think that you must have logging.info at a global level in a module, then you have to configure logging before doing imports. This leads to unpleasant-looking scripts.
This answer from Carlos A. Ibarra is in principle right, however that implementation might break since you are iterating over a list that might be changed by calling removeHandler(). This is unsafe.
Two alternatives are:
while len(logging.root.handlers) > 0:
logging.root.removeHandler(logging.root.handlers[-1])
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
or:
logging.root.handlers = []
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
where the first of these two using the loop is the safest (since any destruction code for the handler can be called explicitly inside the logging framework). Still, this is a hack, since we rely on logging.root.handlers to be a list.
Here's the one piece of the puzzle that the above answers didn't mention... and then it will all make sense: the "root" logger -- which is used if you call, say, logging.info() before logging.basicConfig(level=logging.DEBUG) -- has a default logging level of WARNING.
That's why logging.info() and logging.debug() don't do anything: because you've configured them not to, by... um... not configuring them.
Possibly related (this one bit me): when NOT calling basicConfig, I didn't seem to be getting my debug messages, even though I set my handlers to DEBUG level. After a bit of hair-pulling, I found you have to set the level of the custom logger to be DEBUG as well. If your logger is set to WARNING, then setting a handler to DEBUG (by itself) won't get you any output on logger.info() and logger.debug().
Ran into this same issue today and, as an alternative to the answers above, here's my solution.
import logging
import sys
logging.debug('foo') # IRL, this call is from an imported module
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO, force=True)
logging.info('bar') # without force=True, this is not printed to the console
Here's what the docs say about the force argument.
If this keyword argument is specified as true, any existing handlers
attached to the root logger are removed and closed, before carrying
out the configuration as specified by the other arguments.
A cleaner version of the answer given by #paul-kremer is:
while len(logging.root.handlers):
logging.root.removeHandler(logging.root.handlers[-1])
Note: it is generally safe to assume logging.root.handlers will always be a list (see: https://github.com/python/cpython/blob/cebe9ee988837b292f2c571e194ed11e7cd4abbb/Lib/logging/init.py#L1253)
Here is what I did.
I wanted to log to a file which has a name configured in a config-file and also get the debug-logs of the config-parsing.
TL;DR; This logs into a buffer until everything to configure the logger is available
# Log everything into a MemoryHandler until the real logger is ready.
# The MemoryHandler never flushes (flushLevel 100 is above CRITICAL) automatically but only on close.
# If the configuration was loaded successfully, the real logger is configured and set as target of the MemoryHandler
# before it gets flushed by closing.
# This means, that if the log gets to stdout, it is unfiltered by level
root_logger = logging.getLogger()
root_logger.setLevel(logging.NOTSET)
stdout_logging_handler = logging.StreamHandler(sys.stderr)
tmp_logging_handler = logging.handlers.MemoryHandler(1024 * 1024, 100, stdout_logging_handler)
root_logger.addHandler(tmp_logging_handler)
config: ApplicationConfig = ApplicationConfig.from_filename('config.ini')
# because the records are already logged, unwanted ones need to be removed
filtered_buffer = filter(lambda record: record.levelno >= config.main_config.log_level, tmp_logging_handler.buffer)
tmp_logging_handler.buffer = filtered_buffer
root_logger.removeHandler(tmp_logging_handler)
logging.basicConfig(filename=config.main_config.log_filename, level=config.main_config.log_level, filemode='wt')
logging_handler = root_logger.handlers[0]
tmp_logging_handler.setTarget(logging_handler)
tmp_logging_handler.close()
stdout_logging_handler.close()