Python Logging to head of logfile? - python

I use python logging and would like the latest log entry to be at the head of the log file, rather than the tail.
I cannot find anything in https://docs.python.org/2/howto/logging.html that appears useful for appending a logger record at the head of the logfile instead of the tail.
This is my logger: ("Carl" is my GoPiGo3 robot)
import logging
# create logger
logger = logging.getLogger('lifelog')
logger.setLevel(logging.INFO)
loghandler = logging.FileHandler('/home/pi/Carl/life.log')
logformatter = logging.Formatter('%(asctime)s|%(message)s',"%Y-%m-%d %H:%M")
loghandler.setFormatter(logformatter)
logger.addHandler(loghandler)
#logger.info('-------------')
and I log with:
logger.info('<something to log>')
Is there a python logging module intrinsic solution?
Do I have to write my own handler?

The Answer: I should not want to do this.
Because:
Inserting something at the beginning of a file displaces everything
that comes after it, so the bigger the file, the more an insert will
cost. That would go counter to the efforts of the logging module to be
sparing in its use of resources. – BoarGules Mar 28 at 13:56
Using 'tac' to view the log file is a good solution.

Related

How to get the current python logging config?

I am using the python logging library to configure my loggers with an input dict, like this:
logging.config.dictConfig(config)
I have a special function that should use a new function, and then switch back to the original logger at the end of the function. The original logger can vary, so I do not want to hardcode it. Some psuedocode to describe what I want:
def switch_logger_for_this_code():
old_logging_config = logger.get_current_logging_config(). # This is what I want to accomplish
logging.config.dictConfig(new_logging_config)
logging.info('This log goes to the new config!')
logging.config.dictConfig(old_logging_config)
logging.info('This log goes to the old config!')
return
Is this possible to do with the logging library?
I don't believe there is any built in way of retrieving the config of an initialized logger and storing it into a variable. I'm not sure if this would work for your project, but have you considered just creating a temporary logger object for use within the function's scope?
import logging
from sys import stdout
def switch_logger_for_this_code(msg) -> None:
# Create logging.Formatter for output customization
formatter = logging.Formatter(
fmt="[%(asctime)s.%(msecs)d] %(message)s",
datefmt="%Y/%m/%d %H:%M:%S"
)
handler = logging.StreamHandler(stdout) # Create output handler
handler.setLevel(logging.ERROR) # Specify logging level
handler.setFormatter(formatter) # Apply desired format
temp_log = logging.getLogger("temp") # Get new or existing Logger
temp_log.addHandler(handler) # Add output handler
temp_log.error(msg) # Log the message
switch_logger_for_this_code("hello world") # Log 'hello world'
# Which would output the following
# [2020/10/20 15:09:18.548] hello world
The main idea behind this approach is that the original logger object won't be modified, but you can still use the temporary logger to log what you had originally intended in your desired format.
I just looked through the source code for the logging module, and I don't see any way to get at the internal configuration, which starts as the passed in configuration, but can then be modified by other calls to the logging system.

Python Logging Module Inconsistent Behaviour

I've seen some very odd behaviour with the logger module. It started with a relatively complex project, but now I've seen it with the following script:
import logging
import os
# Uncomment the following line to remove handlers
# logging.getLogger().handlers = []
filePath = os.environ['userprofile'] + r'\Documents\log.txt'
logging.basicConfig(filename=filePath)
logging.debug('Gleep')
logging.shutdown()
This should simply write 'Gleep' to the log.txt file to your documents. Currently it is writing the file but not writing anything to it, however, I've inconsistently seen the following behaviour:
List item
No log file being written at all.
Log file created, but nothing written to it.
Everything working fine.
The only way I've got it working before is to remove existing handlers (commented out in the example above).
This is on several machines in different locations.
So...am I doing something grotesquely wrong here? Why is the logging module acting this way?
I'm not sure how to prove/disprove/debug your 'other' situations, but maybe the following can help clarify what is happening in the code from your question:
First, setting logging.getLogger().handlers = [] should not be necessary, since logging.getLogger() is the root logger by default and has no handlers. Here is a fresh Python 3.7 shell:
>>> import logging
>>> logging.getLogger()
<RootLogger root (WARNING)>
>>> logging.getLogger().handlers
[]
(Note that in the absence of any handlers, a logger will fall back to lastResort, but that should be irrelevant here since you add a handler implicitly via basicConfig().)
Which brings you to logging.basicConfig(filename=filePath): this adds a FileHandler handler to the root logger. It does not touch the level of the root logger, which is WARNING by default, so your message won't pass the 'level test' and won't be emitted as a result.
>>> logging.root.getEffectiveLevel()
30
(This uses .getEffectiveLevel() rather than just the plain attribute because a logger will walk its hierarchy until it finds a level if its level is NOTSET.)
All that is to say: as you currently have it, you are logging from the root logger (level WARNING) a message object that has level DEBUG, so the message will go nowhere.

writing a log file from python program

I want to output some strings to a log file and I want the log file to be continuously updated.
I have looked into the logging module pf python and found out that it is
mostly about formatting and concurrent access.
Please let me know if I am missing something or amy other way of doing it
usually i do the following:
# logging
LOG = "/tmp/ccd.log"
logging.basicConfig(filename=LOG, filemode="w", level=logging.DEBUG)
# console handler
console = logging.StreamHandler()
console.setLevel(logging.ERROR)
logging.getLogger("").addHandler(console)
The logging part initialises logging's basic configurations. In the following I set up a console handler that prints out some logging information separately. Usually my console output is set to output only errors (logging.ERROR) and the detailed output in the LOG file.
Your loggings will now printed to file. For instance using:
logger = logging.getLogger(__name__)
logger.debug("hiho debug message")
or even
logging.debug("next line")
should work.
Doug Hellmann has a nice guide.
To add my 10cents with regards to using logging. I've only recently discovered the Logging module and was put off at first. Maybe just because it initially looks like a lot of work, but it's really simple and incredibly handy.
This is the set up that I use. Similar to Mkinds answer, but includes a timestamp.
# Set up logging
log = "bot.log"
logging.basicConfig(filename=log,level=logging.DEBUG,format='%(asctime)s %(message)s', datefmt='%d/%m/%Y %H:%M:%S')
logging.info('Log Entry Here.')
Which will produce something like:
22/09/2015 14:39:34 Log Entry Here.
You can log to a file with the Logging API.
Example: http://docs.python.org/2/howto/logging.html#logging-to-a-file

Changing logging's 'basicConfig' which is already set

I am using the logging module in python as:
import logging, sys
logger= logging.getLogger(__file__)
logging.basicConfig(stream = sys.stderr, level=logging.DEBUG, format='%(filename)s:%(lineno)s %(levelname)s:%(message)s')
logger.debug("Hello World")
Now, after I have set the basic configuration on line 3, I want to have a command line argument that can change the output stream from sys.stderr to a file.
I have read the doc and it says that if both filename and stream are present at the same time, the stream is ignored.
Now, I wanna know how do change the stream to a file after I have already done the basicConfig thing in line 3?
If you look in the Python sources for logging/__init__.py, you'll see that basicConfig() sets the handlers on the root logger object by calling addHandler(). If you want to start from scratch, you could remove all existing handlers and then call basicConfig() again.
# Example to remove all root logger handlers and reconfigure. (UNTESTED)
import logging
# Remove all handlers associated with the root logger object.
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
# Reconfigure logging again, this time with a file.
logging.basicConfig(filename = 'myfile.log', level=logging.DEBUG, format='%(filename)s:%(lineno)s %(levelname)s:%(message)s')
Attribution to be given in Spack, for his comment. I am just expanding the idea in a proper answer
In 2022 and using Python 3.8, we can use the force input of basicConfig method
As per Python documentation
force
If this keyword argument is specified as true, any existing handlers
attached to the root logger are removed and closed, before carrying
out the configuration as specified by the other arguments.
Based on OP code sample, can just add the following line
logging.basicConfig(filename = 'my_file.log', level = logging.DEBUG, format = '%(filename)s:%(lineno)s %(levelname)s:%(message)s', force = True)

Redefining logging root logger

At my current project there are thousand of code lines which looks like this:
logging.info("bla-bla-bla")
I don't want to change all these lines, but I would change log behavior. My idea is changing root logger to other Experimental logger, which is configured by ini-file:
[loggers]
keys = Experimental
[formatter_detailed]
format = %(asctime)s:%(name)s:%(levelname)s %(module)s:%(lineno)d: %(message)s
[handler_logfile]
class = FileHandler
args = ('experimental.log', 'a')
formatter = detailed
[logger_Experimental]
level = DEBUG
qualname = Experimental
handlers = logfile
propagate = 0
Now setting the new root logger is done by this piece of code:
logging.config.fileConfig(path_to_logger_config)
logging.root = logging.getLogger('Experimental')
Is redefining of root logger safe? Maybe there is more convenient way?
I've tried to use google and looked through stackoverflow questions, but I didn't find the answer.
You're advised not to redefine the root logger in the way you describe. In general you should only use the root logger directly for small scripts - for larger applications, best practice is to use
logger = logging.getLogger(__name__)
in each module where you use logging, and then make calls to logger.info() etc.
If all you want to do is to log to a file, why not just add a file handler to the root logger? You can do using e.g.
if __name__ == '__main__':
logging.basicConfig(filename='experimental.log', filemode='w')
main() # or whatever your main entry point is called
or via a configuration file.
Update: When I say "you're advised", I mean by me, here in this answer ;-) While you may not run into any problems in your scenario, it's not good practice to overwrite a module attribute which hasn't been designed to be overwritten. For example, the root logger is an instance of a different class (which is not part of the public API), and there are other references to it in the logging machinery which would still point to the old value. Either of these facts could lead to hard-to-debug problems. Since the logging package allows a number of ways of achieving what you seem to want (seemingly, logging to a file rather than the console), then you should use those mechanisms that have been provided.
logger = logging.getLogger()
Leaving the name empty will return you the root logger.
logger = logging.getLogger('name')
Gives you another logger.

Categories