I'm messing around with python a bit, and I want to track the date/time of which some events happen. For now, outputting to the console will work just fine.
Currently, I'm handling it like so:
First, get the formatted date/time wrapped in brackets:
def get_date_time():
now = datetime.datetime.now().strftime("%Y/%m/%d %I:%M:%S")
return "[" + now + "]"
Next, whenever I want to "log" an action, call it like so:
print(get_date_time(), "Outputting whatever text here")
Is there anything wrong with doing it this way? Is there maybe a more efficient/clean way to do it? There may not be, it just has me curious.
On every problem; if you are repating stuff, you are doing it wrong.
For your question; just make it a function, something along the lines of;
def log(s):
print(get_date_time(), s)
Or for bigger projects, use logging module.
Yes, use the Python logging module. http://docs.python.org/2/library/logging.html
You can format all your logs to output datetimes if you like.
import logging
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s',
filename='/tmp/myapp.log',
filemode='w')
logging.debug('A debug message')
prints
2004-07-02 13:00:08,743 DEBUG A debug message
as seen here http://docs.python.org/2.4/lib/minimal-example.html
If you wish to formate the datetime, use logging.formatter()
Related
I'm using python's logging library, but I want the debug logs to have a different format than the warning and error logs. Is this possible?
ETA: I want warnings and errors to appear as:
%(levelname)s: %(message)s
but debug statements to appear as
DEBUG: (only Brian cares about this) : %(message)s
all other questions I've seen have been to change the format, but that changes for EVERYTHING.
First of all, double-check if you really need this. A log output with different record formats is prone to be rather hard to read by both humans and machines.
Maybe what you actually need is different formats for different log destinations (console vs file) which will also have different verbosity (the file will have a debug log with additional information).
Now, the way is to use a custom Formatter:
class MultiformatFormatter(logging.Formatter):
def __init__(self,<args>):
<...>
def format(self,record):
if record.levelno <= logging.DEBUG:
s=<generate string one way>
else:
s=<generate string another way>
return s
<...>
#for each handler that this should apply to
handler.setFormatter(MultiformatFormatter(<args>))
I have to print the thread name , but I need to add function call everywhere:
self.logger.debug("{} log".format(currentThread().getName()))
self.logger.error("{} log".format(currentThread().getName()))
I use the logging module, is it possible to add a log prefix which comes from a function call?
Antoher case is that the logger is not from logging module, is it possbile to monkey-patch the function, generating some functions so that I can call it like this:
self.logger.my_debug("log")
self.logger.my_error("log")
I do not want to write many functions manually with duplicated codes for different logging levels, it should be good if the codes are like this
for log_level in ("error", "warn", "debug", "info")
...generating functions...
Take a look at the documentation: https://docs.python.org/2/library/logging.html#logrecord-attributes
Your code could look like this:
FORMAT = '%(asctime)-15s %(threadName)s %(funcName)-8s %(message)s'
logging.basicConfig(format=FORMAT)
logger = logging.getLogger('mylogger')
logger.warning("message here)
To stress a point: Logging is normally configured globally. Especially thinks like "what format has the log" and "where does it get logged?" are decided there.
While I dislike the python logging system b/C it'S rather convuluted, I would say that understanding how it works is imperative if you want to log effectively.
I'm using Python 3.3.5 and the logging module to log information to a local file (from different threads). There are cases where I'd like to output some additional information, without knowing exactly what that information will be (e.g. it might be one single line of text or a dict).
What I'd like to do is add this additional information to my log file, after the log record has been written. Furthermore, the additional info is only necessary when the log level is error (or higher).
Ideally, it would look something like:
2014-04-08 12:24:01 - INFO - CPU load not exceeded
2014-04-08 12:24:26 - INFO - Service is running
2014-04-08 12:24:34 - ERROR - Could not find any active server processes
Additional information, might be several lines.
Dict structured information would be written as follows:
key1=value1
key2=value2
2014-04-08 12:25:16 - INFO - Database is responding
Short of writing a custom log formatter, I couldn't find much which would fit my requirements. I've read about filters and contexts, but again this doesn't seem like a good match.
Alternatively, I could just write to a file using the standard I/O, but most of the functionality already exists in the Logging module, and moreover it's thread-safe.
Any input would be greatly appreciated. If a custom log formatter is indeed necessary, any pointers on where to start would be fantastic.
Keeping in mind that many people consider a multi-line logging message a bad practice (understandably so, since if you have a log processor like DataDog or Splunk which are very prepared to handle single line logs, multi-line logs will be very hard to parse), you can play with the extra parameter and use a custom formatter to append stuff to the message that is going to be shown (take a look to the usage of 'extra' in the logging package documentation).
import logging
class CustomFilter(logging.Filter):
def filter(self, record):
if hasattr(record, 'dct') and len(record.dct) > 0:
for k, v in record.dct.iteritems():
record.msg = record.msg + '\n\t' + k + ': ' + v
return super(CustomFilter, self).filter(record)
if __name__ == "__main__":
logging.getLogger().setLevel(logging.DEBUG)
extra_logger = logging.getLogger('extra_logger')
extra_logger.setLevel(logging.INFO)
extra_logger.addFilter(CustomFilter())
logging.debug("Nothing special here... Keep walking")
extra_logger.info("This shows extra",
extra={'dct': {"foo": "bar", "baz": "loren"}})
extra_logger.debug("You shouldn't be seeing this in the output")
extra_logger.setLevel(logging.DEBUG)
extra_logger.debug("Now you should be seeing it!")
That code outputs:
DEBUG:root:Nothing special here... Keep walking
INFO:extra_logger:This shows extra
foo: bar
baz: loren
DEBUG:extra_logger:Now you should be seeing it!
I still recommend calling the super's filter function in your custom filter, mainly because that's the function that decides whether showing the message or not (for instance, if your logger's level is set to logging.INFO, and you log something using extra_logger.debug, that message shouldn't be seen, as shown in the example above)
I just add \n symbols to the output text.
i'm using a simple line splitter in my smaller applications:
for line in logmessage.splitlines():
writemessage = logtime + " - " + line + "\n"
logging.info(str(writemessage))
Note that this is not thread-safe and should probably only be used in log-volume logging applications.
However you can output to log almost anything, as it will preserve your formatting. I have used it for example to output JSON API responses formatted using: json.dumps(parsed, indent=4, sort_keys=True)
It seems that I made a small typo when defining my LogFormatter string: by accidentally escaping the newline character, I wrongly assumed that writing multi-line output to a log file was not possible.
Cheers to #Barafu for pointing this out (which is why I assigned him the correct answer).
Here's the sample code:
import logging
lf = logging.Formatter('%(levelname)-8s - %(message)s\n%(detail)s')
lh = logging.FileHandler(filename=r'c:\temp\test.log')
lh.setFormatter(lf)
log = logging.getLogger()
log.setLevel(logging.DEBUG)
log.addHandler(lh)
log.debug('test', extra={'detail': 'This is a multi-line\ncomment to test the formatter'})
The resulting output would look like this:
DEBUG - test
This is a multi-line
comment to test the formatter
Caveat:
If there is no detail information to log, and you pass an empty string, the logger will still output a newline. Thus, the remaining question is: how can we make this conditional?
One approach would be to update the logging formatter before actually logging the information, as described here.
I want to output some strings to a log file and I want the log file to be continuously updated.
I have looked into the logging module pf python and found out that it is
mostly about formatting and concurrent access.
Please let me know if I am missing something or amy other way of doing it
usually i do the following:
# logging
LOG = "/tmp/ccd.log"
logging.basicConfig(filename=LOG, filemode="w", level=logging.DEBUG)
# console handler
console = logging.StreamHandler()
console.setLevel(logging.ERROR)
logging.getLogger("").addHandler(console)
The logging part initialises logging's basic configurations. In the following I set up a console handler that prints out some logging information separately. Usually my console output is set to output only errors (logging.ERROR) and the detailed output in the LOG file.
Your loggings will now printed to file. For instance using:
logger = logging.getLogger(__name__)
logger.debug("hiho debug message")
or even
logging.debug("next line")
should work.
Doug Hellmann has a nice guide.
To add my 10cents with regards to using logging. I've only recently discovered the Logging module and was put off at first. Maybe just because it initially looks like a lot of work, but it's really simple and incredibly handy.
This is the set up that I use. Similar to Mkinds answer, but includes a timestamp.
# Set up logging
log = "bot.log"
logging.basicConfig(filename=log,level=logging.DEBUG,format='%(asctime)s %(message)s', datefmt='%d/%m/%Y %H:%M:%S')
logging.info('Log Entry Here.')
Which will produce something like:
22/09/2015 14:39:34 Log Entry Here.
You can log to a file with the Logging API.
Example: http://docs.python.org/2/howto/logging.html#logging-to-a-file
I use the Python logging framework with default settings.
For some data compare reason:I have to compare the log with other data output.
But the python log begin with a default, something like:
INFO:root:post params in transmitter
Can I set the python log output without INFO:root:, like:
post params in transmitter
with my own log only?
Thx a lot!
Sure thing. You could set the format to watever you like:
format: '%(message)s'
Like this:
logging.basicConfig(format='%(message)s', ...)
See the doc for more info: http://docs.python.org/library/logging.config.html
Those "INFO:..." or "DEBUG:..." appear there because some handler defines that. My guess: the default handler is still there.
You can check it by taking a peek at logger.handlers right after you created it.
logger = logging.getLogger()
logger.handlers = [] # This is the key thing for the question!
# Start defining and assigning your handlers here
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(asctime)s: %(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
Also, you could just override the format for that default handler:
if (len(logger.handlers) > 0): # Check here whatever. Or try/except. You get the idea...
formatter = logging.Formatter("%(asctime)s: %(levelname)s - %(message)s")
logger.handlers[0].setFormatter(formatter)
I am not a Python expert so maybe there is a better way to remove or even not create that default handler but this works pretty well for me.
Note: As stated in the docs, the .basicConfig is useful for simple loggers. If you have multiple streams, with multiple formats, it does not work as far as I know and you need to go with the custom handlers.
endDate = '2015-07-24'
logging.basicConfig(filename="win" + strftime("%Y%m%d", localtime()) + ".txt", level=logging.DEBUG,format='%(message)s')
Use a Formatter.
Your really dont need to get into removing INFO word.. . (it shall really help you when your code would more messy and you would be using more stuff than just info like debugging exception etc)
If you want to compare you data with that data you can do something like skipping first 10 character (INFO:ROOT:) and then do whatever you feel like. Umm something like this:
f = open("my.log","a+")
lines = f.readlines()
for line in lines:
if(line[10:] == my_output):
do_whatever_you_feel_like()