We Keep Coding

logging information is being sent to more files than specified by the handler

In an attempt to learn the Python module logging I made the small script below.
However, whenever I use the object my_logger it outputs to the file my_logger.log as specified by the file handler, but it also outputs the same text to the previously specified file in basicConfig log.log. My question is, why is it outputting to both locations instead of just the file specified by the FileHandler?
import logging
logging.basicConfig(level=logging.INFO,
filename='log.log',
filemode="w",
format="%(levelname)s - %(message)s")
logging.debug("debug_message") # Lowest |
logging.info("info_message") # |
logging.warning("warning_message") # |
logging.error("error_message") # |
logging.critical("critical_message") # Highest V
my_logger = logging.getLogger('My_Logger')
my_logger.info("Successfully created my custom logger")
handler = logging.FileHandler("my_logger.log")
formatter = logging.Formatter("%(name)s: %(levelname)s - %(message)s")
handler.setFormatter(formatter)
my_logger.addHandler(handler)
my_logger.info("Successfully created My Logger!")
try:
1/0
except ZeroDivisionError as e:
my_logger.exception("ZeroDivisionError")

You have two loggers in your script. One is the root logger that is setup using basiConfig and accessed using logging keyword itself. Another logger is "My_Logger" which is a child of root logger. Both loggers are active in your script and are instructed to write to specified handlers.
That's why you are getting logs written in both files.
If you want to use a customized handler than don't use basicConfig. Use only one logger in the whole of your module.

Perplexed by Python logging

I started using Python logging to tidy up the messages generated by some code. I have the statement
logging.basicConfig(filename='my_log.log', filemode='w', encoding='utf-8', level=logging.INFO)
in the main Python module and was expecting the log file to be created from scratch every time the program runs. However, Python appends the new messages to the existing file. I wrote the program below to try to test what is going on but now I don't get a log file created and the messages appear in the console (Spyder 5). I am running WinPython 3.9 on a Windows 10 machine.
import datetime
import logging
logging.basicConfig(filename='my_log.log', filemode='w', level=logging.INFO)
t = datetime.datetime.now()
logging.info("Something happened at %s" % t)
All suggestions welcome - thanks!
Update: following #bzu's suggestion, I changed my code to this:
import datetime
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
handler = logging.FileHandler('my_log.log', 'w')
logger.addHandler(handler)
t = datetime.datetime.now()
logging.info("Something happened at %s" % t)
This created the log file but nothing was written to it. If I change the final line to
logger.info("Something happened at %s" % t)
the the message appears in the log file. However, I want to log the messages from different modules which import logging and so know about logging in general but I don't know how they can access the specific logger object

If you run the program with plain python (python log_test_file.py), it should work as expected.
I suspect that the problem with Spyder is, that it overrides the logging module config.
To avoid this kind of problems, the recommended way is to use custom logger per file:
import logging
logger = logging.getLogger(__name__)
# boilerplate: can be converted into a method and reused
logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler = logging.FileHandler(filename='my_log.log', mode='w', encoding='utf-8')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("Something happened")

After some searching on the web, adding force=True to logging.basicConfig seemed to solve the problem ie:
logging.basicConfig(filename='my_log.log', filemode='w', encoding='utf-8', level=logging.INFO, force=True)

How can I get coloredlogs in PyCharm Python Console to format properly?

My logging setup is:
import coloredlogs
import logging
import sys
# Create a logger object.
# logger = logging.getLogger(__name__)
# By default the install() function installs a handler on the root logger,
# this means that log messages from your code and log messages from the
# libraries that you use will all show up on the terminal.
coloredlogs.install(level='DEBUG')
logging.basicConfig(
format='%(asctime)s %(levelname)-8s %(message)s',
level=logging.INFO,
stream=sys.stdout,
datefmt='%Y-%m-%d %H:%M:%S')
If I configure the console to use python for output, all lines start at column 0, but all output is red. But if I specify use terminal, the colors are there, but the lines don't start at column 1. They all start with the end column of the line before.
How can I get all log messages starting at column 0 AND in color?

Try adding isatty to your call to install. This overrides the whatever auto-detection is doing to attempt at determining what type of terminal is being used.
import coloredlogs
import logging
import sys
logger = logging.getLogger(__name__)
coloredlogs.install(level=logging.DEBUG, logger=logger, isatty=True,
fmt="%(asctime)s %(levelname)-8s %(message)s",
stream=sys.stdout,
datefmt='%Y-%m-%d %H:%M:%S')
logger.debug("this is a debugging message")
logger.info("this is an informational message")
logger.warning("this is a warning message")
logger.error("this is an error message")
logger.critical("this is a critical message")
explanation:
If I understood the problem correctly, you are seeing the logging handler default to using sys.stderr instead of sys.stdout. (this is why the text would appear in red)
There are likely two issues going on.
coloredlogs.install needs to be told what type of rules you want in the handler, not logging.basicConfig.
The automatic terminal detection would fail unless you either force the formatter to think it's a terminal or you set pycharm to 'simulate' a terminal. Fixed by passing in stream=sys.stdout

I can't get Python logging module to write to the log file from different module

I have a project that consists of several modules. There is main module (main.py) that creates a TK GUI and loads the data. It passes this data to process.py which processes the data using functions from checks.py. I am trying to implement logging for all the modules to log to a file. In the main.py log messages are written to the log file but in the other modules they are only written to the console. I assume its to do with the import module line executing part of the code before the code in main.py has set up the logger, but i can't work out how to arrange it to avoid that. It seems like a reasonably common question on Stackoverflow, but i couldn't get the other answers to work for me. I am sure I am not missing much. Simplified code is shown below:
Moving the logging code inside and outside of various functions in the modules. The code I used to start me off is the code from Corey Schaffer's Youtube channel.
Main.py
import logging
from process import process_data
def main():
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s:%(name)s:%(message)s')
templogfile = tempfile.gettempdir() + '\\' + 'TST_HA_Debug.log'
file_handler = logging.FileHandler(templogfile)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)
logger.addHandler(file_handler)
logger.addHandler(stream_handler)
logger.debug('Logging has started') # This gets written to the file
process_data(data_object) # call process_data in process.py
process.py
import logging
def process_data(data):
logger = logging.getLogger(__name__)
logger.debug('This message is logged by process') #This wont get written to the log file but get written to the console
#do some stuff with data here and log some msgs
return
Main.py will write to the log file, process.py will only write to the console.

I've rewritten your scripts a little so that this code can stand alone. If I changed this too much let me know and I can revisit it. These two files are an example of having it log to file. Note my comments:
## main.py
import logging
from process import process_data
import os
def main():
# Give this logger a name
logger = logging.getLogger("Example")
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s:%(name)s:%(message)s')
# I changed this to the same directory, for convenience
templogfile = os.path.join(os.getcwd(), 'TST_HA_Debug.log')
file_handler = logging.FileHandler(templogfile)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)
logger.addHandler(file_handler)
logger.addHandler(stream_handler)
logging.getLogger("Example").debug('Logging has started') # This still gets written to the file
process_data() # call process_data in process.py
if __name__ == '__main__':
main()
## process.py
import logging
def process_data(data=None):
# make sure to grab the correct logger
logger = logging.getLogger("Example")
logger.debug('This message is logged by process') # This does get logged to file now
#do some stuff with data here and log some msgs
return
Why does this work? Because the module-level functions use the default root logger, which is not the one you've configured. For more details on this see these docs. There is a similar question that goes more into depth here.
By getting the configured logger before you start logging, you are able to log to the right configuration. Hope this helps!

Logs not printing in file in python

I am trying to print logs using logger module in python.
Following is the code I am keeping on the top of file.
if __name__ == '__main__':
LOG_FILENAME = '/home/akash/exdion-pdf-extracter/doc/epod.log'
logging.basicConfig(
filename=LOG_FILENAME,
level=logging.DEBUG,
)
There are different files with function calls from one another. I have used the following line to display a line in the logger.
#staticmethod
def initiate_pdf_processing(ct_doc, pt_doc, feature, startAndEndKeyList):
logging.info("testing logger")
...
There are multiple instances of the similar above logger function. But I can't receive the logger output in the designated file. The code and files are huge. However there are a few error generated by the code which are getting printed in the log file.

Use below code bit out of the main namespace. This way, you are defining a logger and creating a log file as global file, and you can call the logger anywhere in the code. A logger code bit below is how I usually code.
logfile = '<your_file_name>.log'
if(os.path.isfile(logfile)):
os.remove(logfile)
file_handler = logging.FileHandler(logfile)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter(
'%(asctime)s %(pathname)s [%(process)d]: %(levelname)s:: %(message)s'))
logger = logging.getLogger('wbs-server-log')
logger.setLevel(logging.DEBUG)
logger.addHandler(file_handler)

The issue might be that you have to initialize logging above if __name__ == '__main__' block. That way logging will be initialized when you import this as module.
Suggestion for initializing logging:
import logging
log = logging.getLogger(PACKAGE_NAME)
stream_handler = logging.StreamHandler(stream=open(LOG_FILE_NAME, 'a'))
stream_handler.setLevel(logging.DEBUG)
log.addHandler(stream_handler)
log.debug('your message here')
After this you can tweak log message formatting with logging.Formatter.