We Keep Coding

Logs not printing in file in python

I am trying to print logs using logger module in python.
Following is the code I am keeping on the top of file.
if __name__ == '__main__':
LOG_FILENAME = '/home/akash/exdion-pdf-extracter/doc/epod.log'
logging.basicConfig(
filename=LOG_FILENAME,
level=logging.DEBUG,
)
There are different files with function calls from one another. I have used the following line to display a line in the logger.
#staticmethod
def initiate_pdf_processing(ct_doc, pt_doc, feature, startAndEndKeyList):
logging.info("testing logger")
...
There are multiple instances of the similar above logger function. But I can't receive the logger output in the designated file. The code and files are huge. However there are a few error generated by the code which are getting printed in the log file.

Use below code bit out of the main namespace. This way, you are defining a logger and creating a log file as global file, and you can call the logger anywhere in the code. A logger code bit below is how I usually code.
logfile = '<your_file_name>.log'
if(os.path.isfile(logfile)):
os.remove(logfile)
file_handler = logging.FileHandler(logfile)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter(
'%(asctime)s %(pathname)s [%(process)d]: %(levelname)s:: %(message)s'))
logger = logging.getLogger('wbs-server-log')
logger.setLevel(logging.DEBUG)
logger.addHandler(file_handler)

The issue might be that you have to initialize logging above if __name__ == '__main__' block. That way logging will be initialized when you import this as module.
Suggestion for initializing logging:
import logging
log = logging.getLogger(PACKAGE_NAME)
stream_handler = logging.StreamHandler(stream=open(LOG_FILE_NAME, 'a'))
stream_handler.setLevel(logging.DEBUG)
log.addHandler(stream_handler)
log.debug('your message here')
After this you can tweak log message formatting with logging.Formatter.

Changing log to 'DEBUG' level doesn't output to console like it did before I changed it to 'WARNING; in Pythons Logging module

I am using the following code within a class to initiate a logger in Python.
...
self.logger_name=os.path.join(os.getcwd(),'HMM.log')
self.logger=self._get_logger()
self.initial_log()
def _get_logger(self):
childLogger = logging.getLogger(__name__)
childLogger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
cname=self.logger_name[:-4]+ now.strftime("%Y-%m-%d_%H-%M") +'.log'
formatter = logging.Formatter('%(asctime)s %(levelname)s %(funcName)s: %(message)s')
childHandler=logging.FileHandler(cname,mode='w')
childHandler.setFormatter(formatter)
childLogger.addHandler(childHandler)
childLogger.propagate = True
return childLogger
...
The level was set to logging.DEBUG (second line down in _get_logger()) and worked fine, (i.e. printed the log to the console as well as to file). I then changed it to logging.WARNING for a little while. Now that I want to change it back I no longer get a this information printed to console and only to file. Can anybody tell me why this has happened?

Logging module not writing to file

I'm using logging module, and I've passed in the same parameters that I have on other jobs that are currently working:
import logging
from inst_config import config3
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s [%(levelname)s] - %(message)s',
filename=config3.GET_LOGFILE(config3.REQUESTS_USAGE_LOGFILE))
logging.warning('This should go in the file.')
if __name__ == '__main__':
logging.info('Starting unload.')
Using this method to create the filename:
REQUESTS_USAGE_LOGFILE = r'C:\RunLogs\Requests_Usage\requests_usage_runlog_{}.txt'.format(
CUR_MONTH)
def GET_LOGFILE(logfile):
"""Truncates file and returns."""
with open(logfile, 'w'):
pass
return logfile
When I run it, however, it is creating the file, and then still outputting the logging info to the console. I'm running in Powershell.
Just tried putting it inside the main statement like this:
if __name__ == '__main__':
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s [%(levelname)s] - %(message)s',
filename=config3.GET_LOGFILE(config3.REQUESTS_USAGE_LOGFILE))
logging.warning('This should go in the file.')
Still no luck.

I add the following lines before the logging.basicConfig() and it worked for me.
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)

You can try running this snippet in your main file.
import logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s [%(levelname)s] - %(message)s',
filename='filename.txt') # pass explicit filename here
logger = logging.get_logger() # get the root logger
logger.warning('This should go in the file.')
print logger.handlers # you should have one FileHandler object

In addition to Forge's answer on using logging.basicConfig(), as of Python 3.8 a parameter to basicConfig() got added. To quote the docs:
"""
This function does nothing if the root logger already has handlers
configured, unless the keyword argument *force* is set to ``True``.
...
force If this keyword is specified as true, any existing handlers
attached to the root logger are removed and closed, before
carrying out the configuration as specified by the other
arguments.
"""
This is why yue dong's answer to remove all handlers has worked for some as well as Alexander's answer to reset logger.handlers to [].
Debugging logger.handlers (as Forge's answer suggests) led me to see a single StreamHandler there (so basicConfig() did nothing for me until I used force=True as parameter).
Hope that helps in addition to all the other answers here too!

If you are using 'root' logger which is by default has name "", than you can do this trick:
logging.getLogger().setLevel(logging.INFO)
logger = logging.getLogger('')
logger.handlers = []
In addition you may want to specify logging level as in code above, this level will persist for all descendant loggers.
If instead, you specified particular logger, than do
logger = logging.getLogger('my_service')
logger.handlers = []
fh = logging.FileHandler(log_path)
fh.setLevel(logging.INFO)
# create console handler
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
logger.addHandler(fh)
logger.addHandler(ch)
logger.info('service started')
The above code will create new logger 'my_service'. In case that logger has been already created it clears all handles. That it adds handles for writing in specified file and console. See official documentation as well.
You can also use hierarchical loggers. It is done directly.
logger = logging.getLogger('my_service.update')
logger.info('updated successfully')

Scrapy: logging to a file without ScrapyFileLogObserver()

Apparently, I shouldn't be using ScrapyFileLogObserver anymore (http://doc.scrapy.org/en/1.0/topics/logging.html). But I still want to be able to save my log messages to a file, and I still want all the standard Scrapy console information to be saved to the file too.
From reading up on how to use the logging module, this is the code that I have tried to use:
class BlahSpider(CrawlSpider):
name = 'blah'
allowed_domains = ['blah.com']
start_urls = ['https://www.blah.com/blahblahblah']
rules = (
Rule(SgmlLinkExtractor(allow=r'whatever'), callback='parse_item', follow=True),
)
def __init__(self):
CrawlSpider.__init__(self)
self.logger = logging.getLogger()
self.logger.setLevel(logging.DEBUG)
logging.basicConfig(filename='debug_log.txt', filemode='w', format='%(asctime)s %(levelname)s: %(message)s',
level=logging.DEBUG)
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
simple_format = logging.Formatter('%(levelname)s: %(message)s')
console.setFormatter(simple_format)
self.logger.addHandler(console)
self.logger.info("Something")
def parse_item(self):
i = BlahItem()
return i
It runs fine, and it saves the "Something" to the file. However, all of the stuff that I see in the command prompt window, all of the stuff that used to be saved to the file when I used ScrapyFileLogObserver, is not saved now.
I thought that my "console" handler with "logging.StreamHandler()" was supposed to deal with that, but this is just what I had read and I don't really understand how it works.
Can anyone point out what I am missing or where I have gone wrong?
Thank you.

I think the problem is that you've used both basicConfig and addHandler.
Configure two handlers separately:
self.logger = logging.getLogger()
self.logger.setLevel(logging.DEBUG)
logFormatter = logging.Formatter('%(asctime)s %(levelname)s: %(message)s')
# file handler
fileHandler = logging.FileHandler("debug_log.txt")
fileHandler.setLevel(logging.DEBUG)
fileHandler.setFormatter(logFormatter)
self.logger.addHandler(fileHandler)
# console handler
consoleHandler = logging.StreamHandler()
consoleHandler.setLevel(logging.DEBUG)
consoleHandler.setFormatter(logFormatter)
self.logger.addHandler(consoleHandler)
See also:
logger configuration to log to file and print to stdout

you can log all scrapy logs to file by first disabling root handle in scrapy.utils.log.configure_logging and then adding your own log handler.
In settings.py file of scrapy project add the following code:
import logging
from logging.handlers import RotatingFileHandler
from scrapy.utils.log import configure_logging
LOG_ENABLED = False
# Disable default Scrapy log settings.
configure_logging(install_root_handler=False)
# Define your logging settings.
log_file = '/tmp/logs/CRAWLER_logs.log'
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
rotating_file_log = RotatingFileHandler(log_file, maxBytes=10485760, backupCount=1)
rotating_file_log.setLevel(logging.DEBUG)
rotating_file_log.setFormatter(formatter)
root_logger.addHandler(rotating_file_log)
Also we customize log level (DEBUG to INFO) and formatter as required.
Hope this helps!

Global Python Logger with file Rotator

I have create a global logger using the following:
def logini():
logfile='/var/log/cs_status.log'
import logging
import logging.handlers
global logger
logger = logging.getLogger()
logging.basicConfig(filename=logfile,filemode='a',format='%(asctime)s %(name)s %(levelname)s %(message)s',datefmt='%y%m%d-%H:%M:%S',level=logging.DEBUG,propagate=0)
handler = logging.handlers.RotatingFileHandler(logfile, maxBytes=2000000, backupCount=5)
logger.addHandler(handler)
__builtins__.logger = logger
It works, however I am getting 2 outputs for every log, one with the formatting and one without.
I realize that this is being caused by the file rotater as I can comment out the 2 lines of the handler code and then I get a single outputted correct log entry.
How can I prevent the log rotator from outputting a second entry ?

Currently you're configuring two file loggers that point to the same logfile. To only use the RotatingFileHandler, get rid of the basicConfig call:
logger = logging.getLogger()
handler = logging.handlers.RotatingFileHandler(logfile, maxBytes=2000000,
backupCount=5)
formatter = logging.Formatter(fmt='%(asctime)s %(name)s %(levelname)s %(message)s',
datefmt='%y%m%d-%H:%M:%S')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
All basicConfig does for you is to provide an easy way to instantiate either a StreamHandler (default) or a FileHandler and set its loglevel and formats (see the docs for more information). If you need a handler other than these two, you should instantiate and configure it yourself.