We Keep Coding

Create different log files in python using logging

I am trying to create logs for errors. This is the logger i am using.
import logging
import os
def create_log(source):
logging.basicConfig(filename="logs/"+source+".log",
format='%(asctime)s::%(levelname)s::%(message)s',
filemode='a')
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def info_logger(message):
logger.info(message)
def error_logger(message):
print(message)
logger.error(message)
I am calling this logger in a for loop where i am doing some operation and trying to create logs for each iteration
for i in data["source_id"]:
--Some task here--
log_file_name = str(source_dict["source_id"]) + "_" + source_dict["source_name"] + "_"+str(datetime.today().strftime("%Y-%m-%d_%H_%M_%S"))
create_log(log_file_name)
for the first iteration, log file is getting created. But for other iterations, the same log file is getting appended. I want to make seperate log files for each iteration. Any idea how can i do that?

You can try this
import logging
debug = logging.FileHandler("debug.log")
debug.setLevel(logging.DEBUG)
error = logging.FileHandler("error.log")
error.setLevel(logging.ERROR)
warning = logging.FileHandler("warning.log")
warning.setLevel(logging.WARNING)
console = logging.StreamHandler()
logging.basicConfig( # noqa
level=logging.INFO,
format="[%(asctime)s]:%(levelname)s %(name)s :%(module)s/%(funcName)s,%(lineno)d: %(message)s",
handlers=[debug, error, warning, console]
)
logger = logging.getLogger()
logger.debug("This is debug [error+warning]")
logger.error("This is error [error only]")
logger.warning("This is warn [error+warning]")

Logs not printing in file in python

I am trying to print logs using logger module in python.
Following is the code I am keeping on the top of file.
if __name__ == '__main__':
LOG_FILENAME = '/home/akash/exdion-pdf-extracter/doc/epod.log'
logging.basicConfig(
filename=LOG_FILENAME,
level=logging.DEBUG,
)
There are different files with function calls from one another. I have used the following line to display a line in the logger.
#staticmethod
def initiate_pdf_processing(ct_doc, pt_doc, feature, startAndEndKeyList):
logging.info("testing logger")
...
There are multiple instances of the similar above logger function. But I can't receive the logger output in the designated file. The code and files are huge. However there are a few error generated by the code which are getting printed in the log file.

Use below code bit out of the main namespace. This way, you are defining a logger and creating a log file as global file, and you can call the logger anywhere in the code. A logger code bit below is how I usually code.
logfile = '<your_file_name>.log'
if(os.path.isfile(logfile)):
os.remove(logfile)
file_handler = logging.FileHandler(logfile)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter(
'%(asctime)s %(pathname)s [%(process)d]: %(levelname)s:: %(message)s'))
logger = logging.getLogger('wbs-server-log')
logger.setLevel(logging.DEBUG)
logger.addHandler(file_handler)

The issue might be that you have to initialize logging above if __name__ == '__main__' block. That way logging will be initialized when you import this as module.
Suggestion for initializing logging:
import logging
log = logging.getLogger(PACKAGE_NAME)
stream_handler = logging.StreamHandler(stream=open(LOG_FILE_NAME, 'a'))
stream_handler.setLevel(logging.DEBUG)
log.addHandler(stream_handler)
log.debug('your message here')
After this you can tweak log message formatting with logging.Formatter.

Python Logger object is not callable

I'm trying to write a module to use it in different scripts
import logging
from logging.handlers import RotatingFileHandler
_logger_name = "Nagios"
_print_format = "%(asctime)s - %(levelname)s - %(message)s"
_level = logging.DEBUG
class Log():
def __init__(self,log_file,logger_name=_logger_name,level=_level):
self.log_file = log_file
self.logger_name = logger_name
self.level = level
def getLog(self):
"""
Return the logging object
"""
_logger = logging.getLogger(self.logger_name)
_logger.setLevel(self.level)
_logger.addHandler(self._rotateLog())
return _logger
def _rotateLog(self):
"""
Rotating the log files if it exceed the size
"""
rh = RotatingFileHandler(self.log_file,
maxBytes=20*1024*1024, backupCount=2)
formatter = logging.Formatter(_print_format)
rh.setFormatter(formatter)
return rh
log = Log("kdfnknf").getLog()
log("hello")
I see the following error:
Traceback (most recent call last):
File "nagiosLog.py", line 45, in <module>
log("hello")
TypeError: 'Logger' object is not callable
Any idea why I'm getting this error,
When debugged using pdb I do see it returns the object and printing the dir(log) I don't see the Logger module in it.
Am I missing something here

log("Hello")
This is wrong.
Correct is
log.info("Hello")
log must be printed with logging level i.e. info/error/warning

See the logging docs:
You have to use a function, you can't just call Logger:
Logger.info(msg, *args, **kwargs)
Logs a message with level INFO on
this logger. The arguments are interpreted as for debug().
or
Logger.warning(msg, *args, **kwargs)
Logs a message with level WARNING on this logger. The arguments are >interpreted as for debug().
so instead, do:
log.info("Test info level logging...")

logging error in python

I have been writing simple scripts and I am trying to use logger to generate log for each functions in the scripts.
1) based on the function name I create a logger filehandler and I try to put logs using that handler. I also delete the previous existing file with the same name.
3) at the end of the function I close the handler.
My problem are:
1)even though I close the handler, the next time I run the same function I get an error that the file I am trying to delete is (as a part of setting the logger file handler) is still being used.
2) Also the logger prints everything to console which I dont want, I just want it to write everything to the file.
Here are the logger functions:
def setLogger(path):
"""
#purpose: Intializes basic logging directory and file
"""
LOG_FILENAME = path + "\\" + "log.txt"
#logging.basicConfig(filename=LOG_FILENAME,
# format='%(levelname)s %(asctime)s %(message)s',level=logging.INFO
# )
logger = logging.getLogger()
logger.setLevel(logging.INFO)
file_handler = logging.FileHandler(LOG_FILENAME)
file_handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(levelname)s %(asctime)s %(message)s")
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
return logger
def unsetLogger(logger):
"""
#purpose: performs a basic shutdown of logger
"""
logger.handlers[0].close()
logger.removeHandler(logger.handlers[0])
logging.shutdown
The way i use them is:
for eg:
def fun():
os.remove(path)
logger = setLogger(path)
` logging.info("hi") #this writes to file and prints on the console as well
unsetLogger(logger)
if I run the function fun() once, its all good. but if i run it again, I get that can't delete error for the log file.
Thanks in Advance.
learningNinja

After making some slight modifications, I came up with the following test to try to reproduce your error, but I don't get any errors.
import os
import logging
def setLogger(path):
"""
#purpose: Intializes basic logging directory and file
"""
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Simplified log file path (I just use full value passed in, and don't append "\log.txt")
file_handler = logging.FileHandler(path)
file_handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(levelname)s %(asctime)s %(message)s")
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
return logger
def unsetLogger(logger):
"""
#purpose: performs a basic shutdown of logger
"""
logger.handlers[0].close()
logger.removeHandler(logger.handlers[0])
logging.shutdown
def fun():
try:
# Was getting error trying to remove a file that didn't exist on
# first execution...
os.remove("log.txt")
except:
pass
logger = setLogger("log.txt")
logging.info("hi")
unsetLogger(logger)
fun()
fun()
fun()
See if there is anything I'm doing differently than your actual code and maybe that might help you.

Sharing external module/function between own classes

Question about sharing 'functions' between classes.
Situation:
All my own code is in 1 file
I'm using python-daemon to daemonize my script
That uses a class (Doorcamdaemon) to initiate and run.
It imports another class (Doorcam) which has a looping function
I'm using a sample script for the daemon functions, and it shows how to use the logging module.
The logging works from the main part of the script and in the Doorcamdaemon class, but not in the Doorcam class.
class Doorcamdaemon():
def __init__(self):
#skipping some content, not related to this issue
self.Doorcam=Doorcam()
def run(self):
self.Doorcam.startListening() #looping function
class Doorcam
def __init__(self):
#skipping somecontent, not related to this issue
def startListening(self):
while True:
logger.info('Hello')
app = Doorcamdaemon()
logger = logging.getLogger("DoorcamLog")
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler = logging.FileHandler("/var/log/doorcam.log")
handler.setFormatter(formatter)
logger.addHandler(handler)
daemon_runner = runner.DaemonRunner(app)
daemon_runner.daemon_context.files_preserve=[handler.stream]
daemon_runner.do_action()
The error returned is:
$ ./Doorcam.py start
Traceback (most recent call last):
File "./Doorcam.py", line 211, in <module>
app = Doorcamdaemon()
File "./Doorcam.py", line 159, in __init__
self.doorcam=Doorcam()
File "./Doorcam.py", line 18, in __init__
logger.info('Doorcam started capturing')
NameError: global name 'logger' is not defined
So my obvious question: How can I make it work in the Doorcam class as well?

Try moving the line
app = Doorcamdaemon()
to after the lines that create and set up logger. The traceback is telling you:
logger doesn't exist in line 18 where Doorcam's constructor tries to use it
Doorcamdaemon tries to construct a Doorcam at line 159 in its own constructor
So you can't create a Doorcamdaemon if logger isn't defined yet.
Some of the content you omitted in Doorcam.__init__ was related to this issue :)