I'm running a Celery task that executes a function. This function generates some logging information. Using the get_task_logger logger, I am able print the logging information to the Celery stdout.
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
def my_func_called_inside_a_task():
logger.debug("SOME OUTPUT HERE")
However, I also want to import this function as a normal python script (not using Celery) and log to, for example, stdout. Normally, I might do something like the following:
import logging
logger = logging.getLogger(__name__)
def my_func_called_inside_a_task():
logger.debug("SOME OUTPUT HERE")
How do I combine both approaches so I don't have to do something redundant like the following?
import logging
from celery.utils.log import get_task_logger
logger = logging.getLogger(__name__)
logger_celery = get_task_logger(__name__)
def my_func_called_inside_a_task():
logger.debug("SOME OUTPUT HERE")
logger_celery.debug("SOME OUTPUT HERE")
Summary:
If I call the function from a celery task, I'd like it to log to the celery worker stdout. If I call the function from a normal Python prompt, it would use the normal Python logger. Any help is much appreciated.
You can pass an optional argument to that function.
import logging
from celery.utils.log import get_task_logger
def my_func_called_inside_a_task(use_celery_logger=None):
if use_celery_logger:
logger = get_task_logger(__name__)
else:
logger = logging.getLogger(__name__)
logger.debug("SOME OUTPUT HERE")
and in Your celery task call it as
my_func_called_inside_a_task(use_celery_logger=True)
and for normal logging you can call it as it is
my_func_called_inside_a_task()
Related
I have 3 classes, a main class that is called App, a secondary class that have some business rules and another class that has some useful functions called Utils.
The main class, App has a method that run another method of the secondary class in parallel, this method of the secondary class uses a method of the class Utils to generate logs, but I think that because of the context manager that I use to run processeses in parallel, the logging module loses his configurations. See:
class App:
otherClass = OtherClass()
utils = Utils()
n_cpus = os.cpu_count()
def call_multiprocessing(self):
#When I run this way the logging messages goes to the console and not to the file
#specified in basic config
with concurrent.futures.ProcessPoolExecutor(max_workers=self.n_cpus) as executor:
co_routines = executor.map(self.otherClass.some_method(), self.list_of_parameters)
class OtherClass()
utils = Utils()
utils.setup_log_folder(folder_name)
def some_method(self, some_parameter):
for a in some_parameter:
#DO SOME STUFF
self.utils.generate_log(some_message)
import logging
class Utils:
def setup_log(folder):
# if someone tried to log something before basicConfig is called, Python creates a default handler that
# goes to the console and will ignore further basicConfig calls. So I Remove the handler if there is one.
root = logging.getLogger()
if root.handlers:
for handler in root.handlers:
root.removeHandler(handler)
logging.basicConfig(
format="%(asctime)s [%(levelname)s] %(message)s",
datefmt='%d/%m/%Y %H:%M:%S',
handlers=[
logging.FileHandler(os.path.join(folder, 'log.txt'), 'w', 'utf-8'),
]
)
The file log.txt is created, but the log messages are shown in console. How can I solve that?
Given this code I try to have log statements working but I am not able to. The documentation tells me that I do not have to set a level.
When a logger is created, the level is set to NOTSET (which causes all
messages to be processed when the logger is the root logger, or
delegation to the parent when the logger is a non-root logger).
But it did not work without. Therefore I tried to set it to debug. But still no luck.
"""
Experimental Port Fowarding
"""
import logging
def main(config):
""" entry point"""
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
log.debug("opening config file...")
config_file = open(config, 'r')
log.debug("config found!")
The logger you are getting doesn't have any handlers. You can check this by doing print(log.handlers) and seeing the output is an empty list ([]).
The simplest way to use the logging library is something like this, where you call logging.basicConfig to set everything up, as shown in the logging module basic tutorial:
"""
Experimental Port Fowarding
"""
import logging
logging.basicConfig(level=logging.DEBUG)
def main(config):
""" entry point"""
logging.debug("opening config file...")
config_file = open(config, 'r')
logging.debug("config found!")
main('test.conf')
This works for me from outside and inside IPython.
If you want to avoid basicConfig for some reason you need to register a handler manually, like this:
import logging
def main(config):
""" entry point"""
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
# Minimal change: add StreamHandler to display to stdout
log.addHandler(logging.StreamHandler())
log.debug("opening config file...")
config_file = open(config, 'r')
log.debug("config found!")
By default the logger writes to STDERR stream which usually prints to the console itself.
Basically you can change the log file path by setting:
logging.basicConfig(filename="YourFileName.log")
log = logging.getLogger(__name__)
I want to find out how logging should be organised given that I write many scripts and modules that should feature similar logging. I want to be able to set the logging appearance and the logging level from the script and I want this to propagate the appearance and level to my modules and only my modules.
An example script could be something like the following:
import logging
import technicolor
import example_2_module
def main():
verbose = True
global log
log = logging.getLogger(__name__)
logging.root.addHandler(technicolor.ColorisingStreamHandler())
# logging level
if verbose:
logging.root.setLevel(logging.DEBUG)
else:
logging.root.setLevel(logging.INFO)
log.info("example INFO message in main")
log.debug("example DEBUG message in main")
example_2_module.function1()
if __name__ == '__main__':
main()
An example module could be something like the following:
import logging
log = logging.getLogger(__name__)
def function1():
print("printout of function 1")
log.info("example INFO message in module")
log.debug("example DEBUG message in module")
You can see that in the module there is minimal infrastructure written to import the logging of the appearance and the level set in the script. This has worked fine, but I've encountered a problem: other modules that have logging. This can result in output being printed twice, and very detailed debug logging from modules that are not my own.
How should I code this such that the logging appearance/level is set from the script but then used only by my modules?
You need to set the propagate attribute to False so that the log message does not propagate to ancestor loggers. Here is the documentation for Logger.propagate -- it defaults to True. So just:
import logging
log = logging.getLogger(__name__)
log.propagate = False
I want to create a Python logging object in my main program and have logging in both my main program and in the modules it uses at the same logging level. The basic example given in logging documentation is essentially as follows:
main.py:
import logging
import mylib
def main():
logging.basicConfig(level = logging.INFO)
logging.info('Started')
mylib.do_something()
logging.info('Finished')
if __name__ == '__main__':
main()
mylib.py:
import logging
def do_something():
logging.info('Doing something')
This works fine. I am not sure, however, of how to get a Python logging object doing something similar. The following, for example, does not work:
main.py:
import logging
import mylib
def main():
verbose = True
global log
log = logging.getLogger(__name__)
if verbose:
log.setLevel(logging.INFO)
else:
log.setLevel(logging.DEBUG)
log.info('Started')
mylib.do_something()
log.info('Finished')
if __name__ == '__main__':
main()
mylib.py:
import logging
def do_something():
log.info('Doing something')
It does not work because the global log object is not recognised in the mylib.py module. How should I be doing this? My two main goals are
to have the logging level that is set in my main program propagate through to any modules used and
to be able to use log for logging, not "logging" (i.e. log.info("alert") as opposed to logging.info("alert")).
Your application should configure the logging once (with basicConfig or see logging.config) and then in each file you can have:
import logging
log = logging.getLogger(__name__)
# further down:
log.info("alert")
So you use a logger everywhere and you don't directly access the logging module in your codebase. This allows you to configure all your loggers in your logging configuration (which, as said earlier, is configured once and loaded at the beginning of your application).
You can use a different logger in each module as follows:
import logging
LOG = logging.getLogger(__name__)
# Stuff
LOG.info("A log message")
I have implemented a Celery task which makes use of some external libraries.
Let's say the task code is something like this:
# mypackage.tasks
import logging
from extpackage1 import module1
from extpackage2 import module2
from celery import Celery
from celery.utils.log import get_task_logger
celery = Celery('tasks', broker='amqp://user:pass#host/vhname')
logger = get_task_logger(__name__)
#celery.task
def my_task(my_job_id):
logger.info('My task executed with my_job_id=%s' % my_job_id)
module1.func1()
module2.func2()
while the 2 modules I import are basically:
# extpackage1.module1
import logging
logger = logging.getLogger(__name__)
def func1():
# let's do something here ...
logger.info('Logging inside module1')
and:
# extpackage2.module2
import logging
logger = logging.getLogger(__name__)
def func2():
# let's do something here ...
logger.info('Logging inside module2')
I would like to obtain a log file which include the correct value of my_job_id in each line, depending on the task which is executed.
It would be something such as:
[1][...other info..] My task executed with my_job_id=1
[1][...other info..] Logging inside module1
[1][...other info..] Logging inside module2
[2][...other info..] My task executed with my_job_id=2
[2][...other info..] Logging inside module1
[2][...other info..] Logging inside module2
I could easily include the value of my_job_id into the task logger, correctly obtaining the lines I log from inside the task function (like [1][...other info..] My task executed with my_job_id=1 and [2][...other info..] My task executed with my_job_id=2).
But what about the external libraries? Is there an elegant way to 'wrap' the log produced by the external script and format it according to my requirements?