I have implemented a Celery task which makes use of some external libraries.
Let's say the task code is something like this:
# mypackage.tasks
import logging
from extpackage1 import module1
from extpackage2 import module2
from celery import Celery
from celery.utils.log import get_task_logger
celery = Celery('tasks', broker='amqp://user:pass#host/vhname')
logger = get_task_logger(__name__)
#celery.task
def my_task(my_job_id):
logger.info('My task executed with my_job_id=%s' % my_job_id)
module1.func1()
module2.func2()
while the 2 modules I import are basically:
# extpackage1.module1
import logging
logger = logging.getLogger(__name__)
def func1():
# let's do something here ...
logger.info('Logging inside module1')
and:
# extpackage2.module2
import logging
logger = logging.getLogger(__name__)
def func2():
# let's do something here ...
logger.info('Logging inside module2')
I would like to obtain a log file which include the correct value of my_job_id in each line, depending on the task which is executed.
It would be something such as:
[1][...other info..] My task executed with my_job_id=1
[1][...other info..] Logging inside module1
[1][...other info..] Logging inside module2
[2][...other info..] My task executed with my_job_id=2
[2][...other info..] Logging inside module1
[2][...other info..] Logging inside module2
I could easily include the value of my_job_id into the task logger, correctly obtaining the lines I log from inside the task function (like [1][...other info..] My task executed with my_job_id=1 and [2][...other info..] My task executed with my_job_id=2).
But what about the external libraries? Is there an elegant way to 'wrap' the log produced by the external script and format it according to my requirements?
Related
I know how to suppress log messages up to a certain log level from an imported module:
import logging
logging.getLogger("module_name").setLevel(logging.WARNING)
I want to know if it's possible to suppress log messages from only a specific function/class in an imported module (and keep all other messages in the module).
I tried this:
logging.getLogger("module_name.function_name").setLevel(logging.WARNING)
but it didn't work. Is it possible?
I came up with this in the end based on logging with filters:
Contents of my_module.py:
import logging
logger = logging.getLogger(__name__)
def func1():
logger.debug('Running func1!')
def func2():
logger.debug('Running func2!')
I want to ignore any messages from func2, but keep any messages from func1.
Contents of main.py:
import logging
import my_module
logging.basicConfig(level=logging.DEBUG)
class IgnoreFunc2(logging.Filter):
def filter(self, record):
return not record.funcName == 'func2'
# this relies on knowing the variable name in my_module
my_module.logger.addFilter(IgnoreFunc2())
def main():
my_module.func1()
my_module.func2()
if __name__ == '__main__':
main()
Output:
DEBUG:my_module:Running func1!
I want to test that a specific celery task call logger.info exactly once when the task is invoked with the delay() API.
And I want to do the test by patching logger.info .
I want to test as described here for the Product.order case https://docs.celeryproject.org/en/latest/userguide/testing.html.
The setup is : python 2.7 on ubuntu 16.04 . celery 4.3.0 , pytest 4.0.0, mock 3.0.3.
I have the following file system structure:
poc/
prj/
-celery_app.py
-tests.py
celery_app.py
from __future__ import absolute_import
from celery import Celery
from celery.utils.log import get_task_logger
app = Celery('celery_app')
logger = get_task_logger(__name__)
#app.task(bind=True)
def debug_task(self):
logger.info('Request: {0!r}'.format(self.request))
tests.py
from __future__ import absolute_import
from mock import patch
from prj.celery_app import debug_task
#patch('logging.Logger.info')
def test_log_info_is_called_only_once_when_called_sync(log_info):
debug_task()
log_info.assert_called_once()
#patch('logging.Logger.info')
def test_log_info_is_called_only_once_when_called_async(log_info):
debug_task.delay()
log_info.assert_called_once()
I expect both tests to have success.
Instead the first has success, while the second fails with AssertionError: Expected 'info' to have been called once. Called 0 times.
I expect that the evaluation of the expression logger.info inside debug_task() context evaluates to <MagicMock name='info' id='someinteger'> in both cases, instead it evaluates to <bound method Logger.info of <logging.Logger object at 0x7f894eeb1690>> in the second case, showing no patching.
I know that in the second case the celery worker executes the task inside a thread.
I ask for a way to patch the logger.info call when debug_task.delay() is executed.
Thanks in advance for any answer.
I want to find out how logging should be organised given that I write many scripts and modules that should feature similar logging. I want to be able to set the logging appearance and the logging level from the script and I want this to propagate the appearance and level to my modules and only my modules.
An example script could be something like the following:
import logging
import technicolor
import example_2_module
def main():
verbose = True
global log
log = logging.getLogger(__name__)
logging.root.addHandler(technicolor.ColorisingStreamHandler())
# logging level
if verbose:
logging.root.setLevel(logging.DEBUG)
else:
logging.root.setLevel(logging.INFO)
log.info("example INFO message in main")
log.debug("example DEBUG message in main")
example_2_module.function1()
if __name__ == '__main__':
main()
An example module could be something like the following:
import logging
log = logging.getLogger(__name__)
def function1():
print("printout of function 1")
log.info("example INFO message in module")
log.debug("example DEBUG message in module")
You can see that in the module there is minimal infrastructure written to import the logging of the appearance and the level set in the script. This has worked fine, but I've encountered a problem: other modules that have logging. This can result in output being printed twice, and very detailed debug logging from modules that are not my own.
How should I code this such that the logging appearance/level is set from the script but then used only by my modules?
You need to set the propagate attribute to False so that the log message does not propagate to ancestor loggers. Here is the documentation for Logger.propagate -- it defaults to True. So just:
import logging
log = logging.getLogger(__name__)
log.propagate = False
I'm running a Celery task that executes a function. This function generates some logging information. Using the get_task_logger logger, I am able print the logging information to the Celery stdout.
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
def my_func_called_inside_a_task():
logger.debug("SOME OUTPUT HERE")
However, I also want to import this function as a normal python script (not using Celery) and log to, for example, stdout. Normally, I might do something like the following:
import logging
logger = logging.getLogger(__name__)
def my_func_called_inside_a_task():
logger.debug("SOME OUTPUT HERE")
How do I combine both approaches so I don't have to do something redundant like the following?
import logging
from celery.utils.log import get_task_logger
logger = logging.getLogger(__name__)
logger_celery = get_task_logger(__name__)
def my_func_called_inside_a_task():
logger.debug("SOME OUTPUT HERE")
logger_celery.debug("SOME OUTPUT HERE")
Summary:
If I call the function from a celery task, I'd like it to log to the celery worker stdout. If I call the function from a normal Python prompt, it would use the normal Python logger. Any help is much appreciated.
You can pass an optional argument to that function.
import logging
from celery.utils.log import get_task_logger
def my_func_called_inside_a_task(use_celery_logger=None):
if use_celery_logger:
logger = get_task_logger(__name__)
else:
logger = logging.getLogger(__name__)
logger.debug("SOME OUTPUT HERE")
and in Your celery task call it as
my_func_called_inside_a_task(use_celery_logger=True)
and for normal logging you can call it as it is
my_func_called_inside_a_task()
I want to create a Python logging object in my main program and have logging in both my main program and in the modules it uses at the same logging level. The basic example given in logging documentation is essentially as follows:
main.py:
import logging
import mylib
def main():
logging.basicConfig(level = logging.INFO)
logging.info('Started')
mylib.do_something()
logging.info('Finished')
if __name__ == '__main__':
main()
mylib.py:
import logging
def do_something():
logging.info('Doing something')
This works fine. I am not sure, however, of how to get a Python logging object doing something similar. The following, for example, does not work:
main.py:
import logging
import mylib
def main():
verbose = True
global log
log = logging.getLogger(__name__)
if verbose:
log.setLevel(logging.INFO)
else:
log.setLevel(logging.DEBUG)
log.info('Started')
mylib.do_something()
log.info('Finished')
if __name__ == '__main__':
main()
mylib.py:
import logging
def do_something():
log.info('Doing something')
It does not work because the global log object is not recognised in the mylib.py module. How should I be doing this? My two main goals are
to have the logging level that is set in my main program propagate through to any modules used and
to be able to use log for logging, not "logging" (i.e. log.info("alert") as opposed to logging.info("alert")).
Your application should configure the logging once (with basicConfig or see logging.config) and then in each file you can have:
import logging
log = logging.getLogger(__name__)
# further down:
log.info("alert")
So you use a logger everywhere and you don't directly access the logging module in your codebase. This allows you to configure all your loggers in your logging configuration (which, as said earlier, is configured once and loaded at the beginning of your application).
You can use a different logger in each module as follows:
import logging
LOG = logging.getLogger(__name__)
# Stuff
LOG.info("A log message")