We Keep Coding

Logging into syslog from a new service in systemD

I created a new service in systemD, so far it does not do anything, I am just trying to check that everything is running step by step.
For that, I am adding logs, however, it does not appear in Syslog, which I understand is the default for services.
When I use simple prints in my code, it does appear in Syslog though
import logging
class recoveryService:
def __init__(self):
self.id = 'integ38'
print self.id # prints to log
logging.info("the id is {}".format(self.id)) #does not print to log
def run(self):
print 'reached run' #prints to log
logging.info('reached run log') #does not print to log
if __name__ == '__main__':
recovery = recoveryService()
recovery.run()
How can I make these loggings appear in syslog?

logging.getLogger().setLevel('INFO') call this once before your first logging call. The default level of the root logger is WARNING so no logs below that level are shown. Also be aware that it is good practice to define a handler or use logging.basicConfig to set up your logging.

Logging for a flask app

I am developing a webapp with flask that acts as interface to a python library that makes computations (usually time consuming).
Each call to the server is identified with an identifier, and I want to write logs of calls to the library to a file that depends on the given identifier.
A minimal working example is as follows.
computations.py
import time
import logging
logger = logging.getLogger(__name__)
def long_computation(identifier):
logger.info('called computation with identifier %s', identifier)
for i in range(100):
logger.info('in step %d of identifier %s', i, identifier)
time.sleep(1)
logger.info('finished computation with identifier %s')
server.py
from flask import Flask, request
import logging
import threading
import computations
app = Flask(__name__)
def call_computation(identifier):
fh = logging.FileHandler("computations-%s.log" % identifier)
formatter = logging.Formatter('%(asctime)s %(levelname)-8s %(name)s : %(message)s')
fh.setFormatter(formatter)
fh.setLevel(logging.INFO)
computations.logger.setLevel(logging.INFO)
computations.logger.addHandler(fh)
computations.long_computation(identifier)
computations.logger.removeHandler(fh)
#app.route('/node/<identifier>', methods=['GET','POST'])
def serve_node(identifier):
thread = threading.Thread(target=call_computation, args=(identifier,))
thread.start()
return "I will compute it!"
When I make a call to the server, say http://127.0.0.1:5000/node/A it creates the logfile computations-A.log and logs correctly to this file. However if I make another call to the server, say http://127.0.0.1:5000/node/B before the first computation ends, then it creates the log file computations-B.log, but the logs of both computations, corresponding to the different calls to call_computation go to both files. That is, both files computations-A.log and computations-B.log have, for instance, lines like:
2018-08-02 20:31:57,524 INFO computations : in step 56 of identifier B
2018-08-02 20:31:57,799 INFO computations : in step 97 of identifier A
Could anyone please help me in order to make the calls to the library to go to the appropiate log file? Notice that I cannot, in principle, modify the package that makes the computations, so that I cannot create more loggers inside that package.
Thanks in advance!

The solution is to filter the log records. In the server.py file create a subclass of logging.Filter:
class MyFilter(logging.Filter):
def __init__(self, thread_id):
super(MyFilter, self).__init__()
self.thread_id = thread_id
def filter(self, record):
return record.thread == self.thread_id
and when setting up the handler, add an instance of this class:
myfilter = MyFilter(threading.current_thread().ident)
fh.addFilter(myfilter)
This way, when a log record reaches the filter, if the thread that created the log is the same that created the filter, then it will pass to the next level; otherwise it will get filtered out.

How to create standalone using Flask 0.10 micro framework to be run at a particular time

The 'app' is created in views.py. The folder structure is:
~/tasks/
/tasks
/__init__.py
/cron1.py
/views.py # app object is created here
There are some functions that I wish to import in the standalone cron1.py which I wish to run as cron job. The cron function call I make as follows:
if __name__ == '__main__':
try:
with app.app_context():
# with current_app.test_request_context():
get_recurring_tasklist()
except Exception, e:
print e
On execution from root folder with command $ PYTHONPATH=. python tasks/create_recurring_tasks.py Error I get is: working outside of request context I am using a method call from views which uses request object. How to go about?

If you have a standalone cron job the better practice is to avoid using any sort of logic in the job that involves the Flask request context/object. The test_request_context context is just for testing and shouldn't be used in task setup.
I don't know what your view function looks like, but you should abstract away the "task creation" part of the function into its own method that lives independently. That the cron job and view function can both share it.
#app.route('/task_list')
def get_task_list():
data = request.args.get('tasks')
# some sort of logic happening here
ret = []
for word in data:
ret.append('some task')
return jsonify(data=ret)
After
def convert_data_to_tasks(data):
ret = []
for word in tasks:
ret.append('some task')
return ret
#app.route('/task_list')
def get_task_list():
tasks = request.args.get('tasks')
ret = convert_data_to_tasks(tasks)
return jsonify(ret)

How do I test if a certain log message is logged in a Django test case?

I want to ensure that a certain condition in my code causes a log message to be written to the django log. How would I do this with the Django unit testing framework?
Is there a place where I can check logged messages, similarly to how I can check sent emails? My unit test extends django.test.TestCase.

Using the mock module for mocking the logging module or the logger object. When you've done that, check the arguments with which the logging function is called.
For example, if you code looks like this:
import logging
logger = logging.getLogger('my_logger')
logger.error("Your log message here")
it would look like:
from unittest.mock import patch # For python 2.x use from mock import patch
#patch('this.is.my.module.logger')
def test_check_logging_message(self, mock_logger):
mock_logger.error.assert_called_with("Your log message here")

You can also use assertLogs from django.test.TestCase
When you code is
import logging
logger = logging.getLogger('my_logger')
def code_that_throws_error_log():
logger.error("Your log message here")
This is the test code.
with self.assertLogs(logger='my_logger', level='ERROR') as cm:
code_that_throws_error_log()
self.assertIn(
"ERROR:your.module:Your log message here",
cm.output
)
This lets you avoid patching just for logs.

The common way of mocking out the logger object (see the splendid chap Simeon Visser's answer) is slightly tricky in that it requires the test to mock out the logging in all the places it's done. This is awkward if the logging comes from more than one module, or is in code you don't own. If the module the logging comes from changes name, it will break your tests.
The splendid 'testfixtures' package includes tools to add a logging handler which captures all generated log messages, no matter where they come from. The captured messages can later be interrogated by the test. In its simplest form:
Assuming code-under-test, which logs:
import logging
logger = logging.getLogger()
logger.info('a message')
logger.error('an error')
A test for this would be:
from testfixtures import LogCapture
with LogCapture() as l:
call_code_under_test()
l.check(
('root', 'INFO', 'a message'),
('root', 'ERROR', 'an error'),
)
The word 'root' indicates the logging was sent via a logger created using logging.getLogger() (i.e. with no args.) If you pass an arg to getLogger (__name__ is conventional), that arg will be used in place of 'root'.
The test does not care what module created the logging. It could be a sub-module called by our code-under-test, including 3rd party code.
The test asserts about the actual log message that was generated, as opposed to the technique of mocking, which asserts about the args that were passed. These will differ if the logging.info call uses '%s' format strings with additional arguments that you don't expand yourself (e.g. use logging.info('total=%s', len(items)) instead of logging.info('total=%s' % len(items)), which you should. It's no extra work, and allows hypothetical future logging aggregation services such as 'Sentry' to work properly - they can see that "total=12" and "total=43" are two instances of the same log message. That's the reason why pylint warns about the latter form of logging.info call.)
LogCapture includes facilities for log filtering and the like. Its parent 'testfixtures' package, written by Chris Withers, another splendid chap, includes many other useful testing tools. Documentation is here: http://pythonhosted.org/testfixtures/logging.html

Django has a nice context manager function called patch_logger.
from django.test.utils import patch_logger
then in your test case:
with patch_logger('logger_name', 'error') as cm:
self.assertIn("Error message", cm)
where:
logger_name is the logger name (duh)
error is the log level
cm is the list of all log messages
More details:
https://github.com/django/django/blob/2.1/django/test/utils.py#L638
It should work the same for django < 2.0, independently of python version (as long as it's supported by dj)

If you're using test classes, you can use following solution:
import logger
from django.test import TestCase
class MyTest(TestCase):
#classmethod
def setUpClass(cls):
super(MyTest, cls).setUpClass()
cls.logging_error = logging.error
logging.error = cls._error_log
#classmethod
def tearDownClass(cls):
super(MyTest, cls).tearDownClass()
logging.error = cls.logging_error
#classmethod
def _error_log(cls, msg):
cls.logger = msg
def test_logger(self):
self.assertIn('Message', self.logger)
This method replaces error function of logging module with your custom method only for test purposes and put stdout into cls.logger variable which is available in every test case by calling self.logger. At the end it reverts changes by placing error function from logging module back.

How can I determine if any errors were logged during a python program's execute?

I have a python script which calls log.error() and log.exception() in several places. These exceptions are caught so that the script can continue to run, however, I would like to be able to determine if log.error() and/or log.exception() were ever called so I can exit the script with an error code by calling sys.exit(1). A naive implementation using an "error" variable is included below. It seems to me there must be a better way.
error = False
try:
...
except:
log.exception("Something bad occurred.")
error = True
if error:
sys.exit(1)

I had the same issue as the original poster: I wanted to exit my Python script with an error code if any messages of error or greater severity were logged. For my application, it's desirable for execution to continue as long as no unhandled exceptions are raised. However, continuous integrations builds should fail if any errors are logged.
I found the errorhandler python package, which does just what we need. See the GitHub, PyPI page, and docs.
Below is the code I used:
import logging
import sys
import errorhandler
# Track if message gets logged with severity of error or greater
error_handler = errorhandler.ErrorHandler()
# Also log to stderr
stream_handler = logging.StreamHandler(stream=sys.stderr)
logger = logging.getLogger()
logger.setLevel(logging.INFO) # Set whatever logging level for stderr
logger.addHandler(stream_handler)
# Do your program here
if error_handler.fired:
logger.critical('Failure: exiting with code 1 due to logged errors')
raise SystemExit(1)

You can check logger._cache. It returns a dictionary with keys corresponding to the numeric value of the error level logged. So for checking if an error was logged you could do:
if 40 in logger._cache and logger._cache[40]

I think that your solution is not the best option. Logging is one aspect of your script, returning an error code depending on the control flow is another. Perhaps using exceptions would be a better option.
But if you want to track the calls to log, you can wrap it within a decorator. A simple example of a decorator follows (without inheritance or dynamic attribute access):
class LogWrapper:
def __init__(self, log):
self.log = log
self.error = False
def exception(self, message)
self.error = True
self.log.exception(message)

whenever logger._cache is not a solution (when other packages / modules log on their own which won't end in logger._cache), there's a way to build a ContextFilter which will record the worst called log level:
class ContextFilterWorstLevel(logging.Filter):
def __init__(self):
self.worst_level = logging.INFO
def filter(self, record):
if record.levelno > self.worst_level:
self.worst_level = record.levelno
return True
# Create a logger object and add the filter
logger = logging.getLogger()
logger.addFilter(ContextFilterWorstLevel())
# Check the worst log level called later
for filter in logger.filters:
if isinstance(filter, ContextFilterWorstLevel):
print(filter.worst_level)

You can employ a counter. If you want to track individual exceptions, create a dictionary with the exception as the key and the integer counter as the value.