We Keep Coding

Why is my error not being written to my logfile?

I have an existing program (a python executable) but when I run it, it sometimes stops for no reason. So to try and see where the problem arises I aimed to add the logging concept to the python application so I could exactly see what and when a problem arises.
Below is my code, the file log.txt is being created which is good. Now for testing I created an error on purpose by changing the password of the database. Now from experience I know that it should give me an mysql.connector.errors error, which it does in my terminal. But when I open the logfile.txt I get an empty file...
My aim is to catch all different types of errors thats why I use the Exception method.
What I expected to see in log.txt was for example:
2021-09-16 09:04:29,827 ERROR __main__ 1045 (28000): Access denied for user 'root'#'localhost' (using password: YES)
This is my code:
logging.basicConfig(filename='log.text', level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(name)s %(message)s')
logger=logging.getLogger(__name__)
def main():
mp_list = get_mp_list() #<-- Get the list created
pool = Pool(min(cpu_count(), len(mp_list))) #<-- create a pool
results = pool.imap_unordered(process_mp, mp_list) #<-- Create the pool.imap object
while True:
try:
result = next(results) #<-- run the program
except StopIteration:
break
except Exception as err: #<-- write to the logfile in case of error
logger.error(err)
raise
if __name__ == '__main__':
main()

You seem to be using multiprocessing, and if multiple processes log to the same file, it might not work - for the reasons described in the logging documentation. Basically, you need to use a different strategy to ensure that only one process writes to a particular file. It's too long to reproduce the details here - refer to the linked info for details.

Add filemode it should work then
logging.basicConfig(filename='log.text', level=logging.DEBUG, filemode="w",
format='%(asctime)s %(levelname)s %(name)s %(message)s')

Unexpected python logger output when using several handlers with different log levels

I am trying to log data to stderr and into a file. The file should contain all log messages, and to stderr should go only the log level configured on the command line. This is described several times in the logging howto - but it does not seem to work for me. I have created a small test script which illustrates my problem:
#!/usr/bin/env python
import logging as l
l.basicConfig(level=100)
logger = l.getLogger("me")
# ... --- === SEE THIS LINE === --- ...
logger.setLevel(l.CRITICAL)
sh = l.StreamHandler()
sh.setLevel(l.ERROR)
sh.setFormatter(l.Formatter('%(levelname)-8s CONSOLE %(message)s'))
logger.addHandler(sh)
fh = l.FileHandler("test.dat", "w")
fh.setLevel(l.DEBUG)
fh.setFormatter(l.Formatter('%(levelname)-8s FILE %(message)s'))
logger.addHandler(fh)
logger.info("hi this is INFO")
logger.error("well this is ERROR")
In line 5th code line I can go for logger.setLevel(l.CRITICAL) or logger.setLevel(l.DEBUG). Both results are unsatisfying.
With logger.setLevel(l.CRITICAL) I get ...
$ python test.py
$ cat test.dat
$
Now with logger.setLevel(l.DEBUG) I get ...
$ python test.py
INFO:me:hi this is INFO
ERROR CONSOLE well this is ERROR
ERROR:me:well this is ERROR
$ cat test.dat
INFO FILE hi this is INFO
ERROR FILE well this is ERROR
$
In one case I see nothing nowhere, in the other I see everything everywhere, and one message is being displayed even twice on the console.
Now I get where the ERROR CONSOLE and ERROR FILE outputs come from, those I expect. I don't get where the INFO:me... or ERROR:me... outputs are coming from, and I would like to get rid of them.
Things I already tried:
Creating a filter as described here: https://stackoverflow.com/a/7447596/902327 (does not work)
Emptying handlers from the logger with logger.handlers = [] (also does not work)
Can somebody help me out here? It seems like a straightforward requirement and I really don't seem to get it.

You can set the root level to DEBUG, set propagate to False and then set the appropriate level for the other handlers.
import logging as l
l.basicConfig()
logger = l.getLogger("me")
# ... --- === SEE THIS LINE === --- ...
logger.setLevel(l.DEBUG)
logger.propagate = False
sh = l.StreamHandler()
sh.setLevel(l.ERROR)
sh.setFormatter(l.Formatter('%(levelname)-8s CONSOLE %(message)s'))
logger.addHandler(sh)
fh = l.FileHandler("test.dat", "w")
fh.setLevel(l.INFO)
fh.setFormatter(l.Formatter('%(levelname)-8s FILE %(message)s'))
logger.addHandler(fh)
logger.info("hi this is INFO")
logger.error("well this is ERROR")
Output:
~$ python test.py
ERROR CONSOLE well this is ERROR
~$ cat test.dat
INFO FILE hi this is INFO
ERROR FILE well this is ERROR

Intercept Messages from 3rd Party Code

I am writing a Python script that uses 3rd party modules from GDAL. The GDAL functions do not raise exceptions when an error occurs, but do send messages to stdout. Generally, the errors that occur with GDAL functions do not warrant stopping the process and I don't need to know about the error having occurred.
Is there a way I can intercept the messages that are being sent to stdout before they are printed in the console? The GDAL messages interfere with the messages that I have provided in my own code.

As described in "Python Gotchas", you can turn on Exceptions using gdal.UseExceptions(), e.g.:
from osgeo import gdal
dsrc = gdal.Open('nonexist')
# ... silence
gdal.UseExceptions()
dsrc = gdal.Open('nonexist')
# Traceback (most recent call last):
# File "<interactive input>", line 1, in <module>
# RuntimeError: `nonexist' does not exist in the file system,
# and is not recognised as a supported dataset name.
You could always then use a try except block get the actual error message string:
try:
dsrc = gdal.Open('nonexist')
except RuntimeError as e:
print(str(e))
which will print the error message:
`nonexist' does not exist in the file system,
and is not recognised as a supported dataset name.

Easy application logging/debugging with nginx, uwsgi, flask?

I'm not looking to turn on the dangerous debugging console, but my application is getting a 500 error and doesn't seem to be writing any output for me to investigate more deeply.
I saw this exchange on the mailing list, which led me to this page on logging errors.
However, I still find this very confusing and have a couple of questions:
(1) In which file should the stuff below go?
ADMINS = ['yourname#example.com']
if not app.debug:
import logging
from logging.handlers import SMTPHandler
mail_handler = SMTPHandler('127.0.0.1',
'server-error#example.com',
ADMINS, 'YourApplication Failed')
mail_handler.setLevel(logging.ERROR)
app.logger.addHandler(mail_handler)
...assuming the "getting bigger" file pattern for larger applications? __init__.py? config.py? run.py?
(2) I am overwhelmed by options there, and can't tell which I should use. Which loggers should I turn on, with what settings, to replicate the local python server debug I get to stdout when I run run.py? I find that default, local output stream very useful, more so than the interactive debugger in the page. Does anyone have a pattern they could share on setting up something replicating this with an nginx deployment, outputting to a log?
(3) Is there anything I need to change, not at the flask level, but in nginx, say in my /etc/nginx/sites-available/appname file, to enable logging?
UPDATE
Specifically, I'm looking for information like I get when python runs locally as to why, say, a package isn't working, or where my syntax error might be, or what variable doesn't exist:
$ python run.py
Traceback (most recent call last):
File "run.py", line 1, in <module>
from myappname import app
File "/home/me/myappname/myappname/__init__.py", line 27, in <module>
file_handler.setLevel(logging.debug)
File "/usr/lib/python2.7/logging/__init__.py", line 710, in setLevel
self.level = _checkLevel(level)
File "/usr/lib/python2.7/logging/__init__.py", line 190, in _checkLevel
raise TypeError("Level not an integer or a valid string: %r" % level)
When I run flask on a server, I never see this. I just get a uWSGI error in the browser, and have no idea which code was problematic. I would just like something like the above to be written to a file.
I notice also that setting the following logging didn't really write much to file, even when I turn the log way up to the DEBUG level:
from logging import FileHandler
file_handler = FileHandler('mylog.log')
file_handler.setLevel(logging.DEBUG)
app.logger.addHandler(file_handler)
mylog.log is blank, even when my application errors out.
I'll also add that I've tried to set debug = True in the following ways, in __init__.py:
app = Flask(__name__)
app.debug = True
app.config['DEBUG'] = True
from werkzeug.debug import DebuggedApplication
app.wsgi_app = DebuggedApplication(app.wsgi_app, True)
app.config.from_object('config')
app.config.update(DEBUG=True)
app.config['DEBUG'] = True
if __name__ == '__main__':
app.run(debug=True)
While in my config.py file, I have...
debug = True
Debug = True
DEBUG = True
Yet, no debugging happens, and without logging or debugging, this is rather hard to track down. Errors simply terminate the application with the un-useful browser message:
uWSGI Error
Python application not found

Set config['PROPAGATE_EXCEPTIONS'] to True when running app in production and you want tracebacks to be logged into log files. (I haven't tried with SMTP handler, though..)

The part where you create handlers, add to loggers etc. should be in the if __name__ == '__main__' clause, i.e. your main entry point. I assume that would be run.py.
I'm not sure I can answer this - it depends on what you want. I'd advise looking at the logging tutorial to see the various options available.
I don't believe you need to change anything at the nginx level.
Update: You might want to have an exception clause that covers uncaught exceptions, e.g.
if __name__ == '__main__':
try:
app.run(debug=True)
except Exception:
app.logger.exception('Failed')
which should write the traceback of any exception which occurred in app.run() to the log.

I know that this is a VERY old post, but I ran into the issue now, and it took me a bit to find the solution. Flask sends errors to the server. I was running Gunicorn with an upstart script on Ubuntu 14.04 LTS, and the place where I found the error logs was as follows:
/var/log/upstart/myapp.log
http://docs.gunicorn.org/en/stable/deploy.html#upstart
Just in case some other poor soul ends up in this situation.

e.printStackTrace equivalent in python

I know that print(e) (where e is an Exception) prints the occurred exception
but, I was trying to find the python equivalent of Java's e.printStackTrace() that exactly traces the exception to what line it occurred and prints the entire trace of it.
Could anyone please tell me the equivalent of e.printStackTrace() in Python?

import traceback
traceback.print_exc()
When doing this inside an except ...: block it will automatically use the current exception. See http://docs.python.org/library/traceback.html for more information.

There is also logging.exception.
import logging
...
try:
g()
except Exception as ex:
logging.exception("Something awful happened!")
# will print this message followed by traceback
Output:
ERROR 2007-09-18 23:30:19,913 error 1294 Something awful happened!
Traceback (most recent call last):
File "b.py", line 22, in f
g()
File "b.py", line 14, in g
1/0
ZeroDivisionError: integer division or modulo by zero
(From http://blog.tplus1.com/index.php/2007/09/28/the-python-logging-module-is-much-better-than-print-statements/ via How to print the full traceback without halting the program?)

e.printStackTrace equivalent in python
In Java, this does the following (docs):
public void printStackTrace()
Prints this throwable and its backtrace to the standard error stream...
This is used like this:
try
{
// code that may raise an error
}
catch (IOException e)
{
// exception handling
e.printStackTrace();
}
In Java, the Standard Error stream is unbuffered so that output arrives immediately.
The same semantics in Python 2 are:
import traceback
import sys
try: # code that may raise an error
pass
except IOError as e: # exception handling
# in Python 2, stderr is also unbuffered
print >> sys.stderr, traceback.format_exc()
# in Python 2, you can also from __future__ import print_function
print(traceback.format_exc(), file=sys.stderr)
# or as the top answer here demonstrates, use:
traceback.print_exc()
# which also uses stderr.
Python 3
In Python 3, we can get the traceback directly from the exception object (which likely behaves better for threaded code).
Also, stderr is line-buffered, but the print function gets
a flush argument, so this would be immediately printed to stderr:
print(traceback.format_exception(None, # <- type(e) by docs, but ignored
e, e.__traceback__),
file=sys.stderr, flush=True)
Conclusion:
In Python 3, therefore, traceback.print_exc(), although it uses sys.stderr by default, would buffer the output, and you may possibly lose it. So to get as equivalent semantics as possible, in Python 3, use print with flush=True.

Adding to the other great answers, we can use the Python logging library's debug(), info(), warning(), error(), and critical() methods. Quoting from the docs for Python 3.7.4,
There are three keyword arguments in kwargs which are inspected: exc_info which, if it does not evaluate as false, causes exception information to be added to the logging message.
What this means is, you can use the Python logging library to output a debug(), or other type of message, and the logging library will include the stack trace in its output. With this in mind, we can do the following:
import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
def f():
a = { 'foo': None }
# the following line will raise KeyError
b = a['bar']
def g():
f()
try:
g()
except Exception as e:
logger.error(str(e), exc_info=True)
And it will output:
'bar'
Traceback (most recent call last):
File "<ipython-input-2-8ae09e08766b>", line 18, in <module>
g()
File "<ipython-input-2-8ae09e08766b>", line 14, in g
f()
File "<ipython-input-2-8ae09e08766b>", line 10, in f
b = a['bar']
KeyError: 'bar'