Here's a tiny example. When running under functions framework on a GCE VM with an appropriate service account,
import logging
import google.cloud.logging
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()
def test(request):
logging.warning("testing warning")
return "done"
does not log a warning message.
Changing it to:
import logging
import google.cloud.logging
logging_client = google.cloud.logging.Client()
def test(request):
logging_client.setup_logging()
logging.warning("testing warning")
return "done"
does produce the correct message.
And if I take the version that doesn't work, add a call to the function and run it normally without functions framework, it produces a warning as expected.
import logging
import google.cloud.logging
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()
def test(request):
logging.warning("testing warning")
return "done"
test("foo")
Can someone explain what's going on, either with how the functions frameworks works, or how the logging handler works?
Thanks!
I had a similar issue using gcsfs with functions framework that provided the answer. It looks like when you use functions framework the function call runs in a different process from the initialization.
I mimicked this with:
import logging
import google.cloud.logging
import os
import time
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()
def test(request):
logging.warning("testing warning")
return "done"
pid = os.fork()
if pid > 0:
time.sleep(10)
else:
test("foo")
and it does not work. Move the setup_logging to inside of the function call, and it does. So it seems that you have to do the setup_logging() in the subprocess for it to work, and that's why it has to be done in the function call rather than as part of initialization in function framework.
I'm using PTB library and have many handlers as:
def some_handler(update, context): # Update is new data from user, context is all my data
do_something()
And I want to notify user if error has occured, like:
def some_handler(update, context):
try:
do_something()
except Exception as e:
notify_user(text="Some error occured")
logger.error(e)
To follow DRY and make code more nicely I wrote such decorator:
def bot_logger(text: str = ''):
def decorator(function):
#loguru_logger.catch
#wraps(function)
def wrapper(*args, **kwargs):
try:
return function(*args, **kwargs)
except Exception as e:
notify_user(text=f'Unknown error. {text} Please try againt latter')
loguru_logger.error(e
return wrapper
return decorator
But in most cases I'm getting a pretty obscure log record like
2021-11-26 19:47:32.558 | ERROR | bot:wrapper:46 - There are no messages to send
Question:
How I can make this message a more informative as a standard python error?
What should I fix in bot_logger decorator?
Logger setup:
from loguru import logger as loguru_logger
loguru_logger.add(
sink="log_error.txt",
filter=lambda record: record["level"].name == "ERROR",
backtrace=True,
format="{time} {level} {function}:{line} {message}",
level="ERROR",
rotation="1 MB",
compression="zip",
enqueue=True,
diagnose=True
)
P.S. I checked another similar questions
Best practices for logging in python
Python logging using a decorator
And the others but don't found an asnwer
Also, I tried different logger formats and parameters but it's not change a log record a much
Here is how I capture image with my camera
import logging
import gphoto2 as gp
def main():
def callback(level, domain, string, data=None):
print('Callback: level =', level, ', domain =', domain, ', string =', string)
if data:
print('Callback data:', data)
logging.basicConfig(
format='%(levelname)s: %(name)s: %(message)s', level=logging.WARNING)
callback_obj = gp.check_result(gp.use_python_logging())
camera = gp.Camera()
camera.init()
try:
camera_file_path = gp.check_result(camera.capture(gp.GP_CAPTURE_IMAGE))
except gp.GPhoto2Error as ex:
print("callback: ", ex, ex.code, ex.message, ex.string)
camera.exit()
if __name__ == "__main__" : exit(main())
If the camera is not able to focus it generates following error
...
WARNING: gphoto2: (ptp_usb_getresp [usb.c:466]) PTP_OC 0x90c8 receiving resp failed: Out of Focus (0xa002)
WARNING: gphoto2: (camera_nikon_capture [library.c:3153]) 'ret' failed: 'Out of Focus' (0xa002)
WARNING: gphoto2: (gp_context_error) Out of Focus
WARNING: gphoto2: (gp_camera_capture [gphoto2-camera.c:1340]) 'camera->functions->capture (camera, type, path, context)' failed: -1
('callback: ', GPhoto2Error('[-1] Unspecified error',), -1, '[-1] Unspecified error', 'Unspecified error')
The exception error code is -1 but how can I capture Out of Focus warning?
UPDATE
Filtered out unnecessary errors from logs
import logging
import gphoto2 as gp
from datetime import datetime
def main():
def callback(level, domain, string, data=None):
err_codes = ("(0x2005)", "(0x2019)")
if not string.decode().endswith(err_codes):
print("[{0}] {1}: {2}".format(datetime.utcnow(), domain.decode(), string.decode()))
if data:
print('Callback data:', data)
callback_obj = gp.check_result(gp.gp_log_add_func(gp.GP_LOG_ERROR, callback))
camera = gp.Camera()
try:
camera.init()
except gp.GPhoto2Error as err:
exit(err.code)
try:
camera_file_path = camera.capture(gp.GP_CAPTURE_IMAGE)
except gp.GPhoto2Error as err:
exit(err.code)
camera.exit()
if __name__ == "__main__" : exit(main())
You've defined a callback function, in which you could parse the error string to detect out of focus, but you haven't installed your callback so libgphoto2 isn't using it. Use gp_log_add_func to install your callback.
Also, you're passing the return value of camera.capture to gp.check_result. This is incorrect as camera.capture already checks the result and raises an exception if there's an error.
If you can figure out the name of the logger, you can add your own handler to it and do your own log processing.
Calling "getLogger" with your camera library's name would get you the logger.
Calling AddHandler on that logger with a custom handler would allow you to do your own log processing for logs coming from that logger.
Please see Python's Logging Cookbook for more info
Hope this helps
How can I check errors in my code using NVDA error log? I've created an addon, which seemingly doesn't works due to some error. How can I debug my code using error log or any other method possible?
The error I'm facing is in the following code:
import globalPluginHandler
import ui
import versionInfo
from NVDAObjects.window.scintilla import Scintilla
class GlobalPlugin(globalPluginHandler.GlobalPlugin):
def _messag(self, gesture):
ui.message("notepad++ opened") # speak hello when notepad++ is the current screen
def chooseNVDAObjectOverlayClasses(self, obj, clsList):
if obj.windowClassName == u'Scintilla' and obj.windowControlID == 0:
clsList.insert(0, self._messag)
I don't know anything about NVDA, but when python programs fail, they do it by raising an exception. You could perhaps include a mechanism to catch these exceptions and send them to a file. For example let's take a plugin from the NVDA developer's guide
--- start ---
# Version announcement plugin for NVDA
# Developer guide example 2
import globalPluginHandler
import ui
import versionInfo
class GlobalPlugin(globalPluginHandler.GlobalPlugin):
#MY_EXCEPTION_CATCHER
def script_announceNVDAVersion(self, gesture):
ui.message(versionInfo.version)
__gestures={
"kb:NVDA+shift+v": "announceNVDAVersion",
}
--- end ---
I added a decorator to the code that catches the exceptions that may occur and displays them in an html file that can be displayed in browser for example
import functools
import cgitb
import sys
def MY_EXCEPTION_CATCHER(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception:
with open('exception.html', 'w') as outfile:
outfile.write(cgitb.html(sys.exc_info())
raise
return wrapper
I hope it helps.
How can I log my Python exceptions?
try:
do_something()
except:
# How can I log my exception here, complete with its traceback?
Use logging.exception from within the except: handler/block to log the current exception along with the trace information, prepended with a message.
import logging
LOG_FILENAME = '/tmp/logging_example.out'
logging.basicConfig(filename=LOG_FILENAME, level=logging.DEBUG)
logging.debug('This message should go to the log file')
try:
run_my_stuff()
except:
logging.exception('Got exception on main handler')
raise
Now looking at the log file, /tmp/logging_example.out:
DEBUG:root:This message should go to the log file
ERROR:root:Got exception on main handler
Traceback (most recent call last):
File "/tmp/teste.py", line 9, in <module>
run_my_stuff()
NameError: name 'run_my_stuff' is not defined
Use exc_info options may be better, remains warning or error title:
try:
# coode in here
except Exception as e:
logging.error(e, exc_info=True)
My job recently tasked me with logging all the tracebacks/exceptions from our application. I tried numerous techniques that others had posted online such as the one above but settled on a different approach. Overriding traceback.print_exception.
I have a write up at http://www.bbarrows.com/ That would be much easier to read but Ill paste it in here as well.
When tasked with logging all the exceptions that our software might encounter in the wild I tried a number of different techniques to log our python exception tracebacks. At first I thought that the python system exception hook, sys.excepthook would be the perfect place to insert the logging code. I was trying something similar to:
import traceback
import StringIO
import logging
import os, sys
def my_excepthook(excType, excValue, traceback, logger=logger):
logger.error("Logging an uncaught exception",
exc_info=(excType, excValue, traceback))
sys.excepthook = my_excepthook
This worked for the main thread but I soon found that the my sys.excepthook would not exist across any new threads my process started. This is a huge issue because most everything happens in threads in this project.
After googling and reading plenty of documentation the most helpful information I found was from the Python Issue tracker.
The first post on the thread shows a working example of the sys.excepthook NOT persisting across threads (as shown below). Apparently this is expected behavior.
import sys, threading
def log_exception(*args):
print 'got exception %s' % (args,)
sys.excepthook = log_exception
def foo():
a = 1 / 0
threading.Thread(target=foo).start()
The messages on this Python Issue thread really result in 2 suggested hacks. Either subclass Thread and wrap the run method in our own try except block in order to catch and log exceptions or monkey patch threading.Thread.run to run in your own try except block and log the exceptions.
The first method of subclassing Thread seems to me to be less elegant in your code as you would have to import and use your custom Thread class EVERYWHERE you wanted to have a logging thread. This ended up being a hassle because I had to search our entire code base and replace all normal Threads with this custom Thread. However, it was clear as to what this Thread was doing and would be easier for someone to diagnose and debug if something went wrong with the custom logging code. A custome logging thread might look like this:
class TracebackLoggingThread(threading.Thread):
def run(self):
try:
super(TracebackLoggingThread, self).run()
except (KeyboardInterrupt, SystemExit):
raise
except Exception, e:
logger = logging.getLogger('')
logger.exception("Logging an uncaught exception")
The second method of monkey patching threading.Thread.run is nice because I could just run it once right after __main__ and instrument my logging code in all exceptions. Monkey patching can be annoying to debug though as it changes the expected functionality of something. The suggested patch from the Python Issue tracker was:
def installThreadExcepthook():
"""
Workaround for sys.excepthook thread bug
From
http://spyced.blogspot.com/2007/06/workaround-for-sysexcepthook-bug.html
(https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230540&group_id=5470).
Call once from __main__ before creating any threads.
If using psyco, call psyco.cannotcompile(threading.Thread.run)
since this replaces a new-style class method.
"""
init_old = threading.Thread.__init__
def init(self, *args, **kwargs):
init_old(self, *args, **kwargs)
run_old = self.run
def run_with_except_hook(*args, **kw):
try:
run_old(*args, **kw)
except (KeyboardInterrupt, SystemExit):
raise
except:
sys.excepthook(*sys.exc_info())
self.run = run_with_except_hook
threading.Thread.__init__ = init
It was not until I started testing my exception logging I realized that I was going about it all wrong.
To test I had placed a
raise Exception("Test")
somewhere in my code. However, wrapping a a method that called this method was a try except block that printed out the traceback and swallowed the exception. This was very frustrating because I saw the traceback bring printed to STDOUT but not being logged. It was I then decided that a much easier method of logging the tracebacks was just to monkey patch the method that all python code uses to print the tracebacks themselves, traceback.print_exception.
I ended up with something similar to the following:
def add_custom_print_exception():
old_print_exception = traceback.print_exception
def custom_print_exception(etype, value, tb, limit=None, file=None):
tb_output = StringIO.StringIO()
traceback.print_tb(tb, limit, tb_output)
logger = logging.getLogger('customLogger')
logger.error(tb_output.getvalue())
tb_output.close()
old_print_exception(etype, value, tb, limit=None, file=None)
traceback.print_exception = custom_print_exception
This code writes the traceback to a String Buffer and logs it to logging ERROR. I have a custom logging handler set up the 'customLogger' logger which takes the ERROR level logs and send them home for analysis.
You can log all uncaught exceptions on the main thread by assigning a handler to sys.excepthook, perhaps using the exc_info parameter of Python's logging functions:
import sys
import logging
logging.basicConfig(filename='/tmp/foobar.log')
def exception_hook(exc_type, exc_value, exc_traceback):
logging.error(
"Uncaught exception",
exc_info=(exc_type, exc_value, exc_traceback)
)
sys.excepthook = exception_hook
raise Exception('Boom')
If your program uses threads, however, then note that threads created using threading.Thread will not trigger sys.excepthook when an uncaught exception occurs inside them, as noted in Issue 1230540 on Python's issue tracker. Some hacks have been suggested there to work around this limitation, like monkey-patching Thread.__init__ to overwrite self.run with an alternative run method that wraps the original in a try block and calls sys.excepthook from inside the except block. Alternatively, you could just manually wrap the entry point for each of your threads in try/except yourself.
You can get the traceback using a logger, at any level (DEBUG, INFO, ...). Note that using logging.exception, the level is ERROR.
# test_app.py
import sys
import logging
logging.basicConfig(level="DEBUG")
def do_something():
raise ValueError(":(")
try:
do_something()
except Exception:
logging.debug("Something went wrong", exc_info=sys.exc_info())
DEBUG:root:Something went wrong
Traceback (most recent call last):
File "test_app.py", line 10, in <module>
do_something()
File "test_app.py", line 7, in do_something
raise ValueError(":(")
ValueError: :(
EDIT:
This works too (using python 3.6)
logging.debug("Something went wrong", exc_info=True)
What I was looking for:
import sys
import traceback
exc_type, exc_value, exc_traceback = sys.exc_info()
traceback_in_var = traceback.format_tb(exc_traceback)
See:
https://docs.python.org/3/library/traceback.html
Uncaught exception messages go to STDERR, so instead of implementing your logging in Python itself you could send STDERR to a file using whatever shell you're using to run your Python script. In a Bash script, you can do this with output redirection, as described in the BASH guide.
Examples
Append errors to file, other output to the terminal:
./test.py 2>> mylog.log
Overwrite file with interleaved STDOUT and STDERR output:
./test.py &> mylog.log
Here is a version that uses sys.excepthook
import traceback
import sys
logger = logging.getLogger()
def handle_excepthook(type, message, stack):
logger.error(f'An unhandled exception occured: {message}. Traceback: {traceback.format_tb(stack)}')
sys.excepthook = handle_excepthook
This is how I do it.
try:
do_something()
except:
# How can I log my exception here, complete with its traceback?
import traceback
traceback.format_exc() # this will print a complete trace to stout.
maybe not as stylish, but easier:
#!/bin/bash
log="/var/log/yourlog"
/path/to/your/script.py 2>&1 | (while read; do echo "$REPLY" >> $log; done)
To key off of others that may be getting lost in here, the way that works best with capturing it in logs is to use the traceback.format_exc() call and then split this string for each line in order to capture in the generated log file:
import logging
import sys
import traceback
try:
...
except Exception as ex:
# could be done differently, just showing you can split it apart to capture everything individually
ex_t = type(ex).__name__
err = str(ex)
err_msg = f'[{ex_t}] - {err}'
logging.error(err_msg)
# go through the trackback lines and individually add those to the log as an error
for l in traceback.format_exc().splitlines():
logging.error(l)
Heres a simple example taken from the python 2.6 documentation:
import logging
LOG_FILENAME = '/tmp/logging_example.out'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG,)
logging.debug('This message should go to the log file')