We Keep Coding

How can you catch a custom exception from Celery worker, or stop it being prefixed with `celery.backends.base`?

My Celery task raises a custom exception NonTransientProcessingError, which is then caught by AsyncResult.get(). Tasks.py:
class NonTransientProcessingError(Exception):
pass
#shared_task()
def throw_exception():
raise NonTransientProcessingError('Error raised by POC model for test purposes')
In the Python console:
from my_app.tasks import *
r = throw_exception.apply_async()
try:
r.get()
except NonTransientProcessingError as e:
print('caught NonTrans in type specific except clause')
But my custom exception is my_app.tasks.NonTransientProcessingError, whereas the exception raised by AsyncResult.get() is celery.backends.base.NonTransientProcessingError, so my except clause fails.
Traceback (most recent call last):
File "<input>", line 4, in <module>
File "/...venv/lib/python3.5/site-packages/celery/result.py", line 175, in get
raise meta['result']
celery.backends.base.NonTransientProcessingError: Error raised by POC model for test purposes
If I catch the exception within the task, it works fine. It is only when the exception is raised to the .get() call that it is renamed.
How can I raise a custom exception and catch it correctly?
I have confirmed that the same happens when I define a Task class and raise the custom exception in its on_failure method. The following does work:
try:
r.get()
except Exception as e:
if type(e).__name__ == 'NonTransientProcessingError':
print('specific exception identified')
else:
print('caught generic but not identified')
Outputs:
specific exception identified
But this can't be the best way of doing this? Ideally I'd like to catch exception superclasses for categories of behaviour.
I'm using Django 1.8.6, Python 3.5 and Celery 3.1.18, with a Redis 3.1.18, Python redis lib 2.10.3 backend.

import celery
from celery import shared_task
class NonTransientProcessingError(Exception):
pass
class CeleryTask(celery.Task):
def on_failure(self, exc, task_id, args, kwargs, einfo):
if isinstance(exc, NonTransientProcessingError):
"""
deal with NonTransientProcessingError
"""
pass
def run(self, *args, **kwargs):
pass
#shared_task(base=CeleryTask)
def add(x, y):
raise NonTransientProcessingError
Use a base Task with on_failure callback to catch custom exception.

It might be a bit late, but I have a solution to solve this issue. Not sure if it is the best one, but it solved my problem at least.
I had the same issue. I wanted to catch the exception produced in the celery task, but the result was of the class celery.backends.base.CustomException. The solution is in the following form:
import celery, time
from celery.result import AsyncResult
from celery.states import FAILURE
def reraise_celery_exception(info):
exec("raise {class_name}('{message}')".format(class_name=info.__class__.__name__, message=info.__str__()))
class CustomException(Exception):
pass
#celery.task(name="test")
def test():
time.sleep(10)
raise CustomException("Exception is raised!")
def run_test():
task = test.delay()
return task.id
def get_result(id):
task = AsyncResult(id)
if task.state == FAILURE:
reraise_celery_exception(task.info)
In this case you prevent your program from raising celery.backends.base.CustomException, and force it to raise the right exception.

How to safely handle an exception inside a context manager

I think I've read that exceptions inside a with do not allow __exit__ to be call correctly. If I am wrong on this note, pardon my ignorance.
So I have some pseudo code here, my goal is to use a lock context that upon __enter__ logs a start datetime and returns a lock id, and upon __exit__ records an end datetime and releases the lock:
def main():
raise Exception
with cron.lock() as lockid:
print('Got lock: %i' % lockid)
main()
How can I still raise errors in addition to existing the context safely?
Note: I intentionally raise the base exception in this pseudo-code as I want to exit safely upon any exception, not just expected exceptions.
Note: Alternative/standard concurrency prevention methods are irrelevant, I want to apply this knowledge to any general context management. I do not know if different contexts have different quirks.
PS. Is the finally block relevant?

The __exit__ method is called as normal if the context manager is broken by an exception. In fact, the parameters passed to __exit__ all have to do with handling this case! From the docs:
object.__exit__(self, exc_type, exc_value, traceback)
Exit the runtime context related to this object. The parameters describe the exception that caused the context to be exited. If the context was exited without an exception, all three arguments will be None.
If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method.
Note that __exit__() methods should not reraise the passed-in exception; this is the caller’s responsibility.
So you can see that the __exit__ method will be executed and then, by default, any exception will be re-raised after exiting the context manager. You can test this yourself by creating a simple context manager and breaking it with an exception:
DummyContextManager(object):
def __enter__(self):
print('Entering...')
def __exit__(self, exc_type, exc_value, traceback):
print('Exiting...')
# If we returned True here, any exception would be suppressed!
with DummyContextManager() as foo:
raise Exception()
When you run this code, you should see everything you want (might be out of order since print tends to end up in the middle of tracebacks):
Entering...
Exiting...
Traceback (most recent call last):
File "C:\foo.py", line 8, in <module>
raise Exception()
Exception

The best practice when using #contextlib.contextmanager was not quite clear to me from the above answer. I followed the link in the comment from #BenUsman.
If you are writing a context manager you must wrap the yield in try-finally block:
from contextlib import contextmanager
#contextmanager
def managed_resource(*args, **kwds):
# Code to acquire resource, e.g.:
resource = acquire_resource(*args, **kwds)
try:
yield resource
finally:
# Code to release resource, e.g.:
release_resource(resource)
>>> with managed_resource(timeout=3600) as resource:
... # Resource is released at the end of this block,
... # even if code in the block raises an exception

how to capture a traceback in gevent

I've spawned a Greenlet and linked it to a callable. Some time later, the Greenlet fails with an Exception. The linked callable gets called. That's all great!
Here's the issue:
The traceback for the Exception appears on my console, as you'd expect. But I want do things with that traceback within the linked callable. How do I get access to that traceback within the linked callable?
(My first instinct was to use traceback.extract_stack(), but it turns out that provides a traceback for the linked callable itself and not for the Exception.)

The traceback is intentionally not saved when the Greenlet dies. If it was saved, it would keep a lot of objects alive that are expected to be deleted, which matters especially if the object manages some resource (open file or socket).
If you want to save the traceback you have to do it yourself.

As an alternative to Stephen Diehl's solution using Greenlet.link_exception.
import traceback
import gevent
def job():
raise Exception('ooops')
def on_exception(greenlet):
try:
greenlet.get()
except Exception:
err = traceback.format_exc()
# Do something with `err`
g = gevent.spawn(job)
g.link_exception(on_exception)

The Greenlet object should have an exception property that you can look at:
http://www.gevent.org/gevent.html#gevent.Greenlet.exception

Just make sure you grab the exception value of the Greenlet and throw it outside the Greenlet, for example get returns either the value returned or raises the internal exception.
import traceback
import gevent
def fail():
return 0/0
gl = gevent.spawn(fail)
try:
gl.get()
except Exception as e:
stack_trace = traceback.format_exc() # here's your stacktrace
Should give you what you need.

How can I debug pserve using Eclipse?

I'm getting started with Pyramid development on Windows. I have Python 2.7 installed. I used virtualenv to create a nice sandbox for my Pyramid app. I also created PyDev 2.4 on Eclipse Indigo. I also created a separate PyDev interpreter just for my virutalenv, so it should have access to all the directories.
I set up a new debug configuration.
Project: testapp (the only project in the workspace)
Main module: ${workspace_loc:testapp/Scripts/pserve-script.py}
Args: development.ini
Working dir: Other: ${workspace_loc:testapp/testapp}
When I hit Debug, the output is:
pydev debugger: starting Starting server in PID 2208.
Unhandled exception in thread started by
Traceback (most recent call last):
File "C:\Tools\eclipse-cpp-indigo-SR1-incubation-win32-x86_64\eclipse\plugins\org.python.pydev.debug_2.3.0.2011121518\pysrc\pydevd.py", line 200, in __call__ Unhandled exception in thread started by
Traceback (most recent call last):
Unhandled exception in thread started by
Traceback (most recent call last):
File "C:\Tools\eclipse-cpp-indigo-SR1-incubation-win32-x86_64\eclipse\plugins\org.python.pydev.debug_2.3.0.2011121518\pysrc\pydevd.py", line 200, in __call__ self.original_func(*self.args, **self.kwargs)
Unhandled exception in thread started by
File "C:\Tools\eclipse-cpp-indigo-SR1-incubation-win32-x86_64\eclipse\plugins\org.python.pydev.debug_2.3.0.2011121518\pysrc\pydevd.py", line 200, in __call__
TypeErrorTraceback (most recent call last):
self.original_func(*self.args, **self.kwargs) :
File "C:\Tools\eclipse-cpp-indigo-SR1-incubation-win32-x86_64\eclipse\plugins\org.python.pydev.debug_2.3.0.2011121518\pysrc\pydevd.py", line 200, in __call__ self.original_func(*self.args, **self.kwargs)
TypeErrorThreadedTaskDispatcher object argument after ** must be a mapping, not tuple
TypeError: self.original_func(*self.args, **self.kwargs) : ThreadedTaskDispatcher object argument after ** must be a mapping, not tuple
TypeErrorThreadedTaskDispatcher object argument after ** must be a mapping, not tuple :
ThreadedTaskDispatcher object argument after ** must be a mapping, not tuple
serving on http://0.0.0.0:6543
Even though it says the server is running, it's not. Nothing is listening on that port.
Any idea on how to fix this? Debugging certainly isn't necessary, but I like having a fully set up development environment. Thanks!

Pyramid includes remarkably good debug support in the form of the debug toolbar.
Make sure that the line
pyramid.includes = pyramid_debugtoolbar
in your development.ini isn't commented out to enable it. It doesn't support Eclipse breakpoints, but gives almost everything else you'd want.

Haven't gotten into that error, but usually, on difficult to debug environments, the remote debugger (http://pydev.org/manual_adv_remote_debugger.html) may be used (that way it works kind of like pdb: add code to add a breakpoint, so, until that point, your program runs as usual).

Pyramid's pserve seems to use multiple threads like Fabio suggests might be the case. I found I could make breakpoints work by monkey-patching the ThreadTaskDispatcher before invoking pserve:
# Allow attaching PyDev to the web app
import sys;sys.path.append('..../pydev/2.5.0-2/plugins/org.python.pydev.debug_2.4.0.201208051101/pysrc/')
# Monkey patch the thread task dispatcher, so it sets up the tracer in the worker threads
from waitress.task import ThreadedTaskDispatcher
_prev_start_new_thread = ThreadedTaskDispatcher.start_new_thread
def start_new_thread(ttd, fn, args):
def settrace_and_call(*args, **kwargs):
import pydevd ; pydevd.settrace(suspend=False)
return fn(*args, **kwargs)
from thread import start_new_thread
start_new_thread(settrace_and_call, args)
ThreadedTaskDispatcher.start_new_thread = start_new_thread
Note, I also tried:
set_trace(..., trace_only_current_thread=False)
But this either makes the app unusably slow, or doesn't work for some other reason.
Having done the above, when run the app will automatically register it with pydev debug server running locally. See:
http://pydev.org/manual_adv_remote_debugger.html

Log exception with traceback in Python

How can I log my Python exceptions?
try:
do_something()
except:
# How can I log my exception here, complete with its traceback?

Use logging.exception from within the except: handler/block to log the current exception along with the trace information, prepended with a message.
import logging
LOG_FILENAME = '/tmp/logging_example.out'
logging.basicConfig(filename=LOG_FILENAME, level=logging.DEBUG)
logging.debug('This message should go to the log file')
try:
run_my_stuff()
except:
logging.exception('Got exception on main handler')
raise
Now looking at the log file, /tmp/logging_example.out:
DEBUG:root:This message should go to the log file
ERROR:root:Got exception on main handler
Traceback (most recent call last):
File "/tmp/teste.py", line 9, in <module>
run_my_stuff()
NameError: name 'run_my_stuff' is not defined

Use exc_info options may be better, remains warning or error title:
try:
# coode in here
except Exception as e:
logging.error(e, exc_info=True)

My job recently tasked me with logging all the tracebacks/exceptions from our application. I tried numerous techniques that others had posted online such as the one above but settled on a different approach. Overriding traceback.print_exception.
I have a write up at http://www.bbarrows.com/ That would be much easier to read but Ill paste it in here as well.
When tasked with logging all the exceptions that our software might encounter in the wild I tried a number of different techniques to log our python exception tracebacks. At first I thought that the python system exception hook, sys.excepthook would be the perfect place to insert the logging code. I was trying something similar to:
import traceback
import StringIO
import logging
import os, sys
def my_excepthook(excType, excValue, traceback, logger=logger):
logger.error("Logging an uncaught exception",
exc_info=(excType, excValue, traceback))
sys.excepthook = my_excepthook
This worked for the main thread but I soon found that the my sys.excepthook would not exist across any new threads my process started. This is a huge issue because most everything happens in threads in this project.
After googling and reading plenty of documentation the most helpful information I found was from the Python Issue tracker.
The first post on the thread shows a working example of the sys.excepthook NOT persisting across threads (as shown below). Apparently this is expected behavior.
import sys, threading
def log_exception(*args):
print 'got exception %s' % (args,)
sys.excepthook = log_exception
def foo():
a = 1 / 0
threading.Thread(target=foo).start()
The messages on this Python Issue thread really result in 2 suggested hacks. Either subclass Thread and wrap the run method in our own try except block in order to catch and log exceptions or monkey patch threading.Thread.run to run in your own try except block and log the exceptions.
The first method of subclassing Thread seems to me to be less elegant in your code as you would have to import and use your custom Thread class EVERYWHERE you wanted to have a logging thread. This ended up being a hassle because I had to search our entire code base and replace all normal Threads with this custom Thread. However, it was clear as to what this Thread was doing and would be easier for someone to diagnose and debug if something went wrong with the custom logging code. A custome logging thread might look like this:
class TracebackLoggingThread(threading.Thread):
def run(self):
try:
super(TracebackLoggingThread, self).run()
except (KeyboardInterrupt, SystemExit):
raise
except Exception, e:
logger = logging.getLogger('')
logger.exception("Logging an uncaught exception")
The second method of monkey patching threading.Thread.run is nice because I could just run it once right after __main__ and instrument my logging code in all exceptions. Monkey patching can be annoying to debug though as it changes the expected functionality of something. The suggested patch from the Python Issue tracker was:
def installThreadExcepthook():
"""
Workaround for sys.excepthook thread bug
From
http://spyced.blogspot.com/2007/06/workaround-for-sysexcepthook-bug.html
(https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230540&group_id=5470).
Call once from __main__ before creating any threads.
If using psyco, call psyco.cannotcompile(threading.Thread.run)
since this replaces a new-style class method.
"""
init_old = threading.Thread.__init__
def init(self, *args, **kwargs):
init_old(self, *args, **kwargs)
run_old = self.run
def run_with_except_hook(*args, **kw):
try:
run_old(*args, **kw)
except (KeyboardInterrupt, SystemExit):
raise
except:
sys.excepthook(*sys.exc_info())
self.run = run_with_except_hook
threading.Thread.__init__ = init
It was not until I started testing my exception logging I realized that I was going about it all wrong.
To test I had placed a
raise Exception("Test")
somewhere in my code. However, wrapping a a method that called this method was a try except block that printed out the traceback and swallowed the exception. This was very frustrating because I saw the traceback bring printed to STDOUT but not being logged. It was I then decided that a much easier method of logging the tracebacks was just to monkey patch the method that all python code uses to print the tracebacks themselves, traceback.print_exception.
I ended up with something similar to the following:
def add_custom_print_exception():
old_print_exception = traceback.print_exception
def custom_print_exception(etype, value, tb, limit=None, file=None):
tb_output = StringIO.StringIO()
traceback.print_tb(tb, limit, tb_output)
logger = logging.getLogger('customLogger')
logger.error(tb_output.getvalue())
tb_output.close()
old_print_exception(etype, value, tb, limit=None, file=None)
traceback.print_exception = custom_print_exception
This code writes the traceback to a String Buffer and logs it to logging ERROR. I have a custom logging handler set up the 'customLogger' logger which takes the ERROR level logs and send them home for analysis.

You can log all uncaught exceptions on the main thread by assigning a handler to sys.excepthook, perhaps using the exc_info parameter of Python's logging functions:
import sys
import logging
logging.basicConfig(filename='/tmp/foobar.log')
def exception_hook(exc_type, exc_value, exc_traceback):
logging.error(
"Uncaught exception",
exc_info=(exc_type, exc_value, exc_traceback)
)
sys.excepthook = exception_hook
raise Exception('Boom')
If your program uses threads, however, then note that threads created using threading.Thread will not trigger sys.excepthook when an uncaught exception occurs inside them, as noted in Issue 1230540 on Python's issue tracker. Some hacks have been suggested there to work around this limitation, like monkey-patching Thread.__init__ to overwrite self.run with an alternative run method that wraps the original in a try block and calls sys.excepthook from inside the except block. Alternatively, you could just manually wrap the entry point for each of your threads in try/except yourself.

You can get the traceback using a logger, at any level (DEBUG, INFO, ...). Note that using logging.exception, the level is ERROR.
# test_app.py
import sys
import logging
logging.basicConfig(level="DEBUG")
def do_something():
raise ValueError(":(")
try:
do_something()
except Exception:
logging.debug("Something went wrong", exc_info=sys.exc_info())
DEBUG:root:Something went wrong
Traceback (most recent call last):
File "test_app.py", line 10, in <module>
do_something()
File "test_app.py", line 7, in do_something
raise ValueError(":(")
ValueError: :(
EDIT:
This works too (using python 3.6)
logging.debug("Something went wrong", exc_info=True)

What I was looking for:
import sys
import traceback
exc_type, exc_value, exc_traceback = sys.exc_info()
traceback_in_var = traceback.format_tb(exc_traceback)
See:
https://docs.python.org/3/library/traceback.html

Uncaught exception messages go to STDERR, so instead of implementing your logging in Python itself you could send STDERR to a file using whatever shell you're using to run your Python script. In a Bash script, you can do this with output redirection, as described in the BASH guide.
Examples
Append errors to file, other output to the terminal:
./test.py 2>> mylog.log
Overwrite file with interleaved STDOUT and STDERR output:
./test.py &> mylog.log

Here is a version that uses sys.excepthook
import traceback
import sys
logger = logging.getLogger()
def handle_excepthook(type, message, stack):
logger.error(f'An unhandled exception occured: {message}. Traceback: {traceback.format_tb(stack)}')
sys.excepthook = handle_excepthook

This is how I do it.
try:
do_something()
except:
# How can I log my exception here, complete with its traceback?
import traceback
traceback.format_exc() # this will print a complete trace to stout.

maybe not as stylish, but easier:
#!/bin/bash
log="/var/log/yourlog"
/path/to/your/script.py 2>&1 | (while read; do echo "$REPLY" >> $log; done)

To key off of others that may be getting lost in here, the way that works best with capturing it in logs is to use the traceback.format_exc() call and then split this string for each line in order to capture in the generated log file:
import logging
import sys
import traceback
try:
...
except Exception as ex:
# could be done differently, just showing you can split it apart to capture everything individually
ex_t = type(ex).__name__
err = str(ex)
err_msg = f'[{ex_t}] - {err}'
logging.error(err_msg)
# go through the trackback lines and individually add those to the log as an error
for l in traceback.format_exc().splitlines():
logging.error(l)

Heres a simple example taken from the python 2.6 documentation:
import logging
LOG_FILENAME = '/tmp/logging_example.out'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG,)
logging.debug('This message should go to the log file')