I would like to implement a deferred exception in Python that is OK to store somewhere but as soon as it is used in any way, it raises the exception that was deferred. Something like this:
# this doesn't work but it's a start
class DeferredException(object):
def __init__(self, exc):
self.exc = exc
def __getattr__(self, key):
raise self.exc
# example:
mydict = {'foo': 3}
try:
myval = obtain_some_number()
except Exception as e:
myval = DeferredException(e)
mydict['myval'] = myval
def plus_two(x):
print x+2
# later on...
plus_two(mydict['foo']) # prints 5
we_dont_use_this_val = mydict['myval'] # Always ok to store this value if not used
plus_two(mydict['myval']) # If obtain_some_number() failed earlier,
# re-raises the exception, otherwise prints the value + 2.
The use case is that I want to write code to analyze some values from incoming data; if this code fails but the results are never used, I want it to fail quietly; if it fails but the results are used later, then I'd like the failure to propagate.
Any suggestions on how to do this? If I use my DeferredException class I get this result:
>>> ke = KeyError('something')
>>> de = DeferredException(ke)
>>> de.bang # yay, this works
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in __getattr__
KeyError: 'something'
>>> de+2 # boo, this doesn't
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'DeferredException' and 'int'
Read section 3.4.12 of the docs, "Special method lookup for new-style classes." It explains exactly the problem you have encountered. The normal attribute lookup is bypassed by the interpreter for certain operators, such as addition (as you found out the hard way). Thus the statement de+2 in your code never calls your getattr function.
The only solution, according to that section, is to insure that "the special method must be set on the class object itself in order to be consistently invoked by the interpreter."
Perhaps you'd be better off storing all your deferred exceptions in a global list, wrapping your entire program in a try:finally: statement, and printing out the whole list in the finally block.
Related
When writing defensive code in python (e.g. you're handling some user input or whatever), I find it useful to return Exception objects alongside regular computation results, so they can be discarded/logged or processed in some other way. Consider the following snippet:
import logging
from traceback import TracebackException
from typing import Union
logging.basicConfig(level=logging.INFO)
def _compute(x) -> int:
return len(x)
def compute(x) -> Union[int, Exception]:
try:
return _compute(x)
except Exception as e:
return e
inputs = [
'whatever',
1,
'ooo',
None,
]
outputs = []
for i in inputs:
r = compute(i)
outputs.append(r)
for i, r in zip(inputs, outputs):
logging.info('compute(%s)', i)
if isinstance(r, Exception):
logging.exception(r)
else:
logging.info(r)
This results in the following output
INFO:root:compute(whatever)
INFO:root:8
INFO:root:compute(1)
ERROR:root:object of type 'int' has no len()
NoneType: None
INFO:root:compute(ooo)
INFO:root:3
INFO:root:compute(None)
ERROR:root:object of type 'NoneType' has no len()
NoneType: None
So you can see that useful exception information like stacktrace is lost, which makes it a bit hard to debug the cause of exception.
This can be fixed by logging exception as logging.exception(r, exc_info=r):
INFO:root:compute(whatever)
INFO:root:8
INFO:root:compute(1)
ERROR:root:object of type 'int' has no len()
Traceback (most recent call last):
File "/tmp/test.py", line 15, in compute
return _compute(x)
File "/tmp/test.py", line 10, in _compute
return len(x)
TypeError: object of type 'int' has no len()
INFO:root:compute(ooo)
INFO:root:3
INFO:root:compute(None)
ERROR:root:object of type 'NoneType' has no len()
Traceback (most recent call last):
File "/tmp/test.py", line 15, in compute
return _compute(x)
File "/tmp/test.py", line 10, in _compute
return len(x)
TypeError: object of type 'NoneType' has no len()
My question is -- why doesn't logging.exception method do this by default, if the argument passed to it happens to be an Exception? I tried searching in PEPs/etc but wasn't really fruitful.
My only guess is that logging.exception is essentially just a special case of logging.error, so in principle logging.exception method doesn't know whether is' passed an Exception object or something else. So supporting this would require some code, e.g. checking whether isinstance(msg, Exception), and perhaps the authors of logging library decided it's a bit too specific. But IMO it makes sense considering in practice in most cases logging.exception is passed an Exception object.
logging.exception does log the traceback by default. However, you're using it wrong. As the docs say,
This function should only be called from an exception handler.
logging.exception does not expect to be passed an exception instance, or any sort of exception information whatsoever. It uses sys.exc_info to gather info about the exception currently being handled, which only works if an exception is currently being handled. If you call it outside of an exception handler, it breaks.
I'm seeing some unexpected behavior in my code related to python multiprocessing, and the Manager class in particular. I wrote out a super simple example to try and better understand what's going on:
import multiprocessing as mp
from collections import defaultdict
def process(d):
print('doing the process')
d['a'] = []
d['a'].append(1)
d['a'].append(2)
def main():
pool = mp.Pool(mp.cpu_count())
with mp.Manager() as manager:
d = manager.dict({'c': 2})
result = pool.apply_async(process, args=(d))
print(result.get())
pool.close()
pool.join()
print(d)
if __name__ == '__main__':
main()
This fails, and the stack trace printed from result.get() is as follows:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "<string>", line 2, in __iter__
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 825, in _callmethod
proxytype = self._manager._registry[token.typeid][-1]
AttributeError: 'NoneType' object has no attribute '_registry'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "mp_test.py", line 34, in <module>
main()
File "mp_test.py", line 25, in main
print(result.get())
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
AttributeError: 'NoneType' object has no attribute '_registry'
I'm still unclear on what's happening here. This seems to me to be a very, very straightforward application of the Manager class. It's nearly a copy of the actual example used in the official python documentation, with the only difference being that i'm using a pool and running the process with apply_async. I'm doing this because that's what i'm using in my actual project.
To clarify, I wouldn't get a stack trace if I didn't have the result = and print(result.get()) in there. I just see {'c': 2} printed when I run the script, which indicated to me that something was going wrong and wasn't being shown.
A couple things to start with: first, this isn't the code you ran. The code you posted has
result = pool.apply_async(process2, args=(d))
but there is no process2() defined. Assuming "process` was intended, the next thing is the
args=(d)
part. That's the same as typing
args=d
but that's not what's needed. You need to pass a sequence of the intended arguments. So you need to change that part to
args=(d,) # build a 1-tuple
or
args=[d] # build a list
Then the output changes, to
{'c': 2, 'a': []}
Why aren't 1 and 2 in the the 'a' list? Because it's only the dict itself that lives on the manager server.
d['a'].append(1)
first gets the mapping for 'a' from the server, which is an empty list. But that empty list is not shared in any way - it's local to process(). You append 1 to it, and then it's thrown away - the server knows nothing about it. Same thing for 2.
To get what you want, you need to "do something" to tell the manager server about what you changed; e.g.,
d['a'] = L = []
L.append(1)
L.append(2)
d['a'] = L
I am using python unittest to test a web app using selenium. In my teardownClass, I am calling cls.fail but it returns "AttributeError" saying that "failureException" not found in the "string".
Here is what my teardownClass method looks like:
#classmethod
def tearDownClass(cls):
browser_logs = cls.driver.get_log("browser")
errors = [log_entry['message'] for log_entry in browser_logs if logentry['level'] == 'SEVERE']
if errors:
cls.fail(errors)
this returns the following attributeError:
======================================================================
ERROR: tearDownClass (__main__.unitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/shahrukh/Desktop/apollotest/test_users.py", line 239, in tearDownClass
cls.fail("error")
File "/usr/lib/python3.6/unittest/case.py", line 670, in fail
raise self.failureException(msg)
AttributeError: 'str' object has no attribute 'failureException'
----------------------------------------------------------------------
"fail" method from /usr/lib/python3.6/unittest/case.py file:
def fail(self, msg=None):
"""Fail immediately, with the given message."""
raise self.failureException(msg)
This method is part of "Testcase" class
The problem is that fail method above does not find the reference to self. "errors" in my code is a string object. So when I call cls.fail(errors) it considers self=errors and msg remains None.
I don't get this problem if I change my code statement to cls.fail(cls, errors).
I want to understand why am I experiencing this behavior? Am I missing anything because according to my understanding cls.fail should mean that self in fail method is equal to cls.
Would really appreciate some help.
I am creating tests for a python project. The normal tests work just fine, however I want to test if in a certain condition my function raises a self-defined exception. Therefor I want to use assertRaises(Exception, Function). Any ideas?
The function that raises the exception is:
def connect(comp1, comp2):
if comp1 == comp2:
raise e.InvalidConnectionError(comp1, comp2)
...
The exception is:
class InvalidConnectionError(Exception):
def __init__(self, connection1, connection2):
self._connection1 = connection1
self._connection2 = connection2
def __str__(self):
string = '...'
return string
The test method is the following:
class TestConnections(u.TestCase):
def test_connect_error(self):
comp = c.PowerConsumer('Bus', True, 1000)
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
However I get the following error:
Error
Traceback (most recent call last):
File "C:\Users\t5ycxK\PycharmProjects\ElectricPowerDesign\test_component.py", line 190, in test_connect_error
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
File "C:\Users\t5ycxK\PycharmProjects\ElectricPowerDesign\component.py", line 428, in connect
raise e.InvalidConnectionError(comp1, comp2)
InvalidConnectionError: <unprintable InvalidConnectionError object>
assertRaises expects to actually perform the call. Yet, you already perform it by yourself, thereby throwing the error before assertRaises actually executes.
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
# run this ^ with first static argument ^ and second argument ^ from `c.connect(comp, comp)`
Use either of those instead:
self.assertRaises(e.InvalidConnectionError, c.connect, comp, comp)
with self.assertRaises(e.InvalidConnectionError):
c.connect(comp, comp)
I've got 2 files : main.py and batsol.py
batsol.py contains a class and main.py is creating some instances from the class Batsol.
So I'll show you a concise version of my code...
class Batsol:
def __init__(self, addressCan = None, name = None) :
self.addressCan = addressCan
self.name = name
#other stuff ...
Then my main.py :
from batsol import Batsol
# other import and code ...
print(callable(Batsol))
bs1 = Batsol()
# code...
if len(listener.ring_buffer) == 0 :
for Batsol in tab_BS :
try:
print(tab_BS[Batsol])
except (IndexError):
pass
# code...
while(True) :
# for and if interlocked
print(callable(Batsol))
bs2 = Batsol()
The console shows :
True
False
Traceback (most recent call last):
File "./main.py", line 135, in <module>
bs2 = Batsol()
TypeError: 'int' object is not callable
the second part of the traceback is not linked to other stuff i'm doing in my code (thread not terminated properly... something like this) , in my opinion
Exception ignored in: <module 'threading' from '/usr/lib/python3.4/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.4/threading.py", line 1292, in _shutdown
t = _pickSomeNonDaemonThread()
File "/usr/lib/python3.4/threading.py", line 1300, in _pickSomeNonDaemonThread
if not t.daemon and t.is_alive():
TypeError: 'bool' object is not callable
WHY my object is not callable inside my tests loops ???
It drives me crazy...
Your shadowing occurs in this code fragment:
if len(listener.ring_buffer) == 0 :
for Batsol in tab_BS :
try:
print(tab_BS[Batsol])
except (IndexError):
pass
time.sleep(4)
for-in construct on sequences works as following:
Sequence is asked for next (first, second, ... last) element. Internal pointer keeps track of element in current iteration.
Element gets assigned to name on left side of "in".
Go to 1.
After loop finishes, Batsol is no longer your class, but last element from tab_BS.
I'd suggest getting a better IDE, or using good static code analysis tool (Pylint / Flake8 etc.) as this kind of error is easily detected by e.g. PyCharm (your code shadows name from outer scope).
Related: How bad is shadowing names defined in outer scopes?