We Keep Coding

__getattribute__ running into recursion

I have a filesystem object (let it be fs). The object makes use of jwt token for authentication. When we create the filesystem object, the authentication is done. Once, we have the object, we can call method like ls to list the directory, etc. The token has some expiration time.
The issue is when I call fs.ls('/'), there is no validation in the backend for the token like is token still valid or not. What I want is, when ever there is call to a method on the object, I will intercept the call and check for the token expiration. If it is about to expire will update the token.
Searching and reading on SO, I came to about __getattribute__. But my code is not working as expected. Sometimes I am getting recursion error or sometimes I am getting null values.
This code gives recursion error:
class FileSystem(adlfs.FileSystem):
def __init__(self, *args, **kwargs):
try:
self.name, self.token = self._get_token()
if self.name is not None:
kwargs["name"] = self.name
if self.token is not None:
kwargs["token"] = self.token
self.exp = self.token.token.expires_on
super().__init__(*args, **kwargs)
except Exception as exception:
print(exception)
def _get_token(self) -> (str, 'Credential'):
return name, token
def __getattribute__(self, attr):
attribute = super().__getattribute__(attr)
if callable(attribute):
curr_time = int(time.time())
if curr_time > self.token_exp:
def refresh_token(*args, **kwargs):
self.name, self.token = self._get_token()
self.token_exp = updateTokenExpiration(self.token)
super().updateConnection()
return attribute(*args, **kwargs)
return refresh_token
else:
return attribute
else:
return attribute

Typically, in __getattribute__() all occurrences of self.something need to be replaced with super().__getattribute__('something') (unless they are targets of an assignment or del).
In your case that can be relaxed for non-callables (as for them your implementation of __getattribute__() just calls super().__getattribute__(...), practically without doing anything more), but for callables it still needs to be adjusted, for example:
# before adjustment:
self.name, self.token = self._get_token()
# after adjustment:
self.name, self.token = super().__getattribute__('_get_token')()
Otherwise your implementation calls itself, so that an infinite recursion occurs.
Replacing simple attribute access with such calls can, however, be tedious if you have many such places in your __getattribute__()...
A possible trick is to use in your definition of __getattribute__() only such attributes/methods of self that are specially named, e.g. with _ga_ at the beginning of their names, and filter out such names from the customized behavior of your __getattribute__(), e.g.:
def __getattribute__(self, name):
if name.startswith('_ga_'):
return super().__getattribute__(name)
...here the actual part of your custom implementation
...in which you can freely use `self._ga_whatever...`

How to access the instance of a class from an inner decorator class?

I have a class that handles the API calls to a server. Certain methods within the class require the user to be logged in. Since it is possible for the session to run out, I need some functionality that re-logins the user once the session timed out. My idea was to use a decorator. If I try it like this
class Outer_Class():
class login_required():
def __init__(self, decorated_func):
self.decorated_func = decorated_func
def __call__(self, *args, **kwargs):
try:
response = self.decorated_func(*args, **kwargs)
except:
print('Session probably timed out. Logging in again ...')
args[0]._login()
response = self.decorated_func(*args, **kwargs)
return response
def __init__(self):
self.logged_in = False
self.url = 'something'
self._login()
def _login(self):
print(f'Logging in on {self.url}!')
self.logged_in = True
#this method requires the user to be logged in
#login_required
def do_something(self, param_1):
print('Doing something important with param_1')
if (): #..this fails
raise Exception()
I get an error. AttributeError: 'str' object has no attribute '_login'
Why do I not get a reference to the Outer_Class-instance handed over via *args? Is there another way to get a reference to the instance?
Found this answer How to get instance given a method of the instance? , but the decorated_function doesn't seem to have a reference to it's own instance.
It works fine, when Im using a decorator function outside of the class. This solves the problem, but I like to know, if it is possible to solve the this way.

The problem is that the magic of passing the object as the first hidden parameter only works for a non static method. As your decorator returns a custom callable object which is not a function, it never receives the calling object which is just lost in the call. So when you try to call the decorated function, you only pass it param_1 in the position of self. You get a first exception do_something() missing 1 required positional argument: 'param_1', fall into the except block and get your error.
You can still tie the decorator to the class, but it must be a function to have self magic work:
class Outer_Class():
def login_required(decorated_func):
def inner(self, *args, **kwargs):
print("decorated called")
try:
response = decorated_func(self, *args, **kwargs)
except:
print('Session probably timed out. Logging in again ...')
self._login()
response = decorated_func(self, *args, **kwargs)
return response
return inner
...
#this method requires the user to be logged in
#login_required
def do_something(self, param_1):
print('Doing something important with param_1', param_1)
if (False): #..this fails
raise Exception()
You can then successfully do:
>>> a = Outer_Class()
Logging in on something!
>>> a.do_something("foo")
decorated called
Doing something important with param_1

You have the command of
args[0]._login()
in the except. Since args[0] is a string and it doesn't have a _login method, you get the error message mentioned in the question.

Python How to force object instantiation via Context Manager?

I want to force object instantiation via class context manager. So make it impossible to instantiate directly.
I implemented this solution, but technically user can still instantiate object.
class HessioFile:
"""
Represents a pyhessio file instance
"""
def __init__(self, filename=None, from_context_manager=False):
if not from_context_manager:
raise HessioError('HessioFile can be only use with context manager')
And context manager:
#contextmanager
def open(filename):
"""
...
"""
hessfile = HessioFile(filename, from_context_manager=True)
Any better solution ?

If you consider that your clients will follow basic python coding principles then you can guarantee that no method from your class will be called if you are not within the context.
Your client is not supposed to call __enter__ explicitly, therefore if __enter__ has been called you know your client used a with statement and is therefore inside context (__exit__ will be called).
You just need to have a boolean variable that helps you remember if you are inside or outside context.
class Obj:
def __init__(self):
self._inside_context = False
def __enter__(self):
self._inside_context = True
print("Entering context.")
return self
def __exit__(self, *exc):
print("Exiting context.")
self._inside_context = False
def some_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some stuff with", name)
def some_other_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some other stuff with", name)
with Obj() as inst_a:
inst_a.some_stuff("A")
inst_a.some_other_stuff("A")
inst_b = Obj()
with inst_b:
inst_b.some_stuff("B")
inst_b.some_other_stuff("B")
inst_c = Obj()
try:
inst_c.some_stuff("c")
except Exception:
print("Instance C couldn't do stuff.")
try:
inst_c.some_other_stuff("c")
except Exception:
print("Instance C couldn't do some other stuff.")
This will print:
Entering context.
Doing some stuff with A
Doing some other stuff with A
Exiting context.
Entering context.
Doing some stuff with B
Doing some other stuff with B
Exiting context.
Instance C couldn't do stuff.
Instance C couldn't do some other stuff.
Since you'll probably have many methods that you want to "protect" from being called from outside context, then you can write a decorator to avoid repeating the same code to test for your boolean:
def raise_if_outside_context(method):
def decorator(self, *args, **kwargs):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
return method(self, *args, **kwargs)
return decorator
Then change your methods to:
#raise_if_outside_context
def some_other_stuff(self, name):
print("Doing some other stuff with", name)

I suggest the following approach:
class MainClass:
def __init__(self, *args, **kwargs):
self._class = _MainClass(*args, **kwargs)
def __enter__(self):
print('entering...')
return self._class
def __exit__(self, exc_type, exc_val, exc_tb):
# Teardown code
print('running exit code...')
pass
# This class should not be instantiated directly!!
class _MainClass:
def __init__(self, attribute1, attribute2):
self.attribute1 = attribute1
self.attribute2 = attribute2
...
def method(self):
# execute code
if self.attribute1 == "error":
raise Exception
print(self.attribute1)
print(self.attribute2)
with MainClass('attribute1', 'attribute2') as main_class:
main_class.method()
print('---')
with MainClass('error', 'attribute2') as main_class:
main_class.method()
This will outptut:
entering...
attribute1
attribute2
running exit code...
---
entering...
running exit code...
Traceback (most recent call last):
File "scratch_6.py", line 34, in <module>
main_class.method()
File "scratch_6.py", line 25, in method
raise Exception
Exception

None that I am aware of. Generally, if it exists in python, you can find a way to call it. A context manager is, in essence, a resource management scheme... if there is no use-case for your class outside of the manager, perhaps the context management could be integrated into the methods of the class? I would suggest checking out the atexit module from the standard library. It allows you to register cleanup functions much in the same way that a context manager handles cleanup, but you can bundle it into your class, such that each instantiation has a registered cleanup function. Might help.
It is worth noting that no amount of effort will prevent people from doing stupid things with your code. Your best bet is generally to make it as easy as possible for people to do smart things with your code.

You can think of hacky ways to try and enforce this (like inspecting the call stack to forbid direct calls to your object, boolean attribute that is set upon __enter__ that you check before allowing other actions on the instance) but that will eventually become a mess to understand and explain to others.
Irregardless, you should also be certain that people will always find ways to bypass it if wanted. Python doesn't really tie your hands down, if you want to do something silly it lets you do it; responsible adults, right?
If you need an enforcement, you'd be better off supplying it as a documentation notice. That way if users opt to instantiate directly and trigger unwanted behavior, it's their fault for not following guidelines for your code.

Django Testing: See traceback where wrong Response gets created

This pattern is from the django docs:
class SimpleTest(unittest.TestCase):
def test_details(self):
client = Client()
response = client.get('/customer/details/')
self.assertEqual(response.status_code, 200)
From: https://docs.djangoproject.com/en/1.8/topics/testing/tools/#default-test-client
If the test fails, the error message does not help very much. For example if the status_code is 302, then I see 302 != 200.
The question is now: Where does the wrong HTTPResponse get created?
I would like to see the stacktrace of the interpreter where the wrong HTTPResponse object get created.
I read the docs for the assertions of django but found no matching method.
Update
This is a general question: How to see the wanted information immediately if the assertion fails? Since these assertions (self.assertEqual(response.status_code, 200)) are common, I don't want to start debugging.
Update 2016
I had the same idea again, found the current answer not 100% easy. I wrote a new answer, which has a simple to use solution (subclass of django web client): Django: assertEqual(response.status_code, 200): I want to see useful stack of functions calls

I think it could be achieved by creating a TestCase subclass that monkeypatches django.http.response.HttpResponseBase.__init__() to record a stack trace and store it on the Response object, then writing an assertResponseCodeEquals(response, status_code=200) method that prints the stored stack trace on failure to show where the Response was created.
I could actually really use a solution for this myself, and might look at implementing it.
Update:
Here's a v1 implementation, which could use some refinement (eg only printing relevant lines of the stack trace).
import mock
from traceback import extract_stack, format_list
from django.test.testcases import TestCase
from django.http.response import HttpResponseBase
orig_response_init = HttpResponseBase.__init__
def new_response_init(self, *args, **kwargs):
orig_response_init(self, *args, **kwargs)
self._init_stack = extract_stack()
class ResponseTracebackTestCase(TestCase):
#classmethod
def setUpClass(cls):
cls.patcher = mock.patch.object(HttpResponseBase, '__init__', new_response_init)
cls.patcher.start()
#classmethod
def tearDownClass(cls):
cls.patcher.stop()
def assertResponseCodeEquals(self, response, status_code=200):
self.assertEqual(response.status_code, status_code,
"Response code was '%s', expected '%s'" % (
response.status_code, status_code,
) + '\n' + ''.join(format_list(response._init_stack))
)
class MyTestCase(ResponseTracebackTestCase):
def test_index_page_returns_200(self):
response = self.client.get('/')
self.assertResponseCodeEquals(response, 200)

How do I see the traceback if the assertion fails without debugging
If the assertion fails, there isn't a traceback. The client.get() hasn't failed, it just returned a different response than you were expecting.
You could use a pdb to step through the client.get() call, and see why it is returning the unexpected response.

Maybe this could work for you:
class SimpleTest(unittest.TestCase):
#override_settings(DEBUG=True)
def test_details(self):
client = Client()
response = client.get('/customer/details/')
self.assertEqual(response.status_code, 200, response.content)
Using #override_settings to have DEBUG=True will have the stacktrace just as if you were running an instance in DEBUG mode.
Secondly, in order to provide the content of the response, you need to either print it or log it using the logging module, or add it as your message for the assert method. Without a debugger, once you assert, it is too late to print anything useful (usually).
You can also configure logging and add a handler to save messages in memory, and print all of that; either in a custom assert method or in a custom test runner.

I was inspired by the solution that #Fush proposed but my code was using assertRedirects which is a longer method and was a bit too much code to duplicate without feeling bad about myself.
I spent a bit of time figuring out how I could just call super() for each assert and came up with this. I've included 2 example assert methods - they would all basically be the same. Maybe some clever soul can think of some metaclass magic that does this for all methods that take 'response' as their first argument.
from bs4 import BeautifulSoup
from django.test.testcases import TestCase
class ResponseTracebackTestCase(TestCase):
def _display_response_traceback(self, e, content):
soup = BeautifulSoup(content)
assert False, u'\n\nOriginal Traceback:\n\n{}'.format(
soup.find("textarea", {"id": "traceback_area"}).text
)
def assertRedirects(self, response, *args, **kwargs):
try:
super(ResponseTracebackTestCase, self).assertRedirects(response, *args, **kwargs)
except Exception as e:
self._display_response_traceback(e, response.content)
def assertContains(self, response, *args, **kwargs):
try:
super(ResponseTracebackTestCase, self).assertContains(response, *args, **kwargs)
except Exception as e:
self._display_response_traceback(e, response.content)

I subclassed the django web client, to get this:
Usage
def test_foo(self):
...
MyClient().get(url, assert_status=200)
Implementation
from django.test import Client
class MyClient(Client):
def generic(self, method, path, data='',
content_type='application/octet-stream', secure=False,
assert_status=None,
**extra):
if assert_status:
return self.assert_status(assert_status, super(MyClient, self).generic, method, path, data, content_type, secure, **extra)
return super(MyClient, self).generic(method, path, data, content_type, secure, **extra)
#classmethod
def assert_status(cls, status_code, method_pointer, *args, **kwargs):
assert hasattr(method_pointer, '__call__'), 'Method pointer needed, looks like the result of a method call: %r' % (method_pointer)
def new_init(self, *args, **kwargs):
orig_response_init(self, *args, **kwargs)
if not status_code == self.status_code:
raise HTTPResponseStatusCodeAssertionError('should=%s is=%s' % (status_code, self.status_code))
def reraise_exception(*args, **kwargs):
raise
with mock.patch('django.core.handlers.base.BaseHandler.handle_uncaught_exception', reraise_exception):
with mock.patch.object(HttpResponseBase, '__init__', new_init):
return method_pointer(*args, **kwargs)
Conclusion
This results in a long exception if a http response with a wrong status code was created. If you are not afraid of long exceptions, you see very fast the root of the problem. That's what I want, I am happy.
Credits
This was based on other answers of this question.

python 2.6 exceptions.TypeError: unbound method _init_() ESMTP client instance

I am a completely new user to Python and BuildBot. Currently I am using an e-mail alert
when the BuildBot build status changes (moves from success to fail, or vice versa), and failing will e-mail every time there is a failed build. I am encountering the following Python error when sending an email is attempted.
--- <exception caught here> ---
**ESMTPClient.__init__(self, secret, contextFactory, *args, **kw)
exceptions.TypeError?: unbound method init() must be called with ESMTPClient
instance as first argument (got ESMTPSender instance instead)**
I have found some examples of this error online when searching for an answer, including
You just need to pass 'self' as an argument to 'Thread.init' and
calling the super class
but I am still unsure why there is an error. I would appreciate any guidance/help on why this error has occurred and how to go about resolving the issue. I am not the author of this code so I am unsure of what to be looking for to solve the problem.
The email was working before the following code was changed from gmail account to company account.
c['status'].append(mail.MailNotifier(
fromaddr="load.builder#company.co.uk",
extraRecipients=["example#company.com",
],
sendToInterestedUsers=False,
mode=('change', 'failing'),
relayhost="smtp.company.lan",
useTls=True,
smtpUser="lbuilder",
smtpPassword="password"))
Here's the block of code producing the exception:
class ESMTPSender(SenderMixin, ESMTPClient):
requireAuthentication = True
requireTransportSecurity = True
def __init__(self, username, secret, contextFactory=None, *args, **kw):
self.heloFallback = 0
self.username = username
if contextFactory is None:
contextFactory = self._getContextFactory()
ESMTPClient.__init__(self, secret, contextFactory, *args, **kw)
self._registerAuthenticators()
SSA

This seems like it would be a difficult exception to come by -- Generally you don't call __init__ explicitly unless you're inheriting from some other class. Here's one situation where you could get that error:
class Foo(object):
def __init__(self,*args):
print("In Foo, args:",args,type(self))
class Bar(object):
def __init__(self,*args):
Foo.__init__(self,*args) #Doesn't work. Complains that the object isn't the right type.
To fix this, we can make Bar inherit from Foo:
class Bar(Foo):
#^ Bar now inherits from Foo
def __init__(self,*args):
Foo.__init__(self,*args) #This will work now since a Bar instance is a Foo instance
If it doesn't make sense to have Bar subclassed from Foo, you can factor the common code out into a separate function:
def common_code(instance,*args):
print("Common code: args",args,type(instance))
class Foo(object):
def __init__(self,*args):
common_code(self,*args)
class Bar(object):
def __init__(self,*args):
common_code(self,*args)
Although this kind of problem can be difficult to diagnose without actually seeing the code which produces the error.