I have a filesystem object (let it be fs). The object makes use of jwt token for authentication. When we create the filesystem object, the authentication is done. Once, we have the object, we can call method like ls to list the directory, etc. The token has some expiration time.
The issue is when I call fs.ls('/'), there is no validation in the backend for the token like is token still valid or not. What I want is, when ever there is call to a method on the object, I will intercept the call and check for the token expiration. If it is about to expire will update the token.
Searching and reading on SO, I came to about __getattribute__. But my code is not working as expected. Sometimes I am getting recursion error or sometimes I am getting null values.
This code gives recursion error:
class FileSystem(adlfs.FileSystem):
def __init__(self, *args, **kwargs):
try:
self.name, self.token = self._get_token()
if self.name is not None:
kwargs["name"] = self.name
if self.token is not None:
kwargs["token"] = self.token
self.exp = self.token.token.expires_on
super().__init__(*args, **kwargs)
except Exception as exception:
print(exception)
def _get_token(self) -> (str, 'Credential'):
return name, token
def __getattribute__(self, attr):
attribute = super().__getattribute__(attr)
if callable(attribute):
curr_time = int(time.time())
if curr_time > self.token_exp:
def refresh_token(*args, **kwargs):
self.name, self.token = self._get_token()
self.token_exp = updateTokenExpiration(self.token)
super().updateConnection()
return attribute(*args, **kwargs)
return refresh_token
else:
return attribute
else:
return attribute
Typically, in __getattribute__() all occurrences of self.something need to be replaced with super().__getattribute__('something') (unless they are targets of an assignment or del).
In your case that can be relaxed for non-callables (as for them your implementation of __getattribute__() just calls super().__getattribute__(...), practically without doing anything more), but for callables it still needs to be adjusted, for example:
# before adjustment:
self.name, self.token = self._get_token()
# after adjustment:
self.name, self.token = super().__getattribute__('_get_token')()
Otherwise your implementation calls itself, so that an infinite recursion occurs.
Replacing simple attribute access with such calls can, however, be tedious if you have many such places in your __getattribute__()...
A possible trick is to use in your definition of __getattribute__() only such attributes/methods of self that are specially named, e.g. with _ga_ at the beginning of their names, and filter out such names from the customized behavior of your __getattribute__(), e.g.:
def __getattribute__(self, name):
if name.startswith('_ga_'):
return super().__getattribute__(name)
...here the actual part of your custom implementation
...in which you can freely use `self._ga_whatever...`
Related
I am string with decorators and the first use I have is to wrap a HTTP call to account for failed connections. The working code is as follows:
import requests
class Gotify:
def __init__(self):
self.url = "https://postman-echo.com/status/200"
def ensure_safe_call(caller):
def wrapper(*args, **kwargs):
try:
r = caller(*args, **kwargs)
r.raise_for_status()
except Exception as e:
try:
print(f"cannot reach gotify: {e}: {r.text}")
except NameError:
print(f"cannot reach gotify: {e} (the response r does not exist)")
else:
print("OK notified gotify of result change")
return wrapper
#ensure_safe_call
def send(self, title, message):
return requests.get(self.url)
Gotify().send("hello", "world")
This correct displays OK notified gotify of result change.
When editing this in PyCharm, I get two warning which I do not understand:
and
What do they mean in the context of my decorators (there are none when I do not use decorators)
class Gotify:
def __init__(self):
self.url = "https://postman-echo.com/status/200"
def ensure_safe_call(caller):
Because ensure_safe_call is a class method, the first argument (in your case caller) is actually the self argument, the instance of the clas object--Gotify.
Hence the warning message about the Gotify object not being callable (it's not callable because you have not overridden the __call__ class method in your Gotify class)
Function ensure_safe_call lacks a positional argument -- this is because ensure_safe_call only takes in the self argument, and doesn't specify any actual input arguments (recall that caller == self given the way you have it defined). Thus, your decorator ensure_safe_call cannot wrap anything, because it's accepting no position arguments.
You need to define a positional argument
def ensure_safe_call(self, caller):
...
I have a class that handles the API calls to a server. Certain methods within the class require the user to be logged in. Since it is possible for the session to run out, I need some functionality that re-logins the user once the session timed out. My idea was to use a decorator. If I try it like this
class Outer_Class():
class login_required():
def __init__(self, decorated_func):
self.decorated_func = decorated_func
def __call__(self, *args, **kwargs):
try:
response = self.decorated_func(*args, **kwargs)
except:
print('Session probably timed out. Logging in again ...')
args[0]._login()
response = self.decorated_func(*args, **kwargs)
return response
def __init__(self):
self.logged_in = False
self.url = 'something'
self._login()
def _login(self):
print(f'Logging in on {self.url}!')
self.logged_in = True
#this method requires the user to be logged in
#login_required
def do_something(self, param_1):
print('Doing something important with param_1')
if (): #..this fails
raise Exception()
I get an error. AttributeError: 'str' object has no attribute '_login'
Why do I not get a reference to the Outer_Class-instance handed over via *args? Is there another way to get a reference to the instance?
Found this answer How to get instance given a method of the instance? , but the decorated_function doesn't seem to have a reference to it's own instance.
It works fine, when Im using a decorator function outside of the class. This solves the problem, but I like to know, if it is possible to solve the this way.
The problem is that the magic of passing the object as the first hidden parameter only works for a non static method. As your decorator returns a custom callable object which is not a function, it never receives the calling object which is just lost in the call. So when you try to call the decorated function, you only pass it param_1 in the position of self. You get a first exception do_something() missing 1 required positional argument: 'param_1', fall into the except block and get your error.
You can still tie the decorator to the class, but it must be a function to have self magic work:
class Outer_Class():
def login_required(decorated_func):
def inner(self, *args, **kwargs):
print("decorated called")
try:
response = decorated_func(self, *args, **kwargs)
except:
print('Session probably timed out. Logging in again ...')
self._login()
response = decorated_func(self, *args, **kwargs)
return response
return inner
...
#this method requires the user to be logged in
#login_required
def do_something(self, param_1):
print('Doing something important with param_1', param_1)
if (False): #..this fails
raise Exception()
You can then successfully do:
>>> a = Outer_Class()
Logging in on something!
>>> a.do_something("foo")
decorated called
Doing something important with param_1
You have the command of
args[0]._login()
in the except. Since args[0] is a string and it doesn't have a _login method, you get the error message mentioned in the question.
Ive been on a tear of writing some decorators recently.
One of the ones I just wrote allows you to put the decorator just before a class definition, and it will cause every method of the class to print some logigng info when its run (more for debugging/initial super basic speed tests during a build)
def class_logit(cls):
class NCls(object):
def __init__(self, *args, **kwargs):
self.instance = cls(*args, **kwargs)
#staticmethod
def _class_logit(original_function):
def arg_catch(*args, **kwargs):
start = time.time()
result = original_function(*args, **kwargs)
print('Called: {0} | From: {1} | Args: {2} | Kwargs: {3} | Run Time: {4}'
''.format(original_function.__name__, str(inspect.getmodule(original_function)),
args, kwargs, time.time() - start))
return result
return arg_catch
def __getattribute__(self, s):
try:
x = super(NCls, self).__getattribute__(s)
except AttributeError:
pass
else:
return x
x = self.instance.__getattribute__(s)
if type(x) == type(self.__init__):
return self._class_logit(x)
else:
return x
return NCls
This works great when applied to a very basic class i create.
Where I start to encounter issues is when I apply it to a class that is inheriting another - for instance, using QT:
#scld.class_logit
class TestWindow(QtGui.QDialog):
def __init__(self):
print self
super(TestWindow, self).__init__()
a = TestWindow()
Im getting the following error... and im not entirely sure what to do about it!
self.instance = cls(*args, **kwargs)
File "<string>", line 15, in __init__
TypeError: super(type, obj): obj must be an instance or subtype of type
Any help would be appreciated!
(Apologies in advance, no matter WHAT i do SO is breaking the formatting on my first bit of code... Im even manually spending 10 minutes adding spaces but its coming out incorrectly... sorry!)
You are being a bit too intrusive with your decorator.
While if you want to profile methods defined on the Qt framework itself, a somewhat aggressive approach is needed, your decorator replaces the entire class by a proxy.
Qt bindings are somewhat complicated indeed, and it is hard to tell why it is erroring when being instantiated in this case.
So - first things first - if your intent would be to apply the decorator to a class hierarchy defined by yourself, or at least one defined in pure Python, a good approach there could be using metaclasses: with a metaclass you could decorate each method when a class is created, and do not mess anymore at runtime, when methods are retrieved from each class.
but Qt, as some other libraries, have its methods and classes defined in native code, and that will prevent you from wrapping existing methods in a new class. So, wrapping the methods on attribute retrieval on __getattribute__ could work.
Here is a simpler approach that instead of using a Proxy, just plug-in a foreign __getattribute__ that does the wrap-with-logger thing you want.
Your mileage may vary with it. Specially, it won't be triggered if one method of the class is called by other method in native code - as this won't go through Python's attribute retrieval mechanism (instead, it will use C++ method retrieval directly).
from PyQt5 import QtWidgets, QtGui
def log_dec(func):
def wraper(*args, **kwargs):
print(func.__name__, args, kwargs)
return func(*args, **kwargs)
return wraper
def decorate(cls):
def __getattribute__(self, attr):
attr = super(cls, self).__getattribute__(attr)
if callable(attr):
return log_dec(attr)
return attr
cls.__getattribute__ = __getattribute__
return cls
#decorate
class Example(QtGui.QWindow):
pass
app = QtWidgets.QApplication([])
w = Example()
w.show()
(Of course, just replace the basic logger by your fancy logger above)
So I am working with an API wrapper in python for vk, Europe's Facebook equivalent. The documentation on the vk site has all the API calls that can be used, and the wrapper is able to call them correctly. For example, to get a user's information, you would call api.users.get(id) to get a user by id. My question is this: how can the wrapper correctly handle such a call when neither users or a corresponding users.get() method is defined inside the api object?
I know it involves the __getattr__() and __call__() methods, but I can't find any good documentation on how to use them in this way.
EDIT
the api object is instantiated via api = vk.API(id, email, password)
Let's walk through this together, shall we?
api
To execute api.users.get(), Python first has to know api. And due to your instantiation, it does know it: It's a local variable holding an instance of APISession.
api.users
Then, it has to know api.users. Python first looks at the members of the api instance, at the members of its class (APISession) and at the members of that class' super-classes (only object in the case of APISession). Failing to find a member called users in any of these places, it looks for a member function called __getattr__ in those same places. It will find it right on the instance, because APISession has an (instance) member function of this name.
Python then calls it with 'users' (the name of the so-far missing member) and uses the function's return value as if it were that member. So
api.users
is equivalent to
api.__getattr__('users')
Let's see what that returns.
def __getattr__(self, method_name):
return APIMethod(self, method_name)
Oh. So
api.users # via api.__getattr__('users')
is equivalent to
APIMethod(api, 'users')
creating a new APIMethod instance.
api and 'users' end up as that instance's _api_session and _method_name members. Makes sense, I guess.
api.users.get
Python still hasn't executed our statement. It needs to know api.users.get() to do so. The same game as before repeats, just in the api.users object instead of the api object this time: No member method get() and no member get is found on the APIMethod instance api.users points to, nor on its class or superclasses, so Python turns to the __getattr__ method, which for this class does something peculiar:
def __getattr__(self, method_name):
return APIMethod(self._api_session, self._method_name + '.' + method_name)
A new instance of the same class! Let's plug in the instance members of api.users, and
api.users.get
becomes equivalent to
APIMethod(api, 'users' + '.' + 'get')
So we will have the api object also in api.user.get's _apisession and the string 'users.get' in its _method_name.
api.users.get() (note the ())
So api.users.get is an object. To call it, Python has to pretend it's a function, or more specifically, a method of api.users. It does so, by instead calling api.users.get's __call__ method, which looks like this:
def __call__(self, **method_kwargs):
return self._api_session(self._method_name, **method_kwargs)
Let's work this out:
api.users.get()
# is equivalent to
api.users.get.__call__() # no arguments, because we passed none to `get()`
# will return
api.users.get._api_session(api.users.get._method_name)
# which is
api('users.get')
So now Python is calling the api object as if it were a function. __call__ to the rescue, once more, this time looking like this:
def __call__(self, method_name, **method_kwargs):
response = self.method_request(method_name, **method_kwargs)
response.raise_for_status()
# there are may be 2 dicts in 1 json
# for example: {'error': ...}{'response': ...}
errors = []
error_codes = []
for data in json_iter_parse(response.text):
if 'error' in data:
error_data = data['error']
if error_data['error_code'] == CAPTCHA_IS_NEEDED:
return self.captcha_is_needed(error_data, method_name, **method_kwargs)
error_codes.append(error_data['error_code'])
errors.append(error_data)
if 'response' in data:
for error in errors:
warnings.warn(str(error))
return data['response']
if AUTHORIZATION_FAILED in error_codes: # invalid access token
self.access_token = None
self.get_access_token()
return self(method_name, **method_kwargs)
else:
raise VkAPIMethodError(errors[0])
Now, that's a lot of error handling. For this analysis, I'm only interested in the happy path. I'm only interested in the happy path's result (and how we got there). So lets start at the result.
return data['response']
Where did data come from? It's the first element of response.text interpreted as JSON that does contain a 'response' object. So it seems that from the response object we got, we're extracting the actual response part.
Where did the response object come from? It was returned by
api.method_request('users.get')
Which, for all we care, is a plain normal method call that doesn't require any fancy fallbacks. (Its implementation of course, on some levels, might.)
Assuming the comments are correct, and api is an instance of APISession as defined in this particular commit, then this is a bit of an interesting maze:
So first you want to access api.user. APISession has no such attribute, so it calls __getattr__('user') instead, which is defined as:
def __getattr__(self, method_name):
return APIMethod(self, method_name)
So this constructs an APIMethod(api,'user'). Now you want to call the method get on the APIMethod(api,'user') that is bound to api.users, but an APIMethod doesn't have a get method, so it calls its own __getattr__('get') to figure out what to do:
def __getattr__(self, method_name):
return APIMethod(self._api_session, self._method_name + '.' + method_name)
This returns a APIMethod(api,'users.get') which is then called, invoking the __call__ method of the APIMethod class, which is:
def __call__(self, **method_kwargs):
return self._api_session(self._method_name, **method_kwargs)
So this tries to return api('users.get'), but api is an APISession object, so it invokes the __call__ method of this class, defined as (stripping out the error handling for simplicity):
def __call__(self, method_name, **method_kwargs):
response = self.method_request(method_name, **method_kwargs)
response.raise_for_status()
for data in json_iter_parse(response.text):
if 'response' in data:
return data['response']
So it then calls a method_request('users.get'), which if you follow that method actually does a POST request, and some data comes back as a response, which is then returned.
The users.get() has nothing to do with the api object. As for the users, you are right, if there is no such member defined, then there is certainly some logic inside the __getattr__. So as you can see in the documentation __getattr__ is...
Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name.
So exactly, as there is no users defined for the api's class, then the __getattr__ is being called with 'users' passed as the name parameter. Then, most probably dynamically, depending on the passed parameter, an object is being constructed for the users component and returned, which will be responsible to define or handle in similar way the get() method.
To get the whole idea, try the following:
class A(object):
def __init__(self):
super(A, self).__init__()
self.defined_one = 'I am defined inside A'
def __getattr__(self, item):
print('getting attribute {}'.format(item))
return 'I am ' + item
a = A()
>>> print(a.some_item) # this will call __getattr__ as some_item is not defined
getting attribute some_item
I am some_item
>>> print(a.and_another_one) # this will call __getattr__ as and_another_one is not defined
getting attribute and_another_one
I am and_another_one
>>> print(a.defined_one) # this will NOT call __getattr__ as defined_one is defined in A
I am defined inside A
I'm trying to write a library that will register an arbitrary list of service calls from multiple service endpoints to a container. I intend to implement the service calls in classes written one per service. Is there a way to maintain the boundedness of the methods from the service classes when registering them to the container (so they will still have access to the instance data of their owning object instance), or must I register the whole object then write some sort of pass through in the container class with __getattr__ or some such to access the methods within instance context?
container:
class ServiceCalls(object):
def __init__(self):
self._service_calls = {}
def register_call(self, name, call):
if name not in self._service_calls:
self._service_calls[name] = call
def __getattr__(self, name):
if name in self._service_calls:
return self._service_calls[name]
services:
class FooSvc(object):
def __init__(self, endpoint):
self.endpoint = endpoint
def fooize(self, *args, **kwargs):
#call fooize service call with args/kwargs utilizing self.endpoint
def fooify(self, *args, **kwargs):
#call fooify service call with args/kwargs utilizing self.endpoint
class BarSvc(object):
def __init__(self, endpoint):
self.endpoint = endpoint
def barize(self, *args, **kwargs):
#call barize service call with args/kwargs utilizing self.endpoint
def barify(self, *args, **kwargs):
#call barify service call with args/kwargs utilizing self.endpoint
implementation code:
foosvc = FooSvc('fooendpoint')
barsvc = BarSvc('barendpoint')
calls = ServiceCalls()
calls.register('fooize', foosvc.fooize)
calls.register('fooify', foosvc.fooify)
calls.register('barize', barsvc.barize)
calls.register('barify', barsvc.barify)
calls.fooize(args)
I think this answers your question:
In [2]: f = 1 .__add__
In [3]: f(3)
Out[3]: 4
You won't need the staticmethod function when adding these functions to classes, because they are effectively already "staticed".
What you are trying to do will work fine, as you can see by running your own code. :)
The object foosvc.fooize is called a "bound method" in Python, and it contains both, a reference to foosvc and to the function FooSvc.fooize. If you call the bound method, the reference to self will be passed implicitly as the first paramater.
On a side note, __getattr__() shouldn't silently return None for invalid attribute names. Better use this:
def __getattr__(self, name):
try:
return self._service_calls[name]
except KeyError:
raise AttributeError
I don't understand the use case for this -- it seems to me that the easy, simple, idiomatic way to accomplish this is to just pass in an object.
But: program to the interface, not the implementation. Only assume that the object has the method you need -- don't touch the internals or any other methods.