We Keep Coding

Why are there warnings in these decorated methods?

I am string with decorators and the first use I have is to wrap a HTTP call to account for failed connections. The working code is as follows:
import requests
class Gotify:
def __init__(self):
self.url = "https://postman-echo.com/status/200"
def ensure_safe_call(caller):
def wrapper(*args, **kwargs):
try:
r = caller(*args, **kwargs)
r.raise_for_status()
except Exception as e:
try:
print(f"cannot reach gotify: {e}: {r.text}")
except NameError:
print(f"cannot reach gotify: {e} (the response r does not exist)")
else:
print("OK notified gotify of result change")
return wrapper
#ensure_safe_call
def send(self, title, message):
return requests.get(self.url)
Gotify().send("hello", "world")
This correct displays OK notified gotify of result change.
When editing this in PyCharm, I get two warning which I do not understand:
and
What do they mean in the context of my decorators (there are none when I do not use decorators)

class Gotify:
def __init__(self):
self.url = "https://postman-echo.com/status/200"
def ensure_safe_call(caller):
Because ensure_safe_call is a class method, the first argument (in your case caller) is actually the self argument, the instance of the clas object--Gotify.
Hence the warning message about the Gotify object not being callable (it's not callable because you have not overridden the __call__ class method in your Gotify class)
Function ensure_safe_call lacks a positional argument -- this is because ensure_safe_call only takes in the self argument, and doesn't specify any actual input arguments (recall that caller == self given the way you have it defined). Thus, your decorator ensure_safe_call cannot wrap anything, because it's accepting no position arguments.
You need to define a positional argument
def ensure_safe_call(self, caller):
...

Get result from MotorEngine async query

I'm trying to switch MongoEngine with MotorEngine in my Tornado app for the sake of asynchronous DB access and so far I got nowhere.
query
#gen.coroutine
def get_all_users(self):
users = yield User.objects.find_all()
handler
class IUser(BaseHandler):
#asynchronous
#gen.engine
def get(self,userId=None, *args, **kwargs):
try:
userMethods = UserMethods()
sessionId = self.request.headers.get('sessionId')
ret = userMethods.get_all_users()
except Exception as ex:
print str(ex)
self.finish()
When I print ret variable it says <tornado.concurrent.Future object at 0x7fb0236fe450>. If I try to print ret.result() it gets me nowhere.
Any help is appreciated since I'm struggling with everything I guess...

get_all_users needs to return its value somehow. In Python 2.6 or 2.7, generators aren't allowed to use the "return" statement, so coroutines have a special "Return" exception:
#gen.coroutine
def get_all_users(self):
users = yield User.objects.find_all()
raise gen.Return(users)
In Python 3.3 and later, you can simply "return users".
Now in "get", calling "get_all_users" only gives you a pending Future, not a value. You must wait for the Future to resolve to a value by yielding it:
ret = yield userMethods.get_all_users()
For more about calling coroutines from coroutines, see my "Refactoring Tornado Coroutines".
By the way, you can decorate "get" with just "gen.coroutine", it's more modern than "asynchronous" and "gen.engine", but either style works.

Just a suggestion. If you want to avoid to create an instance of userMethods every time you use its method:
userMethods = UserMethods()
You can use the #classmethod decorator before declaring it:
class UserMethods():
pass
#classmethod
#tornado.gen.coroutine
def get_all_users(self):
users = yield User.objects.find_all()
raise gen.Return(users)
## class IUser
...
try:
# userMethods = UserMethods() --not necesary now--
sessionId = self.request.headers.get('sessionId')
ret = yield userMethods.get_all_users()
except Exception as ex:
print str(ex)
...

Encapsulating retries into `with` block

I'm looking to encapsulate logic for database transactions into a with block; wrapping the code in a transaction and handling various exceptions (locking issues). This is simple enough, however I'd like to also have the block encapsulate the retrying of the code block following certain exceptions. I can't see a way to package this up neatly into the context manager.
Is it possible to repeat the code within a with statement?
I'd like to use it as simply as this, which is really neat.
def do_work():
...
# This is ideal!
with transaction(retries=3):
# Atomic DB statements
...
...
I'm currently handling this with a decorator, but I'd prefer to offer the context manager (or in fact both), so I can choose to wrap a few lines of code in the with block instead of an inline function wrapped in a decorator, which is what I do at the moment:
def do_work():
...
# This is not ideal!
#transaction(retries=3)
def _perform_in_transaction():
# Atomic DB statements
...
_perform_in_transaction()
...

Is it possible to repeat the code within a with statement?
No.
As pointed out earlier in that mailing list thread, you can reduce a bit of duplication by making the decorator call the passed function:
def do_work():
...
# This is not ideal!
#transaction(retries=3)
def _perform_in_transaction():
# Atomic DB statements
...
# called implicitly
...

The way that occurs to me to do this is just to implement a standard database transaction context manager, but allow it to take a retries argument in the constructor. Then I'd just wrap that up in your method implementations. Something like this:
class transaction(object):
def __init__(self, retries=0):
self.retries = retries
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, traceback):
pass
# Implementation...
def execute(self, query):
err = None
for _ in range(self.retries):
try:
return self._cursor.execute(query)
except Exception as e:
err = e # probably ought to save all errors, but hey
raise err
with transaction(retries=3) as cursor:
cursor.execute('BLAH')

As decorators are just functions themselves, you could do the following:
with transaction(_perform_in_transaction, retries=3) as _perf:
_perf()
For the details, you'd need to implement transaction() as a factory method that returns an object with __callable__() set to call the original method and repeat it up to retries number of times on failure; __enter__() and __exit__() would be defined as normal for database transaction context managers.
You could alternatively set up transaction() such that it itself executes the passed method up to retries number of times, which would probably require about the same amount of work as implementing the context manager but would mean actual usage would be reduced to just transaction(_perform_in_transaction, retries=3) (which is, in fact, equivalent to the decorator example delnan provided).

While I agree it can't be done with a context manager... it can be done with two context managers!
The result is a little awkward, and I am not sure whether I approve of my own code yet, but this is what it looks like as the client:
with RetryManager(retries=3) as rm:
while rm:
with rm.protect:
print("Attempt #%d of %d" % (rm.attempt_count, rm.max_retries))
# Atomic DB statements
There is an explicit while loop still, and not one, but two, with statements, which leaves a little too much opportunity for mistakes for my liking.
Here's the code:
class RetryManager(object):
""" Context manager that counts attempts to run statements without
exceptions being raised.
- returns True when there should be more attempts
"""
class _RetryProtector(object):
""" Context manager that only raises exceptions if its parent
RetryManager has given up."""
def __init__(self, retry_manager):
self._retry_manager = retry_manager
def __enter__(self):
self._retry_manager._note_try()
return self
def __exit__(self, exc_type, exc_val, traceback):
if exc_type is None:
self._retry_manager._note_success()
else:
# This would be a good place to implement sleep between
# retries.
pass
# Suppress exception if the retry manager is still alive.
return self._retry_manager.is_still_trying()
def __init__(self, retries=1):
self.max_retries = retries
self.attempt_count = 0 # Note: 1-based.
self._success = False
self.protect = RetryManager._RetryProtector(self)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, traceback):
pass
def _note_try(self):
self.attempt_count += 1
def _note_success(self):
self._success = True
def is_still_trying(self):
return not self._success and self.attempt_count < self.max_retries
def __bool__(self):
return self.is_still_trying()
Bonus: I know you don't want to separate your work off into separate functions wrapped with decorators... but if you were happy with that, the redo package from Mozilla offers the decorators to do that, so you don't have to roll your own. There is even a Context Manager that effective acts as temporary decorator for your function, but it still relies on your retrievable code to be factored out into a single function.

This question is a few years old but after reading the answers I decided to give this a shot.
This solution requires the use of a "helper" class, but I I think it does provide an interface with retries configured through a context manager.
class Client:
def _request(self):
# do request stuff
print("tried")
raise Exception()
def request(self):
retry = getattr(self, "_retry", None)
if not retry:
return self._request()
else:
for n in range(retry.tries):
try:
return self._request()
except Exception:
retry.attempts += 1
class Retry:
def __init__(self, client, tries=1):
self.client = client
self.tries = tries
self.attempts = 0
def __enter__(self):
self.client._retry = self
def __exit__(self, *exc):
print(f"Tried {self.attempts} times")
del self.client._retry
>>> client = Client()
>>> with Retry(client, tries=3):
... # will try 3 times
... response = client.request()
tried once
tried once
tried once
Tried 3 times

Accessing function arguments from decorator

I have a Request handler and a decorator, I would like to work with the self object inside the decorator
class MyHandler(webapp.RequestHandler):
#myDecorator
def get(self):
#code
Update: Please notice the difference between the first and second self
class myDecorator(object):
def __init__(self, f):
self.f = f
def __call__(self):
#work with self
MyHandler > get ( function ) > self ( argument )
myDecorator > __call__ ( function ) > self ( argument )
the self arguments mentioned above are different. My intention is to access the first self from inside __call__ function, or find a way to do something similar.
Hi can I access MyHandlers self argument from get function inside the decorator?
Update2: I want to implement a decorator to work with a custom login in google app engine:
I have a class ( requestHandler ):
class SomeHandler(webapp.RequestHandler):
#only_registered_users
def get(self):
#do stuff here
And I want to decorate the get function in order to check out if the user is logged in or not:
from util.sessions import Session
import logging
class only_registered_users(object):
def __init__(self, f):
self.f = f
def __call__(self):
def decorated_get(self):
logging.debug("request object:", self.request)
session = Session()
if hasattr(session, 'user_key'):
return self.f(self)
else:
self.request.redirect("/login")
return decorated_get
I know if a user is logged in if has the property 'user_key' in a session Object.
That's the main goal I'm interested in on this specific case
Let me know your suggestions / opinions if I'm doing something wrong!
Thanks!

I'm not entirely clear what it is you want, but if you just want to use the decorated function's arguments, then that is exactly what a basic decorator does. So to access say, self.request from a decorator you could do:
def log_request(fn):
def decorated_get(self):
logging.debug("request object:", self.request)
return fn(self)
return decorated_get
class MyHandler(webapp. RequestHandler):
#log_request
def get(self):
self.response.out.write('hello world')
If you are trying to access the class the decorated function is attached to, then it's a bit tricker and you'll have to cheat a bit using the inspect module.
import inspect
def class_printer(fn):
cls = inspect.getouterframes(inspect.currentframe())[1][3]
def decorated_fn(self, msg):
fn(self,cls+" says: "+msg)
return decorated_fn
class MyClass():
#class_printer
def say(self, msg):
print msg
In the example above we fetch the name of the class from the currentframe (during the execution of the decorator) and then store that in the decorated function. Here we are just prepending the class-name to whatever the msg variable is before passing it on to the original say function, but you can use your imagination to do what ever you like once you have the class name.
>>> MyClass().say('hello')
MyClass says: hello

source
def p_decorate(func):
def func_wrapper(name):
return "<p>{0}</p>".format(func(name))
return func_wrapper
#p_decorate
def get_text(name):
return "lorem ipsum, {0} dolor sit amet".format(name)
print get_text("John")
# Outputs <p>lorem ipsum, John dolor sit amet</p>

Try this approach: Can a Python decorator of an instance method access the class?
Not the exact same question but you should be able to use the same approach to create a reference to self or a reference to a dictionary with objects of a certain class in it that you can get out in your decorator.

import random
#decorator to the get function in order to check out if the user is logged in or not
def only_registered_users(func):
def wrapper(handler):
print 'Checking if user is logged in'
if random.randint(0, 1):
print 'User is logged in. Calling the original function.'
func(handler)
else:
print 'User is NOT logged in. Redirecting...'
# redirect code here
return wrapper
class MyHandler(object):
#only_registered_users
def get(self):
print 'Get function called'
m = MyHandler()
m.get()

The self argument to __call__ will be populated with the instance the decorated method is being called on. There's no way to access the decorator object from here - __call__ is a bound method, and what you're asking for would require it to be, in effect, 'doubly bound'. Python doesn't support this, so the decorator instance gets replaced with the decorated function's first argument.
The easiest way to work around this is to use nested functions, as #Chris recommends.

python: closures and classes

I need to register an atexit function for use with a class (see Foo below for an example) that, unfortunately, I have no direct way of cleaning up via a method call: other code, that I don't have control over, calls Foo.start() and Foo.end() but sometimes doesn't call Foo.end() if it encounters an error, so I need to clean up myself.
I could use some advice on closures in this context:
class Foo:
def cleanup(self):
# do something here
def start(self):
def do_cleanup():
self.cleanup()
atexit.register(do_cleanup)
def end(self):
# cleanup is no longer necessary... how do we unregister?
Will the closure work properly, e.g. in do_cleanup, is the value of self bound correctly?
How can I unregister an atexit() routine?
Is there a better way to do this?
edit: this is Python 2.6.5

Make a registry a global registry and a function that calls a function in it, and remove them from there when necessary.
cleaners = set()
def _call_cleaners():
for cleaner in list(cleaners):
cleaner()
atexit.register(_call_cleaners)
class Foo(object):
def cleanup(self):
if self.cleaned:
raise RuntimeError("ALREADY CLEANED")
self.cleaned = True
def start(self):
self.cleaned = False
cleaners.add(self.cleanup)
def end(self):
self.cleanup()
cleaners.remove(self.cleanup)

I think the code is fine. There's no way to unregister, but you can set a boolean flag that would disable cleanup:
class Foo:
def __init__(self):
self.need_cleanup = True
def cleanup(self):
# do something here
print 'clean up'
def start(self):
def do_cleanup():
if self.need_cleanup:
self.cleanup()
atexit.register(do_cleanup)
def end(self):
# cleanup is no longer necessary... how do we unregister?
self.need_cleanup = False
Lastly, bear in mind that atexit handlers don't get called if "the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called."

self is bound correctly inside the callback to do_cleanup, but in fact if all you are doing is calling the method you might as well use the bound method directly.
You use atexit.unregister() to remove the callback, but there is a catch here as you must unregister the same function that you registered and since you used a nested function that means you have to store a reference to that function. If you follow my suggestion of using a bound method then you still have to save a reference to it:
class Foo:
def cleanup(self):
# do something here
def start(self):
self._cleanup = self.cleanup # Need to save the bound method for unregister
atexit.register(self._cleanup)
def end(self):
atexit.unregister(self._cleanup)
Note that it is still possible for your code to exit without calling ther atexit registered functions, for example if the process is aborted with ctrl+break on windows or killed with SIGABRT on linux.
Also as another answer suggests you could just use __del__ but that can be problematic for cleanup while a program is exiting as it may not be called until after other globals it needs to access have been deleted.
Edited to note that when I wrote this answer the question didn't specify Python 2.x. Oh well, I'll leave the answer here anyway in case it helps anyone else.

Since shanked deleted his posting, I'll speak in favor of __del__ again:
import atexit, weakref
class Handler:
def __init__(self, obj):
self.obj = weakref.ref(obj)
def cleanup(self):
if self.obj is not None:
obj = self.obj()
if obj is not None:
obj.cleanup()
class Foo:
def __init__(self):
self.start()
def cleanup(self):
print "cleanup"
self.cleanup_handler = None
def start(self):
self.cleanup_handler = Handler(self)
atexit.register(self.cleanup_handler.cleanup)
def end(self):
if self.cleanup_handler is None:
return
self.cleanup_handler.obj = None
self.cleanup()
def __del__(self):
self.end()
a1=Foo()
a1.end()
a1=Foo()
a2=Foo()
del a2
a3=Foo()
a3.m=a3
This supports the following cases:
objects where .end is called regularly; cleanup right away
objects that are released without .end being called; cleanup when the last
reference goes away
objects living in cycles; cleanup atexit
objects that are kept alive; cleanup atexit
Notice that it is important that the cleanup handler holds a weak reference
to the object, as it would otherwise keep the object alive.
Edit: Cycles involving Foo will not be garbage-collected, since Foo implements __del__. To allow for the cycle being deleted at garbage collection time, the cleanup must be taken out of the cycle.
class Cleanup:
cleaned = False
def cleanup(self):
if self.cleaned:
return
print "cleanup"
self.cleaned = True
def __del__(self):
self.cleanup()
class Foo:
def __init__(self):...
def start(self):
self.cleaner = Cleanup()
atexit.register(Handler(self).cleanup)
def cleanup(self):
self.cleaner.cleanup()
def end(self):
self.cleanup()
It's important that the Cleanup object has no references back to Foo.

Why don't you try it? It only took me a minute to check.
(Answer: Yes)
However, you can simplify it. The closure isn't needed.
class Foo:
def cleanup(self):
pass
def start(self):
atexit.register(self.cleanup)
And to not cleanup twice, just check in the cleanup method if a cleanup is needed or not before you clean up.