Context managers define setup/cleanup functions __enter__ and __exit__. Awesome. I want to keep one around as a member variable. When my class object goes out of scope, I want this cleanup performed. This is basically the behavior that I understand happens automatically with C++ constructors/destructors.
class Animal(object):
def __init__(self):
self.datafile = open("file.txt") # This has a cleanup function
# I wish I could say something like...
with open("file.txt") as self.datafile: # uh...
def makeSound(self):
sound = self.datafile # I'll be using it later
# Usage...
if True:
animal = Animal()
# file should be cleaned up and closed at this point.
I give classes a close function if it makes sense and then use the closing context manager:
class MyClass(object):
def __init__(self):
self.resource = acquire_resource()
def close():
release_resource(self.resource)
And then use it like:
from contextlib import closing
with closing(MyClass()) as my_object:
# use my_object
Python doesn't do C++-style RAII ("Resource Acquisition Is Initialization", meaning anything you acquire in the constructor, you release in the destructor). In fact, almost no languages besides C++ do C++-style RAII. Python's context managers and with statements are a different way to achieve the same thing that C++ does with RAII, and that most other languages do with finally statements, guard statements, etc. (Python also has finally, of course.)
What exactly do you mean by "When my class object goes out of scope"?
Objects don't go out of scope; references (or variables, or names, whatever you prefer) do. Some time after the last reference goes out of scope (for CPython, this is immediately, unless it's involved in a reference cycle; for other implementations, it's usually not), the object will be garbage-collected.
If you want to do some cleanup when your objects are garbage-collected, you use the __del__ method for that. But that's rarely what you actually want. (In fact, some classes have a __del__ method just to warn users that they forgot to clean up, rather than to silently clean up.)
A better solution is to make Animal itself a context manager, so it can manage other context managers—or just manage things explicitly. Then you can write:
if True:
with Animal() as animal:
# do stuff
# now the file is closed
Here's an example:
class Animal(object):
def __init__(self):
self.datafile = open("file.txt")
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
self.datafile.close()
def makeSound(self):
sound = self.datafile # I'll be using it later
(Just falling off the end of __exit__ like that means that, after calling self.datafile.close(), we successfully do nothing if there was no exception, or re-raise the same exception if there was one. So, you don't have to write anything explicit to make that happen.)
But usually, if you're going to make a class into a context manager, you also want to add an explicit close. Just like files do. And once you do that, you really don't need to make Animal into a context manager; you can just use closing.
Related
Context managers define setup/cleanup functions __enter__ and __exit__. Awesome. I want to keep one around as a member variable. When my class object goes out of scope, I want this cleanup performed. This is basically the behavior that I understand happens automatically with C++ constructors/destructors.
class Animal(object):
def __init__(self):
self.datafile = open("file.txt") # This has a cleanup function
# I wish I could say something like...
with open("file.txt") as self.datafile: # uh...
def makeSound(self):
sound = self.datafile # I'll be using it later
# Usage...
if True:
animal = Animal()
# file should be cleaned up and closed at this point.
I give classes a close function if it makes sense and then use the closing context manager:
class MyClass(object):
def __init__(self):
self.resource = acquire_resource()
def close():
release_resource(self.resource)
And then use it like:
from contextlib import closing
with closing(MyClass()) as my_object:
# use my_object
Python doesn't do C++-style RAII ("Resource Acquisition Is Initialization", meaning anything you acquire in the constructor, you release in the destructor). In fact, almost no languages besides C++ do C++-style RAII. Python's context managers and with statements are a different way to achieve the same thing that C++ does with RAII, and that most other languages do with finally statements, guard statements, etc. (Python also has finally, of course.)
What exactly do you mean by "When my class object goes out of scope"?
Objects don't go out of scope; references (or variables, or names, whatever you prefer) do. Some time after the last reference goes out of scope (for CPython, this is immediately, unless it's involved in a reference cycle; for other implementations, it's usually not), the object will be garbage-collected.
If you want to do some cleanup when your objects are garbage-collected, you use the __del__ method for that. But that's rarely what you actually want. (In fact, some classes have a __del__ method just to warn users that they forgot to clean up, rather than to silently clean up.)
A better solution is to make Animal itself a context manager, so it can manage other context managers—or just manage things explicitly. Then you can write:
if True:
with Animal() as animal:
# do stuff
# now the file is closed
Here's an example:
class Animal(object):
def __init__(self):
self.datafile = open("file.txt")
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
self.datafile.close()
def makeSound(self):
sound = self.datafile # I'll be using it later
(Just falling off the end of __exit__ like that means that, after calling self.datafile.close(), we successfully do nothing if there was no exception, or re-raise the same exception if there was one. So, you don't have to write anything explicit to make that happen.)
But usually, if you're going to make a class into a context manager, you also want to add an explicit close. Just like files do. And once you do that, you really don't need to make Animal into a context manager; you can just use closing.
As it is common knowledge, the python __del__ method should not be used to clean up important things, as it is not guaranteed this method gets called. The alternative is the use of a context manager, as described in several threads.
But I do not quite understand how to rewrite a class to use a context manager. To elaborate, I have a simple (non-working) example in which a wrapper class opens and closes a device, and which shall close the device in any case the instance of the class gets out of its scope (exception etc).
The first file mydevice.py is a standard wrapper class to open and close a device:
class MyWrapper(object):
def __init__(self, device):
self.device = device
def open(self):
self.device.open()
def close(self):
self.device.close()
def __del__(self):
self.close()
this class is used by another class myclass.py:
import mydevice
class MyClass(object):
def __init__(self, device):
# calls open in mydevice
self.mydevice = mydevice.MyWrapper(device)
self.mydevice.open()
def processing(self, value):
if not value:
self.mydevice.close()
else:
something_else()
My question: When I implement the context manager in mydevice.py with __enter__ and __exit__ methods, how can this class be handled in myclass.py? I need to do something like
def __init__(self, device):
with mydevice.MyWrapper(device):
???
but how to handle it then? Maybe I overlooked something important? Or can I use a context manager only within a function and not as a variable inside a class scope?
I suggest using the contextlib.contextmanager class instead of writing a class that implements __enter__ and __exit__. Here's how it would work:
class MyWrapper(object):
def __init__(self, device):
self.device = device
def open(self):
self.device.open()
def close(self):
self.device.close()
# I assume your device has a blink command
def blink(self):
# do something useful with self.device
self.device.send_command(CMD_BLINK, 100)
# there is no __del__ method, as long as you conscientiously use the wrapper
import contextlib
#contextlib.contextmanager
def open_device(device):
wrapper_object = MyWrapper(device)
wrapper_object.open()
try:
yield wrapper_object
finally:
wrapper_object.close()
return
with open_device(device) as wrapper_object:
# do something useful with wrapper_object
wrapper_object.blink()
The line that starts with an at sign is called a decorator. It modifies the function declaration on the next line.
When the with statement is encountered, the open_device() function will execute up to the yield statement. The value in the yield statement is returned in the variable that's the target of the optional as clause, in this case, wrapper_object. You can use that value like a normal Python object thereafter. When control exits from the block by any path – including throwing exceptions – the remaining body of the open_device function will execute.
I'm not sure if (a) your wrapper class is adding functionality to a lower-level API, or (b) if it's only something you're including so you can have a context manager. If (b), then you can probably dispense with it entirely, since contextlib takes care of that for you. Here's what your code might look like then:
import contextlib
#contextlib.contextmanager
def open_device(device):
device.open()
try:
yield device
finally:
device.close()
return
with open_device(device) as device:
# do something useful with device
device.send_command(CMD_BLINK, 100)
99% of context manager uses can be done with contextlib.contextmanager. It is an extremely useful API class (and the way it's implemented is also a creative use of lower-level Python plumbing, if you care about such things).
The issue is not that you're using it in a class, it's that you want to leave the device in an "open-ended" way: you open it and then just leave it open. A context manager provides a way to open some resource and use it in a relatively short, contained way, making sure it is closed at the end. Your existing code is already unsafe, because if some crash occurs, you can't guarantee that your __del__ will be called, so the device may be left open.
Without knowing exactly what the device is and how it works, it's hard to say more, but the basic idea is that, if possible, it's better to only open the device right when you need to use it, and then close it immediately afterwards. So your processing is what might need to change, to something more like:
def processing(self, value):
with self.device:
if value:
something_else()
If self.device is an appropriately-written context manager, it should open the device in __enter__ and close it in __exit__. This ensures that the device will be closed at the end of the with block.
Of course, for some sorts of resources, it's not possible to do this (e.g., because opening and closing the device loses important state, or is a slow operation). If that is your case, you are stuck with using __del__ and living with its pitfalls. The basic problem is that there is no foolproof way to leave the device "open-ended" but still guarantee it will be closed even in the event of some unusual program failure.
I'm not quite sure what you're asking. A context manager instance can be a class member - you can re-use it in as many with clauses as you like and the __enter__() and __exit__() methods will be called each time.
So, once you'd added those methods to MyWrapper, you can construct it in MyClass just as you are above. And then you'd do something like:
def my_method(self):
with self.mydevice:
# Do stuff here
That will call the __enter__() and __exit__() methods on the instance you created in the constructor.
However, the with clause can only span a function - if you use the with clause in the constructor then it will call __exit__() before exiting the constructor. If you want to do that, the only way is to use __del__(), which has its own problems as you've already mentioned. You could open and close the device just when you need it using with but I don't know if this fulfills your requirements.
Is there any way to use monitor thread synchronization like java methods synchronization,in python class to ensure thread safety and avoid race condition?
I want a monitor like synchronization mechanism that allows only one method call in my class or object
You might want to have a look at python threading interface. For simple mutual exclusion functionality you might use a Lock object. You can easily do this using the with statement like:
...
lock = Lock()
...
with (lock):
# This code will only be executed by one single thread at a time
# the lock is released when the thread exits the 'with' block
...
See also here for an overview of different thread synchronization mechanisms in python.
There is no python language construct for Java's synchronized (but I guess it could be built using decorators)
I built a simple prototype for it, here's a link to the GitHub repository for all the details : https://github.com/m-a-rahal/monitor-sync-python
I used inheritance instead of decorators, but maybe I'll include that option later
Here's what the 'Monitor' super class looks like:
import threading
class Monitor(object):
def __init__(self, lock = threading.Lock()):
''' initializes the _lock, threading.Lock() is used by default '''
self._lock = lock
def Condition(self):
''' returns a condition bound to this monitor's lock'''
return threading.Condition(self._lock)
init_lock = __init__
Now all you need to do to define your own monitor is to inherit from this class:
class My_Monitor_Class(Monitor):
def __init__(self):
self.init_lock() # just don't forget this line, creates the monitor's _lock
cond1 = self.Condition()
cond2 = self.Condition()
# you can see i defined some 'Condition' objects as well, very simple syntax
# these conditions are bound to the lock of the monitor
you can also pass your own lock instead
class My_Monitor_Class(Monitor):
def __init__(self, lock):
self.init_lock(lock)
check out threading.Condition() documentation
Also you need to protect all the 'public' methods with the monitor's lock, like this:
class My_Monitor_Class(Monitor):
def method(self):
with self._lock:
# your code here
if you want to use 'private' methods (called inside the monitor), you can either NOT protect them with the _lock (or else the threads will get stuck), or use RLock instead for the monitor
EXTRA TIP
sometimes a monitor consists of 'entrance' and 'exit' protocols
monitor.enter_protocol()
<critical section>
monitor.exit_protocol()
in this case, you can exploit python's cool with statement :3
just define the __enter__ and __exit__ methods like this:
class monitor(Monitor):
def __enter__(self):
with self._lock:
# enter_protocol code here
def __exit__(self, type, value, traceback):
with self._lock:
# exit_protocol code here
now all you need to do is call the monitor using with statement:
with monitor:
<critical section>
I am building an event dispatcher framework that decodes messages and calls back into user code. My C++ background suggests that I write:
class Handler:
def onFoo(self):
pass
def onBar(self):
pass
class Dispatcher:
def __init__(self):
self.handler = Handler()
def run():
while True:
msg = decode_message() # magic
if msg == 'foo':
handler.onFoo()
if msg == 'bar':
handler.onBar()
Then the framework user would write something like:
class MyHandler(Handler):
def onFoo(self):
do_something()
dispatcher = Dispatcher()
myHandler = MyHandler()
dispatcher.handler = myHandler
dispatcher.run()
But I can also imagine putting onFoo() and onBar() as methods of Dispatcher and letting the user replace them with other methods. Then the user's code would look like:
def onFoo():
do_something()
dispatcher = Dispatcher()
dispatcher.onFoo = onFoo
I could also make Dispatcher.onFoo a list of callables, so the user could attach more than one handler like in C#.
What is the most Pythonic way to do it?
I wouldn't say that there's anything particularly wrong about doing it the first way, especially if you want to allow Dispatcher objects to be customized in an organized way at runtime (i.e. by passing different kinds of Handler objects to them), if your Handlers need to interact with and maintain complex state.
But you don't really gain anything by defining a base Handler class this way; a class could still subclass Handler without overriding any methods, because this isn't an abstract base class -- it's just a regular base class. So if there are some sensible default behaviors, I would suggest building them in to Handler. Otherwise, your users don't gain anything from Handler at all -- they might as well just define their own Handler class. Though if you just want to provide a no-op placeholder, then this set-up is fine.
In any case, I would personally prefer the first approach to the second that you suggest; I'm not certain that either is more "pythonic," but the first seems like a cleaner approach to me, since it keeps Handler and Dispatcher logic separate. (It avoids situations like the one you see in threading.Thread where you can only safely override some methods -- I've always found that a bit jarring.)
I feel I should point out, though, that if you really want a bonafide abstract base class, you should write one! As of 2.6, Python provides support for flexible abstract base classes with lots of nice functionality. For example, you can define an abstract method with the abstractmethod decorator, which ensures that it must be overridden by the subclass; but you can also define regular methods that don't have to be overridden. If you're seeking a Pythonic version of the c++ idiom, this is probably the best way to go -- it's not the same thing, of course, but it comes closer than what you have now.
If an object relies on a module that is not included with Python (like win32api, gstreamer, gui toolkits, etc.), and a class/function/method from that module may fail, what should the object do?
Here's an example:
import guimodule # Just an example; could be anything
class RandomWindow(object):
def __init__(self):
try:
self.dialog = guimodule.Dialog() # I might fail
except: guimodule.DialogError:
self.dialog = None # This can't be right
def update(self):
self.dialog.prepare()
self.dialog.paint()
self.dialog.update()
# ~30 more methods
This class would only be a tiny (and unnecessary, but useful) part of a bigger program.
Let's assume we have an imaginary module called guimodule, with a class called Dialog, that may fail to instantiate. If our RandomWindow class has say, 30 methods that manipulate this window, checking if self.dialog is not None will be a pain, and will slow down the program when implemented in constantly used methods (like the update method in the example above). Calling .paint() on a NoneType (when the Dialog fails to load) will raise an error, and making a dummy Dialog class with all of the original's methods and attributes would be absurd.
How can I modify my class to handle a failed creation of the Dialog class?
Rather than creating an invalid object, you should have allowed the exception raised in __init__ to propogate out so the error could be handled in an appropriate manner. Or you could have raised a different exception.
See also Python: is it bad form to raise exceptions within __init__?
You may find it useful to have two subclasses of it; one that uses that module and one that does not. A "factory" method could determine which subclass was appropriate, and return an instance of that subclass.
By subclassing, you allow them to share code that is independent of whether that module is available.
I agree that "checking if self.dialog is not None will be pain" but I don't agree that it will slow down things because if self.dialog existed it will be more slower. So forget about slowness for time being. so one way to handle is to create a MockDialog which does nothing on function calls e.g.
class RandomWindow(object):
def __init__(self):
try:
self.dialog = guimodule.Dialog() # I might fail
except: guimodule.DialogError:
self.dialog = DummyDialog() # create a placeholder
class DummyDialog(object):
# either list all methods or override __getattr__ to create a mock object
Making a dummy Dialog class is not as absurd as you might thing if you consider using Pythons __getattr__ feature. This following dummy-implementation would completely fit your needs:
class DummyDialog:
def __getattr__(self, name):
def fct(*args, **kwargs):
pass
return fct