Is there any way to use monitor thread synchronization like java methods synchronization,in python class to ensure thread safety and avoid race condition?
I want a monitor like synchronization mechanism that allows only one method call in my class or object
You might want to have a look at python threading interface. For simple mutual exclusion functionality you might use a Lock object. You can easily do this using the with statement like:
...
lock = Lock()
...
with (lock):
# This code will only be executed by one single thread at a time
# the lock is released when the thread exits the 'with' block
...
See also here for an overview of different thread synchronization mechanisms in python.
There is no python language construct for Java's synchronized (but I guess it could be built using decorators)
I built a simple prototype for it, here's a link to the GitHub repository for all the details : https://github.com/m-a-rahal/monitor-sync-python
I used inheritance instead of decorators, but maybe I'll include that option later
Here's what the 'Monitor' super class looks like:
import threading
class Monitor(object):
def __init__(self, lock = threading.Lock()):
''' initializes the _lock, threading.Lock() is used by default '''
self._lock = lock
def Condition(self):
''' returns a condition bound to this monitor's lock'''
return threading.Condition(self._lock)
init_lock = __init__
Now all you need to do to define your own monitor is to inherit from this class:
class My_Monitor_Class(Monitor):
def __init__(self):
self.init_lock() # just don't forget this line, creates the monitor's _lock
cond1 = self.Condition()
cond2 = self.Condition()
# you can see i defined some 'Condition' objects as well, very simple syntax
# these conditions are bound to the lock of the monitor
you can also pass your own lock instead
class My_Monitor_Class(Monitor):
def __init__(self, lock):
self.init_lock(lock)
check out threading.Condition() documentation
Also you need to protect all the 'public' methods with the monitor's lock, like this:
class My_Monitor_Class(Monitor):
def method(self):
with self._lock:
# your code here
if you want to use 'private' methods (called inside the monitor), you can either NOT protect them with the _lock (or else the threads will get stuck), or use RLock instead for the monitor
EXTRA TIP
sometimes a monitor consists of 'entrance' and 'exit' protocols
monitor.enter_protocol()
<critical section>
monitor.exit_protocol()
in this case, you can exploit python's cool with statement :3
just define the __enter__ and __exit__ methods like this:
class monitor(Monitor):
def __enter__(self):
with self._lock:
# enter_protocol code here
def __exit__(self, type, value, traceback):
with self._lock:
# exit_protocol code here
now all you need to do is call the monitor using with statement:
with monitor:
<critical section>
Related
I am writing a class that has many functionalities(therefore methods), but I require this class to run inside a thread(class opens a subprocess). I want to use the common way of declaring thread based classes of,
class HiStackOverflow(threading.Thread):
# Somethings...
However, as I said, this class of mine has many pseudo-private, regular and static methods. And as I declare them, I want to avoid overriding some necessary threading.Thread method by mistake.
Well I can always check the directory of threading.Thread and see if there are any method names that overlap, however this seemed like a inappropriate way to handle this. It may be impractical as the method count increases.
My question is, is this kind of implementation feasible ? If not, how should I handle this ? Should I write some wrapper class as the Thread handler.
Thanks in advance.
If you're worried about namespace clashes between your class and threading.Thread, I would definitely suggest that you use composition rather than inheritance (or keep the two functionalities separate entirely). There shouldn't be significant overhead to just wrapping the couple threading methods that you need and then name clashes become a non-issue.
It also more cleanly will separate the functionality of your class from the functionality provided by threading. That's likely to be a win in the long run for understanding your code.
There isn't much benefit from inheriting from Thread. You could have a factory method that creates the thread or even have its __init__ do it.
import threading
import time
class MyClass:
def __init__(self):
self._thread = threading.Thread(target=self.run)
self._thread.start()
def run(self):
for i in range(5):
print('worker thread', i)
time.sleep(.5)
def join(self):
self._thread.join()
my_obj = MyClass()
for i in range(3):
print('main thread', i)
time.sleep(.5)
my_obj.join()
print('done')
There seem to be some ideas conflated in this phrase:
but I require this class to run inside a thread(class opens a subprocess)
Classes don't "run". You can start a new thread which executes some class method, or an instance method. That class doesn't have to inherit from Thread. It doesn't even need a reference to the running thread. You just start to execute some function in a new thread and you're done.
Subprocesses are unrelated to threads. You don't need one to do the other.
If you're worried about overriding something, check the documentation (https://docs.python.org/3/library/threading.html#thread-objects). Otherwise, if you want to keep the reference to the thread, you can always do:
class HiStackoverflow:
def run(self):
self.thread = Thread(target=self.entry_point)
self.thread.start()
def entry_point(self):
...
Context managers define setup/cleanup functions __enter__ and __exit__. Awesome. I want to keep one around as a member variable. When my class object goes out of scope, I want this cleanup performed. This is basically the behavior that I understand happens automatically with C++ constructors/destructors.
class Animal(object):
def __init__(self):
self.datafile = open("file.txt") # This has a cleanup function
# I wish I could say something like...
with open("file.txt") as self.datafile: # uh...
def makeSound(self):
sound = self.datafile # I'll be using it later
# Usage...
if True:
animal = Animal()
# file should be cleaned up and closed at this point.
I give classes a close function if it makes sense and then use the closing context manager:
class MyClass(object):
def __init__(self):
self.resource = acquire_resource()
def close():
release_resource(self.resource)
And then use it like:
from contextlib import closing
with closing(MyClass()) as my_object:
# use my_object
Python doesn't do C++-style RAII ("Resource Acquisition Is Initialization", meaning anything you acquire in the constructor, you release in the destructor). In fact, almost no languages besides C++ do C++-style RAII. Python's context managers and with statements are a different way to achieve the same thing that C++ does with RAII, and that most other languages do with finally statements, guard statements, etc. (Python also has finally, of course.)
What exactly do you mean by "When my class object goes out of scope"?
Objects don't go out of scope; references (or variables, or names, whatever you prefer) do. Some time after the last reference goes out of scope (for CPython, this is immediately, unless it's involved in a reference cycle; for other implementations, it's usually not), the object will be garbage-collected.
If you want to do some cleanup when your objects are garbage-collected, you use the __del__ method for that. But that's rarely what you actually want. (In fact, some classes have a __del__ method just to warn users that they forgot to clean up, rather than to silently clean up.)
A better solution is to make Animal itself a context manager, so it can manage other context managers—or just manage things explicitly. Then you can write:
if True:
with Animal() as animal:
# do stuff
# now the file is closed
Here's an example:
class Animal(object):
def __init__(self):
self.datafile = open("file.txt")
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
self.datafile.close()
def makeSound(self):
sound = self.datafile # I'll be using it later
(Just falling off the end of __exit__ like that means that, after calling self.datafile.close(), we successfully do nothing if there was no exception, or re-raise the same exception if there was one. So, you don't have to write anything explicit to make that happen.)
But usually, if you're going to make a class into a context manager, you also want to add an explicit close. Just like files do. And once you do that, you really don't need to make Animal into a context manager; you can just use closing.
Context managers define setup/cleanup functions __enter__ and __exit__. Awesome. I want to keep one around as a member variable. When my class object goes out of scope, I want this cleanup performed. This is basically the behavior that I understand happens automatically with C++ constructors/destructors.
class Animal(object):
def __init__(self):
self.datafile = open("file.txt") # This has a cleanup function
# I wish I could say something like...
with open("file.txt") as self.datafile: # uh...
def makeSound(self):
sound = self.datafile # I'll be using it later
# Usage...
if True:
animal = Animal()
# file should be cleaned up and closed at this point.
I give classes a close function if it makes sense and then use the closing context manager:
class MyClass(object):
def __init__(self):
self.resource = acquire_resource()
def close():
release_resource(self.resource)
And then use it like:
from contextlib import closing
with closing(MyClass()) as my_object:
# use my_object
Python doesn't do C++-style RAII ("Resource Acquisition Is Initialization", meaning anything you acquire in the constructor, you release in the destructor). In fact, almost no languages besides C++ do C++-style RAII. Python's context managers and with statements are a different way to achieve the same thing that C++ does with RAII, and that most other languages do with finally statements, guard statements, etc. (Python also has finally, of course.)
What exactly do you mean by "When my class object goes out of scope"?
Objects don't go out of scope; references (or variables, or names, whatever you prefer) do. Some time after the last reference goes out of scope (for CPython, this is immediately, unless it's involved in a reference cycle; for other implementations, it's usually not), the object will be garbage-collected.
If you want to do some cleanup when your objects are garbage-collected, you use the __del__ method for that. But that's rarely what you actually want. (In fact, some classes have a __del__ method just to warn users that they forgot to clean up, rather than to silently clean up.)
A better solution is to make Animal itself a context manager, so it can manage other context managers—or just manage things explicitly. Then you can write:
if True:
with Animal() as animal:
# do stuff
# now the file is closed
Here's an example:
class Animal(object):
def __init__(self):
self.datafile = open("file.txt")
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
self.datafile.close()
def makeSound(self):
sound = self.datafile # I'll be using it later
(Just falling off the end of __exit__ like that means that, after calling self.datafile.close(), we successfully do nothing if there was no exception, or re-raise the same exception if there was one. So, you don't have to write anything explicit to make that happen.)
But usually, if you're going to make a class into a context manager, you also want to add an explicit close. Just like files do. And once you do that, you really don't need to make Animal into a context manager; you can just use closing.
I'm trying to figure out what the following module is doing.
import Queue
import multiprocessing
import threading
class BufferedReadQueue(Queue.Queue):
def __init__(self, lim=None):
self.raw = multiprocessing.Queue(lim)
self.__listener = threading.Thread(target=self.listen)
self.__listener.setDaemon(True)
self.__listener.start()
Queue.Queue.__init__(self, lim)
def listen(self):
try:
while True:
self.put(self.raw.get())
except:
pass
#property
def buffered(self):
return self.qsize()
It is only instantiated once in the calling code, and the .raw attribute, multiprocessing.Queue, gets sent to another class, which appears to inherit from multiprocessing.Process.
So as I'm seeing it, an attribute of BufferedReadQueue is being used as a Queue, but not the class (nor an instance of it) itself.
What would be a reason that BufferedReadQueue inherits from Queue.Queue and not just object, if it's not actually being used as a queue?
It looks like BufferedReadQueue is meant to be used as a way to convert the read end of a multiprocessing.Queue into a normal Queue.Queue. Note this in __init__:
self.__listener = threading.Thread(target=self.listen)
self.__listener.setDaemon(True)
self.__listener.start()
This starts up a listener thread, which just constantly tries to get items from the internal multiprocessing.Queue, and then puts all those items to self. It looks like the use-case is something like this:
def func(queue):
queue.put('stuff')
...
buf_queue = BufferedReadQueue()
proc = multiprocessing.Process(target=func, args=(buf_queue.raw,))
proc.start()
out = buf_queue.get() # Only get calls in the parent
Now, why would you do this instead of just using the multiprocessing.Queue directly? Probably because multiprocessing.Queue has some shortcomings that Queue.Queue doesn't. For example qsize(), which this BufferedReadQueue uses, is not reliable with multiprocessing.Queue:
Return the approximate size of the queue. Because of multithreading/multiprocessing semantics, this number is not reliable.
Note that this may raise NotImplementedError on Unix platforms like Mac OS X where sem_getvalue() is not implemented.
It's also possible to introspect a Queue.Queue, and peek at its contents without popping them. This isn't possible with a multiprocessing.Queue.
As it is common knowledge, the python __del__ method should not be used to clean up important things, as it is not guaranteed this method gets called. The alternative is the use of a context manager, as described in several threads.
But I do not quite understand how to rewrite a class to use a context manager. To elaborate, I have a simple (non-working) example in which a wrapper class opens and closes a device, and which shall close the device in any case the instance of the class gets out of its scope (exception etc).
The first file mydevice.py is a standard wrapper class to open and close a device:
class MyWrapper(object):
def __init__(self, device):
self.device = device
def open(self):
self.device.open()
def close(self):
self.device.close()
def __del__(self):
self.close()
this class is used by another class myclass.py:
import mydevice
class MyClass(object):
def __init__(self, device):
# calls open in mydevice
self.mydevice = mydevice.MyWrapper(device)
self.mydevice.open()
def processing(self, value):
if not value:
self.mydevice.close()
else:
something_else()
My question: When I implement the context manager in mydevice.py with __enter__ and __exit__ methods, how can this class be handled in myclass.py? I need to do something like
def __init__(self, device):
with mydevice.MyWrapper(device):
???
but how to handle it then? Maybe I overlooked something important? Or can I use a context manager only within a function and not as a variable inside a class scope?
I suggest using the contextlib.contextmanager class instead of writing a class that implements __enter__ and __exit__. Here's how it would work:
class MyWrapper(object):
def __init__(self, device):
self.device = device
def open(self):
self.device.open()
def close(self):
self.device.close()
# I assume your device has a blink command
def blink(self):
# do something useful with self.device
self.device.send_command(CMD_BLINK, 100)
# there is no __del__ method, as long as you conscientiously use the wrapper
import contextlib
#contextlib.contextmanager
def open_device(device):
wrapper_object = MyWrapper(device)
wrapper_object.open()
try:
yield wrapper_object
finally:
wrapper_object.close()
return
with open_device(device) as wrapper_object:
# do something useful with wrapper_object
wrapper_object.blink()
The line that starts with an at sign is called a decorator. It modifies the function declaration on the next line.
When the with statement is encountered, the open_device() function will execute up to the yield statement. The value in the yield statement is returned in the variable that's the target of the optional as clause, in this case, wrapper_object. You can use that value like a normal Python object thereafter. When control exits from the block by any path – including throwing exceptions – the remaining body of the open_device function will execute.
I'm not sure if (a) your wrapper class is adding functionality to a lower-level API, or (b) if it's only something you're including so you can have a context manager. If (b), then you can probably dispense with it entirely, since contextlib takes care of that for you. Here's what your code might look like then:
import contextlib
#contextlib.contextmanager
def open_device(device):
device.open()
try:
yield device
finally:
device.close()
return
with open_device(device) as device:
# do something useful with device
device.send_command(CMD_BLINK, 100)
99% of context manager uses can be done with contextlib.contextmanager. It is an extremely useful API class (and the way it's implemented is also a creative use of lower-level Python plumbing, if you care about such things).
The issue is not that you're using it in a class, it's that you want to leave the device in an "open-ended" way: you open it and then just leave it open. A context manager provides a way to open some resource and use it in a relatively short, contained way, making sure it is closed at the end. Your existing code is already unsafe, because if some crash occurs, you can't guarantee that your __del__ will be called, so the device may be left open.
Without knowing exactly what the device is and how it works, it's hard to say more, but the basic idea is that, if possible, it's better to only open the device right when you need to use it, and then close it immediately afterwards. So your processing is what might need to change, to something more like:
def processing(self, value):
with self.device:
if value:
something_else()
If self.device is an appropriately-written context manager, it should open the device in __enter__ and close it in __exit__. This ensures that the device will be closed at the end of the with block.
Of course, for some sorts of resources, it's not possible to do this (e.g., because opening and closing the device loses important state, or is a slow operation). If that is your case, you are stuck with using __del__ and living with its pitfalls. The basic problem is that there is no foolproof way to leave the device "open-ended" but still guarantee it will be closed even in the event of some unusual program failure.
I'm not quite sure what you're asking. A context manager instance can be a class member - you can re-use it in as many with clauses as you like and the __enter__() and __exit__() methods will be called each time.
So, once you'd added those methods to MyWrapper, you can construct it in MyClass just as you are above. And then you'd do something like:
def my_method(self):
with self.mydevice:
# Do stuff here
That will call the __enter__() and __exit__() methods on the instance you created in the constructor.
However, the with clause can only span a function - if you use the with clause in the constructor then it will call __exit__() before exiting the constructor. If you want to do that, the only way is to use __del__(), which has its own problems as you've already mentioned. You could open and close the device just when you need it using with but I don't know if this fulfills your requirements.