I have to open a file-like object in python (it's a serial connection through /dev/) and then close it. This is done several times in several methods of my class. How I WAS doing it was opening the file in the constructor, and then closing it in the destructor. I'm getting weird errors though and I think it has to do with the garbage collector and such, I'm still not used to not knowing exactly when my objects are being deleted =\
The reason I was doing this is because I have to use tcsetattr with a bunch of parameters each time I open it and it gets annoying doing all that all over the place. So I want to implement an inner class to handle all that so I can use it doing
with Meter('/dev/ttyS2') as m:
I was looking online and I couldn't find a really good answer on how the with syntax is implemented. I saw that it uses the __enter__(self) and __exit(self)__ methods. But is all I have to do implement those methods and I can use the with syntax? Or is there more to it?
Is there either an example on how to do this or some documentation on how it's implemented on file objects already that I can look at?
Those methods are pretty much all you need for making the object work with with statement.
In __enter__ you have to return the file object after opening it and setting it up.
In __exit__ you have to close the file object. The code for writing to it will be in the with statement body.
class Meter():
def __init__(self, dev):
self.dev = dev
def __enter__(self):
#ttysetattr etc goes here before opening and returning the file object
self.fd = open(self.dev, MODE)
return self
def __exit__(self, type, value, traceback):
#Exception handling here
close(self.fd)
meter = Meter('dev/tty0')
with meter as m:
#here you work with the file object.
m.fd.read()
Easiest may be to use standard Python library module contextlib:
import contextlib
#contextlib.contextmanager
def themeter(name):
theobj = Meter(name)
try:
yield theobj
finally:
theobj.close() # or whatever you need to do at exit
# usage
with themeter('/dev/ttyS2') as m:
# do what you need with m
m.read()
This doesn't make Meter itself a context manager (and therefore is non-invasive to that class), but rather "decorates" it (not in the sense of Python's "decorator syntax", but rather almost, but not quite, in the sense of the decorator design pattern;-) with a factory function themeter which is a context manager (which the contextlib.contextmanager decorator builds from the "single-yield" generator function you write) -- this makes it so much easier to separate the entering and exiting condition, avoids nesting, &c.
The first Google hit (for me) explains it simply enough:
http://effbot.org/zone/python-with-statement.htm
and the PEP explains it more precisely (but also more verbosely):
http://www.python.org/dev/peps/pep-0343/
Related
Trying to understand things at a deeper level.
If I am opening a file, or a web request, tensorflow session, or anything that can be handled with a with statement; is there ever a time when I shouldn't use a with statement?
E.g., is there ever a time when I should use the more generic / general try except structure?
My real question is, what is the underlying structure of what with really does? I read some nice helpful hints as well as the documentation itself, but still some of the inner workings of with seem a bit like black magic to me. I am trying to demystify the magical components.
I always go to the Python Enhancement Proposals (PEPs) to understand concepts in python since they focus more on the conceptual reasoning for things compared to the documentation and usually directly address:
the reason for the new feature / change.
how it could be done with existing code / how it affects existing code.
since you are interested in the implementation aspect here is a relevant exert from PEP 343 - the "with" statement:
Specification: The 'with' Statement
A new statement is proposed with the syntax:
with EXPR as VAR:
BLOCK
(paragraph omitted - not really relevent for this question)
The translation of the above statement is:
mgr = (EXPR)
exit = type(mgr).__exit__ # Not calling it yet
value = type(mgr).__enter__(mgr)
exc = True
try:
try:
VAR = value # Only if "as VAR" is present
BLOCK
except:
# The exceptional case is handled here
exc = False
if not exit(mgr, *sys.exc_info()):
raise
# The exception is swallowed if exit() returns true
finally:
# The normal and non-local-goto cases are handled here
if exc:
exit(mgr, None, None, None)
So the internal workings of the with statement are exactly implemented like try: finally constructs, just with a cleaner syntax that makes it harder to forget to close files etc.
There are reasons for not using a with-statement: If a failure/success requires custom cleanup.
Normally you shouldn't need to know what cleanups should be performed when you have finished with a file. The with-statement takes care of closing the file no matter what (except for cases where Python has no chance of entering the __exit__ method of the contextmanager, for example an abrupt system shutdown or similar really exceptional stuff).
But if you need to perform some local and important cleanup then using a try/except/finally might make more sense. An important keyword in this context is: separation of concerns.
Say you call a function that hypothetically creates an object that isn't managed by Pythons GC and open a file and try to write it to a file. In this case you need to perform more cleanup than just opening and closing the file:
def func():
bad_object = create_object_that_cannot_be_cleaned_by_pythons_gc():
try:
with open(filename, 'w') as file:
file.write(bad_object.to_string())
finally:
bad_object.delete()
I had a really hard time to think of an example where it's of advantage to use try/finally instead of just creating a contextmanager and I'm not sure I suceeded (Normally I would implement this example as contextmanager :-) ). The important part should be that contextmanagers perform a default cleanup operation not localized, custom cleanup.
In C#, when an object implements IDisposable, using should be used to guarantee that resources will be cleaned if an exception is thrown. For instance, instead of:
var connection = new SqlConnection(...);
...
connection.Close();
one needs to write:
using (var connection = new SqlConnection(...))
{
...
}
Therefore, just by looking at the signature of the class, I know exactly whether or not I should initialize the object inside a using.
In Python 3, a similar construct is with. Similarly to C#, it ensures that the resources will be cleaned up automatically when exiting the with context, even if a error is raised.
However, I'm not sure how should I determine whether with should be used or not for a specific class. For instance, an example from psycopg doesn't use with, which may mean that:
I shouldn't either, or:
The example is written for Python 2, or:
The authors of the documentation were unaware of with syntax, or:
The authors decided not to handle exceptional cases for the sake of simplicity.
In general, how should I determine whether with should be used when initializing an instance of a specific class (assuming that documentation says nothing on the subject, and that I have access to source code)?
Regarding when you should use it:
No one forces you to use the with statement, it's just syntactic sugar that's there to make your life easier. If you use it or not is totally up to you but, it is generally recommended to do so. (We're forgetful and with ... looks ways better than explicit initialize resource/finalize recourse calls).
When you can use it:
When you can use it boils down to examining if it defines the context manager protocol. This could be as simple as trying to use with and seeing that it fails :-)
If you dynamically need to check if an object is a context manager, you have two options.
First, wait for the stable release of Python 3.6 which defines an ABC for context managers, ContextManager, which can be used in issubclass/isinstance checks:
>>> from typing import ContextManager
>>> class foo:
... def __enter__(self): pass
... def __exit__(self): pass
...
>>> isinstance(foo(), ContextManager)
True
>>> class foo2: pass
...
>>> isinstance(foo2(), ContextManager)
False
Or, create your own little function to check for it:
def iscontext(inst):
cls = type(inst)
return (any("__enter__" in vars(a) for a in cls.__mro__) and
any("__exit__" in vars(a) for a in cls.__mro__))
As a final note, the with statement is present in Python 2 and in 3, the use case you saw probably just wasn't aware of it :-).
with is for use with context managers.
At the code level, a context manager must define two methods:
__enter__(self)
__exit__(self, type, value, traceback).
Be aware that there are class decorators which can turn otherwise simple classes/functions into context managers - see contextlib for some examples
You should use with whenever you need to perform some similar action before and after executing the statement. For example:
Want to execute SQL query? You need to open and close the connections safely.Use with.
Want to perform some action on file? You have to open and close the file safely. Use with
Want to store some data in temporary file to perform some task? You need to create the directory, and clean it up once you are done. Use with, and so on. . .
Everything you want to perform before the query execution, add it to the __enter__() method. And the action to be performed after, add it to the __exit__() method.
One of the nice thing about with is, __exit__ is executed even if the code within with raises any Exception
I really like the syntax "with open('in_file'') as f".
I want to use that syntax for my own resources which must be opened and closed.
However, I do not understand how to change my open() method to enable the 'with' syntax.
I can (and do) use the contextlib.closing() approach but it becomes a bit bulky after repeated use.
So, I will ask my question below in relation to shelve.open().
I am not proposing a change to the shelve module but instead am using it because the source code is readily available to all of you.
There is nothing special about shelve.open() vs. other standard library resources that require closing: socket.socket(), sqlite3.connect(), urllib2.urlopen(), etc.
import contextlib, inspect, shelve, sys
#print(inspect.getsource(open)) # can not see how it was done here :-(
print('-' * 40)
# Given that we can view the source for the shelve module:
print(inspect.getsource(shelve))
print('-' * 40)
# Given that we can view the docs for the shelve module:
print(shelve.__doc__)
#print('-' * 40)
# Given that the desired syntax is Pythonic but is not supported:
#with shelve.open('test_shelve') as my_shelve:
# my_shelve['fact_number_1'] = "There's a dead fish on the landing."
# Given that the required syntax is convoluted and
# takes programmer attention away from the task at hand:
with contextlib.closing(shelve.open('test_shelve')) as my_shelve:
my_shelve['fact_number_2'] = "There's another dead fish on the landing."
# Q: What changes would need to made to shelve.open() to allow the
# 'with shelve.open(x) as y' syntax?
I am not really interested in an extra wrapper with a different name. Using contextlib.closing() is easier, safer, and more intuitive than that. What I am really interested in is creating a single open() method that can be called either with or without 'with'.
So, to successfully answer this question, you need to take the source code for the shelve module and show what changes would need to be made to shelve.open() to have a single method that can be used either with or without 'with' (like the builtin open() or the Python3 urllib.urlopen()).
The biggest problem here is that if you do
shelf = the_function_you_want()
the function you want has to return the shelf, but if you do
with the_function_you_want() as shelf:
the function you want has to return a context manager. That means you need to return a shelf that is also a context manager, which in turn means you either need to make a shelf subclass or monkey-patch Shelf. It's probably better to make a subclass:
class ContextManagerShelf(shelve.DbfilenameShelf):
def __enter__(self):
return self
def __exit__(self, *exc_info):
self.close()
Then you can use ContextManagerShelf as a context manager or not. The signature is the same as shelve.open. If you want, you can also make an open function to go with it.
i am trying to wrap the read and write operation of an instance of a file object (specifically the readline() and write() methods).
normally, i would simply replace those functions by a wrapper, a bit like this:
def log(stream):
def logwrite(write):
def inner(data):
print 'LOG: > '+data.replace('\r','<cr>').replace('\n','<lf>')
return write(data)
return inner
stream.write = logwrite(stream.write)
but the attributes of a file object are read-only ! how could i wrap them properly ?
(note: i am too lazy to wrap the whole fileobject... really, i don't want to miss a feature that i did not wrap properly, or a feature which may be added in a future version of python)
more context :
i am trying to automate the communication with a modem, whose AT command set is made available on the network through a telnet session. once logged in, i shall "grab" the module with which i want to communicate with. after some time without activity, a timeout occurs which releases the module (so that it is available to other users on the network... which i don't care, i am the sole user of this equipment). the automatic release writes a specific line on the session.
i want to wrap the readline() on a file built from a socket (cf. socket.makefile()) so that when the timeout occurs, a specific exception is thrown, so that i can detect the timeout anywhere in the script and react appropriately without complicating the AT command parser...
(of course, i want to do that because the timeout is quite spurious, otherwise i would simply feed the modem with commands without any side effect only to keep the module alive)
(feel free to propose any other method or strategy to achieve this effect)
use __getattr__ to wrap your file object. provide modified methods for the ones that you are concerned with.
class Wrapped(object):
def __init__(self, file_):
self._file = file_
def write(self, data):
print 'LOG: > '+data.replace('\r','<cr>').replace('\n','<lf>')
return self._file.write(data)
def __getattr__(self, attr):
return getattr(self._file, attr)
This way, requests for attributes which you don't explicitly provide will be routed to the attribute on the wrapped object and you can just implement the ones that you want
logged = Wrapped(open(filename))
In the question How do I "cd" in python, the accepted answer recommended wrapping the os.chdir call in a class to make the return to your original dir exception safe. Here was the recommended code:
class Chdir:
def __init__( self, newPath ):
self.savedPath = os.getcwd()
os.chdir(newPath)
def __del__( self ):
os.chdir( self.savedPath )
Could someone elaborate on how this works to make an unsafe call exception safe?
Thread safety and exception safety are not really the same thing at all. Wrapping the os.chdir call in a class like this is an attempt to make it exception safe not thread safe.
Exception safety is something you'll frequently hear C++ developers talk about. It isn't talked about nearly as much in the Python community. From Boost's Exception-Safety in Generic Components document:
Informally, exception-safety in a
component means that it exhibits
reasonable behavior when an exception
is thrown during its execution. For
most people, the term “reasonable”
includes all the usual expectations
for error-handling: that resources
should not be leaked, and that the
program should remain in a
well-defined state so that execution
can continue.
So the idea in the code snippet you supplied is to ensure that in the case of the exception, the program will return to a well-defined state. In this case, the process will be returned in the directory it started from, whether os.chdir itself fails, or something causes an exception to be thrown and the "Chdir" instance to be deleted.
This pattern of using an object that exists merely for cleaning up is a form of "Resource Acquisition Is Initialization", or "RAII". This technique is very popular in C++, but is not so popular in Python for a few reasons:
Python has try...finally, which serves pretty much the same purpose and is the more common idiom in Python.
Destructors (__del__) in Python are unreliable/unpredicatble in some implementations, so using them in this way is somewhat discouraged. In cpython they happen to be very reliable and predictable as long as cycles aren't involved (ie: when deletion is handled by reference counting) but in other implementations (Jython and I believe also IronPython) deletion happens when the garbage collector gets around to it, which could be much later. (Interestingly, this doesn't stop most Python programmers from relying on __del__ to close their opened files.)
Python has garbage collection, so you don't need to be quite as careful about cleanup as you do in C++. (I'm not saying you don't have to be careful at all, just that in the common situations you can rely on the gc to do the right thing for you.)
A more "pythonic" way of writing the above code would be:
saved_path = os.getcwd()
os.chdir(new_path)
try:
# code that does stuff in new_path goes here
finally:
os.chdir(saved_path)
The direct answer to the question is: It doesn't, the posted code is horrible.
Something like the following could be reasonable to make it "exception safe" (but much better is to avoid chdir and use full paths instead):
saved_path = os.getcwd()
try:
os.chdir(newPath)
do_work()
finally:
os.chdir(saved_path)
And this precise behavior can also be written into a context manager.
__del__ is called when the instance is about to be destroyed. So when you instantiate this class, the current working directory is saved to an instance attribute and then, well, os.chdir is called. When the instance is destroyed (for whatever reason) the current directory is changed to its old value.
This looks a bit incorrect to me. As far as I know, you must call parent's __del__ in your overriden __del__, so it should be more like this:
class Chdir(object):
def __init__(self, new_path):
self.saved_path = os.getcwd()
os.chdir(new_path)
def __del__(self):
os.chdir(self.saved_path)
super(Chdir, self).__del__()
That is, unless I am missing something, of course.
(By the way, can't you do the same using contextmanager?)
This code alone is neither thread-safe nor exception-safe. Actually I'm not really sure what you mean by exception-safe. Following code comes to mind:
try:
# something thrilling
except:
pass
And this is a terrible idea. Exceptions are not for guarding against. Well written code should catch exceptions and do something useful with them.