I'm curious if it is considered safe or good practice to depend on python's with...as statement. For example when opening a file:
with open("myfile","w") as myFile:
#do something
So in this example I neglected to explicitly call myFile.close() however I can assume it was called when python exited the with...as statement by calling the objects __exit__() method. Is it good practice/safe to depend upon this or would it be better to always explicitly call file.close()
This is what context managers are for, to rely on them to close the file for you. Context managers are called even if there was an exception.
The alternative is to use an finally block instead:
myFile = open("myfile","w")
try:
# do something with myFile
finally:
myFile.close()
but because the block inside of the try: might be long, by the time you get to the finally statement you have forgotten what you were setting this up for.
Context managers are more powerful still. Because the __exit__ method is informed of any exceptions, they can act as exception handlers as well (ignore the exception, raise another, etc.).
Yes, the with statement is better way. Since Python 2.5, the file object has been equipped with __enter__() and __exit__() methods. The __exit__() method closes the file object.
Python guarantees that it will call the __exit__() method, but there is not guarantee that the __exit__() method will close the resource, especially with 3rd party code. You need to manually verify that.
Related
Trying to understand things at a deeper level.
If I am opening a file, or a web request, tensorflow session, or anything that can be handled with a with statement; is there ever a time when I shouldn't use a with statement?
E.g., is there ever a time when I should use the more generic / general try except structure?
My real question is, what is the underlying structure of what with really does? I read some nice helpful hints as well as the documentation itself, but still some of the inner workings of with seem a bit like black magic to me. I am trying to demystify the magical components.
I always go to the Python Enhancement Proposals (PEPs) to understand concepts in python since they focus more on the conceptual reasoning for things compared to the documentation and usually directly address:
the reason for the new feature / change.
how it could be done with existing code / how it affects existing code.
since you are interested in the implementation aspect here is a relevant exert from PEP 343 - the "with" statement:
Specification: The 'with' Statement
A new statement is proposed with the syntax:
with EXPR as VAR:
BLOCK
(paragraph omitted - not really relevent for this question)
The translation of the above statement is:
mgr = (EXPR)
exit = type(mgr).__exit__ # Not calling it yet
value = type(mgr).__enter__(mgr)
exc = True
try:
try:
VAR = value # Only if "as VAR" is present
BLOCK
except:
# The exceptional case is handled here
exc = False
if not exit(mgr, *sys.exc_info()):
raise
# The exception is swallowed if exit() returns true
finally:
# The normal and non-local-goto cases are handled here
if exc:
exit(mgr, None, None, None)
So the internal workings of the with statement are exactly implemented like try: finally constructs, just with a cleaner syntax that makes it harder to forget to close files etc.
There are reasons for not using a with-statement: If a failure/success requires custom cleanup.
Normally you shouldn't need to know what cleanups should be performed when you have finished with a file. The with-statement takes care of closing the file no matter what (except for cases where Python has no chance of entering the __exit__ method of the contextmanager, for example an abrupt system shutdown or similar really exceptional stuff).
But if you need to perform some local and important cleanup then using a try/except/finally might make more sense. An important keyword in this context is: separation of concerns.
Say you call a function that hypothetically creates an object that isn't managed by Pythons GC and open a file and try to write it to a file. In this case you need to perform more cleanup than just opening and closing the file:
def func():
bad_object = create_object_that_cannot_be_cleaned_by_pythons_gc():
try:
with open(filename, 'w') as file:
file.write(bad_object.to_string())
finally:
bad_object.delete()
I had a really hard time to think of an example where it's of advantage to use try/finally instead of just creating a contextmanager and I'm not sure I suceeded (Normally I would implement this example as contextmanager :-) ). The important part should be that contextmanagers perform a default cleanup operation not localized, custom cleanup.
I would like to handle a NameError exception by injecting the desired missing variable into the frame and then continue the execution from last attempted instruction.
The following pseudo-code should illustrate my needs.
def function():
return missing_var
try:
print function()
except NameError:
frame = inspect.trace()[-1][0]
# inject missing variable
frame.f_globals["missing_var"] = ...
# continue frame execution from last attempted instruction
exec frame.f_code from frame.f_lasti
Read the whole unittest on repl.it
Notes
As pointed out by ivan_pozdeev in his answer, this is known as resumption.
After more research, I found Veedrac's answer to the question Resuming program at line number in the context before an exception using a custom sys.excepthook posted by lc2817 very interesting. It relies on Richie Hindle's work.
Background
The code runs in a slave process, which is controlled by a parent. Tasks (functions really) are written in the parent and latter passed to the slave using dill. I expect some tasks (running in the slave process) to try to access variables from outer scopes in the parent and I'd like the slave to request those variables to the parent on the fly.
p.s.: I don't expect this magic to run in a production environment.
On the contrary to what various commenters are saying, "resume-on-error" exception handling is possible in Python. The library fuckit.py implements said strategy. It steamrollers errors by rewriting the source code of your module at import time, inserting try...except blocks around every statement and swallowing all exceptions. So perhaps you could try a similar sort of tactic?
It goes without saying: that library is intended as a joke. Don't ever use it in production code.
You mentioned that your use case is to trap references to missing names. Have you thought about using metaprogramming to run your code in the context of a "smart" namespace such as a defaultdict? (This is perhaps only marginally less of a bad idea than fuckit.py.)
from collections import defaultdict
class NoMissingNamesMeta(type):
#classmethod
def __prepare__(meta, name, bases):
return defaultdict(lambda: "foo")
class MyClass(metaclass=NoMissingNamesMeta):
x = y + "bar" # y doesn't exist
>>> MyClass.x
'foobar'
NoMissingNamesMeta is a metaclass - a language construct for customising the behaviour of the class statement. Here we're using the __prepare__ method to customise the dictionary which will be used as the class's namespace during creation of the class. Thus, because we're using a defaultdict instead of a regular dictionary, a class whose metaclass is NoMissingNamesMeta will never get a NameError. Any names referred to during the creation of the class will be auto-initialised to "foo".
This approach is similar to #AndréFratelli's idea of manually requesting the lazily-initialised data from a Scope object. In production I'd do that, not this. The metaclass version requires less typing to write the client code, but at the expense of a lot more magic. (Imagine yourself debugging this code in two years, trying to understand why non-existent variables are dynamically being brought into scope!)
The "resumption" exception handling technique has proven to be problematic, that's why it's missing from C++ and later languages.
Your best bet is to use a while loop to not resume where the exception was thrown but rather repeat from a predetermined place:
while True:
try:
do_something()
except NameError as e:
handle_error()
else:
break
You really can't unwind the stack after an exception is thrown, so you'd have to deal with the issue before hand. If your requirement is to generate these variables on the fly (which wouldn't be recommended, but you seem to understand that), then you'd have to actually request them. You can implement a mechanism for that (such as having a global custom Scope class instance and overriding __getitem__, or using something like the __dir__ function), but not as you are asking for it.
(Edited for even more clarity)
I'm reading the Python book (Python Essential Reference by Beazley) and he says:
The with statement allows a series of statements to execute inside a
runtime context that is controlled by an object that serves as a context manager.
Here is an example:
with open("debuglog","a") as f:
f.write("Debugging\n")
statements
f.write("Done\n")
He goes on to say:
The with obj statement accepts an optional as var specifier. If given, the value
returned by obj._ enter _() is placed into var. It is important to emphasize
that obj is not necessarily the value assigned to var.
I understand the mechanics of what a 'with' keyword does: a file-object is returned by open and that object is accessible via f within the body of the block. I also understand that enter() and eventually exit() will be called.
But what exactly is a run-time context? A few low level details would be nice - or, an example in C. Could someone clarify what exactly a "context" is and how it might relate to other languages (C, C++). My understanding of a context was the environment eg: a Bash shell executes ls in the context of all the (env displayed) shell variables.
With the with keyword - yes f is accessible to the body of the block but isn't that just scoping? eg: for x in y: here x is not scoped within the block and retains it's value outside the block - is this what Beazley means when he talks about 'runtime context', that f is scoped only within the block and looses all significance outside the with-block?? Why does he say that the statements "execute inside a runtime context"??? Is this like an "eval"??
I understand that open returns an object that is "not ... assigned to var"??
Why isn't it assigned to var? What does Beazley mean by making a statement like that?
The with statement was introduced in PEP 343. This PEP also introduced a new term, "context manager", and defined what that term means.
Briefly, a "context manager" is an object that has special method functions .__enter__() and .__exit__(). The with statement guarantees that the .__enter__() method will be called to set up the block of code indented under the with statement, and also guarantees that the .__exit__() method function will be called at the time of exit from the block of code (no matter how the block is exited; for example, if the code raises an exception, .__exit__() will still be called).
http://www.python.org/dev/peps/pep-0343/
http://docs.python.org/2/reference/datamodel.html?highlight=context%20manager#with-statement-context-managers
The with statement is now the preferred way to handle any task that has a well-defined setup and teardown. Working with a file, for example:
with open(file_name) as f:
# do something with file
You know the file will be properly closed when you are done.
Another great example is a resource lock:
with acquire_lock(my_lock):
# do something
You know the code won't run until you get the lock, and as soon as the code is done the lock will be released. I don't often do multithreaded coding in Python, but when I did, this statement made sure that the lock was always released, even in the face of an exception.
P.S. I did a Google search online for examples of context managers and I found this nifty one: a context manager that executes a Python block in a specific directory.
http://ralsina.me/weblog/posts/BB963.html
EDIT:
The runtime context is the environment that is set up by the call to .__enter__() and torn down by the call to .__exit__(). In my example of acquiring a lock, the block of code runs in the context of having a lock available. In the example of reading a file, the block of code runs in the context of the file being open.
There isn't any secret magic inside Python for this. There is no special scoping, no internal stack, and nothing special in the parser. You simply write two method functions, .__enter__() and .__exit__() and Python calls them at specific points for your with statement.
Look again at this section from the PEP:
Remember, PEP 310 proposes roughly this syntax (the "VAR =" part is optional):
with VAR = EXPR:
BLOCK
which roughly translates into this:
VAR = EXPR
VAR.__enter__()
try:
BLOCK
finally:
VAR.__exit__()
In both examples, BLOCK is a block of code that runs in a specific runtime context that is set up by the call to VAR.__enter__() and torn down by VAR.__exit__().
There are two main benefits to the with statement and the way it is all set up.
The more concrete benefit is that it's "syntactic sugar". I would much rather write a two-line with statement than a six-line sequence of statements; it's easier two write the shorter one, it looks nicer and is easier to understand, and it is easier to get right. Six lines versus two means more chances to screw things up. (And before the with statement, I was usually sloppy about wrapping file I/O in a try block; I only did it sometimes. Now I always use with and always get the exception handling.)
The more abstract benefit is that this gives us a new way to think about designing our programs. Raymond Hettinger, in a talk at PyCon 2013, put it this way: when we are writing programs we look for common parts that we can factor out into functions. If we have code like this:
A
B
C
D
E
F
B
C
D
G
we can easily make a function:
def BCD():
B
C
D
A
BCD()
E
F
BCD()
G
But we have never had a really clean way to do this with setup/teardown. When we have a lot of code like this:
A
BCD()
E
A
XYZ()
E
A
PDQ()
E
Now we can define a context manager and rewrite the above:
with contextA:
BCD()
with contextA:
XYZ()
with contextA:
PDQ()
So now we can think about our programs and look for setup/teardown that can be abstracted into a "context manager". Raymond Hettinger showed several new "context manager" recipes he had invented (and I'm racking my brain trying to remember an example or two for you).
EDIT: Okay, I just remembered one. Raymond Hettinger showed a recipe, that will be built in to Python 3.4, for using a with statement to ignore an exception within a block. See it here: https://stackoverflow.com/a/15566001/166949
P.S. I've done my best to give the sense of what he was saying... if I have made any mistake or misstated anything, it's on me and not on him. (And he posts on StackOverflow sometimes so he might just see this and correct me if I've messed anything up.)
EDIT: You've updated the question with more text. I'll answer it specifically as well.
is this what Beazley means when he talks about 'runtime context', that f is scoped only within the block and looses all significance outside the with-block?? Why does he say that the statements "execute inside a runtime context"??? Is this like an "eval"??
Actually, f is not scoped only within the block. When you bind a name using the as keyword in a with statement, the name remains bound after the block.
The "runtime context" is an informal concept and it means "the state set up by the .__enter__() method function call and torn down by the .__exit__() method function call." Again, I think the best example is the one about getting a lock before the code runs. The block of code runs in the "context" of having the lock.
I understand that open returns an object that is "not ... assigned to var"?? Why isn't it assigned to var? What does Beazley mean by making a statement like that?
Okay, suppose we have an object, let's call it k. k implements a "context manager", which means that it has method functions k.__enter__() and k.__exit__(). Now we do this:
with k as x:
# do something
What David Beazley wants you to know is that x will not necessarily be bound to k. x will be bound to whatever k.__enter__() returns. k.__enter__() is free to return a reference to k itself, but is also free to return something else. In this case:
with open(some_file) as f:
# do something
The call to open() returns an open file object, which works as a context manager, and its .__enter__() method function really does just return a reference to itself.
I think most context managers return a reference to self. Since it's an object it can have any number of member variables, so it can return any number of values in a convenient way. But it isn't required.
For example, there could be a context manager that starts a daemon running in the .__enter__() function, and returns the process ID number of the daemon from the .__enter__() function. Then the .__exit__() function would shut down the daemon. Usage:
with start_daemon("parrot") as pid:
print("Parrot daemon running as PID {}".format(pid))
daemon = lookup_daemon_by_pid(pid)
daemon.send_message("test")
But you could just as well return the context manager object itself with any values you need tucked inside:
with start_daemon("parrot") as daemon:
print("Parrot daemon running as PID {}".format(daemon.pid))
daemon.send_message("test")
If we need the PID of the daemon, we can just put it in a .pid member of the object. And later if we need something else we can just tuck that in there as well.
The with context takes care that on entry, the __enter__ method is called and the given var is set to whatever __enter__ returns.
In most cases, that is the object which is worked on previously - in the file case, it is - but e.g. on a database, not the connection object, but a cursor object is returned.
The file example can be extended like this:
f1 = open("debuglog","a")
with f1 as f2:
print f1 is f2
which will print True as here, the file object is returned by __enter__. (From its point of view, self.)
A database works like
d = connect(...)
with d as c:
print d is c # False
print d, c
Here, d and c are completely different: d is the connection to the database, c is a cursor used for one transaction.
The with clause is terminated by a call to __exit__() which is given the state of execution of the clause - either success or failure. In this case, the __exit__() method can act appropriately.
In the file example, the file is closed no matter if there was an error or not.
In the database example, normally the transaction is committed on success and rolled back on failure.
The context manager is for easy initialisation and cleanup of things like exactly these - files, databases etc.
There is no direct correspondence in C or C++ that I am aware of.
C knows no concept of exception, so none can be caught in a __exit__(). C++ knows exceptions, and there seems to be ways to do soo (look below at the comments).
I have to open a file-like object in python (it's a serial connection through /dev/) and then close it. This is done several times in several methods of my class. How I WAS doing it was opening the file in the constructor, and then closing it in the destructor. I'm getting weird errors though and I think it has to do with the garbage collector and such, I'm still not used to not knowing exactly when my objects are being deleted =\
The reason I was doing this is because I have to use tcsetattr with a bunch of parameters each time I open it and it gets annoying doing all that all over the place. So I want to implement an inner class to handle all that so I can use it doing
with Meter('/dev/ttyS2') as m:
I was looking online and I couldn't find a really good answer on how the with syntax is implemented. I saw that it uses the __enter__(self) and __exit(self)__ methods. But is all I have to do implement those methods and I can use the with syntax? Or is there more to it?
Is there either an example on how to do this or some documentation on how it's implemented on file objects already that I can look at?
Those methods are pretty much all you need for making the object work with with statement.
In __enter__ you have to return the file object after opening it and setting it up.
In __exit__ you have to close the file object. The code for writing to it will be in the with statement body.
class Meter():
def __init__(self, dev):
self.dev = dev
def __enter__(self):
#ttysetattr etc goes here before opening and returning the file object
self.fd = open(self.dev, MODE)
return self
def __exit__(self, type, value, traceback):
#Exception handling here
close(self.fd)
meter = Meter('dev/tty0')
with meter as m:
#here you work with the file object.
m.fd.read()
Easiest may be to use standard Python library module contextlib:
import contextlib
#contextlib.contextmanager
def themeter(name):
theobj = Meter(name)
try:
yield theobj
finally:
theobj.close() # or whatever you need to do at exit
# usage
with themeter('/dev/ttyS2') as m:
# do what you need with m
m.read()
This doesn't make Meter itself a context manager (and therefore is non-invasive to that class), but rather "decorates" it (not in the sense of Python's "decorator syntax", but rather almost, but not quite, in the sense of the decorator design pattern;-) with a factory function themeter which is a context manager (which the contextlib.contextmanager decorator builds from the "single-yield" generator function you write) -- this makes it so much easier to separate the entering and exiting condition, avoids nesting, &c.
The first Google hit (for me) explains it simply enough:
http://effbot.org/zone/python-with-statement.htm
and the PEP explains it more precisely (but also more verbosely):
http://www.python.org/dev/peps/pep-0343/
In the question How do I "cd" in python, the accepted answer recommended wrapping the os.chdir call in a class to make the return to your original dir exception safe. Here was the recommended code:
class Chdir:
def __init__( self, newPath ):
self.savedPath = os.getcwd()
os.chdir(newPath)
def __del__( self ):
os.chdir( self.savedPath )
Could someone elaborate on how this works to make an unsafe call exception safe?
Thread safety and exception safety are not really the same thing at all. Wrapping the os.chdir call in a class like this is an attempt to make it exception safe not thread safe.
Exception safety is something you'll frequently hear C++ developers talk about. It isn't talked about nearly as much in the Python community. From Boost's Exception-Safety in Generic Components document:
Informally, exception-safety in a
component means that it exhibits
reasonable behavior when an exception
is thrown during its execution. For
most people, the term “reasonable”
includes all the usual expectations
for error-handling: that resources
should not be leaked, and that the
program should remain in a
well-defined state so that execution
can continue.
So the idea in the code snippet you supplied is to ensure that in the case of the exception, the program will return to a well-defined state. In this case, the process will be returned in the directory it started from, whether os.chdir itself fails, or something causes an exception to be thrown and the "Chdir" instance to be deleted.
This pattern of using an object that exists merely for cleaning up is a form of "Resource Acquisition Is Initialization", or "RAII". This technique is very popular in C++, but is not so popular in Python for a few reasons:
Python has try...finally, which serves pretty much the same purpose and is the more common idiom in Python.
Destructors (__del__) in Python are unreliable/unpredicatble in some implementations, so using them in this way is somewhat discouraged. In cpython they happen to be very reliable and predictable as long as cycles aren't involved (ie: when deletion is handled by reference counting) but in other implementations (Jython and I believe also IronPython) deletion happens when the garbage collector gets around to it, which could be much later. (Interestingly, this doesn't stop most Python programmers from relying on __del__ to close their opened files.)
Python has garbage collection, so you don't need to be quite as careful about cleanup as you do in C++. (I'm not saying you don't have to be careful at all, just that in the common situations you can rely on the gc to do the right thing for you.)
A more "pythonic" way of writing the above code would be:
saved_path = os.getcwd()
os.chdir(new_path)
try:
# code that does stuff in new_path goes here
finally:
os.chdir(saved_path)
The direct answer to the question is: It doesn't, the posted code is horrible.
Something like the following could be reasonable to make it "exception safe" (but much better is to avoid chdir and use full paths instead):
saved_path = os.getcwd()
try:
os.chdir(newPath)
do_work()
finally:
os.chdir(saved_path)
And this precise behavior can also be written into a context manager.
__del__ is called when the instance is about to be destroyed. So when you instantiate this class, the current working directory is saved to an instance attribute and then, well, os.chdir is called. When the instance is destroyed (for whatever reason) the current directory is changed to its old value.
This looks a bit incorrect to me. As far as I know, you must call parent's __del__ in your overriden __del__, so it should be more like this:
class Chdir(object):
def __init__(self, new_path):
self.saved_path = os.getcwd()
os.chdir(new_path)
def __del__(self):
os.chdir(self.saved_path)
super(Chdir, self).__del__()
That is, unless I am missing something, of course.
(By the way, can't you do the same using contextmanager?)
This code alone is neither thread-safe nor exception-safe. Actually I'm not really sure what you mean by exception-safe. Following code comes to mind:
try:
# something thrilling
except:
pass
And this is a terrible idea. Exceptions are not for guarding against. Well written code should catch exceptions and do something useful with them.