I really like the syntax "with open('in_file'') as f".
I want to use that syntax for my own resources which must be opened and closed.
However, I do not understand how to change my open() method to enable the 'with' syntax.
I can (and do) use the contextlib.closing() approach but it becomes a bit bulky after repeated use.
So, I will ask my question below in relation to shelve.open().
I am not proposing a change to the shelve module but instead am using it because the source code is readily available to all of you.
There is nothing special about shelve.open() vs. other standard library resources that require closing: socket.socket(), sqlite3.connect(), urllib2.urlopen(), etc.
import contextlib, inspect, shelve, sys
#print(inspect.getsource(open)) # can not see how it was done here :-(
print('-' * 40)
# Given that we can view the source for the shelve module:
print(inspect.getsource(shelve))
print('-' * 40)
# Given that we can view the docs for the shelve module:
print(shelve.__doc__)
#print('-' * 40)
# Given that the desired syntax is Pythonic but is not supported:
#with shelve.open('test_shelve') as my_shelve:
# my_shelve['fact_number_1'] = "There's a dead fish on the landing."
# Given that the required syntax is convoluted and
# takes programmer attention away from the task at hand:
with contextlib.closing(shelve.open('test_shelve')) as my_shelve:
my_shelve['fact_number_2'] = "There's another dead fish on the landing."
# Q: What changes would need to made to shelve.open() to allow the
# 'with shelve.open(x) as y' syntax?
I am not really interested in an extra wrapper with a different name. Using contextlib.closing() is easier, safer, and more intuitive than that. What I am really interested in is creating a single open() method that can be called either with or without 'with'.
So, to successfully answer this question, you need to take the source code for the shelve module and show what changes would need to be made to shelve.open() to have a single method that can be used either with or without 'with' (like the builtin open() or the Python3 urllib.urlopen()).
The biggest problem here is that if you do
shelf = the_function_you_want()
the function you want has to return the shelf, but if you do
with the_function_you_want() as shelf:
the function you want has to return a context manager. That means you need to return a shelf that is also a context manager, which in turn means you either need to make a shelf subclass or monkey-patch Shelf. It's probably better to make a subclass:
class ContextManagerShelf(shelve.DbfilenameShelf):
def __enter__(self):
return self
def __exit__(self, *exc_info):
self.close()
Then you can use ContextManagerShelf as a context manager or not. The signature is the same as shelve.open. If you want, you can also make an open function to go with it.
Related
In C#, when an object implements IDisposable, using should be used to guarantee that resources will be cleaned if an exception is thrown. For instance, instead of:
var connection = new SqlConnection(...);
...
connection.Close();
one needs to write:
using (var connection = new SqlConnection(...))
{
...
}
Therefore, just by looking at the signature of the class, I know exactly whether or not I should initialize the object inside a using.
In Python 3, a similar construct is with. Similarly to C#, it ensures that the resources will be cleaned up automatically when exiting the with context, even if a error is raised.
However, I'm not sure how should I determine whether with should be used or not for a specific class. For instance, an example from psycopg doesn't use with, which may mean that:
I shouldn't either, or:
The example is written for Python 2, or:
The authors of the documentation were unaware of with syntax, or:
The authors decided not to handle exceptional cases for the sake of simplicity.
In general, how should I determine whether with should be used when initializing an instance of a specific class (assuming that documentation says nothing on the subject, and that I have access to source code)?
Regarding when you should use it:
No one forces you to use the with statement, it's just syntactic sugar that's there to make your life easier. If you use it or not is totally up to you but, it is generally recommended to do so. (We're forgetful and with ... looks ways better than explicit initialize resource/finalize recourse calls).
When you can use it:
When you can use it boils down to examining if it defines the context manager protocol. This could be as simple as trying to use with and seeing that it fails :-)
If you dynamically need to check if an object is a context manager, you have two options.
First, wait for the stable release of Python 3.6 which defines an ABC for context managers, ContextManager, which can be used in issubclass/isinstance checks:
>>> from typing import ContextManager
>>> class foo:
... def __enter__(self): pass
... def __exit__(self): pass
...
>>> isinstance(foo(), ContextManager)
True
>>> class foo2: pass
...
>>> isinstance(foo2(), ContextManager)
False
Or, create your own little function to check for it:
def iscontext(inst):
cls = type(inst)
return (any("__enter__" in vars(a) for a in cls.__mro__) and
any("__exit__" in vars(a) for a in cls.__mro__))
As a final note, the with statement is present in Python 2 and in 3, the use case you saw probably just wasn't aware of it :-).
with is for use with context managers.
At the code level, a context manager must define two methods:
__enter__(self)
__exit__(self, type, value, traceback).
Be aware that there are class decorators which can turn otherwise simple classes/functions into context managers - see contextlib for some examples
You should use with whenever you need to perform some similar action before and after executing the statement. For example:
Want to execute SQL query? You need to open and close the connections safely.Use with.
Want to perform some action on file? You have to open and close the file safely. Use with
Want to store some data in temporary file to perform some task? You need to create the directory, and clean it up once you are done. Use with, and so on. . .
Everything you want to perform before the query execution, add it to the __enter__() method. And the action to be performed after, add it to the __exit__() method.
One of the nice thing about with is, __exit__ is executed even if the code within with raises any Exception
I just realized there is something mysterious (at least for me) in the way you can add vertex instructions in Kivy with the with Python statement. For example, the way with is used goes something like this:
... some code
class MyWidget(Widget)
... some code
def some_method (self):
with self.canvas:
Rectangle(pos=self.pos, size=self.size)
At the beginning I thought that it was just the with Python statement that I have used occasionally. But suddenly I realize it is not. Usually it looks more like this (example taken from here):
with open('output.txt', 'w') as f:
f.write('Hi there!')
There is usually an as after the instance and something like and alias to the object. In the Kivy example we don't define and alias which is still ok. But the part that puzzles me is that instruction Rectangle is still associated to the self.canvas. After reading about the with statement, I am quite convinced that the Kivy code should be written like:
class MyWidget(Widget)
... some code
def some_method (self):
with self.canvas as c:
c.add (Rectangle(pos=self.pos, size=self.size))
I am assuming that internally the method add is the one being called. The assumption is based that we can simply add the rectangles with self.add (Rectangle(pos=self.pos, size=self.size))
Am I missing something about the with Python statement? or is this somehow something Kivy implements?
I don't know Kivy, but I think I can guess how this specific construction work.
Instead of keeping a handle to the object you are interacting with (the canvas?), the with statement is programmed to store it in some global variable, hidden to you. Then, the statements you use inside with use that global variable to retrieve the object. At the end of the block, the global variable is cleared as part of cleanup.
The result is a trade-off: code is less explicit (which is usually a desired feature in Python). However, the code is shorter, which might lead to easier understanding (with the assumption that the reader knows how Kivy works). This is actually one of the techniques of making embedded DSLs in Python.
There are some technicalities involved. For example, if you want to be able to nest such constructions (put one with inside another), instead of a simple global variable you would want to use a global variable that keeps a stack of such objects. Also, if you need to deal with threading, you would use a thread-local variable instead of a global one. But the generic mechanism is still the same—Kivy uses some state which is kept in a place outside your direct control.
There is nothing extra magical with the with statement, but perhaps you are unaware of how it works?
In order for any object to be used in a with statement it must implement two methods: __enter__ and __exit__. __enter__ is called when the with block is entered, and __exit__ is called when the block is exited for any reason.
What the object does in its __enter__ method is, of course, up to it. Since I don't have the Kivy code I can only guess that its canvas.__enter__ method sets a global variable somewhere, and that Rectangle checks that global to see where it should be drawing.
I just realized there is something mysterious (at least for me) in the way you can add vertex instructions in Kivy with the with Python statement. For example, the way with is used goes something like this:
... some code
class MyWidget(Widget)
... some code
def some_method (self):
with self.canvas:
Rectangle(pos=self.pos, size=self.size)
At the beginning I thought that it was just the with Python statement that I have used occasionally. But suddenly I realize it is not. Usually it looks more like this (example taken from here):
with open('output.txt', 'w') as f:
f.write('Hi there!')
There is usually an as after the instance and something like and alias to the object. In the Kivy example we don't define and alias which is still ok. But the part that puzzles me is that instruction Rectangle is still associated to the self.canvas. After reading about the with statement, I am quite convinced that the Kivy code should be written like:
class MyWidget(Widget)
... some code
def some_method (self):
with self.canvas as c:
c.add (Rectangle(pos=self.pos, size=self.size))
I am assuming that internally the method add is the one being called. The assumption is based that we can simply add the rectangles with self.add (Rectangle(pos=self.pos, size=self.size))
Am I missing something about the with Python statement? or is this somehow something Kivy implements?
I don't know Kivy, but I think I can guess how this specific construction work.
Instead of keeping a handle to the object you are interacting with (the canvas?), the with statement is programmed to store it in some global variable, hidden to you. Then, the statements you use inside with use that global variable to retrieve the object. At the end of the block, the global variable is cleared as part of cleanup.
The result is a trade-off: code is less explicit (which is usually a desired feature in Python). However, the code is shorter, which might lead to easier understanding (with the assumption that the reader knows how Kivy works). This is actually one of the techniques of making embedded DSLs in Python.
There are some technicalities involved. For example, if you want to be able to nest such constructions (put one with inside another), instead of a simple global variable you would want to use a global variable that keeps a stack of such objects. Also, if you need to deal with threading, you would use a thread-local variable instead of a global one. But the generic mechanism is still the same—Kivy uses some state which is kept in a place outside your direct control.
There is nothing extra magical with the with statement, but perhaps you are unaware of how it works?
In order for any object to be used in a with statement it must implement two methods: __enter__ and __exit__. __enter__ is called when the with block is entered, and __exit__ is called when the block is exited for any reason.
What the object does in its __enter__ method is, of course, up to it. Since I don't have the Kivy code I can only guess that its canvas.__enter__ method sets a global variable somewhere, and that Rectangle checks that global to see where it should be drawing.
I have to open a file-like object in python (it's a serial connection through /dev/) and then close it. This is done several times in several methods of my class. How I WAS doing it was opening the file in the constructor, and then closing it in the destructor. I'm getting weird errors though and I think it has to do with the garbage collector and such, I'm still not used to not knowing exactly when my objects are being deleted =\
The reason I was doing this is because I have to use tcsetattr with a bunch of parameters each time I open it and it gets annoying doing all that all over the place. So I want to implement an inner class to handle all that so I can use it doing
with Meter('/dev/ttyS2') as m:
I was looking online and I couldn't find a really good answer on how the with syntax is implemented. I saw that it uses the __enter__(self) and __exit(self)__ methods. But is all I have to do implement those methods and I can use the with syntax? Or is there more to it?
Is there either an example on how to do this or some documentation on how it's implemented on file objects already that I can look at?
Those methods are pretty much all you need for making the object work with with statement.
In __enter__ you have to return the file object after opening it and setting it up.
In __exit__ you have to close the file object. The code for writing to it will be in the with statement body.
class Meter():
def __init__(self, dev):
self.dev = dev
def __enter__(self):
#ttysetattr etc goes here before opening and returning the file object
self.fd = open(self.dev, MODE)
return self
def __exit__(self, type, value, traceback):
#Exception handling here
close(self.fd)
meter = Meter('dev/tty0')
with meter as m:
#here you work with the file object.
m.fd.read()
Easiest may be to use standard Python library module contextlib:
import contextlib
#contextlib.contextmanager
def themeter(name):
theobj = Meter(name)
try:
yield theobj
finally:
theobj.close() # or whatever you need to do at exit
# usage
with themeter('/dev/ttyS2') as m:
# do what you need with m
m.read()
This doesn't make Meter itself a context manager (and therefore is non-invasive to that class), but rather "decorates" it (not in the sense of Python's "decorator syntax", but rather almost, but not quite, in the sense of the decorator design pattern;-) with a factory function themeter which is a context manager (which the contextlib.contextmanager decorator builds from the "single-yield" generator function you write) -- this makes it so much easier to separate the entering and exiting condition, avoids nesting, &c.
The first Google hit (for me) explains it simply enough:
http://effbot.org/zone/python-with-statement.htm
and the PEP explains it more precisely (but also more verbosely):
http://www.python.org/dev/peps/pep-0343/
I am developing a medium size program in python spread across 5 modules. The program accepts command line arguments using OptionParser in the main module e.g. main.py. These options are later used to determine how methods in other modules behave (e.g. a.py, b.py). As I extend the ability for the user to customise the behaviour or the program I find that I end up requiring this user-defined parameter in a method in a.py that is not directly called by main.py, but is instead called by another method in a.py:
main.py:
import a
p = some_command_line_argument_value
a.meth1(p)
a.py:
meth1(p):
# some code
res = meth2(p)
# some more code w/ res
meth2(p):
# do something with p
This excessive parameter passing seems wasteful and wrong, but has hard as I try I cannot think of a design pattern that solves this problem. While I had some formal CS education (minor in CS during my B.Sc.), I've only really come to appreciate good coding practices since I started using python. Please help me become a better programmer!
Create objects of types relevant to your program, and store the command line options relevant to each in them. Example:
import WidgetFrobnosticator
f = WidgetFrobnosticator()
f.allow_oncave_widgets = option_allow_concave_widgets
f.respect_weasel_pins = option_respect_weasel_pins
# Now the methods of WidgetFrobnosticator have access to your command-line parameters,
# in a way that's not dependent on the input format.
import PlatypusFactory
p = PlatypusFactory()
p.allow_parthenogenesis = option_allow_parthenogenesis
p.max_population = option_max_population
# The platypus factory knows about its own options, but not those of the WidgetFrobnosticator
# or vice versa. This makes each class easier to read and implement.
Maybe you should organize your code more into classes and objects? As I was writing this, Jimmy showed a class-instance based answer, so here is a pure class-based answer. This would be most useful if you only ever wanted a single behavior; if there is any chance at all you might want different defaults some of the time, you should use ordinary object-oriented programming in Python, i.e. pass around class instances with the property p set in the instance, not the class.
class Aclass(object):
p = None
#classmethod
def init_p(cls, value):
p = value
#classmethod
def meth1(cls):
# some code
res = cls.meth2()
# some more code w/ res
#classmethod
def meth2(cls):
# do something with p
pass
from a import Aclass as ac
ac.init_p(some_command_line_argument_value)
ac.meth1()
ac.meth2()
If "a" is a real object and not just a set of independent helper methods, you can create an "p" member variable in "a" and set it when you instantiate an "a" object. Then your main class will not need to pass "p" into meth1 and meth2 once "a" has been instantiated.
[Caution: my answer isn't specific to python.]
I remember that Code Complete called this kind of parameter a "tramp parameter". Googling for "tramp parameter" doesn't return many results, however.
Some alternatives to tramp parameters might include:
Put the data in a global variable
Put the data in a static variable of a class (similar to global data)
Put the data in an instance variable of a class
Pseudo-global variable: hidden behind a singleton, or some dependency injection mechanism
Personally, I don't mind a tramp parameter as long as there's no more than one; i.e. your example is OK for me, but I wouldn't like ...
import a
p1 = some_command_line_argument_value
p2 = another_command_line_argument_value
p3 = a_further_command_line_argument_value
a.meth1(p1, p2, p3)
... instead I'd prefer ...
import a
p = several_command_line_argument_values
a.meth1(p)
... because if meth2 decides that it wants more data than before, I'd prefer if it could extract this extra data from the original parameter which it's already being passed, so that I don't need to edit meth1.
With objects, parameter lists should normally be very small, since most appropriate information is a property of the object itself. The standard way to handle this is to configure the object properties and then call the appropriate methods of that object. In this case set p as an attribute of a. Your meth2 should also complain if p is not set.
Your example is reminiscent of the code smell Message Chains. You may find the corresponding refactoring, Hide Delegate, informative.