Related
Ruby can add methods to the Number class and other core types to get effects like this:
1.should_equal(1)
But it seems like Python cannot do this. Is this true? And if so, why? Does it have something to do with the fact that type can't be modified?
Rather than talking about different definitions of monkey patching, I would like to just focus on the example above. I have already concluded that it cannot be done as a few of you have answered. But I would like a more detailed explanation of why it cannot be done, and maybe what feature, if available in Python, would allow this.
To answer some of you: The reason I might want to do this is simply aesthetics/readability.
item.price.should_equal(19.99)
This reads more like English and clearly indicates which is the tested value and which is the expected value, as supposed to:
should_equal(item.price, 19.99)
This concept is what Rspec and some other Ruby frameworks are based on.
No, you cannot. In Python, all data (classes, methods, functions, etc) defined in C extension modules (including builtins) are immutable. This is because C modules are shared between multiple interpreters in the same process, so monkeypatching them would also affect unrelated interpreters in the same process. (Multiple interpreters in the same process are possible through the C API, and there has been some effort towards making them usable at Python level.)
However, classes defined in Python code may be monkeypatched because they are local to that interpreter.
What exactly do you mean by Monkey Patch here? There are several slightly different definitions.
If you mean, "can you change a class's methods at runtime?", then the answer is emphatically yes:
class Foo:
pass # dummy class
Foo.bar = lambda self: 42
x = Foo()
print x.bar()
If you mean, "can you change a class's methods at runtime and make all of the instances of that class change after-the-fact?" then the answer is yes as well. Just change the order slightly:
class Foo:
pass # dummy class
x = Foo()
Foo.bar = lambda self: 42
print x.bar()
But you can't do this for certain built-in classes, like int or float. These classes' methods are implemented in C and there are certain abstractions sacrificed in order to make the implementation easier and more efficient.
I'm not really clear on why you would want to alter the behavior of the built-in numeric classes anyway. If you need to alter their behavior, subclass them!!
You can do this, but it takes a little bit of hacking. Fortunately, there's a module now called "Forbidden Fruit" that gives you the power to patch methods of built-in types very simply. You can find it at
http://clarete.github.io/forbiddenfruit/?goback=.gde_50788_member_228887816
or
https://pypi.python.org/pypi/forbiddenfruit/0.1.0
With the original question example, after you write the "should_equal" function, you'd just do
from forbiddenfruit import curse
curse(int, "should_equal", should_equal)
and you're good to go! There's also a "reverse" function to remove a patched method.
def should_equal_def(self, value):
if self != value:
raise ValueError, "%r should equal %r" % (self, value)
class MyPatchedInt(int):
should_equal=should_equal_def
class MyPatchedStr(str):
should_equal=should_equal_def
import __builtin__
__builtin__.str = MyPatchedStr
__builtin__.int = MyPatchedInt
int(1).should_equal(1)
str("44").should_equal("44")
Have fun ;)
Python's core types are immutable by design, as other users have pointed out:
>>> int.frobnicate = lambda self: whatever()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't set attributes of built-in/extension type 'int'
You certainly could achieve the effect you describe by making a subclass, since user-defined types in Python are mutable by default.
>>> class MyInt(int):
... def frobnicate(self):
... print 'frobnicating %r' % self
...
>>> five = MyInt(5)
>>> five.frobnicate()
frobnicating 5
>>> five + 8
13
There's no need to make the MyInt subclass public, either; one could just as well define it inline directly in the function or method that constructs the instance.
There are certainly a few situations where Python programmers who are fluent in the idiom consider this sort of subclassing the right thing to do. For instance, os.stat() returns a tuple subclass that adds named members, precisely in order to address the sort of readability concern you refer to in your example.
>>> import os
>>> st = os.stat('.')
>>> st
(16877, 34996226, 65024L, 69, 1000, 1000, 4096, 1223697425, 1223699268, 1223699268)
>>> st[6]
4096
>>> st.st_size
4096
That said, in the specific example you give, I don't believe that subclassing float in item.price (or elsewhere) would be very likely to be considered the Pythonic thing to do. I can easily imagine somebody deciding to add a price_should_equal() method to item if that were the primary use case; if one were looking for something more general, perhaps it might make more sense to use named arguments to make the intended meaning clearer, as in
should_equal(observed=item.price, expected=19.99)
or something along those lines. It's a bit verbose, but no doubt it could be improved upon. A possible advantage to such an approach over Ruby-style monkey-patching is that should_equal() could easily perform its comparison on any type, not just int or float. But perhaps I'm getting too caught up in the details of the particular example that you happened to provide.
You can't patch core types in python.
However, you could use pipe to write a more human readable code:
from pipe import *
#Pipe
def should_equal(obj, val):
if obj==val: return True
return False
class dummy: pass
item=dummy()
item.value=19.99
print item.value | should_equal(19.99)
If you really really really want to do a monkey patch in Python, you can do a (sortof) hack with the "import foo as bar" technique.
If you have a class such as TelnetConnection, and you want to extend it, subclass it in a separate file and call it something like TelnetConnectionExtended.
Then, at the top of your code, where you would normally say:
import TelnetConnection
change that to be:
import TelnetConnectionExtended as TelnetConnection
and then everywhere in your code that you reference TelnetConnection will actually be referencing TelnetConnectionExtended.
Sadly, this assumes that you have access to that class, and the "as" only operates within that particular file (it's not a global-rename), but I've found it to be useful from time to time.
Here's an example of implementing item.price.should_equal, although I'd use Decimal instead of float in a real program:
class Price(float):
def __init__(self, val=None):
float.__init__(self)
if val is not None:
self = val
def should_equal(self, val):
assert self == val, (self, val)
class Item(object):
def __init__(self, name, price=None):
self.name = name
self.price = Price(price)
item = Item("spam", 3.99)
item.price.should_equal(3.99)
No but you have UserDict UserString and UserList which were made with exactly this in mind.
If you google you will find examples for other types, but this are builtin.
In general monkey patching is less used in Python than in Ruby.
What does should_equal do? Is it a boolean returning True or False? In that case, it's spelled:
item.price == 19.99
There's no accounting for taste, but no regular python developer would say that's less readable than your version.
Does should_equal instead set some sort of validator? (why would a validator be limited to one value? Why not just set the value and not update it after that?) If you want a validator, this could never work anyway, since you're proposing to modify either a particular integer or all integers. (A validator that requires 18.99 to equal 19.99 will always fail.) Instead, you could spell it like this:
item.price_should_equal(19.99)
or this:
item.should_equal('price', 19.99)
and define appropriate methods on item's class or superclasses.
It seems what you really wanted to write is:
assert item.price == 19.99
(Of course comparing floats for equality, or using floats for prices, is a bad idea, so you'd write assert item.price == Decimal(19.99) or whatever numeric class you were using for the price.)
You could also use a testing framework like py.test to get more info on failing asserts in your tests.
No, you can't do that in Python. I consider it to be a good thing.
No, sadly you cannot extend types implemented in C at runtime.
You can subclass int, although it is non-trivial, you may have to override __new__.
You also have a syntax issue:
1.somemethod() # invalid
However
(1).__eq__(1) # valid
Here is how I made custom string/int/float...etc. methods:
class MyStrClass(str):
def __init__(self, arg: str):
self.arg_one = arg
def my_str_method(self):
return self.arg_one
def my_str_multiple_arg_method(self, arg_two):
return self.arg_one + arg_two
class MyIntClass(int):
def __init__(self, arg: int):
self.arg_one = arg
def my_int_method(self):
return self.arg_one * 2
myString = MyStrClass("StackOverflow")
myInteger = MyIntClass(15)
print(myString.count("a")) # Output: 1
print(myString.my_str_method()) # Output: StackOverflow
print(myString.my_str_multiple_arg_method(" is cool!")) # Output: StackOverflow is cool!
print(myInteger.my_int_method()) # Output: 30
It's maybe not the best solution, but it works just fine.
Here's how I achieve the .should_something... behavior:
result = calculate_result('blah') # some method defined somewhere else
the(result).should.equal(42)
or
the(result).should_NOT.equal(41)
I included a decorator method for extending this behavior at runtime on a stand-alone method:
#should_expectation
def be_42(self)
self._assert(
action=lambda: self._value == 42,
report=lambda: "'{0}' should equal '5'.".format(self._value)
)
result = 42
the(result).should.be_42()
You have to know a bit about the internals but it works.
Here's the source:
https://github.com/mdwhatcott/pyspecs
It's also on PyPI under pyspecs.
I have an API in Python which can return an object, or None if no object is found. I want to avoid run-time exceptions/crashes, etc., hence I want to force the users of my API, to do an is not None test.
For example:
x = getObject(...)
if x is not None:
print x.getName() #should be o.k.
y = getObject(...)
print y.getName() # print an error to the log
How can I achieve that?
In comparable code in C++, I can add a flag that will be checked when I call the getName(); the flag is set only upon comparing the object to NULL.
In Python, however, I am unable to overload the is operator. Are there any other ways I can achieve that functionality in Python?
You cannot force the use of if x is not None because you cannot override the behavior of id(). The is operator internally compares the ids of the two objects being compared, and you have no way of controlling that behavior.
However, you can force the use of if x != None or if not x == Noneby overriding the __eq__ and __ne__ methods of your class, respectively.
This is not good practice, however. As #Kevin has noted in the comments, is is the preferred operator to use when comparing to None.
What I would do is write clear and organized documentation for this API, and then clearly warn users that the instantiation could fail and return None. Then, gently nudge users towards good practices by providing an example with the built-in getattr function or an example with the is not None check.
Like it was already said, you can't override is behavior.
To do what you want, basically you can create a surrogate object that has a getName() function. To let the user check if the function failed, you can have the object evaluate to False. (This is a standard practice and I think this is better than making the object equal to None with the __eq__ operator). To do this, you can override override __nonzero__() having it return False.
Example:
class GetObjectFailed(object):
def __nonzero__():
return False
def getName():
return "An error has occurred" # You could specify a specific error message here...
x = getObject(...)
print x # prints "An error has occurred"
if x:
# This is the recommended way of doing things
# Do something with the object
x.proccess()
if x is not None:
# This will never work
x.proccess()
if x != None:
# This is possible but not recommended
x.proccess()
I'm creating instances of a class Foo, and I'd like to be able to instantiate these in a general way from a variety of types. You can't pass Foo a dict or list. Note that Foo is from a 3rd party code base - I can't change Foo's code.
I know that type checking function arguments in Python is considered bad form. Is there a more Pythonic way to write the function below (i.e. without type checking)?
def to_foo(arg):
if isinstance(arg, dict):
return dict([(key,to_foo(val)) for key,val in arg.items()])
elif isinstance(arg, list):
return [to_foo(i) for i in arg]
else:
return Foo(arg)
Edit: Using try/except blocks is possible. For instance, you could do:
def to_foo(arg):
try:
return Foo(arg)
except ItWasADictError:
return dict([(key,to_foo(val)) for key,val in arg.items()])
except ItWasAListError:
return [to_foo(i) for i in arg]
I'm not totally satisfied by this for two reasons: first, type checking seems like it addresses more directly the desired functionality, whereas the try/except block here seems like it's getting to the same place but less directly. Second, what if the errors don't cleanly map like this? (e.g. if passing either a list or dict throws a TypeError)
Edit: a third reason I'm not a huge fan of the try/except method here is I need to go and find what exceptions Foo is going to throw in those cases, rather than being able to code it up front.
If you're using python 3.4 you can use functools.singledispatch, or a backport for a different python version
from functools import singledispatch
#singledispatch
def to_foo(arg):
return Foo(arg)
#to_foo.register(list)
def to_foo_list(arg):
return [Foo(i) for i in arg]
#to_foo.register(dict)
def to_foo_dict(arg):
return {key: Foo(val) for key, val in arg.items()}
This is a fairly new construct for python, but a common pattern in other languages. I'm not sure you'd call this pythonic or not, but it does feel better than writing isinstances everywhere. Though, in practise, the singledispatch is probably just doing the isinstance checks for you internally.
The pythonic way to deal with your issue is to go ahead and assume (first) that arg is Foo and except any error:
try:
x = Foo(arg)
except NameError:
#do other things
The phrase for this idea is "duck typing", and it's a popular pattern in python.
This question already has answers here:
What is the best (idiomatic) way to check the type of a Python variable? [duplicate]
(10 answers)
Closed 5 years ago.
Sometimes checking of arguments in Python is necessary. e.g. I have a function which accepts either the address of other node in the network as the raw string address or class Node which encapsulates the other node's information.
I use type() function as in:
if type(n) == type(Node):
do this
elif type(n) == type(str)
do this
Is this a good way to do this?
Update 1: Python 3 has annotation for function parameters. These can be used for type checks using tool: http://mypy-lang.org/
Use isinstance(). Sample:
if isinstance(n, unicode):
# do this
elif isinstance(n, Node):
# do that
...
>>> isinstance('a', str)
True
>>> isinstance(n, Node)
True
Sounds like you're after a "generic function" - one which behaves differently based on the arguments given. It's a bit like how you'll get a different function when you call a method on a different object, but rather than just using the first argument (the object/self) to lookup the function you instead use all of the arguments.
Turbogears uses something like this for deciding how to convert objects to JSON - if I recall correctly.
There's an article from IBM on using the dispatcher package for this sort of thing:
From that article:
import dispatch
#dispatch.generic()
def doIt(foo, other):
"Base generic function of 'doIt()'"
#doIt.when("isinstance(foo,int) and isinstance(other,str)")
def doIt(foo, other):
print "foo is an unrestricted int |", foo, other
#doIt.when("isinstance(foo,str) and isinstance(other,int)")
def doIt(foo, other):
print "foo is str, other an int |", foo, other
#doIt.when("isinstance(foo,int) and 3<=foo<=17 and isinstance(other,str)")
def doIt(foo, other):
print "foo is between 3 and 17 |", foo, other
#doIt.when("isinstance(foo,int) and 0<=foo<=1000 and isinstance(other,str)")
def doIt(foo, other):
print "foo is between 0 and 1000 |", foo, other
You can also use a try catch to type check if necessary:
def my_function(this_node):
try:
# call a method/attribute for the Node object
if this_node.address:
# more code here
pass
except AttributeError, e:
# either this is not a Node or maybe it's a string,
# so behavior accordingly
pass
You can see an example of this in Beginning Python in the second about generators (page 197 in my edition) and I believe in the Python Cookbook. Many times catching an AttributeError or TypeError is simpler and apparently faster. Also, it may work best in this manner because then you are not tied to a particular inheritance tree (e.g., your object could be a Node or it could be something other object that has the same behavior as a Node).
No, typechecking arguments in Python is not necessary. It is never
necessary.
If your code accepts addresses as rawstring or as a Node object, your
design is broken.
That comes from the fact that if you don't know already the type of an
object in your own program, then you're doing something wrong already.
Typechecking hurts code reuse and reduces performance. Having a function
that performs different things depending on the type of the object passed
is bug-prone and has a behavior harder to understand and maintain.
You have following saner options:
Make a Node object constructor that accepts rawstrings, or a function
that converts strings in Node objects. Make your function assume the
argument passed is a Node object. That way, if you need to pass a
string to the function, you just do:
myfunction(Node(some_string))
That's your best option, it is clean, easy to understand and maintain.
Anyone reading the code immediatelly understands what is happening,
and you don't have to typecheck.
Make two functions, one that accepts Node objects and one that accepts
rawstrings. You can make one call the other internally, in the most
convenient way (myfunction_str can create a Node object and call
myfunction_node, or the other way around).
Make Node objects have a __str__ method and inside your function,
call str() on the received argument. That way you always get a string
by coercion.
In any case, don't typecheck. It is completely unnecessary and has only
downsides. Refactor your code instead in a way you don't need to typecheck.
You only get benefits in doing so, both in short and long run.
Ruby can add methods to the Number class and other core types to get effects like this:
1.should_equal(1)
But it seems like Python cannot do this. Is this true? And if so, why? Does it have something to do with the fact that type can't be modified?
Rather than talking about different definitions of monkey patching, I would like to just focus on the example above. I have already concluded that it cannot be done as a few of you have answered. But I would like a more detailed explanation of why it cannot be done, and maybe what feature, if available in Python, would allow this.
To answer some of you: The reason I might want to do this is simply aesthetics/readability.
item.price.should_equal(19.99)
This reads more like English and clearly indicates which is the tested value and which is the expected value, as supposed to:
should_equal(item.price, 19.99)
This concept is what Rspec and some other Ruby frameworks are based on.
No, you cannot. In Python, all data (classes, methods, functions, etc) defined in C extension modules (including builtins) are immutable. This is because C modules are shared between multiple interpreters in the same process, so monkeypatching them would also affect unrelated interpreters in the same process. (Multiple interpreters in the same process are possible through the C API, and there has been some effort towards making them usable at Python level.)
However, classes defined in Python code may be monkeypatched because they are local to that interpreter.
What exactly do you mean by Monkey Patch here? There are several slightly different definitions.
If you mean, "can you change a class's methods at runtime?", then the answer is emphatically yes:
class Foo:
pass # dummy class
Foo.bar = lambda self: 42
x = Foo()
print x.bar()
If you mean, "can you change a class's methods at runtime and make all of the instances of that class change after-the-fact?" then the answer is yes as well. Just change the order slightly:
class Foo:
pass # dummy class
x = Foo()
Foo.bar = lambda self: 42
print x.bar()
But you can't do this for certain built-in classes, like int or float. These classes' methods are implemented in C and there are certain abstractions sacrificed in order to make the implementation easier and more efficient.
I'm not really clear on why you would want to alter the behavior of the built-in numeric classes anyway. If you need to alter their behavior, subclass them!!
You can do this, but it takes a little bit of hacking. Fortunately, there's a module now called "Forbidden Fruit" that gives you the power to patch methods of built-in types very simply. You can find it at
http://clarete.github.io/forbiddenfruit/?goback=.gde_50788_member_228887816
or
https://pypi.python.org/pypi/forbiddenfruit/0.1.0
With the original question example, after you write the "should_equal" function, you'd just do
from forbiddenfruit import curse
curse(int, "should_equal", should_equal)
and you're good to go! There's also a "reverse" function to remove a patched method.
def should_equal_def(self, value):
if self != value:
raise ValueError, "%r should equal %r" % (self, value)
class MyPatchedInt(int):
should_equal=should_equal_def
class MyPatchedStr(str):
should_equal=should_equal_def
import __builtin__
__builtin__.str = MyPatchedStr
__builtin__.int = MyPatchedInt
int(1).should_equal(1)
str("44").should_equal("44")
Have fun ;)
Python's core types are immutable by design, as other users have pointed out:
>>> int.frobnicate = lambda self: whatever()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't set attributes of built-in/extension type 'int'
You certainly could achieve the effect you describe by making a subclass, since user-defined types in Python are mutable by default.
>>> class MyInt(int):
... def frobnicate(self):
... print 'frobnicating %r' % self
...
>>> five = MyInt(5)
>>> five.frobnicate()
frobnicating 5
>>> five + 8
13
There's no need to make the MyInt subclass public, either; one could just as well define it inline directly in the function or method that constructs the instance.
There are certainly a few situations where Python programmers who are fluent in the idiom consider this sort of subclassing the right thing to do. For instance, os.stat() returns a tuple subclass that adds named members, precisely in order to address the sort of readability concern you refer to in your example.
>>> import os
>>> st = os.stat('.')
>>> st
(16877, 34996226, 65024L, 69, 1000, 1000, 4096, 1223697425, 1223699268, 1223699268)
>>> st[6]
4096
>>> st.st_size
4096
That said, in the specific example you give, I don't believe that subclassing float in item.price (or elsewhere) would be very likely to be considered the Pythonic thing to do. I can easily imagine somebody deciding to add a price_should_equal() method to item if that were the primary use case; if one were looking for something more general, perhaps it might make more sense to use named arguments to make the intended meaning clearer, as in
should_equal(observed=item.price, expected=19.99)
or something along those lines. It's a bit verbose, but no doubt it could be improved upon. A possible advantage to such an approach over Ruby-style monkey-patching is that should_equal() could easily perform its comparison on any type, not just int or float. But perhaps I'm getting too caught up in the details of the particular example that you happened to provide.
You can't patch core types in python.
However, you could use pipe to write a more human readable code:
from pipe import *
#Pipe
def should_equal(obj, val):
if obj==val: return True
return False
class dummy: pass
item=dummy()
item.value=19.99
print item.value | should_equal(19.99)
If you really really really want to do a monkey patch in Python, you can do a (sortof) hack with the "import foo as bar" technique.
If you have a class such as TelnetConnection, and you want to extend it, subclass it in a separate file and call it something like TelnetConnectionExtended.
Then, at the top of your code, where you would normally say:
import TelnetConnection
change that to be:
import TelnetConnectionExtended as TelnetConnection
and then everywhere in your code that you reference TelnetConnection will actually be referencing TelnetConnectionExtended.
Sadly, this assumes that you have access to that class, and the "as" only operates within that particular file (it's not a global-rename), but I've found it to be useful from time to time.
Here's an example of implementing item.price.should_equal, although I'd use Decimal instead of float in a real program:
class Price(float):
def __init__(self, val=None):
float.__init__(self)
if val is not None:
self = val
def should_equal(self, val):
assert self == val, (self, val)
class Item(object):
def __init__(self, name, price=None):
self.name = name
self.price = Price(price)
item = Item("spam", 3.99)
item.price.should_equal(3.99)
No but you have UserDict UserString and UserList which were made with exactly this in mind.
If you google you will find examples for other types, but this are builtin.
In general monkey patching is less used in Python than in Ruby.
What does should_equal do? Is it a boolean returning True or False? In that case, it's spelled:
item.price == 19.99
There's no accounting for taste, but no regular python developer would say that's less readable than your version.
Does should_equal instead set some sort of validator? (why would a validator be limited to one value? Why not just set the value and not update it after that?) If you want a validator, this could never work anyway, since you're proposing to modify either a particular integer or all integers. (A validator that requires 18.99 to equal 19.99 will always fail.) Instead, you could spell it like this:
item.price_should_equal(19.99)
or this:
item.should_equal('price', 19.99)
and define appropriate methods on item's class or superclasses.
It seems what you really wanted to write is:
assert item.price == 19.99
(Of course comparing floats for equality, or using floats for prices, is a bad idea, so you'd write assert item.price == Decimal(19.99) or whatever numeric class you were using for the price.)
You could also use a testing framework like py.test to get more info on failing asserts in your tests.
No, you can't do that in Python. I consider it to be a good thing.
No, sadly you cannot extend types implemented in C at runtime.
You can subclass int, although it is non-trivial, you may have to override __new__.
You also have a syntax issue:
1.somemethod() # invalid
However
(1).__eq__(1) # valid
Here is how I made custom string/int/float...etc. methods:
class MyStrClass(str):
def __init__(self, arg: str):
self.arg_one = arg
def my_str_method(self):
return self.arg_one
def my_str_multiple_arg_method(self, arg_two):
return self.arg_one + arg_two
class MyIntClass(int):
def __init__(self, arg: int):
self.arg_one = arg
def my_int_method(self):
return self.arg_one * 2
myString = MyStrClass("StackOverflow")
myInteger = MyIntClass(15)
print(myString.count("a")) # Output: 1
print(myString.my_str_method()) # Output: StackOverflow
print(myString.my_str_multiple_arg_method(" is cool!")) # Output: StackOverflow is cool!
print(myInteger.my_int_method()) # Output: 30
It's maybe not the best solution, but it works just fine.
Here's how I achieve the .should_something... behavior:
result = calculate_result('blah') # some method defined somewhere else
the(result).should.equal(42)
or
the(result).should_NOT.equal(41)
I included a decorator method for extending this behavior at runtime on a stand-alone method:
#should_expectation
def be_42(self)
self._assert(
action=lambda: self._value == 42,
report=lambda: "'{0}' should equal '5'.".format(self._value)
)
result = 42
the(result).should.be_42()
You have to know a bit about the internals but it works.
Here's the source:
https://github.com/mdwhatcott/pyspecs
It's also on PyPI under pyspecs.