Related
I know that Python does not support method overloading, but I've run into a problem that I can't seem to solve in a nice Pythonic way.
I am making a game where a character needs to shoot a variety of bullets, but how do I write different functions for creating these bullets? For example suppose I have a function that creates a bullet travelling from point A to B with a given speed. I would write a function like this:
def add_bullet(sprite, start, headto, speed):
# Code ...
But I want to write other functions for creating bullets like:
def add_bullet(sprite, start, direction, speed):
def add_bullet(sprite, start, headto, spead, acceleration):
def add_bullet(sprite, script): # For bullets that are controlled by a script
def add_bullet(sprite, curve, speed): # for bullets with curved paths
# And so on ...
And so on with many variations. Is there a better way to do it without using so many keyword arguments cause its getting kinda ugly fast. Renaming each function is pretty bad too because you get either add_bullet1, add_bullet2, or add_bullet_with_really_long_name.
To address some answers:
No I can't create a Bullet class hierarchy because thats too slow. The actual code for managing bullets is in C and my functions are wrappers around C API.
I know about the keyword arguments but checking for all sorts of combinations of parameters is getting annoying, but default arguments help allot like acceleration=0
What you are asking for is called multiple dispatch. See Julia language examples which demonstrates different types of dispatches.
However, before looking at that, we'll first tackle why overloading is not really what you want in Python.
Why Not Overloading?
First, one needs to understand the concept of overloading and why it's not applicable to Python.
When working with languages that can discriminate data types at
compile-time, selecting among the alternatives can occur at
compile-time. The act of creating such alternative functions for
compile-time selection is usually referred to as overloading a
function. (Wikipedia)
Python is a dynamically typed language, so the concept of overloading simply does not apply to it. However, all is not lost, since we can create such alternative functions at run-time:
In programming languages that defer data type identification until
run-time the selection among alternative
functions must occur at run-time, based on the dynamically determined
types of function arguments. Functions whose alternative
implementations are selected in this manner are referred to most
generally as multimethods. (Wikipedia)
So we should be able to do multimethods in Python—or, as it is alternatively called: multiple dispatch.
Multiple dispatch
The multimethods are also called multiple dispatch:
Multiple dispatch or multimethods is the feature of some
object-oriented programming languages in which a function or method
can be dynamically dispatched based on the run time (dynamic) type of
more than one of its arguments. (Wikipedia)
Python does not support this out of the box1, but, as it happens, there is an excellent Python package called multipledispatch that does exactly that.
Solution
Here is how we might use multipledispatch2 package to implement your methods:
>>> from multipledispatch import dispatch
>>> from collections import namedtuple
>>> from types import * # we can test for lambda type, e.g.:
>>> type(lambda a: 1) == LambdaType
True
>>> Sprite = namedtuple('Sprite', ['name'])
>>> Point = namedtuple('Point', ['x', 'y'])
>>> Curve = namedtuple('Curve', ['x', 'y', 'z'])
>>> Vector = namedtuple('Vector', ['x','y','z'])
>>> #dispatch(Sprite, Point, Vector, int)
... def add_bullet(sprite, start, direction, speed):
... print("Called Version 1")
...
>>> #dispatch(Sprite, Point, Point, int, float)
... def add_bullet(sprite, start, headto, speed, acceleration):
... print("Called version 2")
...
>>> #dispatch(Sprite, LambdaType)
... def add_bullet(sprite, script):
... print("Called version 3")
...
>>> #dispatch(Sprite, Curve, int)
... def add_bullet(sprite, curve, speed):
... print("Called version 4")
...
>>> sprite = Sprite('Turtle')
>>> start = Point(1,2)
>>> direction = Vector(1,1,1)
>>> speed = 100 #km/h
>>> acceleration = 5.0 #m/s**2
>>> script = lambda sprite: sprite.x * 2
>>> curve = Curve(3, 1, 4)
>>> headto = Point(100, 100) # somewhere far away
>>> add_bullet(sprite, start, direction, speed)
Called Version 1
>>> add_bullet(sprite, start, headto, speed, acceleration)
Called version 2
>>> add_bullet(sprite, script)
Called version 3
>>> add_bullet(sprite, curve, speed)
Called version 4
1. Python 3 currently supports single dispatch
2. Take care not to use multipledispatch in a multi-threaded environment or you will get weird behavior.
Python does support "method overloading" as you present it. In fact, what you just describe is trivial to implement in Python, in so many different ways, but I would go with:
class Character(object):
# your character __init__ and other methods go here
def add_bullet(self, sprite=default, start=default,
direction=default, speed=default, accel=default,
curve=default):
# do stuff with your arguments
In the above code, default is a plausible default value for those arguments, or None. You can then call the method with only the arguments you are interested in, and Python will use the default values.
You could also do something like this:
class Character(object):
# your character __init__ and other methods go here
def add_bullet(self, **kwargs):
# here you can unpack kwargs as (key, values) and
# do stuff with them, and use some global dictionary
# to provide default values and ensure that ``key``
# is a valid argument...
# do stuff with your arguments
Another alternative is to directly hook the desired function directly to the class or instance:
def some_implementation(self, arg1, arg2, arg3):
# implementation
my_class.add_bullet = some_implementation_of_add_bullet
Yet another way is to use an abstract factory pattern:
class Character(object):
def __init__(self, bfactory, *args, **kwargs):
self.bfactory = bfactory
def add_bullet(self):
sprite = self.bfactory.sprite()
speed = self.bfactory.speed()
# do stuff with your sprite and speed
class pretty_and_fast_factory(object):
def sprite(self):
return pretty_sprite
def speed(self):
return 10000000000.0
my_character = Character(pretty_and_fast_factory(), a1, a2, kw1=v1, kw2=v2)
my_character.add_bullet() # uses pretty_and_fast_factory
# now, if you have another factory called "ugly_and_slow_factory"
# you can change it at runtime in python by issuing
my_character.bfactory = ugly_and_slow_factory()
# In the last example you can see abstract factory and "method
# overloading" (as you call it) in action
You can use "roll-your-own" solution for function overloading. This one is copied from Guido van Rossum's article about multimethods (because there is little difference between multimethods and overloading in Python):
registry = {}
class MultiMethod(object):
def __init__(self, name):
self.name = name
self.typemap = {}
def __call__(self, *args):
types = tuple(arg.__class__ for arg in args) # a generator expression!
function = self.typemap.get(types)
if function is None:
raise TypeError("no match")
return function(*args)
def register(self, types, function):
if types in self.typemap:
raise TypeError("duplicate registration")
self.typemap[types] = function
def multimethod(*types):
def register(function):
name = function.__name__
mm = registry.get(name)
if mm is None:
mm = registry[name] = MultiMethod(name)
mm.register(types, function)
return mm
return register
The usage would be
from multimethods import multimethod
import unittest
# 'overload' makes more sense in this case
overload = multimethod
class Sprite(object):
pass
class Point(object):
pass
class Curve(object):
pass
#overload(Sprite, Point, Direction, int)
def add_bullet(sprite, start, direction, speed):
# ...
#overload(Sprite, Point, Point, int, int)
def add_bullet(sprite, start, headto, speed, acceleration):
# ...
#overload(Sprite, str)
def add_bullet(sprite, script):
# ...
#overload(Sprite, Curve, speed)
def add_bullet(sprite, curve, speed):
# ...
Most restrictive limitations at the moment are:
methods are not supported, only functions that are not class members;
inheritance is not handled;
kwargs are not supported;
registering new functions should be done at import time thing is not thread-safe
A possible option is to use the multipledispatch module as detailed here:
http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch
Instead of doing this:
def add(self, other):
if isinstance(other, Foo):
...
elif isinstance(other, Bar):
...
else:
raise NotImplementedError()
You can do this:
from multipledispatch import dispatch
#dispatch(int, int)
def add(x, y):
return x + y
#dispatch(object, object)
def add(x, y):
return "%s + %s" % (x, y)
With the resulting usage:
>>> add(1, 2)
3
>>> add(1, 'hello')
'1 + hello'
In Python 3.4 PEP-0443. Single-dispatch generic functions was added.
Here is a short API description from PEP.
To define a generic function, decorate it with the #singledispatch decorator. Note that the dispatch happens on the type of the first argument. Create your function accordingly:
from functools import singledispatch
#singledispatch
def fun(arg, verbose=False):
if verbose:
print("Let me just say,", end=" ")
print(arg)
To add overloaded implementations to the function, use the register() attribute of the generic function. This is a decorator, taking a type parameter and decorating a function implementing the operation for that type:
#fun.register(int)
def _(arg, verbose=False):
if verbose:
print("Strength in numbers, eh?", end=" ")
print(arg)
#fun.register(list)
def _(arg, verbose=False):
if verbose:
print("Enumerate this:")
for i, elem in enumerate(arg):
print(i, elem)
The #overload decorator was added with type hints (PEP 484).
While this doesn't change the behaviour of Python, it does make it easier to understand what is going on, and for mypy to detect errors.
See: Type hints and PEP 484
This type of behaviour is typically solved (in OOP languages) using polymorphism. Each type of bullet would be responsible for knowing how it travels. For instance:
class Bullet(object):
def __init__(self):
self.curve = None
self.speed = None
self.acceleration = None
self.sprite_image = None
class RegularBullet(Bullet):
def __init__(self):
super(RegularBullet, self).__init__()
self.speed = 10
class Grenade(Bullet):
def __init__(self):
super(Grenade, self).__init__()
self.speed = 4
self.curve = 3.5
add_bullet(Grendade())
def add_bullet(bullet):
c_function(bullet.speed, bullet.curve, bullet.acceleration, bullet.sprite, bullet.x, bullet.y)
void c_function(double speed, double curve, double accel, char[] sprite, ...) {
if (speed != null && ...) regular_bullet(...)
else if (...) curved_bullet(...)
//..etc..
}
Pass as many arguments to the c_function that exist, and then do the job of determining which c function to call based on the values in the initial c function. So, Python should only ever be calling the one c function. That one c function looks at the arguments, and then can delegate to other c functions appropriately.
You're essentially just using each subclass as a different data container, but by defining all the potential arguments on the base class, the subclasses are free to ignore the ones they do nothing with.
When a new type of bullet comes along, you can simply define one more property on the base, change the one python function so that it passes the extra property, and the one c_function that examines the arguments and delegates appropriately. It doesn't sound too bad I guess.
It is impossible by definition to overload a function in python (read on for details), but you can achieve something similar with a simple decorator
class overload:
def __init__(self, f):
self.cases = {}
def args(self, *args):
def store_function(f):
self.cases[tuple(args)] = f
return self
return store_function
def __call__(self, *args):
function = self.cases[tuple(type(arg) for arg in args)]
return function(*args)
You can use it like this
#overload
def f():
pass
#f.args(int, int)
def f(x, y):
print('two integers')
#f.args(float)
def f(x):
print('one float')
f(5.5)
f(1, 2)
Modify it to adapt it to your use case.
A clarification of concepts
function dispatch: there are multiple functions with the same name. Which one should be called? two strategies
static/compile-time dispatch (aka. "overloading"). decide which function to call based on the compile-time type of the arguments. In all dynamic languages, there is no compile-time type, so overloading is impossible by definition
dynamic/run-time dispatch: decide which function to call based on the runtime type of the arguments. This is what all OOP languages do: multiple classes have the same methods, and the language decides which one to call based on the type of self/this argument. However, most languages only do it for the this argument only. The above decorator extends the idea to multiple parameters.
To clear up, assume that we define, in a hypothetical static language, the functions
void f(Integer x):
print('integer called')
void f(Float x):
print('float called')
void f(Number x):
print('number called')
Number x = new Integer('5')
f(x)
x = new Number('3.14')
f(x)
With static dispatch (overloading) you will see "number called" twice, because x has been declared as Number, and that's all overloading cares about. With dynamic dispatch you will see "integer called, float called", because those are the actual types of x at the time the function is called.
By passing keyword args.
def add_bullet(**kwargs):
#check for the arguments listed above and do the proper things
Python 3.8 added functools.singledispatchmethod
Transform a method into a single-dispatch generic function.
To define a generic method, decorate it with the #singledispatchmethod
decorator. Note that the dispatch happens on the type of the first
non-self or non-cls argument, create your function accordingly:
from functools import singledispatchmethod
class Negator:
#singledispatchmethod
def neg(self, arg):
raise NotImplementedError("Cannot negate a")
#neg.register
def _(self, arg: int):
return -arg
#neg.register
def _(self, arg: bool):
return not arg
negator = Negator()
for v in [42, True, "Overloading"]:
neg = negator.neg(v)
print(f"{v=}, {neg=}")
Output
v=42, neg=-42
v=True, neg=False
NotImplementedError: Cannot negate a
#singledispatchmethod supports nesting with other decorators such as
#classmethod. Note that to allow for dispatcher.register,
singledispatchmethod must be the outer most decorator. Here is the
Negator class with the neg methods being class bound:
from functools import singledispatchmethod
class Negator:
#singledispatchmethod
#staticmethod
def neg(arg):
raise NotImplementedError("Cannot negate a")
#neg.register
def _(arg: int) -> int:
return -arg
#neg.register
def _(arg: bool) -> bool:
return not arg
for v in [42, True, "Overloading"]:
neg = Negator.neg(v)
print(f"{v=}, {neg=}")
Output:
v=42, neg=-42
v=True, neg=False
NotImplementedError: Cannot negate a
The same pattern can be used for other similar decorators:
staticmethod, abstractmethod, and others.
I think your basic requirement is to have a C/C++-like syntax in Python with the least headache possible. Although I liked Alexander Poluektov's answer it doesn't work for classes.
The following should work for classes. It works by distinguishing by the number of non-keyword arguments (but it doesn't support distinguishing by type):
class TestOverloading(object):
def overloaded_function(self, *args, **kwargs):
# Call the function that has the same number of non-keyword arguments.
getattr(self, "_overloaded_function_impl_" + str(len(args)))(*args, **kwargs)
def _overloaded_function_impl_3(self, sprite, start, direction, **kwargs):
print "This is overload 3"
print "Sprite: %s" % str(sprite)
print "Start: %s" % str(start)
print "Direction: %s" % str(direction)
def _overloaded_function_impl_2(self, sprite, script):
print "This is overload 2"
print "Sprite: %s" % str(sprite)
print "Script: "
print script
And it can be used simply like this:
test = TestOverloading()
test.overloaded_function("I'm a Sprite", 0, "Right")
print
test.overloaded_function("I'm another Sprite", "while x == True: print 'hi'")
Output:
This is overload 3
Sprite: I'm a Sprite
Start: 0
Direction: Right
This is overload 2
Sprite: I'm another Sprite
Script:
while x == True: print 'hi'
You can achieve this with the following Python code:
#overload
def test(message: str):
return message
#overload
def test(number: int):
return number + 1
Either use multiple keyword arguments in the definition, or create a Bullet hierarchy whose instances are passed to the function.
I think a Bullet class hierarchy with the associated polymorphism is the way to go. You can effectively overload the base class constructor by using a metaclass so that calling the base class results in the creation of the appropriate subclass object. Below is some sample code to illustrate the essence of what I mean.
Updated
The code has been modified to run under both Python 2 and 3 to keep it relevant. This was done in a way that avoids the use Python's explicit metaclass syntax, which varies between the two versions.
To accomplish that objective, a BulletMetaBase instance of the BulletMeta class is created by explicitly calling the metaclass when creating the Bullet baseclass (rather than using the __metaclass__= class attribute or via a metaclass keyword argument depending on the Python version).
class BulletMeta(type):
def __new__(cls, classname, bases, classdict):
""" Create Bullet class or a subclass of it. """
classobj = type.__new__(cls, classname, bases, classdict)
if classname != 'BulletMetaBase':
if classname == 'Bullet': # Base class definition?
classobj.registry = {} # Initialize subclass registry.
else:
try:
alias = classdict['alias']
except KeyError:
raise TypeError("Bullet subclass %s has no 'alias'" %
classname)
if alias in Bullet.registry: # unique?
raise TypeError("Bullet subclass %s's alias attribute "
"%r already in use" % (classname, alias))
# Register subclass under the specified alias.
classobj.registry[alias] = classobj
return classobj
def __call__(cls, alias, *args, **kwargs):
""" Bullet subclasses instance factory.
Subclasses should only be instantiated by calls to the base
class with their subclass' alias as the first arg.
"""
if cls != Bullet:
raise TypeError("Bullet subclass %r objects should not to "
"be explicitly constructed." % cls.__name__)
elif alias not in cls.registry: # Bullet subclass?
raise NotImplementedError("Unknown Bullet subclass %r" %
str(alias))
# Create designated subclass object (call its __init__ method).
subclass = cls.registry[alias]
return type.__call__(subclass, *args, **kwargs)
class Bullet(BulletMeta('BulletMetaBase', (object,), {})):
# Presumably you'd define some abstract methods that all here
# that would be supported by all subclasses.
# These definitions could just raise NotImplementedError() or
# implement the functionality is some sub-optimal generic way.
# For example:
def fire(self, *args, **kwargs):
raise NotImplementedError(self.__class__.__name__ + ".fire() method")
# Abstract base class's __init__ should never be called.
# If subclasses need to call super class's __init__() for some
# reason then it would need to be implemented.
def __init__(self, *args, **kwargs):
raise NotImplementedError("Bullet is an abstract base class")
# Subclass definitions.
class Bullet1(Bullet):
alias = 'B1'
def __init__(self, sprite, start, direction, speed):
print('creating %s object' % self.__class__.__name__)
def fire(self, trajectory):
print('Bullet1 object fired with %s trajectory' % trajectory)
class Bullet2(Bullet):
alias = 'B2'
def __init__(self, sprite, start, headto, spead, acceleration):
print('creating %s object' % self.__class__.__name__)
class Bullet3(Bullet):
alias = 'B3'
def __init__(self, sprite, script): # script controlled bullets
print('creating %s object' % self.__class__.__name__)
class Bullet4(Bullet):
alias = 'B4'
def __init__(self, sprite, curve, speed): # for bullets with curved paths
print('creating %s object' % self.__class__.__name__)
class Sprite: pass
class Curve: pass
b1 = Bullet('B1', Sprite(), (10,20,30), 90, 600)
b2 = Bullet('B2', Sprite(), (-30,17,94), (1,-1,-1), 600, 10)
b3 = Bullet('B3', Sprite(), 'bullet42.script')
b4 = Bullet('B4', Sprite(), Curve(), 720)
b1.fire('uniform gravity')
b2.fire('uniform gravity')
Output:
creating Bullet1 object
creating Bullet2 object
creating Bullet3 object
creating Bullet4 object
Bullet1 object fired with uniform gravity trajectory
Traceback (most recent call last):
File "python-function-overloading.py", line 93, in <module>
b2.fire('uniform gravity') # NotImplementedError: Bullet2.fire() method
File "python-function-overloading.py", line 49, in fire
raise NotImplementedError(self.__class__.__name__ + ".fire() method")
NotImplementedError: Bullet2.fire() method
You can easily implement function overloading in Python. Here is an example using floats and integers:
class OverloadedFunction:
def __init__(self):
self.router = {int : self.f_int ,
float: self.f_float}
def __call__(self, x):
return self.router[type(x)](x)
def f_int(self, x):
print('Integer Function')
return x**2
def f_float(self, x):
print('Float Function (Overloaded)')
return x**3
# f is our overloaded function
f = OverloadedFunction()
print(f(3 ))
print(f(3.))
# Output:
# Integer Function
# 9
# Float Function (Overloaded)
# 27.0
The main idea behind the code is that a class holds the different (overloaded) functions that you would like to implement, and a Dictionary works as a router, directing your code towards the right function depending on the input type(x).
PS1. In case of custom classes, like Bullet1, you can initialize the internal dictionary following a similar pattern, such as self.D = {Bullet1: self.f_Bullet1, ...}. The rest of the code is the same.
PS2. The time/space complexity of the proposed solution is fairly good as well, with an average cost of O(1) per operation.
Use keyword arguments with defaults. E.g.
def add_bullet(sprite, start=default, direction=default, script=default, speed=default):
In the case of a straight bullet versus a curved bullet, I'd add two functions: add_bullet_straight and add_bullet_curved.
Overloading methods is tricky in Python. However, there could be usage of passing the dict, list or primitive variables.
I have tried something for my use cases, and this could help here to understand people to overload the methods.
Let's take your example:
A class overload method with call the methods from different class.
def add_bullet(sprite=None, start=None, headto=None, spead=None, acceleration=None):
Pass the arguments from the remote class:
add_bullet(sprite = 'test', start=Yes,headto={'lat':10.6666,'long':10.6666},accelaration=10.6}
Or
add_bullet(sprite = 'test', start=Yes, headto={'lat':10.6666,'long':10.6666},speed=['10','20,'30']}
So, handling is being achieved for list, Dictionary or primitive variables from method overloading.
Try it out for your code.
Plum supports it in a straightforward pythonic way. Copying an example from the README below.
from plum import dispatch
#dispatch
def f(x: str):
return "This is a string!"
#dispatch
def f(x: int):
return "This is an integer!"
>>> f("1")
'This is a string!'
>>> f(1)
'This is an integer!'
You can also try this code. We can try any number of arguments
# Finding the average of given number of arguments
def avg(*args): # args is the argument name we give
sum = 0
for i in args:
sum += i
average = sum/len(args) # Will find length of arguments we given
print("Avg: ", average)
# call function with different number of arguments
avg(1,2)
avg(5,6,4,7)
avg(11,23,54,111,76)
I want to give user API for my library with easier way to distinguish different types of parameters which I pass to function. All groups of arguments are defined earlier (for now I have 3 groups), but attributes of them need to be constructed on run. I can do this in Django ORM style, where double underscore separates 2 parts of parameter. But it is very unreadable. Example:
def api_function(**kwargs):
""" Separate passed arguments """
api_function(post__arg1='foo', api__arg1='bar', post_arg2='foo2')
Better way do this SQLAlchemy, but only to compare attributes and all args are defined earlier. Example:
class API(object):
arg1 = Arg()
arg2 = Arg()
class Post(object): #...
def api_function(*args):
""" Separate passed arguments """
api_function(POST.arg1=='foo', API.arg1=='bar', POST.arg2=='foo2')
What I would like to achive is behaviour like this:
class API(object): # Magic
class POST(object): # Magic
def api_function(*args):
""" Separate passed arguments """
api_function(POST.arg1='foo', API.arg1='bar', POST.arg2='foo2')
What have I tried:
declare metamodel with defined __setattr__, but it rise on evaluation SyntaxError: keyword can't be an expression
declare __set__, but it is designed for known attributes
My questions are:
Is it even possible in Python to work like in third snippet?
If not, is there any really close solution to look like in third snippet? The best way should use assignment operator API.arg1='foo', the worst API(arg1='foo')
Requirements -- should work at least at Python 2.7. Good to work on Python 3.2.
EDIT1
My first test, which is using equality operator (but it NEVER should be use in this way):
class APIMeta(type):
def __getattr__(cls, item):
return ApiData(item, None)
class API(object):
__metaclass__ = APIMeta
def __init__(self, key, value):
self.key = key
self.value = value
def __str__(self):
return "{0}={1}".format(self.key, self.value)
def __eq__(self, other):
self.value = other
return self
def print_api(*api_data):
for a in api_data:
print(str(a))
print_api(API.page=='3', API=='bar')
It is working right, but using == is suggesting that I want to compare something and I want to assign value.
NOTE: I don't know how much I like this schema you want. But I know one annoying thing will be all the imports to call api_function. E.G. from api import POST, API, api_function
As I said in the comments, the first way is not possible. This is because assignment (=) is a statement not an expression, so it can't return a value. Source
But the other way you asked for certainly is:
class POST(object):
def __init__(self, **kwargs):
self.args = kwargs
# You'll also probably want to make this function a little safer.
def __getattr__(self, name):
return self.args[name]
def api_function(*args):
# Update this to how complicated the handling needs to be
# but you get the general idea...
post_data = None
for a in args:
if isinstance(a, POST):
post_data = a.args
if post_data is None:
raise Exception('This function needs a POST object passed.')
print post_data
Using it:
>>> api_function('foo')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in api_function
Exception: This function needs a POST object passed.
>>> api_function(POST(arg1='foo'))
{'arg1': 'foo'}
>>> api_function(POST(arg1='foo',
... arg2='bar'
... )
... )
{'arg1': 'foo', 'arg2': 'bar'}
Here's my solution. It's not the best in design, as the structure of the argument grouper is nested quite deep, so I'd appreciate feedback on it:
class ArgumentGrouper(object):
"""Transforms a function so that you can apply arguments in named groups.
This system isn't tested as thoroughly as something with so many moving
parts should be. Use at own risk.
Usage:
#ArgumentGrouper("foo", "bar")
def method(regular_arg, foo__arg1, bar__arg2):
print(regular_arg + foo__arg1 + bar__arg2)
method.foo(", ").bar("world!")("Hello")() # Prints "Hello, world!"
"""
def __call__(self, func):
"""Decorate the function."""
return self.Wrapper(func, self.argument_values)
def __init__(self, *argument_groups):
"""Constructor.
argument_groups -- The names of argument groups in the function.
"""
self.argument_values = {i: {} for i in argument_groups}
class Wrapper(object):
"""This is the result of decorating the function. You can call group
names as function to supply their keyword arguments.
"""
def __call__(self, *args):
"""Execute the decorated function by passing any given arguments
and predefined group arguments.
"""
kwargs = {}
for group, values in self.argument_values.items():
for name, value in values.items():
# Add a new argument in the form foo__arg1 to kwargs, as
# per the supplied arguments.
new_name = "{}__{}".format(
group,
name
)
kwargs[new_name] = value
# Invoke the function with the determined arguments.
return self.func(*args, **kwargs)
def __init__(self, func, argument_values):
"""Constructor.
func -- The decorated function.
argument_values -- A dict with the current values for group
arguments. Must be a reference to the actual dict, since each
WrappedMethod uses it.
"""
self.func = func
self.argument_values = argument_values
def __getattr__(self, name):
"""When trying to call `func.foo(arg1="bar")`, provide `foo`. TODO:
This would be better handled at initialization time.
"""
if name in self.argument_values:
return self.WrappedMethod(name, self, self.argument_values)
else:
return self.__dict__[name]
class WrappedMethod(object):
"""For `func.foo(arg1="bar")`, this is `foo`. Pretends to be a
function that takes the keyword arguments to be supplied to the
decorated function.
"""
def __call__(self, **kwargs):
"""`foo` has been called, record the arguments passed."""
for k, v in kwargs.items():
self.argument_values[self.name][k] = v
return self.wrapper
def __init__(self, name, wrapper, argument_values):
"""Constructor.
name -- The name of the argument group. (This is the string
"foo".)
wrapper -- The decorator. We need this so that we can return it
to chain calls.
argument_values -- A dict with the current values for group
arguments. Must be a reference to the actual dict, since
each WrappedMethod uses it.
"""
self.name = name
self.wrapper = wrapper
self.argument_values = argument_values
# Usage:
#ArgumentGrouper("post", "api")
def api_function(regular_arg, post__arg1, post__arg2, api__arg3):
print("Got regular args {}".format(regular_arg))
print("Got API args {}, {}, {}".format(post__arg1, post__arg2, api__arg3))
api_function.post(
arg1="foo", arg2="bar"
).api(
arg3="baz"
)
api_function("foo")
Then, usage:
#ArgumentGrouper("post", "api")
def api_function(regular_arg, post__arg1, post__arg2, api__arg3):
print("Got regular args {}".format(regular_arg))
print("Got API args {}, {}, {}".format(post__arg1, post__arg2, api__arg3))
api_function.post(
arg1="foo", arg2="bar"
).api(
arg3="baz"
)
api_function("foo")
Output:
Got regular args foo
Got API args foo, bar, baz
It should be simple to scrape argument group names by introspection.
You'll notice the argument naming convention is hardcoded into the WrappedMethod, so you'll have to make sure you're okay with that.
You can also invoke it in one statement:
api_function.post(
arg1="foo", arg2="bar"
).api(
arg3="baz"
)("foo")
Or you could add a dedicated run method which would invoke it, which would just take the place of Wrapper.__call__.
Python don't allow to use assignment operator inside any other code, so:
(a=1)
func((a=1))
will rise SyntaxError. This means that it is not possible to use it in this way. Moreover:
func(API.arg1=3)
Will be treated that left side of assignment is argument API.arg1 which is not valid name in Python for variables. Only solution is to make this in SQLAlchemy style:
func({
API.arg1: 'foo',
API.arg2: 'bar',
DATA.arg1: 'foo1',
})
or
func(**{
API.arg1: 'foo',
API.arg2: 'bar',
DATA.arg1: 'foo1',
})
or just only:
func( API(arg1='foo', arg2='bar'), POST(arg1='foo1'), POST(arg2='bar1'))
Thank you for your interest and answers.
I'm developing a documentation testing framework -- basically unit tests for PDFs. Tests are (decorated) methods of instances of classes defined by the framework, and these are located and instantiated at runtime and the methods are invoked to execute the tests.
My goal is to cut down on the amount of quirky Python syntax that the people who will write tests need to be concerned about, as these people may or may not be Python programmers, or even very much programmers at all. So I would like them to be able to write "def foo():" instead of "def foo(self):" for methods, but still be able to use "self" to access members.
In an ordinary program I would consider this a horrible idea, but in a domain-specific-languagey kind of program like this one, it seems worth a try.
I have successfully eliminated the self from the method signature by using a decorator (actually, since I am using a decorator already for the test cases, I would just roll it into that), but "self" does not then refer to anything in the test case method.
I have considered using a global for self, and even come up with an implementation that more or less works, but I'd rather pollute the smallest namespace possible, which is why I would prefer to inject the variable directly into the test case method's local namespace. Any thoughts?
My accepted answer to this question was pretty dumb but I was just starting out. Here's a much better way. This is only scantily tested but it's good for a demonstration of the proper way to do this thing which is improper to do. It works on 2.6.5 for sure. I haven't tested any other versions but no opcodes are hardcoded into it so it should be about as portable as most other 2.x code.
add_self can be applied as a decorator but that would defeat the purpose (why not just type 'self'?) It would be easy to adapt the metaclass from my other answer to apply this function instead.
import opcode
import types
def instructions(code):
"""Iterates over a code string yielding integer [op, arg] pairs
If the opcode does not take an argument, just put None in the second part
"""
code = map(ord, code)
i, L = 0, len(code)
extended_arg = 0
while i < L:
op = code[i]
i+= 1
if op < opcode.HAVE_ARGUMENT:
yield [op, None]
continue
oparg = code[i] + (code[i+1] << 8) + extended_arg
extended_arg = 0
i += 2
if op == opcode.EXTENDED_ARG:
extended_arg = oparg << 16
continue
yield [op, oparg]
def write_instruction(inst):
"""Takes an integer [op, arg] pair and returns a list of character bytecodes"""
op, oparg = inst
if oparg is None:
return [chr(op)]
elif oparg <= 65536L:
return [chr(op), chr(oparg & 255), chr((oparg >> 8) & 255)]
elif oparg <= 4294967296L:
# The argument is large enough to need 4 bytes and the EXTENDED_ARG opcode
return [chr(opcode.EXTENDED_ARG),
chr((oparg >> 16) & 255),
chr((oparg >> 24) & 255),
chr(op),
chr(oparg & 255),
chr((oparg >> 8) & 255)]
else:
raise ValueError("Invalid oparg: {0} is too large".format(oparg))
def add_self(f):
"""Add self to a method
Creates a new function by prepending the name 'self' to co_varnames, and
incrementing co_argcount and co_nlocals. Increase the index of all other locals
by 1 to compensate. Also removes 'self' from co_names and decrease the index of
all names that occur after it by 1. Finally, replace all occurrences of
`LOAD_GLOBAL i,j` that make reference to the old 'self' with 'LOAD_FAST 0,0'.
Essentially, just create a code object that is exactly the same but has one more
argument.
"""
code_obj = f.func_code
try:
self_index = code_obj.co_names.index('self')
except ValueError:
raise NotImplementedError("self is not a global")
# The arguments are just the first co_argcount co_varnames
varnames = ('self', ) + code_obj.co_varnames
names = tuple(name for name in code_obj.co_names if name != 'self')
code = []
for inst in instructions(code_obj.co_code):
op = inst[0]
if op in opcode.haslocal:
# The index is now one greater because we added 'self' at the head of
# the tuple
inst[1] += 1
elif op in opcode.hasname:
arg = inst[1]
if arg == self_index:
# This refers to the old global 'self'
if op == opcode.opmap['LOAD_GLOBAL']:
inst[0] = opcode.opmap['LOAD_FAST']
inst[1] = 0
else:
# If `self` is used as an attribute, real global, module
# name, module attribute, or gets looked at funny, bail out.
raise NotImplementedError("Abnormal use of self")
elif arg > self_index:
# This rewrites the index to account for the old global 'self'
# having been removed.
inst[1] -= 1
code += write_instruction(inst)
code = ''.join(code)
# type help(types.CodeType) at the interpreter prompt for this one
new_code_obj = types.CodeType(code_obj.co_argcount + 1,
code_obj.co_nlocals + 1,
code_obj.co_stacksize,
code_obj.co_flags,
code,
code_obj.co_consts,
names,
varnames,
'<OpcodeCity>',
code_obj.co_name,
code_obj.co_firstlineno,
code_obj.co_lnotab,
code_obj.co_freevars,
code_obj.co_cellvars)
# help(types.FunctionType)
return types.FunctionType(new_code_obj, f.func_globals)
class Test(object):
msg = 'Foo'
#add_self
def show(msg):
print self.msg + msg
t = Test()
t.show('Bar')
little upgrade for aaronasterling's solution( i haven't enough reputation to comment it ):
def wrap(f):
#functools.wraps(f)
def wrapper(self,*arg,**kw):
f.func_globals['self'] = self
return f(*arg,**kw)
return wrapper
but both this solutions will work unpredictable if f function will be called recursively for different instance, so you have to clone it like this:
import types
class wrap(object):
def __init__(self,func):
self.func = func
def __get__(self,obj,type):
new_globals = self.func.func_globals.copy()
new_globals['self'] = obj
return types.FunctionType(self.func.func_code,new_globals)
class C(object):
def __init__(self,word):
self.greeting = word
#wrap
def greet(name):
print(self.greeting+' , ' + name+ '!')
C('Hello').greet('kindall')
Here's a one line method decorator that seems to do the job without modifying any Special attributes of Callable types* marked Read-only:
# method decorator -- makes undeclared 'self' argument available to method
injectself = lambda f: lambda self: eval(f.func_code, dict(self=self))
class TestClass:
def __init__(self, thing):
self.attr = thing
#injectself
def method():
print 'in TestClass::method(): self.attr = %r' % self.attr
return 42
test = TestClass("attribute's value")
ret = test.method()
print 'return value:', ret
# output:
# in TestClass::method(): self.attr = "attribute's value"
# return value: 42
Note that unless you take precautions to prevent it, a side-effect of the eval() function may be it adding a few entries -- such as a reference to the __builtin__ module under the key __builtins__ -- automatically to the dict passed to it.
#kendall: Per your comment about how you're using this with methods being in container classes (but ignoring the injection of additional variables for the moment) -- is the following something like what you're doing? It's difficult for me to understand how things are split up between the framework and what the users write. It sounds like an interesting design pattern to me.
# method decorator -- makes undeclared 'self' argument available to method
injectself = lambda f: lambda self: eval(f.func_code, dict(self=self))
class methodclass:
def __call__():
print 'in methodclass::__call__(): self.attr = %r' % self.attr
return 42
class TestClass:
def __init__(self, thing):
self.attr = thing
method = injectself(methodclass.__call__)
test = TestClass("attribute's value")
ret = test.method()
print 'return value:', ret
# output
# in methodclass::__call__(): self.attr = "attribute's value"
# return value: 42
The trick is to add 'self' to f.func_globals. This works in python2.6. I really should get around to installing other versions to test stuff like this on. Sorry for the wall of code but I cover two cases: doing it with a metaclass and doing it with a decorator. For your usecase, I think the metaclass is better since the whole point of this exercise is to shield users from syntax.
import new, functools
class TestMeta(type):
def __new__(meta, classname, bases, classdict):
for item in classdict:
if hasattr(classdict[item], '__call__'):
classdict[item] = wrap(classdict[item])
return type.__new__(meta, classname, bases, classdict)
def wrap(f):
#functools.wraps(f)
def wrapper(self):
f.func_globals['self'] = self
return f()
return wrapper
def testdec(f):
#functools.wraps(f)
def wrapper():
return f()
return wrapper
class Test(object):
__metaclass__ = TestMeta
message = 'You can do anything in python'
def test():
print self.message
#testdec
def test2():
print self.message + ' but the wrapper funcion can\'t take a self argument either or you get a TypeError'
class Test2(object):
message = 'It also works as a decorator but (to me at least) feels better as a metaclass'
#wrap
def test():
print self.message
t = Test()
t2 = Test2()
t.test()
t.test2()
t2.test()
This might be a use case for decorators - you give them a small set of lego bricks to build functions with, and the complicated framework stuff is piped in via #testcase or somesuch.
Edit: You didn't post any code, so this is going to be sketchy, but they don't need to write methods. They can write ordinary functions without "self", and you could use decorators like in this example from the article I linked:
class myDecorator(object):
def __init__(self, f):
print "inside myDecorator.__init__()"
f() # Prove that function definition has completed
def __call__(self):
print "inside myDecorator.__call__()"
#myDecorator
def aFunction():
print "inside aFunction()"
I would like to create a class that effectively does this (mixing a little PHP with Python)
class Middle(object) :
# self.apply is a function that applies a function to a list
# e.g self.apply = [] ... self.apply.append(foobar)
def __call(self, name, *args) :
self.apply(name, *args)
Thus allowing for code to say:
m = Middle()
m.process_foo(a, b, c)
In this case __call() is the PHP __call() method which is invoked when a method is not found on an object.
You need to define __getattr__, it is called if an attribute is not otherwise found on your object.
Notice that getattr is called for any failed lookup, and that you don't get it like a function all, so you have to return the method that will be called.
def __getattr__(self, attr):
def default_method(*args):
self.apply(attr, *args)
return default_method
Consider passing arguments to your methods as arguments, not encoded into the method name which will then be magically used as an argument.
Where are you writing code that doesn't know what methods it will be calling?
Why call c.do_Something(x) and then unpack the method name instead of just calling c.do('Something', x) ?
In any case it's easy enough to handle unfound attributes:
class Dispatcher(object):
def __getattr__(self, key):
try:
return object.__getattr__(self, key)
except AttributeError:
return self.dispatch(key)
def default(self, *args, **kw):
print "Assuming default method"
print args, kw
def dispatch(self, key):
print 'Looking for method: %s'%(key,)
return self.default
A test:
>>> d = Dispatcher()
>>> d.hello()
Looking for method: hello
Assuming default method
() {}
This seems to be fraught with "gotchas" - the thing returned by getattr is going to be presumed to be not just a function, but a bound method on that instance. So be sure to return that.
I actually did this recently. Here's an example of how I solved it:
class Example:
def FUNC_1(self, arg):
return arg - 1
def FUNC_2(self, arg):
return arg - 2
def decode(self, func, arg):
try:
exec( "result = self.FUNC_%s(arg)" % (func) )
except AttributeError:
# Call your default method here
result = self.default(arg)
return result
def default(self, arg):
return arg
and the output:
>>> dude = Example()
>>> print dude.decode(1, 0)
-1
>>> print dude.decode(2, 10)
8
>>> print dude.decode(3, 5)
5
I'm dreaming of a Python method with explicit keyword args:
def func(a=None, b=None, c=None):
for arg, val in magic_arg_dict.items(): # Where do I get the magic?
print '%s: %s' % (arg, val)
I want to get a dictionary of only those arguments the caller actually passed into the method, just like **kwargs, but I don't want the caller to be able to pass any old random args, unlike **kwargs.
>>> func(b=2)
b: 2
>>> func(a=3, c=5)
a: 3
c: 5
So: is there such an incantation? In my case, I happen to be able to compare each argument against its default to find the ones that are different, but this is kind of inelegant and gets tedious when you have nine arguments. For bonus points, provide an incantation that can tell me even when the caller passes in a keyword argument assigned its default value:
>>> func(a=None)
a: None
Tricksy!
Edit: The (lexical) function signature has to remain intact. It's part of a public API, and the primary worth of the explicit keyword args lies in their documentary value. Just to make things interesting. :)
I was inspired by lost-theory's decorator goodness, and after playing about with it for a bit came up with this:
def actual_kwargs():
"""
Decorator that provides the wrapped function with an attribute 'actual_kwargs'
containing just those keyword arguments actually passed in to the function.
"""
def decorator(function):
def inner(*args, **kwargs):
inner.actual_kwargs = kwargs
return function(*args, **kwargs)
return inner
return decorator
if __name__ == "__main__":
#actual_kwargs()
def func(msg, a=None, b=False, c='', d=0):
print msg
for arg, val in sorted(func.actual_kwargs.iteritems()):
print ' %s: %s' % (arg, val)
func("I'm only passing a", a='a')
func("Here's b and c", b=True, c='c')
func("All defaults", a=None, b=False, c='', d=0)
func("Nothin'")
try:
func("Invalid kwarg", e="bogon")
except TypeError, err:
print 'Invalid kwarg\n %s' % err
Which prints this:
I'm only passing a
a: a
Here's b and c
b: True
c: c
All defaults
a: None
b: False
c:
d: 0
Nothin'
Invalid kwarg
func() got an unexpected keyword argument 'e'
I'm happy with this. A more flexible approach is to pass the name of the attribute you want to use to the decorator, instead of hard-coding it to 'actual_kwargs', but this is the simplest approach that illustrates the solution.
Mmm, Python is tasty.
Here is the easiest and simplest way:
def func(a=None, b=None, c=None):
args = locals().copy()
print args
func(2, "egg")
This give the output: {'a': 2, 'c': None, 'b': 'egg'}.
The reason args should be a copy of the locals dictionary is that dictionaries are mutable, so if you created any local variables in this function args would contain all of the local variables and their values, not just the arguments.
More documentation on the built-in locals function here.
One possibility:
def f(**kw):
acceptable_names = set('a', 'b', 'c')
if not (set(kw) <= acceptable_names):
raise WhateverYouWantException(whatever)
...proceed...
IOW, it's very easy to check that the passed-in names are within the acceptable set and otherwise raise whatever you'd want Python to raise (TypeError, I guess;-). Pretty easy to turn into a decorator, btw.
Another possibility:
_sentinel = object():
def f(a=_sentinel, b=_sentinel, c=_sentinel):
...proceed with checks `is _sentinel`...
by making a unique object _sentinel you remove the risk that the caller might be accidentally passing None (or other non-unique default values the caller could possibly pass). This is all object() is good for, btw: an extremely-lightweight, unique sentinel that cannot possibly be accidentally confused with any other object (when you check with the is operator).
Either solution is preferable for slightly different problems.
How about using a decorator to validate the incoming kwargs?
def validate_kwargs(*keys):
def entangle(f):
def inner(*args, **kwargs):
for key in kwargs:
if not key in keys:
raise ValueError("Received bad kwarg: '%s', expected: %s" % (key, keys))
return f(*args, **kwargs)
return inner
return entangle
###
#validate_kwargs('a', 'b', 'c')
def func(**kwargs):
for arg,val in kwargs.items():
print arg, "->", val
func(b=2)
print '----'
func(a=3, c=5)
print '----'
func(d='not gonna work')
Gives this output:
b -> 2
----
a -> 3
c -> 5
----
Traceback (most recent call last):
File "kwargs.py", line 20, in <module>
func(d='not gonna work')
File "kwargs.py", line 6, in inner
raise ValueError("Received bad kwarg: '%s', expected: %s" % (key, keys))
ValueError: Received bad kwarg: 'd', expected: ('a', 'b', 'c')
This is easiest accomplished with a single instance of a sentry object:
# Top of module, does not need to be exposed in __all__
missing = {}
# Function prototype
def myFunc(a = missing, b = missing, c = missing):
if a is not missing:
# User specified argument a
if b is missing:
# User did not specify argument b
The nice thing about this approach is that, since we're using the "is" operator, the caller can pass an empty dict as the argument value, and we'll still pick up that they did not mean to pass it. We also avoid nasty decorators this way, and keep our code a little cleaner.
There's probably better ways to do this, but here's my take:
def CompareArgs(argdict, **kwargs):
if not set(argdict.keys()) <= set(kwargs.keys()):
# not <= may seem weird, but comparing sets sometimes gives weird results.
# set1 <= set2 means that all items in set 1 are present in set 2
raise ValueError("invalid args")
def foo(**kwargs):
# we declare foo's "standard" args to be a, b, c
CompareArgs(kwargs, a=None, b=None, c=None)
print "Inside foo"
if __name__ == "__main__":
foo(a=1)
foo(a=1, b=3)
foo(a=1, b=3, c=5)
foo(c=10)
foo(bar=6)
and its output:
Inside foo
Inside foo
Inside foo
Inside foo
Traceback (most recent call last):
File "a.py", line 18, in
foo(bar=6)
File "a.py", line 9, in foo
CompareArgs(kwargs, a=None, b=None, c=None)
File "a.py", line 5, in CompareArgs
raise ValueError("invalid args")
ValueError: invalid args
This could probably be converted to a decorator, but my decorators need work. Left as an exercise to the reader :P
Perhaps raise an error if they pass any *args?
def func(*args, **kwargs):
if args:
raise TypeError("no positional args allowed")
arg1 = kwargs.pop("arg1", "default")
if kwargs:
raise TypeError("unknown args " + str(kwargs.keys()))
It'd be simple to factor it into taking a list of varnames or a generic parsing function to use. It wouldn't be too hard to make this into a decorator (python 3.1), too:
def OnlyKwargs(func):
allowed = func.__code__.co_varnames
def wrap(*args, **kwargs):
assert not args
# or whatever logic you need wrt required args
assert sorted(allowed) == sorted(kwargs)
return func(**kwargs)
Note: i'm not sure how well this would work around already wrapped functions or functions that have *args or **kwargs already.
Magic is not the answer:
def funky(a=None, b=None, c=None):
for name, value in [('a', a), ('b', b), ('c', c)]:
print name, value