We Keep Coding

How to use function annotations to verify function call types

I recently just discovered there's a thing called function annotation but I'm not very sure of how to use it. This is what I have so far:
def check_type(f):
def decorated(*args, **kwargs):
counter=0
for arg, type in zip(args, f.__annotations__.items()):
if not isinstance(arg, type[1]):
msg = 'Not the valid type'
raise ValueError(msg)
counter+=1
return f(*args, **kwargs)
return decorated
#check_type
def foo(a: int, b: list, c: str): #a must be int, b must be list, c must be str
print(a,b,c)
foo(12, [1,2], '12') #This works
foo(12, 12, 12) #This raises a value error just as I wanted to
foo(a=12, b=12, c=12) #But this works too:(
As you can see, I'm trying to check the types of a,b and c using annotations and a decorator and that raises ValueError if it's not the right type. It works fine when I dont use keyword arguments when calling. But if I used keyword arguments, the types don't get checked. I'm trying to make it work but I had no luck.
My code doesn't support keyword argument. Because I don't have anything that checks that. Nor I have any idea on how to check it. Here's where I need help.
I also did it this way:
def check_type(f):
def decorated(*args, **kwargs):
for name, type in f.__annotations__.items():
if not isinstance(kwargs[name], type):
msg = 'Not the valid type'
raise ValueError(msg)
return f(*args, **kwargs)
return decorated
#But now they have to be assigned using keyword args
#so only foo(a=3,b=[],c='a') works foo(3,[],'a') results in a keyerror
#How can I combine them?

As Paul suggested, it is best to use the bind method of Signature objects (located in inspect) to bind the *args and **kwargs that are to be provided to f and then check if the type matches:
from inspect import signature
from typing import get_type_hints
def check_range(f):
def decorated(*args, **kwargs):
counter=0
# use get_type_hints instead of __annotations__
annotations = get_type_hints(f)
# bind signature to arguments and get an
# ordered dictionary of the arguments
b = signature(f).bind(*args, **kwargs).arguments
for name, value in b.items():
if not isinstance(value, annotations[name]):
msg = 'Not the valid type'
raise ValueError(msg)
counter+=1
return f(*args, **kwargs)
return decorated
Your first case was actually succeeding randomly. dicts have a random order in Python < 3.6 that will most likely change when you fire-up the Python interpreter again, that means the zipping you do isn't deterministic.
Instead of iterating through f.__annotations__, grab it via get_type_hints and then, by getting the names and values through b.items() (which is an OrderedDict and guarantees order) index it with name.

python: dispatch method with string input

I need to write a method that takes in 3 arguments:
a string with the name of a function
an ordered list of arguments to that function. This includes arguments with default values and *varargs, but does not include **kwargs
a dict representing any additional keyword arguments, or None if there are none
And I need to use this input to retrieve a function and call it. For example:
def dispatch(name, args, kwargs=None):
do_magic_here(name, args, kwargs)
def meth1():
print "meth1"
def meth2(a, b):
print "meth2: %s %s" % (a, b)
def meth3(a, **kwargs):
print "meth3: " + a
for k,v in kwargs.iteritems():
print "%s: %s" % (k,v)
And I need to be able to call things like this:
>>> dispatch("meth1", [])
meth1
>>> dispatch("meth2", [1, 3])
meth2: 1 3
>>> dispatch("meth3", [1], {"hello":2, "there":3})
meth3: 1
hello: 2
there: 3
I could do this:
def do_magic_here(name, args, kwargs=None):
if name=="meth1":
meth1()
if name=="meth2":
meth2(args[0], args[1])
if name=="meth3":
meth3(args[0], **kwargs)
But I'm trying to dispatch like 40 methods, and that number may expand, so I'm hoping there's a more programmatic way to do it. I'm looking at something with getattr, but I can't quite figure it out.

I would just use
def dispatch(name, *args, **kwargs):
func_name_dict[name](*args, **kwargs)
with
func_name_dict = {'meth1':meth1,
'meth2':meth2,
...}
Allowing you to pass args and kwargs through more naturally and transparently:
>>> dispatch("meth2", 1, 3)
meth2: 1 3
You can of course use globals() or locals() in place of the dict, but you might need to be careful about which functions in each namespace you do or don't want to expose to the caller

Indeed, getattr will get you there.
class X:
def a(self):
print('a called')
def b(self, arg):
print('b called with ' + arg)
x = X()
getattr(x, 'a')()
# a called
getattr(x, 'b')('foo')
# b called with foo
Just like getattr handles methods and fields the same way, you can handle
functions and variables not associated with a class by referencing locals() or globals().
If you want to refer to a function in the global scope:
globals()['meth'](args)
For example:
def dispatch(name, *args, **kwargs):
globals()[name](*args, **kwargs)
dispatch('meth3', 'hello', foo='bar')
# meth3: hello
# foo: bar
Remember in Python you can always pass a list of arguments or dict of keyword arguments using the **:
dispatch('meth3', *['hello'], **{'foo':'bar'})
If you truly prefer to pass arguments as list/dict to dispatch:
def dispatch(name, args, kwargs):
globals()[name](*args, **kwargs)
dispatch('meth3', ['hello'], {'foo': 'bar'})

Magic assign for custom parameters

I want to give user API for my library with easier way to distinguish different types of parameters which I pass to function. All groups of arguments are defined earlier (for now I have 3 groups), but attributes of them need to be constructed on run. I can do this in Django ORM style, where double underscore separates 2 parts of parameter. But it is very unreadable. Example:
def api_function(**kwargs):
""" Separate passed arguments """
api_function(post__arg1='foo', api__arg1='bar', post_arg2='foo2')
Better way do this SQLAlchemy, but only to compare attributes and all args are defined earlier. Example:
class API(object):
arg1 = Arg()
arg2 = Arg()
class Post(object): #...
def api_function(*args):
""" Separate passed arguments """
api_function(POST.arg1=='foo', API.arg1=='bar', POST.arg2=='foo2')
What I would like to achive is behaviour like this:
class API(object): # Magic
class POST(object): # Magic
def api_function(*args):
""" Separate passed arguments """
api_function(POST.arg1='foo', API.arg1='bar', POST.arg2='foo2')
What have I tried:
declare metamodel with defined __setattr__, but it rise on evaluation SyntaxError: keyword can't be an expression
declare __set__, but it is designed for known attributes
My questions are:
Is it even possible in Python to work like in third snippet?
If not, is there any really close solution to look like in third snippet? The best way should use assignment operator API.arg1='foo', the worst API(arg1='foo')
Requirements -- should work at least at Python 2.7. Good to work on Python 3.2.
EDIT1
My first test, which is using equality operator (but it NEVER should be use in this way):
class APIMeta(type):
def __getattr__(cls, item):
return ApiData(item, None)
class API(object):
__metaclass__ = APIMeta
def __init__(self, key, value):
self.key = key
self.value = value
def __str__(self):
return "{0}={1}".format(self.key, self.value)
def __eq__(self, other):
self.value = other
return self
def print_api(*api_data):
for a in api_data:
print(str(a))
print_api(API.page=='3', API=='bar')
It is working right, but using == is suggesting that I want to compare something and I want to assign value.

NOTE: I don't know how much I like this schema you want. But I know one annoying thing will be all the imports to call api_function. E.G. from api import POST, API, api_function
As I said in the comments, the first way is not possible. This is because assignment (=) is a statement not an expression, so it can't return a value. Source
But the other way you asked for certainly is:
class POST(object):
def __init__(self, **kwargs):
self.args = kwargs
# You'll also probably want to make this function a little safer.
def __getattr__(self, name):
return self.args[name]
def api_function(*args):
# Update this to how complicated the handling needs to be
# but you get the general idea...
post_data = None
for a in args:
if isinstance(a, POST):
post_data = a.args
if post_data is None:
raise Exception('This function needs a POST object passed.')
print post_data
Using it:
>>> api_function('foo')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in api_function
Exception: This function needs a POST object passed.
>>> api_function(POST(arg1='foo'))
{'arg1': 'foo'}
>>> api_function(POST(arg1='foo',
... arg2='bar'
... )
... )
{'arg1': 'foo', 'arg2': 'bar'}

Here's my solution. It's not the best in design, as the structure of the argument grouper is nested quite deep, so I'd appreciate feedback on it:
class ArgumentGrouper(object):
"""Transforms a function so that you can apply arguments in named groups.
This system isn't tested as thoroughly as something with so many moving
parts should be. Use at own risk.
Usage:
#ArgumentGrouper("foo", "bar")
def method(regular_arg, foo__arg1, bar__arg2):
print(regular_arg + foo__arg1 + bar__arg2)
method.foo(", ").bar("world!")("Hello")() # Prints "Hello, world!"
"""
def __call__(self, func):
"""Decorate the function."""
return self.Wrapper(func, self.argument_values)
def __init__(self, *argument_groups):
"""Constructor.
argument_groups -- The names of argument groups in the function.
"""
self.argument_values = {i: {} for i in argument_groups}
class Wrapper(object):
"""This is the result of decorating the function. You can call group
names as function to supply their keyword arguments.
"""
def __call__(self, *args):
"""Execute the decorated function by passing any given arguments
and predefined group arguments.
"""
kwargs = {}
for group, values in self.argument_values.items():
for name, value in values.items():
# Add a new argument in the form foo__arg1 to kwargs, as
# per the supplied arguments.
new_name = "{}__{}".format(
group,
name
)
kwargs[new_name] = value
# Invoke the function with the determined arguments.
return self.func(*args, **kwargs)
def __init__(self, func, argument_values):
"""Constructor.
func -- The decorated function.
argument_values -- A dict with the current values for group
arguments. Must be a reference to the actual dict, since each
WrappedMethod uses it.
"""
self.func = func
self.argument_values = argument_values
def __getattr__(self, name):
"""When trying to call `func.foo(arg1="bar")`, provide `foo`. TODO:
This would be better handled at initialization time.
"""
if name in self.argument_values:
return self.WrappedMethod(name, self, self.argument_values)
else:
return self.__dict__[name]
class WrappedMethod(object):
"""For `func.foo(arg1="bar")`, this is `foo`. Pretends to be a
function that takes the keyword arguments to be supplied to the
decorated function.
"""
def __call__(self, **kwargs):
"""`foo` has been called, record the arguments passed."""
for k, v in kwargs.items():
self.argument_values[self.name][k] = v
return self.wrapper
def __init__(self, name, wrapper, argument_values):
"""Constructor.
name -- The name of the argument group. (This is the string
"foo".)
wrapper -- The decorator. We need this so that we can return it
to chain calls.
argument_values -- A dict with the current values for group
arguments. Must be a reference to the actual dict, since
each WrappedMethod uses it.
"""
self.name = name
self.wrapper = wrapper
self.argument_values = argument_values
# Usage:
#ArgumentGrouper("post", "api")
def api_function(regular_arg, post__arg1, post__arg2, api__arg3):
print("Got regular args {}".format(regular_arg))
print("Got API args {}, {}, {}".format(post__arg1, post__arg2, api__arg3))
api_function.post(
arg1="foo", arg2="bar"
).api(
arg3="baz"
)
api_function("foo")
Then, usage:
#ArgumentGrouper("post", "api")
def api_function(regular_arg, post__arg1, post__arg2, api__arg3):
print("Got regular args {}".format(regular_arg))
print("Got API args {}, {}, {}".format(post__arg1, post__arg2, api__arg3))
api_function.post(
arg1="foo", arg2="bar"
).api(
arg3="baz"
)
api_function("foo")
Output:
Got regular args foo
Got API args foo, bar, baz
It should be simple to scrape argument group names by introspection.
You'll notice the argument naming convention is hardcoded into the WrappedMethod, so you'll have to make sure you're okay with that.
You can also invoke it in one statement:
api_function.post(
arg1="foo", arg2="bar"
).api(
arg3="baz"
)("foo")
Or you could add a dedicated run method which would invoke it, which would just take the place of Wrapper.__call__.

Python don't allow to use assignment operator inside any other code, so:
(a=1)
func((a=1))
will rise SyntaxError. This means that it is not possible to use it in this way. Moreover:
func(API.arg1=3)
Will be treated that left side of assignment is argument API.arg1 which is not valid name in Python for variables. Only solution is to make this in SQLAlchemy style:
func({
API.arg1: 'foo',
API.arg2: 'bar',
DATA.arg1: 'foo1',
})
or
func(**{
API.arg1: 'foo',
API.arg2: 'bar',
DATA.arg1: 'foo1',
})
or just only:
func( API(arg1='foo', arg2='bar'), POST(arg1='foo1'), POST(arg2='bar1'))
Thank you for your interest and answers.

Pythonic way of getting method arguments into a dict

I'm writing an api and was wondering what's the most pythonic way to do the following.
I'm writing a bunch of methods to do various web calls, the arguments mostly translate into post data keys and values.
The way I've been writing it so far is mostly like this;
def doSomething(self,param1,param2,param3):
payload={"param1":param1,
"param2":param2,
"param3":param3}
return self.request("do/something",payload)
This already has the draw back of having to repeat the parameter names which are subject to change, but this pattern isn't too bad.
The following case is what got me trying to think of a better way. In this case there are 4 optional arguments for the call
def doSomethingElse(self,param1,param2=None,param3=None,param4=None,param5=None):
payload= {"param1":param1}
if param2:
payload["param2"]= param2
if param3:
payload["param3"]= param3
# ... etc ...
self.request("do/something/else",payload)
My first thought was to do this:
def doSomethingElse(self,param1,**params):
payload = {"param1":param1}
payload.update(params)
self.request("do/something/else",payload)
or even:
def doSomethingElse(self,**payload):
self.request("do/something/else",payload)
Although the second one is nice and simple, the method can be called without the non-default argument. But in both cases I lose the method signature when using the api and the user won't know what the parameters are (I know I could write the expected signature in a docstring but I'd rather prevent misspelt keywords getting sent).
I'm thinking there must be a nice pythonic solution to do this, any ideas?
EDIT
I think a key point which I didn't make clear enough is that the arguments are getting sent in post data in a call, and I want to make sure only those keys can get sent, like in the first example of doSomethingElse, you can't send anything other than those 5 named parameters.

The Pythonic way is to name the parameters when you call the function, not in the function signature:
def doSomething(self, **kwargs):
self.request("do/something/else", kwargs)
doSomething(param1=3, param2='one', param3=4)

How about simply
def get_payload(ldict):
return {k:v for k,v in ldict.iteritems() if k != 'self' and v is not None}
class fred(object):
some_class_var = 17
def method(self, a, b=2):
payload = get_payload(locals())
print payload
which gives
>>> f = fred()
>>> f.method()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method() takes at least 2 arguments (1 given)
>>> f.method(2)
{'a': 2, 'b': 2}
>>> f.method(2, b=3)
{'a': 2, 'b': 3}
>>> f.method(5, b=None)
{'a': 5}
>>> f.method(2, b=3, c=19)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method() got an unexpected keyword argument 'c'
>>> help(f.method)
Help on method method in module __main__:
method(self, a, b=2) method of __main__.fred instance
which I think matches your criteria. The next step would be to use a decorator (probably with either wraps or the decorator module to preserve the signature) so that payload was computed and then passed, but I don't know if #payload would be all that much better than payload = get_payload(locals()). Note that using locals() this way, it needs to be done at the start.
I second the feeling that this isn't exactly the best way to prevent unwanted nuclear attacks, though.

Something like this, perhaps:
def doSomethingElse(self, param1, **params):
payload = {"param1": param1}
for name, value in params.items():
if value is not None:
payload[name] = value
self.request("do/something/else", payload)

If you have several such functions, you can do as following:
class Requester(object):
def __init__(self, tobecalled, *allowed):
self.tobecalled = tobecalled
self.allowed = set(allowed)
def __call__(self, otherobj, **k):
for kw in k.iterkeys():
if kw not in self.allowed:
raise ValueError("unknown argument(s) given: %s" % kw)
otherobj.request(self.tobecalled, **k)
def __get__(self, outside, outsideclass):
return lambda **k: self(outside, **k)
class Outside(object):
def request(self, method, **k):
print method, k
do_one_thing = Requester("do/one/thing", 'param1', 'param2')
do_nonsense = Requester("do/nonsense", 'param3')
simple = Requester("simple")
o = Outside()
o.do_one_thing(param1=1, param2=2)
o.do_nonsense(param3=12)
o.simple()
try: o.do_one_thing(rparam1=1, param2=2)
except ValueError, e: print e
try: o.do_nonsense(gparam3=12)
except ValueError, e: print e
try: o.simple(whatever=12)
except ValueError, e: print e
What happens here? We create a Requester object which plays the role of a method: if we put it in another class (here: Outside), it can be called in a way that it also gets a reference of an object which it is called on. What I call outside here is "the outer self", as I call it now. And then, it returns a lambda which calls the object itself, just like a function does. And there, the arguments are checked for validity, and if that passes, we do the call on the "outside"'s request() method.

method overloading in python

I need to call unparameterised method first, but also parameterized first, but it is giving an error.
>>> class A:
... def first(self):
... print 'first method'
... def first(self,f):
... print 'first met',f
...
>>> a=A()
>>> a.first()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: first() takes exactly 2 arguments (1 given)
Is it possible to do method overloading in Python like in Java?

Your second first method is overriding the original first method. In Python, it is not possible to create overloaded methods the same way as in Java.
However, you can create methods with optional and/or keyword-based arguments and process those accordingly. Here's an example:
class A:
def first(self, f=None):
if f is not None:
print 'first met', f
else:
print 'first method'
Usage:
a = A()
a.first()
a.first('something')

Python doesn't do function overloading. This is a consequence of it being a loosely-typed language. Instead you can specify an unknown number of arguments and deal with their interpretation in the function logic.
There are a couple ways you can do this. You can specify specific optional arguments:
def func1(arg1, arg2=None):
if arg2 != None:
print "%s %s" % (arg1, arg2)
else:
print "%s" % (arg1)
Calling it we get:
>>> func1(1, 2)
1 2
Or you can specify an unknown number of unnamed arguments (i.e. arguments passed in an array):
def func2(arg1, *args):
if args:
for item in args:
print item
else:
print arg1
Calling it we get:
>>> func2(1, 2, 3, 4, 5)
2
3
4
5
Or you can specify an unknown number of named arguments (i.e. arguments passed in a dictionary):
def func3(arg1, **args):
if args:
for k, v in args.items():
print "%s %s" % (k, v)
else:
print arg1
Calling it we get:
>>> func3(1, arg2=2, arg3=3)
arg2 2
arg3 3
You can use these constructions to produce the behaviour you were looking for in overloading.

Usually you can only define one method in a class with a given name. In your example the 2 argument first() method overwrote the 1 argument first(). If you want two methods with the same name, in python 3 you have to use functools.singledispatch, and map the instance method name to your static method dispatcher, Ouch!
That said, I really like implicit dynamic dispatch in OO programming, and I find it cleaner than writing manual dispatch logic in some kind of 'master' first() function, which is repetitive and brittle to extension.
Challenge question: add another method like A.first(A arg).
You'll probably learn a lot about the python type system if you try to do this!
#!/opt/local/bin/python3.4
from functools import singledispatch;
class A(object):
# default method handles dispatch for undefined types
# note reversed positional args to match single dispatch functools
#singledispatch
def _first(self,arg):
raise TypeError("no match for A._first(%s)" % type(arg));
# adapter maps instance call to (reversed) static method call
def first(self, arg = None): return A._first(arg, self);
# def first()
#_first.register(type(None))
def _(self,none):
print("A.first() called");
# def first(float f)
#_first.register(float)
def _(self,f):
print("A.first(float %s) called" % f);
a = A();
a.first(); # A.first() called
a.first(None); # A.first() called
a.first(3.14); # A.first(float 3.14) called
class B(object): pass;
b = B();
try: a.first(b); # no match for A._first(<class '__main__.B'>)
except TypeError as ex: print(ex);

Python is not C++ or Java; you can't overload methods in the same way.
Really, the only way to do what you want is to test for the presence or absence of the second parameter:
class A:
def first(self, f=None):
if f is None:
print 'first method'
else:
print 'first met',f
You can be yet more sophisticated and check the type of f but that can be dangerous, and is not always "pythonic". (However, it should be mentioned that one use case for function annotations in Python 3 is to allow this sort of "Generic Programming".)

While it is possible to create a system that appears to use overloaded methods, it's a bit involved, and not usually needed.
The usual idiom is to have possibly unneeded parameters default to None, like so:
class A:
def first(self, f=None):
if f is None:
print 'first method'
else:
print 'first met',f
In your case, where you want different behavior based upon whether or not this is the first call to that method, this is what I would do:
class A:
def first(self):
print 'first method'
self.first = self._first
def _first(self, f): # '_' is convention for private name
print 'first met',f
and sample output:
a = A()
a.first()
a.first(3)
prints:
first method
first met 3

Check this code if it is helpful:
from math import pi
class Geometry:
def area(self,length = None,breadth = None,side = None,radius = None):
self.length = length
self.breadth = breadth
self.side = side
self.radius = radius
if length != None and breadth != None:
return length * breadth
elif side != None:
return side * side
else:
return pi * radius * radius
obj1 = Geometry()
print('Area of rectangle is {0}.'.format(obj1.area(length=5,breadth=4)))
print('Area of square is {0}.'.format(obj1.area(side=5)))
print('Area of circle is {0:.6}.'.format(obj1.area(radius=10)))