Background:
I mostly run python scripts from the command line in pipelines and so my arguments are always strings that need to be type casted to the appropriate type. I make a lot of little scripts each day and type casting each parameter for every script takes more time than it should.
Question:
Is there a canonical way to automatically type cast parameters for a function?
My Way:
I've developed a decorator to do what I want if there isn't a better way. The decorator is the autocast fxn below. The decorated fxn is fxn2 in the example. Note that at the end of the code block I passed 1 and 2 as strings and if you run the script it will automatically add them. Is this a good way to do this?
def estimateType(var):
#first test bools
if var == 'True':
return True
elif var == 'False':
return False
else:
#int
try:
return int(var)
except ValueError:
pass
#float
try:
return float(var)
except ValueError:
pass
#string
try:
return str(var)
except ValueError:
raise NameError('Something Messed Up Autocasting var %s (%s)'
% (var, type(var)))
def autocast(dFxn):
'''Still need to figure out if you pass a variable with kw args!!!
I guess I can just pass the dictionary to the fxn **args?'''
def wrapped(*c, **d):
print c, d
t = [estimateType(x) for x in c]
return dFxn(*t)
return wrapped
#autocast
def fxn2(one, two):
print one + two
fxn2('1', '2')
EDIT: For anyone that comes here and wants the updated and concise working version go here:
https://github.com/sequenceGeek/cgAutoCast
And here is also quick working version based on above:
def boolify(s):
if s == 'True' or s == 'true':
return True
if s == 'False' or s == 'false':
return False
raise ValueError('Not Boolean Value!')
def estimateType(var):
'''guesses the str representation of the variables type'''
var = str(var) #important if the parameters aren't strings...
for caster in (boolify, int, float):
try:
return caster(var)
except ValueError:
pass
return var
def autocast(dFxn):
def wrapped(*c, **d):
cp = [estimateType(x) for x in c]
dp = dict( (i, estimateType(j)) for (i,j) in d.items())
return dFxn(*cp, **dp)
return wrapped
######usage######
#autocast
def randomFunction(firstVar, secondVar):
print firstVar + secondVar
randomFunction('1', '2')
If you want to auto-convert values:
def boolify(s):
if s == 'True':
return True
if s == 'False':
return False
raise ValueError("huh?")
def autoconvert(s):
for fn in (boolify, int, float):
try:
return fn(s)
except ValueError:
pass
return s
You can adjust boolify to accept other boolean values if you like.
You could just use plain eval to input string if you trust the source:
>>> eval("3.2", {}, {})
3.2
>>> eval("True", {}, {})
True
But if you don't trust the source, you could use literal_eval from ast module.
>>> ast.literal_eval("'hi'")
'hi'
>>> ast.literal_eval("(5, 3, ['a', 'b'])")
(5, 3, ['a', 'b'])
Edit:
As Ned Batchelder's comment, it won't accept non-quoted strings, so I added a workaround, also an example about autocaste decorator with keyword arguments.
import ast
def my_eval(s):
try:
return ast.literal_eval(s)
except ValueError: #maybe it's a string, eval failed, return anyway
return s #thanks gnibbler
def autocaste(func):
def wrapped(*c, **d):
cp = [my_eval(x) for x in c]
dp = {i: my_eval(j) for i,j in d.items()} #for Python 2.6+
#you can use dict((i, my_eval(j)) for i,j in d.items()) for older versions
return func(*cp, **dp)
return wrapped
#autocaste
def f(a, b):
return a + b
print(f("3.4", "1")) # 4.4
print(f("s", "sd")) # ssd
print(my_eval("True")) # True
print(my_eval("None")) # None
print(my_eval("[1, 2, (3, 4)]")) # [1, 2, (3, 4)]
I'd imagine you can make a type signature system with a function decorator, much like you have, only one that takes arguments. For example:
#signature(int, str, int)
func(x, y, z):
...
Such a decorator can be built rather easily. Something like this (EDIT -- works!):
def signature(*args, **kwargs):
def decorator(fn):
def wrapped(*fn_args, **fn_kwargs):
new_args = [t(raw) for t, raw in zip(args, fn_args)]
new_kwargs = dict([(k, kwargs[k](v)) for k, v in fn_kwargs.items()])
return fn(*new_args, **new_kwargs)
return wrapped
return decorator
And just like that, you can now imbue functions with type signatures!
#signature(int, int)
def foo(x, y):
print type(x)
print type(y)
print x+y
>>> foo('3','4')
<type: 'int'>
<type: 'int'>
7
Basically, this is an type-explicit version of #utdemir's method.
If you're parsing arguments from the command line, you should use the argparse module (if you're using Python 2.7).
Each argument can have an expected type so knowing what to do with it should be relatively straightforward. You can even define your own types.
...quite often the command-line string should instead be interpreted as another type, like a float or int. The type keyword argument of add_argument() allows any necessary type-checking and type conversions to be performed. Common built-in types and functions can be used directly as the value of the type argument:
parser = argparse.ArgumentParser()
parser.add_argument('foo', type=int)
parser.add_argument('bar', type=file)
parser.parse_args('2 temp.txt'.split())
>>> Namespace(bar=<open file 'temp.txt', mode 'r' at 0x...>, foo=2)
There are couple of problems in your snippet.
#first test bools
if var == 'True':
return True
elif var == 'False':
return False
This would always check for True because you are testing against the strings 'True' and 'False'.
There is not an automatic coercion of types in python. Your arguments when you receive via *args and **kwargs can be anything. First will look for list of values (each of which can be any datatype, primitive and complex) and second will look for a mapping (with any valid mapping possible). So if you write a decorator, you will end up with a good list of error checks.
Normally, if you wish to send in str, just when the function is invoked, typecast it to string via (str) and send it.
I know I arrived late at this game, but how about eval?
def my_cast(a):
try:
return eval(a)
except:
return a
or alternatively (and more safely):
from ast import literal_eval
def mycast(a):
try:
return literal_eval(a)
except:
return a
Related
Often I encounter the scenario of functions which accept a finite set of values only. I know how to reflect this behavior in the type annotations, using typing.Literal like so:
import typing
def func(a: typing.Literal['foo', 'bar']):
pass
I would like to have a decorator #validate_literals which validates that the parameters to the are consistent with their type:
#validate_literals
def picky_typed_function(
binary: typing.Literal[0, 1],
char: typing.Literal['a', 'b']
) -> None:
pass
so that the input is validated against the restrictions defined by the arguments's types, and a ValueError is raised in case of a violation:
picky_typed_function(0, 'a') # should pass
picky_typed_function(2, 'a') # should raise "ValueError: binary must be one of (0, 1)')"
picky_typed_function(0, 'c') # should raise "ValueError: char must be one of ('a', 'b')"
picky_typed_function(0, char='c') # should raise "ValueError: char must be one of ('a', 'b')"
picky_typed_function(binary=2, char='c') # should raise "ValueError: binary must be one of (0, 1)"
typing type checks are designed to be static, and not happen during runtime. How can I leverage the typing definition for runtime validation?
We can inspect the decorated (validated) function's signature by using inspect.signature, check which of the parameters of the function is typed as a Literal alias by getting the "origin" of the parameter's annotation through typing.get_origin() (or, for python versions < 3.8, using __origin__) and retrieve the valid values by using [typing.get_args()] (https://stackoverflow.com/a/64522240/3566606) (and iterating recursively over nested Literal definitions) from the Literal alias.
In order to do that, all that is left to do, is to figure out which parameters have been passed as positional arguments and map the corresponding values to the parameter's name, so the value can be compared against the valid values of the parameter.
Finally, we build the decorator using the standard recipe with functools.wraps. In the end, this is the code:
import inspect
import typing
import functools
def args_to_kwargs(func: typing.Callable, *args: list, **kwargs: dict) -> dict:
args_dict = {
list(inspect.signature(func).parameters.keys())[i]: arg
for i, arg in enumerate(args)
}
return {**args_dict, **kwargs}
def valid_args_from_literal(annotation: _GenericAlias) -> Set[Any]:
args = get_args(annotation)
valid_values = []
for arg in args:
if typing.get_origin(annotation) is Literal:
valid_values += valid_args_from_literal(arg)
else:
valid_values += [arg]
return set(valid_values)
def validate_literals(func: typing.Callable) -> typing.Callable:
#functools.wraps(func)
def validated(*args, **kwargs):
kwargs = args_to_kwargs(func, *args, **kwargs)
for name, parameter in inspect.signature(func).parameters.items():
# use parameter.annotation.__origin__ for Python versions < 3.8
if typing.get_origin(parameter.annotation) is typing.Literal:
valid_values = valid_args_from_literal(parameter.annotation)
if kwargs[name] not in valid_values:
raise ValueError(
f"Argument '{name}' must be one of {valid_values}"
)
return func(**kwargs)
return validated
This gives the results specified in the question.
I have also published the alpha version of a python package runtime-typing to perform runtime typechecking: https://pypi.org/project/runtime-typing/ (documentation:https://runtime-typing.readthedocs.io) which handles more cases than just typing.Literal, such as typing.TypeVar and typing.Union.
from typing import Literal
from valdec.dec import validate
#validate
def func(a: Literal["foo", "bar"]) -> str:
return a
assert func("bar") == "bar"
#validate("return", exclude=True)
def func(binary: Literal[0, 1], char: Literal["a", "b"]):
return binary, char
assert func(0, "a") == (0, "a")
func(2, "x")
# valdec.utils.ValidationArgumentsError: Validation error <class
# 'valdec.val_pydantic.ValidationError'>: 2 validation errors for argument
# with the name of:
# binary
# unexpected value; permitted: 0, 1 (type=value_error.const; given=2;
# permitted=(0, 1))
# char
# unexpected value; permitted: 'a', 'b' (type=value_error.const; given=x;
# permitted=('a', 'b')).
valdec: https://github.com/EvgeniyBurdin/valdec
Suppose I have a datatype, for example:
List = Datatype('List')
List.declare('cons', ('car', IntSort()), ('cdr', List))
List.declare('nil')
List = List.create()
And I have an instance, by:
s.check()
instance = s.model()[var]
How can i ask which constructor my instance has?
I know I could do something like this:
for i in range(List.num_constructors()):
if List.recognizer(i)(instance):
break
return List.constructor(i)
But this is not a very practical way when the number of constructors is large.
How can i ask this?
Python interface defines recognizers is_cons, is_nil for you, which you can then use for simplifying your code. Something like:
from z3 import *
List = Datatype('List')
List.declare('cons', ('car', IntSort()), ('cdr', List))
List.declare('nil')
List = List.create()
# constructors
cons = List.cons
car = List.car
cdr = List.cdr
nil = List.nil
# recognizers
def is_cons (x): return simplify(List.is_cons(x))
def is_nil (x): return simplify(List.is_nil(x))
def valToConstructor(l):
if is_cons(l):
return "Found cons"
elif is_nil(l):
return "Found nil"
else:
raise Exception('Unexpected list value: {}'.format(l))
# example:
ex1 = nil
ex2 = cons (0, cons (1, ex1))
print valToConstructor(ex1)
print valToConstructor(ex2)
When run, this produces:
Found nil
Found cons
Note that you cannot skip the exception case in valToConstructor since this function might be called with a symbolic value as well; and the recognizers would not fire for those values. But if you always call it with values you get out of a model, it should work just fine.
Directly accessing the constructor
You can also use the decl function for the underlying AST:
>>> ex1.decl()
nil
>>> ex2.decl()
cons
But you have to be careful when the object you're passing is a symbolic value; as in that case decl returns its name:
>>> ex3 = Const('ex3_name', List)
>>> ex3.decl()
ex3_name
But perhaps this is precisely what you're looking for.
Is there a way to do something like that in python?
def foo(str(arg).lower()):
print arg
foo('LOL')
I wanted to create decorator in the first place but whatever i try i could not get it to work as I wanted, i either get generator returned or list.
Edit:
This is what I wanted to accomplish.
def to_lowercase(function):
def wrapper(*args):
return function(*[x.lower() if isinstance(x, str) else x for x in args])
return wrapper
its possible to write a decorator to do what you seek:
decorators are functions that accept a function as the first argument and arguments as next arguments. Then a different function is returned, or the arguments are modified in some way before being applied to the input-function.
Here we can modify the arguments to lower case if they are instances of str and leave unaltered otherwise using comprehensions:
def tolower(f, *args, **kwargs):
def newfun(*args, **kwargs):
as_ = [a.lower() if isinstance(a, str) else a for a in args]
kws_ = {k: (v.lower() if isinstance(v, str) else v) for k,v in kwargs.items()}
return f(*as_, **kws_)
return newfun
#tolower
def print_lower(x, y, z):
print(x)
print(y)
print(z)
print_lower('A', 'B', 1)
# outputs
a
b
1
I am attempting to write exception raising code blocks into my Python code in order to ensure that the parameters passed to the function meet appropriate conditions (i.e. making parameters mandatory, type-checking parameters, establishing boundary values for parameters, etc...). I understand satisfactorily how to manually raise exceptions as well as handling them.
from numbers import Number
def foo(self, param1 = None, param2 = 0.0, param3 = 1.0):
if (param1 == None):
raise ValueError('This parameter is mandatory')
elif (not isinstance(param2, Number)):
raise ValueError('This parameter must be a valid Numerical value')
elif (param3 <= 0.0):
raise ValueError('This parameter must be a Positive Number')
...
This is an acceptable (tried and true) way of parameter checking in Python, but I have to wonder: Since Python does not have a way of writing Switch-cases besides if-then-else statements, is there a more efficient or proper way to perform this task? Or is implementing long stretches of if-then-else statements my only option?
You could create a decorator function and pass the expected types and (optional) ranges as parameters. Something like this:
def typecheck(types, ranges=None):
def __f(f):
def _f(*args, **kwargs):
for a, t in zip(args, types):
if not isinstance(a, t):
raise TypeError("Expected %s got %r" % (t, a))
for a, r in zip(args, ranges or []):
if r and not r[0] <= a <= r[1]:
raise ValueError("Should be in range %r: %r" % (r, a))
return f(*args, **kwargs)
return _f
return __f
Instead of if ...: raise you could also invert the conditions and use assert, but as noted in comments those might not always be executed.
You could also extend this to allow e.g. open ranges (like (0., None)) or to accept arbitrary (lambda) functions for more specific checks.
Example:
#typecheck(types=[int, float, str], ranges=[None, (0.0, 1.0), ("a", "f")])
def foo(x, y, z):
print("called foo with ", x, y, z)
foo(10, .5, "b") # called foo with 10 0.5 b
foo([1,2,3], .5, "b") # TypeError: Expected <class 'int'>, got [1, 2, 3]
foo(1, 2.,"e") # ValueError: Should be in range (0.0, 1.0): 2.0
I think you can use decorator to check the parameters.
def parameterChecker(input,output):
... def wrapper(f):
... assert len(input) == f.func_code.co_argcount
... def newfun(*args, **kwds):
... for (a, t) in zip(args, input):
... assert isinstance(a, t), "arg {} need to match {}".format(a,t)
... res = f(*args, **kwds)
... if not isinstance(res,collections.Iterable):
... res = [res]
... for (r, t) in zip(res, output):
... assert isinstance(r, t), "output {} need to match {}".format(r,t)
... return f(*args, **kwds)
... newfun.func_name = f.func_name
... return newfun
... return wrapper
example:
#parameterChecker((int,int),(int,))
... def func(arg1, arg2):
... return '1'
func(1,2)
AssertionError: output 1 need to match <type 'int'>
func(1,'e')
AssertionError: arg e need to match <type 'int'>
This has been bugging me for a while about Python, there is no standard way to output if a provided param is None or has missing value, nor handle a JSON/Dict object gracefully,
for example I want to output the actual parameter name in the error message,
username = None
if not username:
log.error("some parameter is missing value")
There is no way to pass the actual parameter name, unless you do this artificially and messily by hard coding the parameter in the error output message, ie,
if not username:
log.error("username is missing value")
but this is both messy and prone to syntax errors, and pain in butt to maintain.
For this reason, I wrote up a "Dictator" function,
https://medium.com/#mike.reider/python-dictionaries-get-nested-value-the-sane-way-4052ab99356b
If you add your parameters into a dict, or read your parameters from a YAML or JSON config file, you can tell Dictator to raise a ValueError if a param is null,
for example,
config.yaml
skills:
sports:
- hockey
- baseball
now your py program reads in this config file and the parameters, as a JSON dict,
with open(conf_file, 'r') as f:
config = yaml.load(f)
now set your parameters and also check if theyre NULL
sports = dictator(config, "skills.sports", checknone=True)
if sports is None, it will raise a ValueError, telling you exactly what parameter is missing
ValueError("missing value for ['skills']['sports']")
you can also provide a fallback value to your parameter, so in case it is None, give it a default fallback value,
sports = dictator(config, "skills.sports", default="No sports found")
This avoids ugly Index/Value/Key error exceptions.
Its a flexible, graceful way to handle large dictionary data structures and also gives you the ability to check your program's parameters for Null values and output the actual parameter names in the error message
In C++ two functions with the same name can be created as long as the signature is different.
So for example myfunc(int x) is different from myfunc(float x).
In python you cannot do this, so, do you need to define functions with different names, or is there a better way to handle this situation?
In Python3.4+ you can use the functools.singledispatch decorator, which allows you to define a generic function and then register typed implementations against it.
From the docs
Generic function:
>>> from functools import singledispatch
>>> #singledispatch
... def fun(arg, verbose=False):
... if verbose:
... print("Let me just say,", end=" ")
... print(arg)
Typed functions:
>>> #fun.register(int)
... def _(arg, verbose=False):
... if verbose:
... print("Strength in numbers, eh?", end=" ")
... print(arg)
...
>>> #fun.register(list)
... def _(arg, verbose=False):
... if verbose:
... print("Enumerate this:")
... for i, elem in enumerate(arg):
... print(i, elem)
There's no built-in solution for earlier releases of Python, but Guido van Rossum blogged about a solution for python2 using decorators. (Edit: there is also a backport of the 3.4 functionality for pythons 2.6 - 3.3 on pypi)
Edit:
Of course, one of the advantages of using Python is that the the same code can often handle ints and floats without explicit dispatching on type, which is one of the reasons why the functionality has only recently been added.
Python doesn't really care whether an argument is an integer or a float. It's dynamically typed. You can do, for example, this:
def SquareMe(num):
return num**2
And you can call this function with any number (int, float, complex, ...).
It's also possible to do this:
def MultMe(data):
return data*2
This will work with numbers, strings (!), lists (!!), NumPy arrays and anything that can be multiplied by a number (if some class provides a method for this).
In python, you have to create only one method, but you can check what arguments can get passed, and if they are different arguments (ie: one is a float and another is an int) then you can differentiate two functions. In code this would look like:
def myfunc(*args):
# do something
# when you call the method
myfunc(a1, a2, k1=a3, k2=a4)
# you get:
args = (a1, a2)
kwds = {'k1':a3, 'k2':a4}
#So now lets recreate myfunc to check arguments
def myfunc(*args):
if isinstance(args[0], str): #This is where you determine argument type
# do what you want to do if argument is string
elif isinstance(args[1], int):
# do what you want to do if argument is an int
As ForceBru said Python dosen't realy care about parameter type , so if you do , you can handle it yourself:
def myfunc(x):
if(isinstance(x,int)):
print (x, 'int') # as myfunc(int x)
if(isinstance(x,float)):
print (x, 'float') # as myfunc(float x)
myfunc(10) # 10 int
myfunc(10.2) # 10.2 float
myfunc ("A") #
You could have the function itself do different things based on the types and number of parameters.
def f (a):
if type (a) == 'float' or type (a) == 'int':
...
if type (a) == 'list':
...
if type (a) == 'dict':
...