We Keep Coding

How to handle different input types with the same actions pythonically?

I have written some code that allow both torch.Tensor and np.ndarray data types.
The functions will perform the same operations but since they are both different APIs for some code, I need to always do an isinstance check in order to perform the operation.
An example:
import torch
import numpy as np
from typing import Union
torch_tensor = torch.tensor([1, 2, 3])
numpy_array = np.array([1, 2, 3])
def expand(inputs: Union[torch.Tensor, np.ndarray]) -> Union[torch.Tensor, np.ndarray]:
if isinstance(inputs, torch.Tensor):
return inputs.unsqueeze(0)
elif isinstance(inputs, np.ndarray):
return np.expand_dims(inputs, 0)
else:
raise TypeError("inputs must be either a torch.Tensor or a numpy.ndarray")
One or two functions with this kind of isinstance check is fine, but it is not very "neat" when I add in more functions with the same kind of interface as expand. Any tips or design that could handle this more elegantly, so that I do not need to perform the isinstance checks in every function.

Expanding on my comment, instead of
from typing import Union
import random
def add_to_a(value: Union[int, str]) -> Union[int, str]:
if isinstance(value, int):
return value + 1
elif isinstance(value, str):
return str(int(value) + 1)
a = random.choice([0, "0"])
print(add_to_a(a))
I would structure the code to something alike this, where each type has a separate pipeline starting from the top level when the variable enters.
import random
def add_to_int_a(value: int) -> int:
return value + 1
def add_to_str_a(value: str) -> str:
return str(int(value) + 1)
a = random.choice([0, "0"])
if isinstance(a, int):
print(add_to_int_a(a))
# Other int type processing
elif isinstance(a, str):
print(add_to_str_a(a))
# Other str type processing
This is the case mostly if there are going to be more of these "Union" functions in the processing pipeline. For one Union here and other there it doesn't really matter personally.
This comes down to DRY (Don't repeat yourself) with the abundance of isinstance checks in the processing functions if we're following the first example with every operation.
Also worth noting, it could be worthwile to construct classes to help differentiate and group the functionalities if it helps the readability at all.

Python typing for function that returns None if list arg only contains None

I'm working with a function that is a bit like this (super simplified, as an example):
def foo(*stuff: None | int):
stuff_not_none = [x for x in stuff if x is not None]
if len(stuff_not_none) is 0:
return None
return sum(stuff_not_none)
If I call the function using:
foo(*[1, 2, 3]), I'd want the return type to be inferred to int.
foo(*[None, None]), I'd want the return type to be inferred to None.
foo(*[1, None]), the dream would be inferred to int, but ok if None | int.
I've tried with generics / overloads, but I couldn't figure out this puzzle. How can I achieve this?

The solution:
from typing import overload
#overload
def foo(*stuff: None) -> None: ... # type: ignore[misc]
#overload
def foo(*stuff: int | None) -> int: ...
def foo(*stuff: int | None) -> int | None:
stuff_not_none = [x for x in stuff if x is not None]
if len(stuff_not_none) is 0:
return None
return sum(stuff_not_none)
reveal_type(foo(None, None)) # revealed type is None
reveal_type(foo(1, 2, 3)) # revealed type is int
reveal_type(foo(None, 2, None, 4)) # revealed type is int
foo('a', 'b') # error: no matching overload
Mypy hates this kind of thing, because the overloads overlap. But you'll find that if you add a type: ignore comment in the right place, it's perfectly able to infer the correct types anyway. (I'm a typeshed maintainer, and we do this kind of thing at typeshed all the time.)
Note that the order of the overloads is very important: type checkers will always try the first overload first, and then, only if that doesn't match, will they try the second overload. This is how we get the int revealed type when we pass in a mixture of ints and Nones: the first overload doesn't match, because of the presence of ints, so the type checker is forced to try the second overload.
Mypy playground demo: https://mypy-play.net/?mypy=latest&python=3.10&gist=ff07808e0a314208fdfa6291dcf9f717

Can mypy recognise custom unpacking?

I have a class that provides __getitem__ - which python is happy to use for unpacking, but when I run mypy on the code I get List or tuple expected as variable arguments.
Here's a minimal reproducer
from typing import Any
class Foo:
def __getitem__(self, idx: int) -> Any:
if idx == 0:
return 1
if idx == 1:
return "bye"
else:
raise IndexError
f = Foo()
t = ("hello", *f)
print(t) # prints ("hello", 1, "bye")
I don't want to have to add an error suppression to each point that I do *f, that defeats the whole purpose of the class.
Is there some way to make mypy understand that unpacking a Foo is OK?
If it matters, I'm currently using mypy 0.800, and Python 3.7.6.

It looks like MyPy is expecting unpackable objects to have an __iter__ method — which is fair enough, in a way, since it's fairly rare for an object to implement __getitem__ and not implement __iter__. You can get the MyPy error to go away through a little bit of lying: tell MyPy there's an __iter__ method even though you have no intention of implementing one. Seems to work on python 3.7/MyPy 0.800 as well as python 3.10/MyPy 0.910.
from typing import Any, Callable, Iterator
class Foo:
__iter__: Callable[["Foo"], Iterator[Any]]
def __getitem__(self, idx: int) -> Any:
if idx == 0:
return 1
if idx == 1:
return "bye"
else:
raise IndexError
f = Foo()
t = ("hello", *f)
print(t) # prints ("hello", 1, "bye")

Python Typing: Validation Decorator for Literal typed Arguments

Often I encounter the scenario of functions which accept a finite set of values only. I know how to reflect this behavior in the type annotations, using typing.Literal like so:
import typing
def func(a: typing.Literal['foo', 'bar']):
pass
I would like to have a decorator #validate_literals which validates that the parameters to the are consistent with their type:
#validate_literals
def picky_typed_function(
binary: typing.Literal[0, 1],
char: typing.Literal['a', 'b']
) -> None:
pass
so that the input is validated against the restrictions defined by the arguments's types, and a ValueError is raised in case of a violation:
picky_typed_function(0, 'a') # should pass
picky_typed_function(2, 'a') # should raise "ValueError: binary must be one of (0, 1)')"
picky_typed_function(0, 'c') # should raise "ValueError: char must be one of ('a', 'b')"
picky_typed_function(0, char='c') # should raise "ValueError: char must be one of ('a', 'b')"
picky_typed_function(binary=2, char='c') # should raise "ValueError: binary must be one of (0, 1)"
typing type checks are designed to be static, and not happen during runtime. How can I leverage the typing definition for runtime validation?

We can inspect the decorated (validated) function's signature by using inspect.signature, check which of the parameters of the function is typed as a Literal alias by getting the "origin" of the parameter's annotation through typing.get_origin() (or, for python versions < 3.8, using __origin__) and retrieve the valid values by using [typing.get_args()] (https://stackoverflow.com/a/64522240/3566606) (and iterating recursively over nested Literal definitions) from the Literal alias.
In order to do that, all that is left to do, is to figure out which parameters have been passed as positional arguments and map the corresponding values to the parameter's name, so the value can be compared against the valid values of the parameter.
Finally, we build the decorator using the standard recipe with functools.wraps. In the end, this is the code:
import inspect
import typing
import functools
def args_to_kwargs(func: typing.Callable, *args: list, **kwargs: dict) -> dict:
args_dict = {
list(inspect.signature(func).parameters.keys())[i]: arg
for i, arg in enumerate(args)
}
return {**args_dict, **kwargs}
def valid_args_from_literal(annotation: _GenericAlias) -> Set[Any]:
args = get_args(annotation)
valid_values = []
for arg in args:
if typing.get_origin(annotation) is Literal:
valid_values += valid_args_from_literal(arg)
else:
valid_values += [arg]
return set(valid_values)
def validate_literals(func: typing.Callable) -> typing.Callable:
#functools.wraps(func)
def validated(*args, **kwargs):
kwargs = args_to_kwargs(func, *args, **kwargs)
for name, parameter in inspect.signature(func).parameters.items():
# use parameter.annotation.__origin__ for Python versions < 3.8
if typing.get_origin(parameter.annotation) is typing.Literal:
valid_values = valid_args_from_literal(parameter.annotation)
if kwargs[name] not in valid_values:
raise ValueError(
f"Argument '{name}' must be one of {valid_values}"
)
return func(**kwargs)
return validated
This gives the results specified in the question.
I have also published the alpha version of a python package runtime-typing to perform runtime typechecking: https://pypi.org/project/runtime-typing/ (documentation:https://runtime-typing.readthedocs.io) which handles more cases than just typing.Literal, such as typing.TypeVar and typing.Union.

from typing import Literal
from valdec.dec import validate
#validate
def func(a: Literal["foo", "bar"]) -> str:
return a
assert func("bar") == "bar"
#validate("return", exclude=True)
def func(binary: Literal[0, 1], char: Literal["a", "b"]):
return binary, char
assert func(0, "a") == (0, "a")
func(2, "x")
# valdec.utils.ValidationArgumentsError: Validation error <class
# 'valdec.val_pydantic.ValidationError'>: 2 validation errors for argument
# with the name of:
# binary
# unexpected value; permitted: 0, 1 (type=value_error.const; given=2;
# permitted=(0, 1))
# char
# unexpected value; permitted: 'a', 'b' (type=value_error.const; given=x;
# permitted=('a', 'b')).
valdec: https://github.com/EvgeniyBurdin/valdec

Complex variable arguments checking at runtime

Let's consider a function like this:
def f(*args, **kwargs):
...
*args are a variable number of arguments that must be:
(an object, a float, a float), possibly repeated N times,
followed by 0 to N' objects.
For example this is a valid call:
f(my_obj, 0, 1, other_obj, 2, 3, obj3, obj4)
but this is invalid:
f(my_obj, other_obj, 2, 3)
This function is exposed to users through a Python shell.
So, there is value in checking user input -- I am using
the typeguard library that works with type annotations
(like mypy).
I am trying to use typing module to write the proper annotations...
I thought I could at least express the constraint on the groups of 3 args
like this:
#typeguard.typechecked
f(*args:Tuple[Any,float,float])
But it does not work.
And in anyways I have no idea how to add the constraint on the following objects.
Of course I can craft myself some code to check arguments, but I am sure something better exists for cases of complex variable arguments sequences (either a clever use of the typing module or another Python lib ?)

What I meant by making the validation:
def _validate(a, b, c):
assert isinstance(b, float), f"{b} is not a float!"
assert isinstance(c, float), f"{c} is not a float"
def _validate_args(args):
if (len(args) % 3 != 0): # wrong number of args
raise ValueError("Arguments must be passed in pack of 3")
for idx in range(0, len(args), 3):
a, b, c = args[idx: idx + 3]
_validate(a, b, c)
def func(*args, **kwargs):
_validate_args(args)
func(1, 2.0, 3, 1, 2, 3)
AssertionError: 3 is not a float
You can make any message you want.