We Keep Coding

Type checking wrapped functions with protocol arguments

I'm working on some code where I have wrapper functions that lift functions from type T to Optional[T], but when I use a type that is a protocol, something goes wrong.
For example, I could have something like this:
from typing import (
TypeVar,
Callable as Fn,
Optional as Opt,
Protocol,
Any,
)
from functools import (
wraps
)
_T = TypeVar('_T')
# Lifting a general function of type T x T -> T
def lift_op(f: Fn[[_T, _T], _T]) -> Fn[[Opt[_T], Opt[_T]], Opt[_T]]:
"""Lift op."""
#wraps(f)
def w(x: Opt[_T], y: Opt[_T]) -> Opt[_T]:
return x if y is None else y if x is None else f(x, y)
return w
to lift an operator op: T x T -> T to Opt[T] x Opt[T] -> Opt[T]. (I've shortened Optional and Callable to Opt and Fn to get shorter lines, nothing more).
This seems to work okay, mostly, but if I have a function that works on a generic type restricted to a protocol, something breaks.
Say I have a function that needs my type to support <. I can then use the protocol
# Protocol for types supporting <
class Ordered(Protocol):
"""Types that support < comparison."""
def __lt__(self: Ord, other: Any) -> bool:
"""Determine if self is < other."""
...
Ord = TypeVar('Ord', bound=Ordered)
and define a min function as
# Min for two optionals
def min_direct(x: Opt[Ord], y: Opt[Ord]) -> Opt[Ord]:
return x if y is None else y if x is None else \
y if y < x else x # on ties choose x
and if I call it with two integers
# mypy accepts that ints are Opt[Ord]
min_direct(1, 2) # No problem here
mypy will accept that int is an Opt[Ord].
But if I use the lift function, it breaks:
#lift_op
def lift_min(x: Ord, y: Ord) -> Ord:
"""Return min of x and y."""
return y if y < x else x
# Now int is no longer an Opt[Ord]!
lift_min(1, 2) # Type error for both args.
I get the errors
error: Argument 1 to "lift_min" has incompatible type "int"; expected "Optional[Ord]"
error: Argument 2 to "lift_min" has incompatible type "int"; expected "Optional[Ord]"
So apparently int isn't an Opt[Ord] in this context.
It is fine if I write a min function specifically for int
# Lifting a function with a concrete type (int) instead
# of a protocol
#lift_op
def imin(x: int, y: int) -> int:
"""Hardwire type to int."""
return x if x <= y else y
# Again int is Opt[Ord]
imin(1, 2) # All is fine here...
or if I specify the type of the wrapped function explicitly:
# Try setting the type of the lifted function explicitly
def lift_min_(x: Ord, y: Ord) -> Ord:
"""Return min of x and y."""
return y if y < x else x
f: Fn[[Opt[Ord],Opt[Ord]], Opt[Ord]] = lift_op(lift_min_)
f(1, 2) # No problem here
I suspect that the return type of the lift_op wrapper isn't the same Fn[[Opt[Ord],Opt[Ord]],Opt[Ord]] as the type annotation for f above, but I'm not sure in what way. It isn't the wraps() call, that doesn't make a difference. But perhaps the Ord type gets bound somehow and is then interpreted differently?
I don't know, and I don't know how to figure it out. What do I need to do to make the wrapper function work so it will accept an int, say, as satisfying the protocol Opt[Ord]?
If you want the code in context, here is a playground

This looks like mypy bug, unfortunately. Your Ord type var gets resolved on decorator level, because T appears both on left and right sides of decorator def. It is partially correct (you do really want to confirm that T is the same in original and transformed functions), but makes this variable unbound. That's why your attempt with assignment works: when you do
f: Fn[[Opt[Ord], Opt[Ord]], Opt[Ord]] = lift_op(lift_min_)
you make Ord bound again. I think you should recheck mypy issue tracker and submit this bug if it was not done before (I was unable to find).
To reproduce this bug, you can use even simple type variables without restrictions:
from typing import TypeVar, Callable as Fn, Optional as Opt
_T = TypeVar('_T')
def allow_none(f: Fn[[_T], _T]) -> Fn[[Opt[_T]], Opt[_T]]:
return f # type: ignore
#allow_none
def simple(x: _T) -> _T:
return x
reveal_type(simple) # N: Revealed type is "def (Union[_T`-1, None]) -> Union[_T`-1, None]"
simple(1) # E: Argument 1 to "simple" has incompatible type "int"; expected "Optional[_T]"
Now almost any type (except for Any and specific type vars) can not be passed as simple argument, because it is not _T. Bound variables are represented like def [_T] (Union[_T`-2, None]) -> Union[_T`-2, None].
Here is related mypy issue, but it doesn't cover exactly your case, so it would be great to report this separately (it may lead to raising priority and faster fix from maintainers). This bug reproduces with pyre too, so probably I'm misunderstanding something - but it really looks odd, type variable that should be bound within Callable is not bound any more.
I have tried a few solutions involving Generic and Protocol, but none of them seem to work: type var binding in function definition is strict enough and callable with type variable from Generic behaves really odd, resolving to <nothing> in Union context (don't kill me for lowercase class naming, it is too similar to classmethod or property and left lowercase intentionally):
class _F(Protocol[_T]):
def __call__(self, __x: _T) -> _T:
...
# Fails
class allow_none(Generic[_T]):
def __call__(self, f: _F[_T], /) -> _F[_T | None]:
return f # type: ignore
reveal_type(allow_none[Ord]().__call__) # N: Revealed type is "def (__main__._F[Ord?]) -> __main__._F[None]"
# But this works for some reason
class allow_none(Generic[_T]):
def __call__(self, f: _F[_T], /) -> _F[_T]:
return f # type: ignore
reveal_type(allow_none[Ord]().__call__) # N: Revealed type is "def (__main__._F[Ord?]) -> __main__._F[Ord?]"
It is probably a bug too, I'll report it asap.
For me your initial attempt with 2-step definition looks like the best workaround, unfortunately.
I also suggest to have a look at returns library by Nikita Sobolev - this link points to Maybe container that deals with your use case. Although I don't really like it, some people consider it better alternatives to chain of is None checks. It comes with typing for such cases (almost) for free.

How to pass Union-typed values to generic function without "isinstance"?

I'm confused about how to store generics/unions in collections. Consider T below that can be bool or str. We can represent this as a bound TypeVar or as a Union. The difference matters when semantically requiring that the generic is a single type through multiple namings:
from typing import TypeVar, Union, List, NoReturn
T = TypeVar("T", bool, str)
T_Union = Union[bool, str]
def generic_function(event: T) -> T:
# must return same type as passed in
...
def union_function(event: T_Union) -> T_Union:
# can return any type in T_Union, not necessarily what was passed in
...
Now, if I want to store in a list some valid values that could be passed to these functions, I see that the list type cannot contain an unbound generic, so I store these values in a Union where each element is one of the types of T:
unbound_list_of_event_types: List[T] = [True, "foo"] # error: Type variable "__main__.T" is unbound
list_of_event_types: List[T_Union] = [True, "foo"]
But, I cannot call the generic function with the Union-typed value:
generic_function(list_of_event_types[0]) # error: Value of type variable "T" of "generic_function" cannot be "Union[bool, str]"
union_function(list_of_event_types[0])
This strikes me as odd, because if we exhaustively check the types in the Union, every function call takes the same form, which looks like needless isinstance-checking:
def assert_never(value: NoReturn) -> NoReturn: ...
# Exhaustively checking the types and calling the function with the *same signature*
if isinstance(list_of_event_types[0], bool):
generic_function(list_of_event_types[0])
elif isinstance(list_of_event_types[0], str):
generic_function(list_of_event_types[0])
else:
assert_never(list_of_event_types[0])
# Seems redundant when we could do this:
generic_function(list_of_event_types[0]) # type: ignore[type-var]
https://mypy-play.net/?mypy=0.931&python=3.10&gist=6133504b68fa1e74e844d0fc280ee42f
Is there a better way to store these values in a collection so they can be passed to generic functions?
Simon Hawe's suggestion to use Union allows this to type check:
T2 = TypeVar("T2", bound=Union[bool, str])
def generic_union_bound_function(event: T2) -> T2:
# must return same type as passed in
...
list_of_event_types: List[Union[bool, str]] = [True, "foo"]
reveal_type(generic_union_bound_function(list_of_event_types[0])) # bool | str
reveal_type(generic_union_bound_function(True)) # bool
But what if we want to use Type[T] to deduce the generic? (this example more closely looks like callback registration for various event types). I opened a mypy bug for this case: https://github.com/python/mypy/issues/12115

The problem here is that list[Union[t1,t2]] means that the type is either t1 or t2 or t1 and t2. t1 and t2 would not be a valid type for something typed with TypeVar('T', t1, t2), as this mean 't1' or 't2' but not both. In your example of bool and str, there is probably nothing that is both bool and str, but in general, you could have something like
class X:
pass
class Y:
pass
class XY(X,Y):
pass
T_U = Union[X,Y]
x : T_U = X()
y : T_U = Y()
xy : T_U = XY()
Now, what you could use for your example would be a TypeVar that is bound to the Union of bool and str. If you type a function with that, it would still resolve to input type is equal to the output type. It would additionally allow the potential type that is a type of both types from the union. This, however, is anyways not really possible for bool and str. So if you try that code
T = TypeVar("T", bound= Union[bool, str])
def some_function(e:TR) -> TR:
...
a = some_function"a")
b = some_function(1)
d = a.split() # works
c = b.split() # works error: "int" has no attribute "split"
it will exactly check for what you wanted.

How can identical types be incompatible in MyPy?

With the following example:
from typing import Callable, Generic, Type, TypeVar
XType = TypeVar('XType', bound=int)
class C(Generic[XType]):
def f(self, x_init: XType) -> XType:
return x_init
def combinator(c_cls: Type[C[XType]]) -> Callable[[C[XType], XType], XType]:
old_f = c_cls.f
def new_f(c: C[XType], x_init: XType) -> XType:
return old_f(c, x_init)
return new_f
MyPy says:
a.py:15: error: Incompatible return value type (got "XType", expected "XType")
a.py:15: error: Argument 1 has incompatible type "C[XType]"; expected "C[XType]"
a.py:15: error: Argument 2 has incompatible type "XType"; expected "XType"

I am not sure I agree with the premise of this question.
Here’s part of the docstring from 3.8
class TypeVar(_Final, _Immutable, _root=True):
"""Type variable.
Usage::
T = TypeVar('T') # Can be anything
A = TypeVar('A', str, bytes) # Must be str or bytes
....
def __init__(self, name, *constraints, bound=None,
covariant=False, contravariant=False):
....
Now, if you had just
ThetaType = TypeVar('ThetaType')
XType = TypeVar('XType')
would you be arguing that uses of ThetaType should be considered uses of XType, even though 2 different typevars were setup? Why would adding the bound optional argument automatically collapse them back together? The source does not enforce presence of bound, or any arguments beside name, in any way.
I don't think it’s typing/mypy’s job to infer your intentions in type declarations, only to check your code vs your declared type intentions. If you mean them to be the same then declare only 1 TypeVar. Considering them the same could lose some semantic meaning if you had actual reasons to have 2.
I’ll add to that bound allows more flexibility than constraints as it matches on subclasses. Let’s say you’ve user-defined 4 subclasses of int. Int1(int), Int2, Int3, Int4.... Now you’ve decided to partition your code where some of it should only accept Int1 and Int2. Typevarint12 could somewhat express that, even though your subclasses all match bound=int.

I support #JL Peyret's point that this is intended behavior.
A couple of additional thoughts:
this isn't specific to TypeVar: if you create two identical classes which have different names, mypy will raise an error if you use them interchangeably
if I understand what you're trying to do, you should be using NewType instead of TypeVar.
The point of using TypeVar combined with Generic is to be able to tell mypy (or anyone reading code really) "the same type will go in and out. It might be of type int, float, ...".
Example:
from typing import Generic, TypeVar
T = TypeVar("T", int, str)
def double(value: T) -> T:
# functional version
return value * 2
class Doubler(Generic[T]):
# class version - not sure why someone would do this,
# it's just an example.
def __init__(self, value: T):
# tip: use dataclasses or attrs instead
self.value = value
#property
def double(self) -> T:
return self.value * 2 # note this works on ints and strings
I think what you're trying to do is actually having types that are integers (and only integers), but you want to be able to track them independently to avoid accidentally switching them around:
from typing import NewType, Tuple
ThetaType = NewType("ThetaType", int)
XType = NewType("XType", int)
def switch_arguments_order(theta: ThetaType, x: XType) -> Tuple[XType, ThetaType]:
# If you accidentally got the wrong order here, mypy would catch it.
# You can also think of a function that only returns one of the two,
# if you accidentally return the wrong one mypy will catch that.
return x, theta
Now, if you want any piece of logic to handle ThetaType or Xtype without making a distinction, then you can have it return int, and mypy won't complain.
def mypy_passes(x: Xtype) -> int:
return x
def mypy_fails(n: int) -> Xtype:
return n
def mypy_passes_with_wrapping(n: int) -> Xtype:
return Xtype(n)

Why does mypy ignore a generic-typed variable that contains a type incompatible with the TypeVar?

Below I define type variable, generic type alias, and a dot product function. mypy doesn't raise an error. Why not?
I would expect it to raise an error for v3 because it's a vector of strings and I've specified that T must be an int, float, or complex.
from typing import Any, Iterable, Tuple, TypeVar
T = TypeVar('T', int, float, complex)
Vector = Iterable[T]
def dot_product(a: Vector[T], b: Vector[T]) -> T:
return sum(x * y for x, y in zip(a, b))
v1: Vector[int] = [] # same as Iterable[int], OK
v2: Vector[float] = [] # same as Iterable[float], OK
v3: Vector[str] = [] # no error - why not?

I think the problem here is that when you're constructing type alias, you're not actually constructing a new type -- you're just giving a nickname or alternate spelling to an existing one.
And if all you're doing is providing an alternative spelling to a type, that means that it it ought to be impossible to add any extra behavior while doing so. That's exactly what's happening here: you're trying to add additional information (your three type constraints) to Iterable, and mypy is ignoring them. There's a note saying basically this at the bottom of the mypy docs on generic type aliases.
The fact that mypy is just silently using your TypeVar without warning that its additional constraints are being ignored feels like a bug, actually. Specifically, it feels like a usability bug: Mypy ought to have raised a warning here and disallowed using anything other then unrestricted typevars inside your type alias.
So what can you do to type your code?
Well, one clean solution would be to not bother creating the Vector type alias -- or create it, but not worry about constraining what it can be parameterized with.
This means a user can create a Vector[str] (aka an Iterable[str]), but that's really no big deal: they'll get a type error the moment they try actually passing it into any function like your dot_product function that does use type aliases.
A second solution would be to create a custom vector subclass. If you do so, you'd be creating a new type and so can actually add new constraints -- but you'd no longer be able to pass lists and such directly into your dot_product classes: you'd need to wrap them in your custom Vector class.
This can be a little clunky, but you may end up drifting to this solution anyways: it gives you the opportunity to add custom methods to your new Vector class, which could perhaps help improve the overall readability of your code, depending on what exactly you're doing.
The third and final solution is to define a custom "Vector" Protocol. This would let us avoid having to wrap our lists in some custom class -- and we're creating a new type so we can add whatever constraints we want. For example:
from typing import Iterable, TypeVar, Iterator, List
from typing_extensions import Protocol
T = TypeVar('T', int, float, complex)
# Note: "class Vector(Protocol[T])" here means the same thing as
# "class Vector(Protocol, Generic[T])".
class Vector(Protocol[T]):
# Any object that implements these three methods with a compatible signature
# is considered to be compatible with "Vector".
def __iter__(self) -> Iterator[T]: ...
def __getitem__(self, idx: int) -> T: ...
def __setitem__(self, idx: int, val: T) -> None: ...
def dot_product(a: Vector[T], b: Vector[T]) -> T:
return sum(x * y for x, y in zip(a, b))
v1: Vector[int] = [] # OK: List[int] is compatible with Vector[int]
v2: Vector[float] = [] # OK: List[float] is compatible with Vector[int]
v3: Vector[str] = [] # Error: Value of type variable "T" of "Vector" cannot be "str"
dot_product(v3, v3) # Error: Value of type variable "T" of "dot_product" cannot be "str"
nums: List[int] = [1, 2, 3]
dot_product(nums, nums) # OK: List[int] is compatible with Vector[int]
The main disadvantage to this approach is that you can't really add any methods with actual logic to your protocol that you can reuse between anything that might be considered a "Vector". (Well, you sort of can, but not in any way that'll be useful in your example).

Unwrapping a list or map as function arguments in Scala

Is it possible to dynamically unwrap a list/tuple/map items as arguments to a function in Scala? I am looking for a Scala equivalent of Python's args/kwargs.
For instance, in Python if a function is defined as def foo(bar1, bar2, bar3=None, bar4=1) then given a list x=[1,7] and a dictionary y={'bar3':True, 'bar4':9} you can call foo as foo(*x, **y).

Just to be clear, the following is valid Python code:
def foo(bar1, bar2, bar3=None, bar4=1): print("bar1="+str(bar1)+" bar2="+str(bar2)+" bar3="+str(bar3)+" bar4="+str(bar4))
x=[1,7]
y={'bar3':True, 'bar4':9}
foo(*x,**y)
However, there is no analogous Scala syntax. There are some similar things, but the main reason this is never going to be possible is that it would violate the compile-time type checking that Scala requires. Let's look more closely.
The reasons
First, think about the varargs portion. Here you want to be able to pass in an arbitrary-length list of arguments and have it fill in the relevant function parameters. This will never work in Scala because the type checker requires that the parameters passed into a function be valid. In your scenario, foo() can accept a parameter list x of length two, but no less. But since any Seq can have an arbitrary number of parameters, how would the type checker know that the x being pass it is valid at compile time?
Second, think about the keywword arguments. Here you are asking for the function to accept an arbitrary Map of arguments and values. But you get the same problem: How can the compile-time type checker know that you are passing in all of the necessary arguments? Or, further, that they are the right types? After all, they example you give is a Map containing both a Boolean and an Int, which would have the type Map[String, Any], so how would the type checker know that this would match your parameter types?
Some solutions
Scala's varargs
You can do some similar things, but not this exactly. For example, if you defined your function to explicitly use varargs, you can pass in a Seq:
def foo(bar1: Int*) = println(f"bar1=$bar1")
val x = Seq(1, 2)
foo(x:_*)
This works because Scala knows that it only needs a sequence of zero or more arguments, and a Seq will always contain zero or more items, so it matches. Further, it only works if the types match as well; here it's expecting a sequence of Ints, and gets it.
tupled
The other thing you can do is to pass in a tuple of arguments:
def foo(bar1: Int, bar2: Int, bar3: Boolean = false, bar4: Int = 1) = println(f"bar1=$bar1 bar2=$bar2 bar3=$bar3 bar4=$bar4")
val x = (1, 2, true, 9)
(foo _).tupled(x)
Again, this works because Scala's type checker can verify that the arguments are valid. The function requires four arguments, of types Int, Int, Boolean, and Int, and since a tuple in Scala has a fixed length and known (and possibly different) types for each position, the type-checker can verify that the arguments match the expected parameters.

Sort of an edge case for the OQ but if you want to pass a Map of arguments for a case class this seems to work:
scala> case class myCC(foo: String = "bar", negInt: Int = -1)
scala> val row = myCC()
scala> println(row)
myCC(bar,-1)
scala> val overrides = Map("foo" -> "baz")
scala> row.getClass.getDeclaredFields foreach { f =>
f.setAccessible(true)
overrides.foreach{case (k,v) => if (k == f.getName) f.set(row, v)}
}
scala> println(row)
myCC(baz,-1)
(borrowed from Scala: How to access a class property dynamically by name?)

Original answers don't mention handling Map as list of pairs - it can be easily converted to map (even -> operator is just shorthand for pair).
def parse(options: (String, String)*) = println (options.toMap)

You can use varargs syntax:
def printAll(strings: String*) {
strings.map(println)
}
No you can use this function so:
printAll("foo")
so:
printAll("foo", "bar")
or so:
printAll("foo", "bar", "baz")