I am trying to figure out how can I provide type hints for a dictionary argument being passed to a function without using Dict[str, str] as that doesnt provide what the keys will be.
I have tried two approaches so far, one with using typing_extensions so that I can have compatibility with 3.6, and also with pydantic, but I cant get the hits to show.
Consider this example code:
from typing_extensions import TypedDict
from pydantic import BaseModel
class Some(TypedDict):
"""keya is some key"""
keya: str
"""another_key is another key"""
another_key: str
def some(a: Some) -> None:
print(a.get('keya'))
return None
some({'keya': 'key', 'another_key': 'nonething'})
As expected, the type hints for the some function shows the type Some, but not its keys.
What I am trying to accomplish is 2 things.
Provide key hints when the function argument is a dict (most important)
Generate documentation with sphinx so that the keys reflect in the documentation.
Edit
As on of the comments pointed out, I can accomplish this with **kwargs to some extent, but that is not the intention. Setting **kwargs does not give me type hints either.
I think in this case, it might actually be better to file a feature request/submit a pull request to your editor improving the quality of its type hinting. Similarly, with sphinx, you could submit a pull request that ensures that the docs either properly link to the definition of Some or include a more detailed description within the function signature itself.
After all, the problem you're facing is a limitation of your editor/sphinx, not with Python or type hints, and you might get better long-term results if you tackle the problem at the source.
You may also get better results if you use the "constructor" of Some instead of passing in a dict literal. At least for me, doing this gets me full key hints for Some when using PyCharm. Not sure if that'll also be the case for your editor:
some(Some(keya='key', another_key='nonething'))
Note that doing Some(...) will actually just return a plain old regular dict at runtime, so this shouldn't lead to any difference in behavior.
It could also be worth trying to do:
x: Some = {
"keya": "key",
"another_key": "nonething",
}
some(x)
...to see if your editor might give better hints with that form.
I have somewhat narrowed down to a possible solution using the following code. It meets most of the requirements:
mypy type checks passes
shows keys
documentation also shows keys and types
The main caviate to this solution is that mypy thinks the value is optional because of a default value so the solution is not quite complete.
Using the validate_items function, I can validate that the values are there. Please see the comments on the code snippet and offer suggestions.
from typing_extensions import TypedDict
from typing import Union
class Some(TypedDict):
keya: str
another_key: str
def validate_items(v: dict) -> None:
for key, value in v.items():
assert isinstance(value,str), '{} is required'.format(key)
# Would love to pull the type of a key from the Some class here or
# would love to put the validation in the Some class itself
def some(a: Some={'keya': '', 'another_key': ''}) -> None:
"""[summary]
Args:
a (Some, optional): [description]. Defaults to {'keya': '', 'another_key': ''}.
Returns:
[type]: [description]
"""
validate_items(dict(a))
print(a.get('keya'))
return None
In the screenshot, I can see that mypy is complaining about the None value which is expected, and in the popup help, we can also see the keys that are required in the dictionary being passed along with the type that is being set to it.
The solution feels quite hacky, and would appreciate any corrections to make it more pythonic.
Related
I'm new to Python development, and trying to get a fix on things. I am using Pycharm for development. I am currently trying to annotate the types of variables for easier access with autocompletion and suggestions. I've tried iterations of the code with mixed results.
This is the code that has a problem:
path = os.path.dirname(os.path.realpath(__file__)) # type: str
components = path.split(os.sep) # type: list[str]
The first problem displayed is at the opening brace of the type annotation of the second line. It says:
Class 'type' does not define '__getitem__', so the '[]' operator cannot be used on its instances.
I've googled around, and although the problem seems clear, opening the code for the list class clearly shows a method __getitem__:
class list(object):
"""
list() -> new empty list
list(iterable) -> new list initialized from iterable's items
"""
....
def __getitem__(self, y): # real signature unknown; restored from __doc__
""" x.__getitem__(y) <==> x[y] """
pass
Okay, maybe that's not trivial to understand and there's some other loading mechanism at work. And besides, the "problem" seems to be that I used list[str] and not List[str]. So I altered the code:
path = os.path.dirname(os.path.realpath(__file__)) # type: str
components = path.split(os.sep) # type: List[str]
Now everything breaks: The second line now complains about this:
Expected type 'List[str]', got 'List[str]' instead`
The previous problem regarding __getitem__ yet persists.
Is there a way to annotate these variables without causing problems for the checker? I am not very happy with the Python documentation in that regard, not explicitly stating the return types of their built-in methods. I have to rely on the information Pycharm provides in the documentation pop-ups (Ctrl+q).
It was a right solution to use List[str] instead of list[str] since builtin types could not be used in type hints, the corresponding PEP has not been accepted yet.
From what module do you import List? I'm unable to reproduce an issue in 2019.3.
Let's say I have a method like the following
def validate(self, item:dict, attrs:dict)-> list:
If I want to be more specific and tell that my return type is a list of ValidationMessages?
How should I / Can I achieve that?
(I would check off the mark as a duplicate, since this is not about extending a list or flattening) I'm asking about how to be more specific on identifying the return type of a method...
With Python 3.6, the built-in typing package will do the job.
from typing import List
def validate(self, item:dict, attrs:dict)-> List[str]:
...
The notation is a bit weird, since it uses brackets but works out pretty well.
Edit: With the new 3.9 version of Python, you can annotate types without importing from the typing module. The only difference is that you use real type names instead of defined types in the typing module.
def validate(self, item:dict, attrs:dict)-> list[str]:
...
NOTE: Type hints are just hints that help IDE. Those types are not enforced. You can add a hint for a variable as str and set an int to it like this:
a:str = 'variable hinted as str'
a = 5 # See, we can set an int
Your IDE will warn you but you will still be able to run the code. Because those are just hints. Python is not a type strict language. Instead, it employs dynamic typing.
I have a function that accepts a class that derives from NamedTuple and converts it into a schema. However when I run MyPy on the following code it fails with Argument 1 to "to_schema" has incompatible type "Type[Foo]"; expected "Type[NamedTuple]"
from typing import NamedTuple, Type
def to_schema(named_tuple: Type[NamedTuple]):
pass
class Foo(NamedTuple):
pass
to_schema(Foo)
Is there a way to properly type the code so that it typechecks with MyPy?
Edit:
Python documentation states that Type[Foo] accepts any subclasses of Foo (https://docs.python.org/3/library/typing.html#typing.Type). I have multiple subclasses of NamedTuple, for entities in our data model, so I'm looking for a way to annotate the function in a way that would typecheck.
The root issue with your code is that NamedTuple is not an actual type -- it's actually just a special sort of "type constructor" that synthesizes an entirely new class and type. E.g. if you try printing out the value of Foo.__mro__, you'll see (<class '__main__.Foo'>, <class 'tuple'>, <class 'object'>) -- NamedTuple is not present there at all.
That means that NamedTuple isn't actually a valid type to use at all -- in that regard, it's actually a little surprising to me that mypy just silently lets you construct Type[NamedTuple] to begin with.
To work around this, you have several potential approaches:
Rather then using Type[NamedTuple], use either Type[tuple] or Type[Tuple[Any]].
Your Foo genuinely is a subtype of a tuple, after all.
If you need methods or fields that are specifically present only in namedtuples, use a custom protocol. For example, if you particularly need the _asdict method in namedtuples, you could do:
from typing_extensions import Protocol
class NamedTupleProto(Protocol):
def _asdict(self) -> Dict[str, Any]: ...
def to_schema(x: Type[NamedTupleProto]) -> None: pass
class Foo(NamedTuple):
pass
to_schema(Foo)
Note that you will need to install the typing_extensions third party library to use this, though there are plans to formalize Protocols and add it to Python at some point. (I forget if the plan was Python 3.7 or 3.8).
Add a type ignore or a cast on the call to to_schema to silence mypy. This isn't the greatest solution, but is also the quickest.
For related discussion, see this issue. Basically, there's consensus on the mypy team that somebody ought to do something about this NamedTuple thing, whether it's by adding an error message or by adding an officially sanctioned protocol, but I think people are too busy with other tasks/bugs to push this forward. (So if you're bored and looking for something to do...)
We have a method in our Python 3.5 application where one of the in-parameters (new_value in the example below) can be of any type, and we're wondering if there is a type hint that we can use for this case?
def update(self, col_name: str, new_value) -> None:
(We would like the type hint for documentation purposes, to make the code easier to read)
Grateful for help!
Depending on what exactly you want to use, there are two different options:
If you want to basically opt-out of type-checking any_value completely and indicate it could be literally any type with zero restrictions, use typing.Any. Example:
from typing import Any
class MyThing:
def update(self, col_name: str, new_value: Any) -> None:
# Note: this typechecks; new_value can be anything, and
# that object might have a foo method
new_value.foo()
# ...snip...
If you want to indicate that new_value can be of any type, but also want to ensure that the update method only ever uses new_value in a fully typesafe way, I would use object, which is the base type of every type in Python:
class MyThing:
def update(self, col_name: str, new_value: object) -> None:
# Note: this does not typecheck since new_value is of
# type 'object', and 'object' is not guaranteed to have
# a method named 'foo'
new_value.foo()
# ...snip...
I personally bias towards using object -- Any is designed specifically as a "bridge" that lets you mix the typed and untyped worlds within your program. I personally think it's easier to reason about code if you keep those two worlds fairly distinct (e.g. by having a fully typed "core" possibly with a bunch of untyped code wrapping it) instead of a method that's both statically and dynamically typed.
Of course, it's not always possible to do this...
(We would like the type hint for documentation purposes, to make the code easier to read)
As an aside/as a suggestion, if you're going to use type hints, I would also strongly recommend going all the way and actually type-check your code using tools like mypy as a part of your development process.
Using type hints for documentation is great, but it can be very confusing if your code doesn't conform to the documentation. Since we have tools to automatically typecheck our code using those type hints, we might as well (and gain that extra guarantee).
(Pragmatically, trying to typecheck your entire codebase all at once can lead to a lot of noise -- what you can try and do instead is progressively typecheck your codebase. For example, you can configure mypy to typecheck only a set list of files (that you maybe grow over time?), make certain checks stricter or looser (perhaps on a per-file basis), and so forth. This works well with the "typed core, untyped wrapper" strategy mentioned above.)
I have a helper function that converts a %Y-%m-%d %H:%M:%S-formatted string to a datetime.datetime:
def ymdt_to_datetime(ymdt: str) -> datetime.datetime:
return datetime.datetime.strptime(ymdt, '%Y-%m-%d %H:%M:%S')
I can validate the ymdt format in the function itself, but it'd be more useful to have a custom object to use as a type hint for the argument, something like
from typing import NewType, Pattern
ymdt_pattern = '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]'
YmdString = NewType('YmdString', Pattern[ymdt_pattern])
def ymdt_to_datetime(ymdt: YmdString)...
Am I going down the wrong rabbit hole? Should this be an issue in mypy or someplace? Or can this be accomplished with the current type hint implementation (3.61)?
There currently is no way for types to statically verify that your string matches a precise format, unfortunately. This is partially because checking at compile time the exact values a given variable can hold is exceedingly difficult to implement (and in fact, is NP-hard in some cases), and partially because the problem becomes impossible in the face of things like user input. As a result, it's unlikely that this feature will be added to either mypy or the Python typing ecosystem in the near future, if at all.
One potential workaround would be to leverage NewType, and carefully control when exactly you construct a string of that format. That is, you could do:
from typing import NewType
YmdString = NewType('YmdString', str)
def datetime_to_ymd(d: datetime) -> YmdString:
# Do conversion here
return YmdStr(s)
def verify_is_ymd(s: str) -> YmdString:
# Runtime validation checks here
return YmdString(s)
If you use only functions like these to introduce values of type YmdString and do testing to confirm that your 'constructor functions' are working perfectly, you can more or less safely distinguish between strings and YmdString at compile time. You'd then want to design your program to minimize how frequently you call these functions to avoid incurring unnecessary overhead, but hopefully, that won't be too onerous to do.
Using type-hints does nothing in Python and acts as an indication of the type in static checkers. It is not meant to perform any actions, merely annotate a type.
You can't do any validation, all you can do, with type-hints and a checker, is make sure the argument passed in is actually of type str.
🌷🌷🌷🌷🌷
Okay, here we are five years later and the answer is now yes, at least if you're willing to take a third-party library on board and decorate the functions you want to be checked at runtime:
$ pip install beartype
import re
from typing import Annotated # python 3.9+
from beartype import beartype
from beartype.vale import Is
YtdString = Annotated[str, Is[lambda string: re.match('[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]', string) is not None]]
#beartype
def just_print_it(ytd_string: YtdString) -> None:
print(ytd_string)
> just_print_it("hey")
BeartypeCallHintParamViolation: #beartyped just_print_it() parameter ytd_string='hey' violates type hint typing.Annotated[str, Is[<lambda>]], as 'hey' violates validator Is[<lambda>]:
False == Is[<lambda>].
> just_print_it("2022-12-23 09:09:23")
2022-12-23 09:09:23
> just_print_it("2022-12-23 09:09:2")
BeartypeCallHintParamViolation: #beartyped just_print_it() parameter ytd_string='2022-12-23 09:09:2' violates type hint typing.Annotated[str, Is[<lambda>]], as '2022-12-23 09:09:2' violates validator Is[<lambda>]:
False == Is[<lambda>].
Please note that I'm using the very imperfect regex pattern I originally included in the question, not production-ready.
Then, a hopeful note: The maintainer of beartype is hard at work on an automagical import hook which will eliminate the need for decorating functions in order to achieve the above.