I'm adding sphinx documentation to a couple of projects, but I'm concerned that down the line future developers aren't going to be rigorous in updating documentation when the method interface changes. Is there a way to have sphinx validate the list of arguments being described against what is actually specified in the method definition?
For example, let's say I set up sphinx using sphinx-quickstart, added autodoc, and am trying to build the following "module":
def product(arg_one, arg_two):
'''
:param arg_one: an object to be multiplied by arg_two
:type arg_one: Object
:param arg_two: an integer which defines the number of arg_one to return
:type arg_two: integer
:returns: The product of arg_one and arg_two
:raises: TypeError
'''
return arg_one * arg_two
And, a week from now, Bob Doe updates product as follows:
def product(value, number, catch_errors=False):
'''
:param arg_one: an object to be multiplied by arg_two
:type arg_one: Object
:param arg_two: an integer which defines the number of arg_one to return
:type arg_two: integer
:returns: The product of arg_one and arg_two
:raises: TypeError
'''
try:
return value * number
except TypeError as exc:
if catch_errors:
return None
raise
Now, the sphinx documentation isn't correct - it's missing the new catch_errors field and the variables have been renamed. But, running
sphinx-build . ./_build
a second time doesn't catch the problems - it just reports
Running Sphinx v1.6.5
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
build succeeded.
I would expect (or hope) that Sphinx is capable of validating the docstring against the actual code of the method, and failing (or at least indicating in some way) when the documentation and the implementation aren't in-sync.
If sphinx isn't capable of this, is there an alternative which is? I know there's function annotations in Python 3, but we're currently supporting both Python 2 and Python 3, so this isn't an option.
After some looking, I think the closest tool I could find easily is darglint but it was recently put into archive-mode on Github (December 2022) so long term support is not guaranteed or even likely.
One option, inspired by this discussion from torchvision is to use Sphinx-Napoleon and pydocstyle with almost everything turned off, except D417 which is a check for missing arguments in the docstring.
So pydocstyle can check for missing arguments but it cannot currently check for extra arguments.
Edit: seems pylint can do this with an extension https://pylint.pycqa.org/en/latest/user_guide/checkers/extensions.html#pylint-extensions-docparams but there is some functional overlap between the various linters and style checkers so you might need to just pick which ones you go with.
For catching re-named arguments, you could also enforce 100% coverage in your tests, but that wouldn't catch refactors to non-keyword arguments.
Of some other packages available:
doctest is good for checking if your docstring examples run correctly, but it doesn't do any signature validation. So if you set a default parameter like the example in the question, it will still pass doctest. If you used keyword arguments for all your examples, you could use this as a check, but it's a bit verbose.
interrogate is capable of checking docstring coverage, but not much else (as far as I can tell?) I'm not sure whether other tools actually check coverage though, so this is potentially a good tool to use in tandem.
Related
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints
In JetBrain's example for docstring in Pycharm, it is mentioned that:
Note that for reStructuredText it's possible to specify types in two formats:
:param param_type param_name: parameter description (type description is on the same line as the parameter description).
:type param_name: param_type (type description is on a separate line)
I want to make my pycharm autogenerate the second variant, not the first one?
Currently, it is even not even adding the type
def foo(a: str, b: int = 1) -> str:
"""
:param a:
:param b:
:return:
"""
return a + str(b)
a related question is: How to customize docstring generation in pycharm and share the template through git?
Currently, in spite of the documentation mentioning both formats, it appears it is not possible to switch between the two styles. https://youtrack.jetbrains.com/issue/PY-12327
You can enable the second style (which I think you want) by checking the following option:
Editor - General - Smart Keys - Python check Insert type placeholders in the documentation comment stub. (or use the Settings search and search for comment stub)
Very unintuitive and many people have been requesting improvements to this functionality in PyCharm. Ideally, you'd want to be able to provide a custom sphinx.mustache docstring template of this style:
{{! Sphinx Docstring Template }}
{{summaryPlaceholder}}
{{extendedSummaryPlaceholder}}
{{#args}}
:param {{var}}: {{descriptionPlaceholder}}
{{/args}}
{{#kwargs}}
:param {{var}}: {{descriptionPlaceholder}}
{{/kwargs}}
{{#exceptions}}
:raises {{type}}: {{descriptionPlaceholder}}
{{/exceptions}}
{{#returns}}
:return: {{descriptionPlaceholder}}
{{/returns}}
{{#yields}}
:yield: {{descriptionPlaceholder}}
{{/yields}}
(example taken from here https://github.com/executablebooks/markdown-it-py/blob/master/docstring.fmt.mustache)
But again, that is currently not supported in PyCharm - feel free to comment and vote on the linked issue, and perhaps it gets some attention.
I am trying to use a factory function to generate some type annotations – specifically for tuple types. I have one version of the factory that works fine (as in, it compiles, runs, and checks out satisfactorily in MyPy):
import typing as tx
HomogenousTypeVar = tx.TypeVar('HomogenousTypeVar')
TupleTypeReturnType = tx.Type[tx.Tuple[HomogenousTypeVar, ...]]
def TupleType(length: int,
tuptyp: tx.Type[HomogenousTypeVar] = str) -> TupleTypeReturnType:
""" Create a type annotation for a tuple of a given type and length """
assert length > 0
return tx.Tuple[tuple(tuptyp for idx in range(length))]
… for which the usage is like e.g.:
class Thing(object):
__slots__: TupleType(2) = ('yo', 'dogg')
other_fields: TupleType(4) = ('i', 'heard',
'you', 'like')
# etc, or what have you
… however, I was unsuccessful when I tried to add support for the typing.ClassVar annotation, which looked like this:
import typing as tx
HomogenousTypeVar = tx.TypeVar('HomogenousTypeVar')
TupleTypeReturnType = tx.Union[tx.Type[tx.Tuple[HomogenousTypeVar, ...]],
tx.Type[tx.ClassVar[tx.Tuple[HomogenousTypeVar, ...]]]]
def TupleType(length: int,
tuptyp: tx.Type[HomogenousTypeVar] = str,
clsvar: bool = False) -> TupleTypeReturnType:
""" Create a type annotation for a tuple of a given type and length,
specifying additionally whether or not it is a ClassVar """
assert length > 0
out = tx.Tuple[tuple(tuptyp for idx in range(length))]
return clsvar and tx.ClassVar[out] or out
… after this change, the code won’t even initially compile – it fails to do so with a TypeError from deep within the typing module:
TypeError: typing.ClassVar[typing.Tuple[~HomogenousTypeVar, ...]] is
not valid as type argument
… which, as errors go, strikes me as a little phoned-in; I mean, is not everything in typing supposed to be a valid type argument in some fashion, give-or-take?
In the typing source code related to ClassVar, there are a handful of restrictions to its use mentioned in the docstring – but this is not one of them. Is there something obvious I am missing? Is my attempt to use this annotation in this fashion a quixotic one? What else might I try?
Are you sure your original code snippet actually type-checks with mypy? When I try running it using either Mypy 0.620 or the latest version from github, I get the following errors:
test.py:13: error: invalid type comment or annotation
test.py:13: note: Suggestion: use TupleType[...] instead of TupleType(...)
test.py:14: error: invalid type comment or annotation
test.py:14: note: Suggestion: use TupleType[...] instead of TupleType(...)
I'm also not able to reproduce the error you get with your ClassVar code -- when I try running it, I get the following errors:
test.py:4: error: Invalid type: ClassVar nested inside other type
test.py:6: error: Incompatible default for argument "tuptyp" (default has type "Type[str]", argument has type "Type[HomogenousTypeVar]")
test.py:12: error: Invalid type alias
test.py:13: warning: Returning Any from function declared to return "Union[Type[Tuple[HomogenousTypeVar?, ...]], Type[Tuple[HomogenousTypeVar?, ...]]]"
test.py:15: error: Name 'Thing' is not defined
test.py:16: error: Revealed type is 'Any'
Are you sure you're actually running mypy, as opposed to just running the code? E.g. if you only run python3 test.py, you're basically skipping all type checks (apart from some bare minimum sanity checks built into the typing module).
If you want to type-check your code, you need to pip-install mypy and run python3 -m mypy test.py.
In any case, all of these error messages are expected behavior -- mypy (and any other PEP 484 compliant type checker) can only analyze your code statically and will not attempt to run or analyze any factory functions/any type-hint generation functions you may try and write.
So, this means that unfortunately you won't be able to use your generated type hint with ClassVars if you want PEP 484 compliant tools to be able to analyze your code -- they can't understand/interpret your original set of type hints, and adding ClassVars certainly won't help.
If you want to generate type hints, the only real option I can think of is to invent some sort of mini-language or macro system on top of Python that when run, will generate Python code. You would then run and typecheck that generated code instead of your macrofied Python language.
But I really don't recommend doing this -- it's a very fragile hack.
More broadly, whenever you start running into these sorts of type-related limitations, I think it's a sign that your code is too complicated. I would either look into simplifying your code or (if that's not possible) switch to a language like Haskell or Idris which would let you use a more expressive (albeit more complex) type system.
For example, in this case, you're trying to generalize the Tuple type -- that leads me to infer that your codebase contains many different kinds of tuples of different arities and types.
That strikes me as being a bit suspicious -- I would instead look into converting some of those tuples into either regular classes or (if you still need tuple-like functionality) a namedtuple. Dataclasses (which are new as of Python 3.7) could also be convenient here.
Those solutions would also help make your code a little more readable -- you can now give concrete names and meanings to each distinct kind of tuple.
Alternatively, if you have only a few distinct types of tuples but use those tuples all over the place, you could try using type aliases so you don't have to repeatedly re-type the same (long) type over and over. E.g. instead of doing:
def foo(x: Tuple[int, int, int, int]) -> None: ...
...you could do:
IpAddress = Tuple[int, int, int, int]
def foo(x: IpAddress) -> None: ...
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints
To give details: this is a constrained environment with Python 2.7.14, Pylint 1.8.1, and Astroid 1.6.0 and various other modules, but I can't easily install new modules such as mypy (not even with virtualenv) or make major changes to the codebase.
Due to https://github.com/PyCQA/pylint/issues/400 I'm getting errors from pylint on some of my code. I've worked around this issue in a hacky way (by removing the integer argument to the wait() method) and now I don't get errors, but no checking is done (pylint can't determine what the type of the variable is at all).
What I'd really like to do is tell pylint what the return type is. I've tried reading the Astroid docs, perusing the various astroid/brains files, and looking at other SO questions and answers such as Set multiple inferred types based on arguments for pylint plugin and pylint, coroutines, decorators and type inferencing but I guess I'm just not getting it. I have added my own load-plugins and I know it's loaded during pylint (because it had syntax errors at first :)). But I'm just not sure what it needs to do.
Suppose I had this code:
class Peer(object):
def get_domain(self):
return "foo"
class TestListener(object):
def __init__(self):
self._latest = None
def get(self, timeout):
# tricks here
return self._latest
def peer_joined(self, peer):
self._latest = peer
peer = TestListener().get(3)
print(peer.get_domain())
print(peer.get_foo())
During # tricks here we are really waiting on a threading.Event() during which another thread will invoke peer_joined() and pass an object of type Peer(), but pylint doesn't grok this.
What I'd like to do is annotate the TestListener.get() method to tell pylint that it will return a Peer object, so that pylint will catch the error in the second print call.
I've tried this but clearly I'm missing something fundamental, since it appears my transform method is never even invoked (if I put a bogus method call there no error is printed):
from astroid import MANAGER, register_module_extender
from astroid.builder import AstroidBuilder
def TestListener_transform():
return AstroidBuilder(MANAGER).string_build('''
class TestListener(object):
def get(self, timeout):
return Peer()
''')
register_module_extender(MANAGER, 'TestListener', TestListener_transform)
# for pylint load-plugins
def register(linter):
pass
def get(self, timeout):
"""
:param timeout: How many seconds to wait
:type timeout: int
:rtype: threading.Event
"""
# tricks here
return self._latest
This is a reStructuredText format for writing a python docstring (a multiline string that appear right after function definition.
You can document there all parameter types, and what the function return along with their datatypes.
pylint can parse that docstring to figure out the datatype of that function's return value.
This stackoverflow answer has more comprehensive explanation
What is the standard Python docstring format?
One thing to mention is that pylint currently has limited capabilities for type-checking, as those seen in mypy. It relies more on inferring values and types without type hints, although the intent is to have support for PEP-484 typing at some point.
Now regarding the question, I see that you are using a module extender, while in fact you should use a class transform, as in:
astroid.MANAGER.register_transform(astroid.ClassDef, _class_transform,
optional_transform_predicate)