We Keep Coding

Linting classes created at runtime in Python

For context, I am using the Python ctypes library to interface with a C library. It isn't necessary to be familiar with C or ctypes to answer this question however. All of this is taking place in the context of a python module I am creating.
In short, my question is: how can I allow Python linters (e.g. PyCharm or plugin for neovim) to lint objects that are created at runtime? "You can't" is not an answer ;). Of course there is always a way, with scripting and the like. I want to know what I would be looking at for the easiest way.
First I introduce my problem and the current approach I am taking. Second, I will describe what I want to do, and ask how.
Within this C library, a whole bunch of error codes are defined. I translated this information from the .h header file into a Python enum:
# CustomErrors.py
from enum import Enum
class CustomErrors(Enum):
ERROR_BROKEN = 1
ERROR_KAPUTT = 2
ERROR_BORKED = 3
Initially, my approach is to have a single exception class containing a type field which described the specific error:
# CustomException.py
from CustomErrors import CustomErrors
class CustomException(Exception):
def __init__(self, customErr):
assert type(customErr) is CustomError
self.type = customErr
super().__init__()
Then, as needed I can raise CustomException(CustomErrors.ERROR_KAPUTT).
Now, what I want to do is create a separate exception class corresponding to each of the enum items in CustomErrors. I believe it is possible to create types at runtime with MyException = type('MyException', (Exception,), {'__doc__' : 'Docstring for ABC class.'}).
I can create the exception classes at runtime like so:
#CustomException.py
from CustomErrors import CustomErrors
...
for ce in CustomErrors:
n = ce.name
vars()[n] = type(n, (Exception,), {'__doc__' : 'Docstring for {0:s} class.'.format(n)})
Note: the reason I want to create these at runtime is to avoid hard-coding of an Exception list that change in the future. I already have the problem of extracting the C enum automatically on the backburner.
This is all well and good, but I have a problem: static analysis cannot resolve the names of these exceptions defined in CustomException. This means PyCharm and other editors for Python will not be able to automatically resolve the names of the exceptions as a suggested autocomplete list when the user types CustomException.. This is not acceptable, as this is code for the end user, who will need to access the exception names for use in try-except constructs.
Here is the only solution I have been able to think of: writing a script which generates the .py files containing the exception names. I can do this using bash. Maybe people will tell me this is really the only option. But I would like to know what other approaches are suggested for solving this problem. Thanks for reading.

You can add a comment to tell mypy to ignore dynamically defined attribute errors. Perhaps the linters that you use share a similar way to silence such errors.
mypy docs on silencing errors based on error codes
This example shows how to ignore an error about an imported name mypy thinks is undefined:
# 'foo' is defined in 'foolib', even though mypy can't see the
# definition.
from foolib import foo # type: ignore[attr-defined]

Keeping alias types simple in Python documentation?

I'm trying to use the typing module to document my Python package, and I have a number of situations where several different types are allowable for a function parameter. For instance, you can either pass a number, an Envelope object (one of the classes in my package), or a list of numbers from which an Envelope is constructed, or a list of lists of numbers from which an envelope is constructed. So I make an alias type as follows:
NumberOrEnvelope = Union[Sequence[Real], Sequence[Sequence[Real]], Real, Envelope]
Then I write the function:
def example_function(parameter: NumberOrEnvelope):
...
And that looks great to me. However, when I create the documentation using Sphinx, I end up with this horrifically unreadable function signature:
example_function(parameter: Union[Sequence[numbers.Real], Sequence[Sequence[numbers.Real]], numbers.Real, expenvelope.envelope.Envelope])
Same thing also with the hints that pop up when I start to try to use the function in PyCharm.
Is there some way I can have it just leave it as "NumberOrEnvelope". Ideally that would also link in the documentation to a clarification of what "NumberOrEnvelope" is, though even if it didn't it would be way better than what's appearing now.

I had the same issue and used https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_type_aliases, introduced in version 3.3.
In your sphinx conf.py, insert this section. It does not seem to make much sense at the first sight, but does the trick:
autodoc_type_aliases = dict(NumberOrEnvelope='NumberOrEnvelope')
Warning: It only works in modules that start with from __future__ import annotation
Note: If there is a target in the documentation, type references even have a hyperlink to the definition. I have classes, documented elsewhere with autoclass, which are used as types of function parameters, and the docs show the nice names of the types with links.

Support for this appears to be in the works.
See Issue #6518.
That issue can be closed by the recent updates to Pull Request #8007 (under review).
If you want the fix ASAP, you can perhaps try using that build.

EDIT: This doesn't quite work, sadly.
Turns out after a little more searching, I found what I was looking for. Instead of:
NumberOrEnvelope = Union[Sequence[Real], Sequence[Sequence[Real]], Real, Envelope]
I found that you can create your own compound type that does the same thing:
NumberOrEnvelope = TypeVar("NumberOrEnvelope", Sequence[Real], Sequence[Sequence[Real]], Real, Envelope)
This displays in documentation as "NumberOrEnvelope", just as I wanted.

Extending stub file for a third-party library/module

I am using the yarl library's URL object.
It has a quasi-private attribute, ._val, which is a urllib.parse.SplitResult object but has no type annotation in yarl/__init__.pyi. (Understandably so, if the developer does not want to formally make it part of the public API.)
However, I have chosen to use URL._val at my own risk. A dummy example:
# urltest.py
from urllib.parse import SplitResult
from typing import Tuple
from yarl import URL
def foo(u: URL) -> Tuple[str, str, str]:
sr: SplitResult = u._val
return sr[:3]
But mypy doesn't like this, because it complains:
$ mypy urltest.py
"URL" has no attribute "_val"
So, how can I, within my own project, "tack on" (or extend) an instance attribute annotation to URL so that it can be used through the rest of my project? I.e.
from yarl import URL
URL._val: SplitResult
# ...
(mypy does not like this either; "Type cannot be declared in assignment to non-self attribute.")
Update
I've tried creating a new stub file, in stubs/yarl/__init__.pyi:
from urllib.parse import SplitResult
class URL:
_val: SplitResult
And then setting export MYPYPATH='.../stubs' as described in stub files. However, this overrides, not extends, the existing annotations, so everything but ._val throws and error:
error: "URL" has no attribute "with_scheme"
error: "URL" has no attribute "host"
error: "URL" has no attribute "fragment"
...and so on.

Unfortunately, I don't think there's really a way of making "partial" changes to the type hints for some 3rd party library -- at least, not with mypy.
I would instead try one of the following three options:
Just # type: ignore the attribute access:
def foo(u: URL) -> Tuple[str, str, str]:
sr: SplitResult = u._val # type: ignore
return sr[:3]
This type-ignore will suppress any error messages that are generated on that line. If you're going to take this approach, I'd also recommend running mypy with the --warn-unused-ignores flag, which will report any redundant and unused # type: ignore statements. It's unlikely this particular # type: ignore will become redundant as mypy updates/as the stubs for your third party library updates, but it's a nice flag to enable just in general.
Talk to the maintainer of this library and see if they're willing to either add a type hint for this attribute (even if it's private), or to expose this information via some new API.
If it helps, there is some precedent for adding type hints even for private or undocumented attributes in Typeshed, the repository of types for the standard library -- see the "What to include" section in their contribution guidelines.
If the library maintainer isn't willing to add this attribute, you could always just fork the stubs for this library, make the change to the forked stubs, and start using that.
I would personally try solution 2 first, followed by solution 1, but that's just me.

One option is to create a new class, based on the class you want to 'extend'. I do this for Pandas DataFrame objects when I want autocomplete for the data I'm working with.
import pandas as pd
class TitanicDataFrame(pd.DataFrame):
PassengerId: pd.Series
Survived: pd.Series
Name: pd.Series
Sex: pd.Series
Age: pd.Series
df: TitanicDataFrame = pd.read_csv('data/titanic.csv')
mean_age = df.Age.mean()
Note that the TitanicDataFrame class isn't actually used (as a class), it's only used as the type (thus ignored at runtime).

Unfortunately, I don't think there's really a way of making "partial" changes to the type hints for some 3rd party library -- at least, not with mypy.
Actually, there is. Per PEP 561, the first place type checkers "SHOULD" look for stubs is in the $PATH:
Stubs or Python source manually put in the beginning of the path. Type checkers SHOULD provide this to allow the user complete control of which stubs to use, and to patch broken stubs/inline types from packages. In mypy the $MYPYPATH environment variable can be used for this.
Hence, fill $MYPYPATH with a list of paths to extra directories where mypy should look for stubs and put your fixes there. You "SHOULD" be able to simply overwrite the section that is failing with proper types. Per the mypy docs:
These stub files do not need to be complete! A good strategy is to use stubgen, a program that comes bundled with mypy, to generate a first rough draft of the stubs. You can then iterate on just the parts of the library you need.
You "SHOULDN'T" even have to use stubgen, but try it out (you may have to use stubgen if you need the other type hints from the package, though I'm not sure). Even if you do, worst case, run stubgen on the file and overwrite the part of the stub that's broken.

One possibility is to simply ignore the type of u for this assignment:
def foo(u: URL) -> Tuple[str, str, str, str]:
sr: SplitResult = typing.cast(typing.Any, u)._val
return sr[:3]
mypy will assume you know what you are doing, and that u has a _val attribute with type str.

How to get rid of delays on double_click_input() actions?

Is there a way I can get rid of delays on double_click_input() actions?
What I'm trying to do is double click the edit box and then type keys here. Maybe both of these actions have some delay, so the whole process performing looks very slow.
Code:
myApp = Desktop(backend='uia').window(title_re='myTitle_re')
myApp.window(auto_id='myAutoId').window(title='myTitle').double_click_input()
myApp.descendants(title='myTitle', control_type='Edit')[1].type_keys('myKeys')
And an additional question: I tried to use double_click() here, but it always throws an exception:
AttributeError: WindowSpecification class has no 'double_click'
method.
Then I tried myApp.window(auto_id='myAutoId').window(title='myTitle').wrapper_object().double_click()
And got:
AttributeError: 'ListItemWrapper' object has no attribute
'double_click'
What should I change to get this work?
I'm using pywinauto 0.6.3.

Answering your first question, you can set some timings to null using global settings. For double_click_input:
from pywinauto.timings import Timings
Timings.after_clickinput_wait = 0.0
Timings.after_setcursorpos_wait = 0.0
For real user input (*_input methods) changing timings may cause modified sequence not to work. But you may experiment for your own risk. Sometimes it's better to use silent methods using window messages like WM_CLICK (for "win32" backend) or UIAutomation Patterns like Invoke Pattern (for "uia" backend).
double_click is not implemented for "uia" because it's unclear which UIAutomation Pattern should be interpreted as double click action. We have method .invoke() and ButtonWrapper.click = invoke alias. But for non-buttons InvokePattern may have different meaning. That's why we left it as .invoke().
P.S. Regarding legacy propery text... It can be obtained by .legacy_properties()[u'Value'] for your case (or other value from returned dict). There are methods set_window_text/set_edit_text using ValuePattern so the text can be set silently without any tricks.

Raising SyntaxError inside a Python-based parser

I'm writing a Python-based parser that can understand some configuration files that we use. The files will basically consist of (name, type) and (name, value) pairs:
Parameter file:
# defines a field called some_bool of type boolean
some_bool : bool
Config file:
# assigns True to some_bool
some_bool = bool
I'm not sure what to do when I encounter a syntax error inside a file I am parsing:
# bol instead of bool
some_bool : bol
Is it bad form to raise a SyntaxError exception in that case or are SyntaxError exceptions better left to show problems in Python code?

using SyntaxError might be confusing. Either I'd create some special exception type called eg. ParseError or ignore given value and just log in as a warning

SyntaxError may be confusing, but I require further argumentation before I am persuaded that this is a good reason not to use it in these cases. IMO it's equally or more confusing to use a different kind of exception.
Even if it might be confusing about when parsing (e.g.) domain-specific code, surely we can -in the error message- make it unambiguous that it is the domain-specific language that has the error, rather than the Python source.
In addition, a SyntaxError offers ready-made fields for line number and filename. Rolling your own SyntaxError would require reinventing those wheels.
The accepted answer here says
Use the most specific Exception constructor that semantically fits your issue.
... and an error encountered while parsing is (semantically) a SyntaxError.
My gut feeling is that an unambiguously constructed SyntaxError is the right thing in such cases. It's also the accepted answer here.
Can anyone convince me otherwise?