We Keep Coding

Keeping alias types simple in Python documentation?

I'm trying to use the typing module to document my Python package, and I have a number of situations where several different types are allowable for a function parameter. For instance, you can either pass a number, an Envelope object (one of the classes in my package), or a list of numbers from which an Envelope is constructed, or a list of lists of numbers from which an envelope is constructed. So I make an alias type as follows:
NumberOrEnvelope = Union[Sequence[Real], Sequence[Sequence[Real]], Real, Envelope]
Then I write the function:
def example_function(parameter: NumberOrEnvelope):
...
And that looks great to me. However, when I create the documentation using Sphinx, I end up with this horrifically unreadable function signature:
example_function(parameter: Union[Sequence[numbers.Real], Sequence[Sequence[numbers.Real]], numbers.Real, expenvelope.envelope.Envelope])
Same thing also with the hints that pop up when I start to try to use the function in PyCharm.
Is there some way I can have it just leave it as "NumberOrEnvelope". Ideally that would also link in the documentation to a clarification of what "NumberOrEnvelope" is, though even if it didn't it would be way better than what's appearing now.

I had the same issue and used https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_type_aliases, introduced in version 3.3.
In your sphinx conf.py, insert this section. It does not seem to make much sense at the first sight, but does the trick:
autodoc_type_aliases = dict(NumberOrEnvelope='NumberOrEnvelope')
Warning: It only works in modules that start with from __future__ import annotation
Note: If there is a target in the documentation, type references even have a hyperlink to the definition. I have classes, documented elsewhere with autoclass, which are used as types of function parameters, and the docs show the nice names of the types with links.

Support for this appears to be in the works.
See Issue #6518.
That issue can be closed by the recent updates to Pull Request #8007 (under review).
If you want the fix ASAP, you can perhaps try using that build.

EDIT: This doesn't quite work, sadly.
Turns out after a little more searching, I found what I was looking for. Instead of:
NumberOrEnvelope = Union[Sequence[Real], Sequence[Sequence[Real]], Real, Envelope]
I found that you can create your own compound type that does the same thing:
NumberOrEnvelope = TypeVar("NumberOrEnvelope", Sequence[Real], Sequence[Sequence[Real]], Real, Envelope)
This displays in documentation as "NumberOrEnvelope", just as I wanted.

Locally patch missing Python type annotations?

Python now supports type hinting, so... yay! It seems like a great method to avoid some of the more obscure runtime bugs.
Sadly, third-party library support remains an issue. Though partially solved by the typeshed project, which is also used by mypy, when trying to port some of my code to use type hints, I ran into issues due to missing stubs.
E.g.
# file:mypytest.py
import lxml.etree as et
tree = et.fromstring('<root><a>1</a><b>2</b><a>3</a></root>')
items = tree.xpath('/root/a')
print([i.text for i in items])
will work perfectly well, but mypy will produce the spurious error message
>>> mypy mypytest.py
mypytest.py:3: error: "_Element" has no attribute "xpath"
because the stub is currently incomplete.
For a larger project, downloading the stub from typeshed, adding the missing entries, and maybe even submitting the corresponding pull request is a no-brainer.
But is there some method to monkey-patch the missing information in quick-and-dirty scenarios?
Bad workaround
The best I was able to come up with was
items = tree.xpath('/root/a') # type: ignore
which silences the error, but also disables type-checking where the variable items is used afterwards. E.g. items[0] + 1 will not cause a warning anymore.
In order to preserve type-checking it is possible to use
items_tmp = tree.xpath('/root/a') # type: ignore
items = items_tmp # type: List[et._Element]
but this seems hackish; It also has to be repeated everywhere the .xpath method is used.
Update from 2017-09-12: Alternatively one can use the syntax
items_tmp : List[et._Element] = tree.xpath('/root/a') # type: ignore

Python wmi parameters reversed

Using python's wmi module to create a vss snapshot, I've found that the parameters don't work unless I reverse them:
import wmi
def vss_create():
shadow_copy_service = wmi.WMI(moniker='winmgmts:\\\\.\\root\\cimv2:Win32_ShadowCopy')
res = shadow_copy_service.Create('ClientAccessible', 'C:\\')
In the msdn docs, the function is instead supposed to be used this way:
Win32_ShadowCopy.Create("C:\\", "ClientAccessible");
Why is this the case, and is there a way to use the intended order?

Summary
It looks like the parameter ordering for wmi object's methods is reversed from normal by the PyWin32 layer, and this behaviour has been present for at least five years. The relevant wmi spec says that a wmi client can pass the parameters in any order, so PyWin32 is not 'wrong' to do this, although I can't tell if it's deliberate or accident. I speculate that it's unlikely to change, for backwards compatibility reasons, but you can work around this and put the parameters in the order you want by specifying them as keyword parameters: Create(Volume=, Context=).
Details
NB. In the below details, I'm trying to go down in layers from the Python WMI module code to WMI objects accessed by COM in PyWin32 code, to WMI objects as documented and used in other languages, to WMI object specification by MOF files, to specification documents. There's several layers and I write "WMI" a lot, meaning different things at different layers.
When you say "Python's wmi module" do you mean Tim Golden's Python WMI module (link to source code) that builds on PyWin32?
When you get a Python WMI object from the wmi module, the initialization steps it goes through are inside the class _wmi_object, and include querying the underlying wmi object for its available methods:
for m in ole_object.Methods_:
self.methods[m.Name] = None
I'm going to skip beneath Python's wmi module, and use PyWin32 directly to look at what you get when querying a WMI COM object for its available methods:
>>> from win32com.client import GetObject
>>> vss = GetObject('winmgmts:\\\\.\\root\\cimv2:Win32_ShadowCopy')
>>> [method.Name for method in list(vss.Methods_)]
[u'Create', u'Revert']
And we see the Win32_ShadowCopy object has the methods Create and Revert. So that's where the Python wmi wrapper first learns about the Create method you are using.
From there, the Python WMI wrapper class does some setup work which I haven't traced through fully, but it seems to initialize class _wmi_method once for each available method of the COM object. This class includes these initialization steps:
self.method = ole_object.Methods_ (method_name)
self.in_parameter_names = [(i.Name, i.IsArray) for i in self.in_parameters.Properties_]
A list comprehension to get the available parameters for each method. Going back to my testing to explore that without the Python WMI layer, it gives output like this:
>>> CreateMethod = vss.Methods_('Create')
>>> [n.Name for n in list(CreateMethod.InParameters.Properties_)]
[u'Context', u'Volume']
This example test shows the PyWin32 later, the COM object for Win32_ShadowCopy, the Create method - lists its available parameters in the order you are seeing - the "wrong" order. The Python WMI layer is picking up that ordering.
When you call the Win32_ShadowCopy object's Create() method through Python WMI's wrapper, the _wmi_method does this:
def __call__ (self, *args, **kwargs):
for n_arg in range (len (args)):
arg = args[n_arg]
parameter = parameters.Properties_[n_arg]
parameter.Value = arg
In other words; it pairs up the parameters you pass in (*args) with the stored parameter list, one by one, taking the parameters in the order you pass them, and pairing them with the method parameters in the order WMI returned them - i.e. it's not intelligent, it just links the first parameter you enter with 'Context' and the second with 'Volume' and gets them backwards, and your code crashes.
The call method also includes Python's **kwargs parameter which takes all given keywords, suggesting you can do
Create(Volume='C:\\', Context="ClientAccessible")
and put them in the order you want by using them as keyword arguments. (I haven't tried).
I have tried tracing the .Properties_ lookup through PyWin32com to try and identify where the ordering comes from at the lower layers, and it goes through a long chain of dynamic and cached lookups. I can't see what happens and I don't understand enough COM or PyWin32 to know what kinds of things to be looking for, so that's a dead end for me.
Taking a different approach and trying to find out from the WMI object setup files where the ordering comes from: run mofcomp.exe which ships with Windows and processes Managed Object Format (MOF) files... click Connect, Create Class "Win32_ShadowCopy"; Click the "Create" method in the methods list, then click the "Edit Method" button; then click "Edit Input Arguments" then click "Show MOF", and get this result:
[abstract]
class __PARAMETERS
{
[in, ID(0): DisableOverride ToInstance] string Volume;
[in, ID(1): DisableOverride ToInstance] string Context = "ClientAccessible";
};
That's the "correct" order of the parameters coming out of the Windows MOF files, with numeric IDs for the parameters - implying they have a correct ordering 0, 1, etc.
c:\windows\system32\wbem\vss.mof, the MOF file which appears to cover the Volume Shadow Copy objects contains this:
[static,implemented,constructor] uint32 Create([in] string Volume,[in] string Context = "ClientAccessible",[out] string ShadowID);
and the PowerShell example in the comments at this MSDN link includes $class.create("C:\", "ClientAccessible").
So those three things all tie up with the same ordering and implies there is a correct or standard ordering.
That leaves me thinking of these possibilities:
There is ordering information coming out of PythonCOM and the wmi module should look at it, but doesn't. - I have looked around quickly, and can't find ID / ordering data with the parameters list, so that seems unlikely.
There is ordering information somewhere unknown to me which the PyWin32 COM layer should be looking at but doesn't. - Not sure here.
There is no official ordering. Trying to confirm this point, we get a fun chain:
What is WMI? Microsoft's implementation of standard management frameworks WBEM and CIM, specified by the DTMF. (DTMF = Distributed Management Task Force, WBEM is Web Based Enterprise Management and CIM is the Common Information Model).
MOF is the Managed Object Format, a text representation of CIM
This document: http://www.dmtf.org/sites/default/files/standards/documents/DSP0221_3.0.0.pdf appears to be the MOF specification. Check section 7.3.3 Class Declaration, from page 18:
line 570:
"A method can have zero or more parameters".
lines 626 through 628:
Method parameters are identified by name and not by position and
clients invoking a method can pass the corresponding arguments in
any order. Therefore parameters with default values can be added to
the method signature at any position.
I don't know for sure if that's an authoritative and current specification, nor have I read all of it looking for exceptions, but it sounds like you should use named parameters.
The WMI objects and methods have a MOF definition, and the MOF specification says you shouldn't rely on the parameter ordering; however, accessing the WMI objects via COM via PyWin32 is showing a different ordering to (MSDN docs, the MOF file and the PowerShell examples). I still don't know why.
And Googling that leads me to this mailing list post by Tim Golden, author of the Python wmi module, saying basically the same thing as I've just found, except five years ago:
the method definition picks up the parameters in the order in which WMI returns them [..]
I've got no idea if there is any guarantee about the order of parameters [..]
Glancing at a few other method definitions, it does seem as though WMI is consistently returning params in the reverse order of their definition in the MOF.
At this point, it looks like PyWin32 is returning a reversed list to the typical Windows parameter order, but is that a bug if the CIM managed object method parameter list specification document explicitly says don't rely on the parameter ordering?

Is there any way to make pydev (or whatever Python IDE) understand module imported by using __import__ or exec

Introduction
Pydev is a great eclipse plugin that let us write python code easily.
It can even give autocompletion suggestion when I do this:
from package.module import Random_Class
x = Random_Class()
x. # the autocompletion will be popped up,
# showing every method & attribute inside Random_Class
That is great !!!
The Problem (And My Question)
However, when I don't use explicit import, and use __import__ for example, I can't have the same autocompletion effect.
import_location = ".".join(('package', 'module'))
__import__(import_location, globals(), locals(), ['*'])
My_Class = getattr(sys.modules[import_location], 'Random_Class')
x = My_Class()
x. # I expect autocompletion, but nothing happened
Question: is there any way (in pydev or any IDE) to make the second one also
show autocompletion?
Why do I need to do this?
Well, I make a simple MVC framework, and I want to provide something like load_model, load_controller, and load_view which is still work with autocompletion (or at least possible to work)
So, instead of leave users do this (although I don't forbid them to do so):
from applications.the_application.models.the_model import The_Model
x = The_Model()
x. # autocompletion works
I want to let users do this:
x = load_model('the_application', 'the_model')()
x. # autocompletion still works
The "applications" part is actually configurable by another script, and I don't want users to change all of their importing model/controller part everytime they change the configuration. Plus, I think load_model, load_controller, and load_view make MVC pattern shown more obvious.
Unexpected Answer
I know some tricks such as doing this (as what people do with
web2py):
import_location = ".".join(('package', 'module'))
__import__(import_location, globals(), locals(), ['*'])
My_Class = getattr(sys.modules[import_location], 'Random_Class')
x = My_Class()
if 0:
from package.module import Random_Class
x = Random_Class()
x. # Now autocompletion is going to work
and I don't expect to do this, since it will only add unnecessary
extra work.
I don't expect any don't try to be clever comments. I have enough of them
I don't expect dynamic import is evil comments. I'm not a purist.
I don't expect any just use django, or pylons, or whatever comments. Such as comments even unrelated to my question.

I have done this before. This may be slightly different from your intended method, so let me know if it doesn't apply.
I dynamically import different modules that all subclass a master class, using similar code to your example. Because the subclassing module already imports the master, I don't need to import it in the main module.
To get highlighting, the solution was to import the master class into the main module first, even though it wasn't used directly. In my case it was a good fallback if the particular subclass didn't exist, but that's an implementation detail.
This only works if your classes all inherit from one parent.

Not really an answer to my own question. However, I can change the approach. So, instead of provide "load_model()", I can use relative import. Something like this:
from ..models.my_model import Model_Class as Great_Model
m = Great_Model()

Pylint best practices

Pylint looks like a good tool for running analysis of Python code.
However, our main objective is to catch any potential bugs and not coding conventions. Enabling all Pylint checks seems to generate a lot of noise. What is the set of Pylint features you use and is effective?

You can block any warnings/errors you don't like, via:
pylint --disable=error1,error2
I've blocked the following (description from http://www.logilab.org/card/pylintfeatures):
W0511: Used when a warning note as FIXME or XXX is detected
W0142: Used * or * magic*. Used when a function or method is called using *args or **kwargs to dispatch arguments. This doesn't improve readability and should be used with care.
W0141: Used builtin function %r. Used when a black listed builtin function is used (see the bad-function option). Usual black listed functions are the ones like map, or filter, where Python offers now some cleaner alternative like list comprehension.
R0912: Too many branches (%s/%s). Used when a function or method has too many branches, making it hard to follow.
R0913: Too many arguments (%s/%s). Used when a function or method takes too many arguments.
R0914: Too many local variables (%s/%s). Used when a function or method has too many local variables.
R0903: Too few public methods (%s/%s). Used when class has too few public methods, so be sure it's really worth it.
W0212: Access to a protected member %s of a client class. Used when a protected member (i.e. class member with a name beginning with an underscore) is access outside the class or a descendant of the class where it's defined.
W0312: Found indentation with %ss instead of %ss. Used when there are some mixed tabs and spaces in a module.
C0111: Missing docstring. Used when a module, function, class or method has no docstring. Some special methods like __init__ don't necessarily require a docstring.
C0103: Invalid name "%s" (should match %s). Used when the name doesn't match the regular expression associated to its type (constant, variable, class...).

To persistently disable warnings and conventions:
Create a ~/.pylintrc file by running pylint --generate-rcfile > ~/.pylintrc
Edit ~/.pylintrc
Uncomment disable= and change that line to disable=W,C

Pyflakes should serve your purpose well.

-E will only flag what Pylint thinks is an error (i.e., no warnings, no conventions, etc.)

Using grep like:
pylint my_file.py | grep -v "^C"
Edit :
As mentionned in the question, to remove the conventions advices from pylint output, you remove the lines that start with an uppercase C.
From the doc of pylint, the output consists in lines that fit the format
MESSAGE_TYPE: LINE_NUM:[OBJECT:] MESSAGE
and the message type can be:
[R]efactor for a “good practice” metric violation
[C]onvention for coding standard violation
[W]arning for stylistic problems, or minor programming issues
[E]rror for important programming issues (i.e. most probably bug)
[F]atal for errors which prevented further processing
Only the first letter is displayed, so you can play with grep to select/remove the level of message type you want.
I didn't use Pylint recently, but I would probably use a parameter inside Pylint to do so.