Pylint best practices - python

Pylint looks like a good tool for running analysis of Python code.
However, our main objective is to catch any potential bugs and not coding conventions. Enabling all Pylint checks seems to generate a lot of noise. What is the set of Pylint features you use and is effective?

You can block any warnings/errors you don't like, via:
pylint --disable=error1,error2
I've blocked the following (description from http://www.logilab.org/card/pylintfeatures):
W0511: Used when a warning note as FIXME or XXX is detected
W0142: Used * or * magic*. Used when a function or method is called using *args or **kwargs to dispatch arguments. This doesn't improve readability and should be used with care.
W0141: Used builtin function %r. Used when a black listed builtin function is used (see the bad-function option). Usual black listed functions are the ones like map, or filter, where Python offers now some cleaner alternative like list comprehension.
R0912: Too many branches (%s/%s). Used when a function or method has too many branches, making it hard to follow.
R0913: Too many arguments (%s/%s). Used when a function or method takes too many arguments.
R0914: Too many local variables (%s/%s). Used when a function or method has too many local variables.
R0903: Too few public methods (%s/%s). Used when class has too few public methods, so be sure it's really worth it.
W0212: Access to a protected member %s of a client class. Used when a protected member (i.e. class member with a name beginning with an underscore) is access outside the class or a descendant of the class where it's defined.
W0312: Found indentation with %ss instead of %ss. Used when there are some mixed tabs and spaces in a module.
C0111: Missing docstring. Used when a module, function, class or method has no docstring. Some special methods like __init__ don't necessarily require a docstring.
C0103: Invalid name "%s" (should match %s). Used when the name doesn't match the regular expression associated to its type (constant, variable, class...).

To persistently disable warnings and conventions:
Create a ~/.pylintrc file by running pylint --generate-rcfile > ~/.pylintrc
Edit ~/.pylintrc
Uncomment disable= and change that line to disable=W,C

Pyflakes should serve your purpose well.

-E will only flag what Pylint thinks is an error (i.e., no warnings, no conventions, etc.)

Using grep like:
pylint my_file.py | grep -v "^C"
Edit :
As mentionned in the question, to remove the conventions advices from pylint output, you remove the lines that start with an uppercase C.
From the doc of pylint, the output consists in lines that fit the format
MESSAGE_TYPE: LINE_NUM:[OBJECT:] MESSAGE
and the message type can be:
[R]efactor for a “good practice” metric violation
[C]onvention for coding standard violation
[W]arning for stylistic problems, or minor programming issues
[E]rror for important programming issues (i.e. most probably bug)
[F]atal for errors which prevented further processing
Only the first letter is displayed, so you can play with grep to select/remove the level of message type you want.
I didn't use Pylint recently, but I would probably use a parameter inside Pylint to do so.

Related

Python: How to deprecate a function alias

As the title states, what I want to achieve is the following:
I have a python code base for which I want to rename all the functions from camelCasing to underscore_naming. In order to maintain backwards compatibility, I have renamed all the functions, but have created function aliases for all the old names. So far, so good.
Now what I want to do is add deprecation warnings to the function aliases, preferably in a fashion like this:
def do_something():
...
#deprecated(deprecated_in='2.0',
details='All functions have been adapted to fit the PEP8 standard. Please use "do_something" instead')
doSomething = do_something
So that if somebody uses the old API call, they will get a deprecation warning. I've taken a look at deprecation and Deprecated, but neither of them seem to work on aliases.
I realise that I can create a function definition for every deprecated name and decorate that, but that loses the elegance of the function alias, and makes for more mucky code. Does anybody have a good suggestion to achieve what I want?
I actually ended up going for a different solution, similar to what's suggested in Method and property aliases with custom docstring in Python.
I modified my code to
#alias('doSomething', deprecated=True)
def do_something():
...
and added a deprecation warning to the alias decorator.

Python - Should I alias imports with underscores?

This is a conceptual question rather than an actual problem, I wanted to ask the great big Internet crowd for feedback.
We all know imported modules end up in the namespace of that module:
# Module a:
import b
__all__ = ['f']
f = lambda: None
That allows you to do this:
import a
a.b # <- Valid attribute
Sometimes that's great, but most imports are side effects of the feature your module provides. In the example above I don't mean to expose b as a valid interface for callers of a.
To counteract that we could do:
import b as _b
This marks the import as private. But I can't find that practice described anywhere nor does PEP8 talk about using aliasing to mark imports as private. So I take it it's not common practice. But from a certain angle I'd say it's definitely semantically clearer, because it cleans up the exposed bits of your module leaving only the relevant interfaces you actually mean to expose. Working with an IDE with autocomplete it makes the suggested list much slimmer.
My question boils down to if you've seen that pattern in use? Does it have a name? What arguments would go against using it?
I have not had success using the the __all__ functionality to hide the b import. I'm using PyCharm and do not see the autocomplete list change.
E.g. from some module I can do:
import a
And the autocomplete box show both b and f.
While Martijn Pieters says that no one actually uses underscore-hiding module imports, that's not exactly true. The traces of this technique can be easily seen in the Python's standard library itself (see a related question). Let's check it:
$ git clone --depth 1 git#github.com:python/cpython.git
$ cd cpython/Lib
$ find -iname '*.py' | xargs grep 'as \+_' | wc -l
183
$ find -iname '*.py' | xargs grep '^import' | wc -l
4578
So, about 4% of all imports are underscore-prefixed — not a majority, but yet far from “no one”. There also are some examples in numpy and matplotlib packages.
For me, this import-underscoring is the only right way to import module without exposing it at public. Unfortunately, it totally ruins code appearance, so many developers avoid using it. But it has some advantages over the __all__ approach:
Library user can decide whether a name is private or not without consulting documentation by just looking at the name. Looking to just __all__ is not enough to tell private from public as some public names may be not listed there.
No need to maintain a refactoring-unfriendly list of code entity names.
To the conclusion, both _name and __all__ are just plain evil, but the thing which actually needs fixing is the Python's module system, designed under an impression of “simple is better than complex” mantra. Compare to, for example, the way how modules behave in Haskell.
UPD:
It looks like PEP-8 has already answered this question in its “Public and internal-interfaces” section:
Even with __all__ set appropriately, internal interfaces (packages, modules, classes, functions, attributes or other names) should still be prefixed with a single leading underscore.
No one uses that pattern, and it is not named.
That's because the proper method to use is to explicitly mark your exported names with the __all__ variable. IDEs will honour this variable, as do tools like help().
Quoting the import statement documentation:
The public names defined by a module are determined by checking the module’s namespace for a variable named __all__; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in __all__ are all considered public and are required to exist. If __all__ is not defined, the set of public names includes all names found in the module’s namespace which do not begin with an underscore character ('_'). __all__ should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module).
(Emphasis mine).
Also see Can someone explain __all__ in Python?

How to document a module constant in Python?

I have a module, errors.py in which several global constants are defined (note: I understand that Python doesn't have constants, but I've defined them by convention using UPPERCASE).
"""Indicates some unknown error."""
API_ERROR = 1
"""Indicates that the request was bad in some way."""
BAD_REQUEST = 2
"""Indicates that the request is missing required parameters."""
MISSING_PARAMS = 3
Using reStructuredText how can I document these constants? As you can see I've listed a docstring above them, but I haven't found any documentation that indicates to do that, I've just done it as a guess.
Unfortunately, variables (and constants) do not have docstrings. After all, the variable is just a name for an integer, and you wouldn't want to attach a docstring to the number 1 the way you would to a function or class object.
If you look at almost any module in the stdlib, like pickle, you will see that the only documentation they use is comments. And yes, that means that help(pickle) only shows this:
DATA
APPEND = b'a'
APPENDS = b'e'
…
… completely ignoring the comments. If you want your docs to show up in the built-in help, you have to add them to the module's docstring, which is not exactly ideal.
But Sphinx can do more than the built-in help can. You can configure it to extract the comments on the constants, or use autodata to do it semi-automatically. For example:
#: Indicates some unknown error.
API_ERROR = 1
Multiple #: lines before any assignment statement, or a single #: comment to the right of the statement, work effectively the same as docstrings on objects picked up by autodoc. Which includes handling inline rST, and auto-generating an rST header for the variable name; there's nothing extra you have to do to make that work.
As a side note, you may want to consider using an enum instead of separate constants like this. If you're not using Python 3.4 (which you probably aren't yet…), there's a backport.enum package for 3.2+, or flufl.enum (which is not identical, but it is similar, as it was the main inspiration for the stdlib module) for 2.6+.
Enum instances (not flufl.enum, but the stdlib/backport version) can even have docstrings:
class MyErrors(enum.Enum):
"""Indicates some unknown error."""
API_ERROR = 1
"""Indicates that the request was bad in some way."""
BAD_REQUEST = 2
"""Indicates that the request is missing required parameters."""
MISSING_PARAMS = 3
Although they unfortunately don't show up in help(MyErrors.MISSING_PARAMS), they are docstrings that Sphinx autodoc can pick up.
If you put a string after the variable, then sphinx will pick it up as the variable's documentation. I know it works because I do it all over the place. Like this:
FOO = 1
"""
Constant signifying foo.
Blah blah blah...
""" # pylint: disable=W0105
The pylint directive tells pylint to avoid flagging the documentation as being a statement with no effect.
This is an older question, but I noted that a relevant answer was missing.
Or you can just include a description of the constants in the docstring of the module via .. py:data::. That way the documentation is also made available via the interactive help. Sphinx will render this nicely.
"""
Docstring for my module.
.. data:: API_ERROR
Indicates some unknown error.
.. data:: BAD_REQUEST
Indicates that the request was bad in some way.
.. data:: MISSING_PARAMS
Indicates that the request is missing required parameters.
"""
You can use hash + colon to document attributes (class or module level).
#: Use this content as input for moo to do bar
MY_CONSTANT = "foo"
This will be picked up by some document generators.
An example here, could not find a better one: Sphinx document module properties
the following worked for me with Sphinx 2.4.4:
in foo.py :
API_ERROR = 1
"""int: Indicates some unknown error."""
then to document it:
.. automodule:: foo.py
:members:
I think you're out of luck here.
Python don't support directly docstrings on variables: there is no attribute that can be attached to variables and retrieved interactively like the __doc__ attribute on modules, classes and functions.
Source.
The Sphinx Napoleon Python documentation extension allows to document module-level variables in an Attributes section.
Per https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html :
Attributes
----------
module_level_variable1 : int
Module level variables may be documented in either the ``Attributes``
section of the module docstring, or in an inline docstring immediately
following the variable.
Either form is acceptable, but the two should not be mixed. Choose
one convention to document module level variables and be consistent
with it.
Writing only because I haven't seen this option in the answers so far:
You can also define your constants as functions that simply return the desired constant value when called, so for example:
def get_const_my_const() -> str:
"""Returns 'my_const'."""
return "my_const"
This way they'll be a bit "more constant" on one hand (less worrying about reassignment) and they'll also provide the opportunity for regular documentation, as with any other function.

Should namedtuples follow constant name conventions in python?

I've written a small python module where I use a couple of namedtuples to pass info around because I find them very expressive. I considered these types and named them following the PEP8 convention for class names (CamelCased). However pylint sees the line:
PersonData = collections.namedtuple('PersonData', 'name surname age')
at the module's global scope and goes: Invalid constant name 'PersonData'.
Am I misusing namedtuples? What's the pythonic recommendation? I can only think of suppressing the warning, renaming the structure to PERSON_DATA, or making it a full class. Note that, in my case, it wouldn't make sense for it to have methods though.
If the answer is to suppress the warning. Wouldn't this be a recurring problem with pylint vs named tuples?
(using pylint-0.26.0, python-2.7.4)
This issue has been solved in newer versions of pylint.
My system was picking the version from the ubuntu (13.04) repositories (pylint-0.26.0).
Using a virtualenv I could pip install pylint-1.0.0 which recognizes named tuples and will actually enforce using the same format as classes.
Note that it won't pick it up if you're doing string manipulation on the fields string. For example:
PersonData = collections.namedtuple('PersonData', 'name ' + 'surname age')
will still cause pylint-1.0.0 to spit out the error code for invalid constant name. In this case the only workaround is to disable the warning as per Martijn Pieters suggestion.
You can ignore pylint here, you are using the camel case naming convention exactly right.
You can suppress the warning:
PersonData = collections.namedtuple('PersonData', 'name surname age') # pylint: disable-msg=C0103
namedtuple is a class factory, so use the naming conventions for a class.
This has been fixed in pylint-1.0.0 (see Pylint's Changelog):
For toplevel name assignment, the class name regex will be used if pylint can detect that value on the right-hand side is a class (like collections.namedtuple()).
Note that the namedtuple must be defined on the toplevel, if defined e.g. within a function pylint will still give a invalid-name warning.
in such case pylint should detect the name is assigned to a class an use the class name regexp.
Please submit a ticket on http://bitbucket.org/logilab/pylint

Mark specific variables as known

I am developing Python scripts which run inside a Jython interpreter. This interpreter sets certain global variables, which I use inside the script.
Pylint of course does not know these variables, so it reports errors all over the place.
Is there a way of making pylint aware that there are certain variables defined outside of its scope?
Alternatively, is there a way that I can define the unknown variables to pylint?
I tried something like
if not globals().has_key('SOME_EXTERNAL_GLOBAL'):
globals()['SOME_EXTERNAL_GLOBAL'] = None
But that did not help (pylint seems to ignore black magic done to globals()).
You have several options:
add variable names to additional-builtins
additional-builtins:
List of additional names supposed to be
defined in builtins. Remember that you should avoid to define new
builtins when possible.
add # pylint: disable=E0602 comment on top of the file to disable undefined-variable check in the file
add # pylint: disable=E0602 comment in the code where the variable is used
run pylint with --disable-msg=E0602 option
Also see:
Pylint ignore specific names
Howto ignore specific undefined variables in Pydev Eclipse
How to disable pylint 'Undefined variable' error for a specific variable in a file?

Categories