We Keep Coding

Python PDM + pre-commit using pylint: imports cannot be found

Background
I am wrangling some legacy code into shape.
I use PDM to manage dependencies, which places all dependent packages in a __pypackages__ folder directly under the repo root level. PDM also uses the relatively new pyproject.toml package config file.
I am trying to adopt pre-commit Git hooks so that I can have automated checks for formatting and style before trying to commit, merge, and/or create PRs.
I am asking pre-commit to use only a few Python tools for now: pylint and black.
Issue
Most of that toolset works great together. However, pylint cannot find any of the modules that are stored in the __pypackages__ folder. Most of what I have read suggests that I alter my $PYTHONPATH to find the modules.
This solution seems very outdated. But also, I am not sure how I can do this in a robust way across the team. I can alter the Git hooks, but the $PYTHONPATH may be different for each engineer, so this will only work for my machine.
I would like to be able to add something in the pyproject.toml file to have pylint find it. I am not sure what to write, though, so that it generically works across the whole team. Something like
[tools.pylint]
pypackages = "./__pypackages__"
Any ideas how I can do this?
Details
I am not sure more details are needed, but here it is:
My actions:
> pre-commit run --all-files # The --all-files flag is just to allow me to test without a commit
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...........................................(no files to check)Skipped
Check for added large files..............................................Passed
black....................................................................Passed
pylint...................................................................Failed
- hook id: pylint
- exit code: 30
************* Module testfile
testfile.py:18:0: E0401: Unable to import 'boto3' (import-error)
boto3 is in the __pypackages__ mentioned above. None of the modules can be imported, but I limited the output for clarity.
I can pdm run ... everything correctly and VS Code sees the modules fine. But pylint is not finding it because it cannot find this __pypackages__ folder.

You can get around this by updating the PYTHONPATH environment variable used by the extension, by creating a file named .env in your workspace (project folder) and adding the following entry:
PYTHONPATH=D:/commonScripts
Note: Relative paths are also supported.
Further info on .env files can be found here https://code.visualstudio.com/docs/python/environments#_environment-variable-definitions-file

How to configure Codacy Python linters?

I recently enabled GitHub Codacy scans on my repo. The Pylint* and Prospector modules (if that is the right terminology) report a lot of warnings:
I have to believe there's a way to configure what they flag, perhaps via an rc file or a .yml placed somewhere, but I haven't figured out what the config files should be named, where they should be placed, and what the allowable syntax(es) are. I'd be happy to RTFM if I could figure out the FM to R.
How do I configure the linters invoked by Codacy code scanners on GitHub?

Codacy will pick up the default config files for each linter.
Prospector (http://prospector.landscape.io/en/master/profiles.html#profiles-configuration)
.landscape.yml, .landscape.yaml, landscape.yml, landscape.yaml,
.prospector.yml, .prospector.yaml, prospector.yml, prospector.yaml
Pylint (http://pylint.pycqa.org/en/latest/faq.html#how-do-i-find-the-option-name-for-pylintrc-corresponding-to-a-specific-command-line-option)
pylintrc, .pylintrc
You can check Codacy's docs here for the details of more tools:
https://docs.codacy.com/repositories-configure/code-patterns/#i-have-my-own-tool-configuration-file
Also, if you set up your project in Codacy's APP, you can also configure those patterns in the settings instead of using a config files.

How to get Read the Docs to generate py-modindex.html?

I'm trying to get Read the Docs to generate the py-modindex.html file. Research into this question lead me to the following setup:
setup.py in the project directory has the following contents, which were the minimum needed to get pytest to work and haven't been changed since I got that part of my project working:
import setuptools
setuptools.setup(
name='polygons',
packages=setuptools.find_packages(),
)
docs/requirements.txt contains a single line:
sphinx-autodoc-annotation
The Read the Docs repository URL points to my GitHub repository.
The RtD setting for "Install your project inside a virtualenv using setup.py install" is checked.
The RtD setting for "Requirements file" points to docs/requirements.txt.
The "Module Index" link gets included in index.html, but the py-modindex.html file is missing.
My understanding is that with the virtualenv setting above, RtD will use the setup.py file to install the project so that Sphinx can read the documentation found in the Python docstrings. I'm using function annotations and would like sphyinx-autodoc-annotation to make use of those when creating the built docs files. All of this works splendidly on my local machine when I run make html while in the docs folder. Now I'm trying to get it to work on Read the Docs.
Note: This is an exercise I'm going through to understand everything before I apply it to my real project, which the polygons project is a placeholder for.

How to force non-default GAE module to update to new version?

So I have the recommended setup for smaller projects, where you have multiple module YAML files all in the main file, all sharing source. Like here: https://cloud.google.com/appengine/docs/python/modules/#devserver
I only have 2 modules: the default module, and my backend module for running tasks, pipeline, etc.
Default is on version 22, backend is on version 'uno' (the first and only version of this module).
I cannot get backend to update to version 'dos'. Whenever I test things I am getting 404's, like the source files don't exist on the backend module. The requests make it to the correct module, but error out.
I have tried to update using: appcfg.py update main_directory app.yaml backend.yaml
But it always looks like it is only doing a 'default module' update. I never see anything about the backend module. Even when I try the above command minus the app.yaml (which is acting as my default module YAML).
In the developer console I can only see the single version for my backend module. It has not added a 2nd version despite my attempts to add a 'dos' version, and a 'v2' version' - both never "worked".
Anyone else have problems updating a 'backend' module to a new version? Is it the 'all in one directory' setup giving me problems? Am I just not using the right appcfg incantation?
Update 1: My directory structure looks like this
where module1.yaml is app.yaml and module2.yaml is backend.yaml.

Drop the main_directory from the update command:
appcfg.py update app.yaml backend.yaml
Specifying a directory only works for single-module apps, for uploading modules only the respective modules' .yaml files should be specified:
You can also update a single module or a subset of the apps modules by specifying only the .yaml files for the desired module(s).

How can I exclude South migrations from coverage reports using coverage.py

I use coverage.py to check the test coverage of my django application. However since I use South for my database migrations, all those files show up with 0% and mess up the overall percentage.
I already tried using --omit=*migrations* in both run and report (and both) but that didn't work.
I tried versions 3.4 and latest revision from Bitbucket as of Dec 20th 2010 with the same result.
Any ideas how I can get coverage.py to actually ignore the migrations folders?

The solution was:
[run]
omit = ../*migrations*

You should be able to match against the migrations directory to omit those files. Have you tried quoting the argument? Depending on your OS and shell, it may be expanding those asterisks prematurely. Try it like this:
--omit='*migrations*'
Alternately, you could put the switch into a .coveragerc file:
[run]
omit = *migrations*

Latest version of django-jenkins has new option COVERAGE_WITH_MIGRATIONS that would exclude migrations. It's not in PyPI yet so you need to install it with pip/easy_install specyfing url git url as source.

Have you tried django_coverage. I think it handles this kind of problem.

This worked for me:
coverage run --source='.' --omit='*/migrations/*.py' manage.py test

try:
coverage run --source=. manage.py test app_name
this ignores third party code and fixes your % problem