We Keep Coding

Sphinx ignores py-file starting with a _ in filename

I am playing around with Sphinx 4.4.0 in a test project. I use sphinx.ext.autodoc and sphinx-apidoc.
But not all py files are recognized. It seams like that files with a starting _ in filename are ignored (e.g. _mypackage.py). But i am not sure if that character is the cause or something else.
Are you aware of an option to modify that behavior?
This is the project structure
sphinx_versuch
├── docs
│   ├── make.bat
│   ├── Makefile
│   └── source
│   ├── conf.py
│   ├── index.rst
│   ├── modules.rst
│   ├── mypackage.rst
│   ├── _static
│   └── _templates
├── LICENSE
├── README.md
├── src
│   ├── mypackage
│   │   ├── a.py
│   │   ├── b.py
│   │   ├── __init__.py
│   │   └── _mypackage.py
│   ├── setup.cfg
│   └── setup.py
└── tests
├── __init__.py
├── test_a.py
├── test_b.py
└── test_mypackage.py

The option --private or -P for sphinx-apidoc make sure that private files (which start with an underscore by convention) are included also.
See also
https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html#cmdoption-sphinx-apidoc-P

How to import locally in a proper way in PyCharm OSX

Here's how the directory looks like
├── Caches
│   ├── 11\ Centroids\ representing\ relative\ anchor\ sizes..png
│   ├── 9\ Centroids\ representing\ relative\ anchor\ sizes..png
│   ├── Generated\ anchors\ relative\ to\ sample\ image\ size.png
│   ├── Relative\ width\ and\ height\ for\ 10107\ boxes..png
│   └── data_set_labels.csv
├── Config
│   ├── feature_map.py
│   ├── set_annotation_conf.py
│   └── voc_conf.json
├── Helpers
│   ├── __pycache__
│   │   ├── annotation_parsers.cpython-37.pyc
│   │   └── visual_tools.cpython-37.pyc
│   ├── anchors.py
│   ├── annotation_parsers.py
│   ├── dataset_handlers.py
│   ├── models.py
│   └── visual_tools.py
├── README.md
├── sample_img.png
└── structure_requirements.txt
in dataset_handlers.py I need to import the following:
from ..Config.feature_map import get_feature_map
and in anchors.py I need to import the following:
from .visual_tools import visualization_wrapper
Both imports do resolve in PyCharm or in other words do not have syntax errors however when are run they give the following error:
ImportError: attempted relative import with no known parent package
If I do this:
from visual_tools import visualization_wrapper
the name visual_tools is unresolved however it runs without errors and it imports the requested things
How to be able to make relative imports from .some_module import something without getting errors and being resolved without marking the directory as sources root.

Using setuptools to copy non .py files

My python project installs via setup.py. The project structure looks like:
├── Makefile
├── README.rst
├── circle.yml
├── docs
│   ├── Makefile
│   ├── conf.py
│   ├── deps.txt
│   ├── guide_installation.rst
│   ├── guide_model.rst
│   ├── guide_transliteration.rst
│   ├── index.rst
│   ├── make.bat
│   └── module_trans.rst
├── indictrans
│   ├── __init__.py
│   ├── _decode
│   ├── _utils
│   ├── base.py
│   ├── iso_code_transformer.py
│   ├── libindic_
│   ├── mappings
│   ├── models
│   ├── polyglot_tokenizer
│   ├── script_transliterate.py
│   ├── test.py
│   ├── tests
│   ├── transliterator.py
│   ├── trunk
│   └── unicode_marks.py
├── requirements.txt
├── setup.cfg
├── setup.py
├── test-requirements.txt
└── tox.ini
where the subfolder indictrans/models looks like
├── ben-eng
│   ├── classes.npy
│   ├── coef.npy
│   ├── intercept_final.npy
│   ├── intercept_init.npy
│   ├── intercept_trans.npy
│   └── sparse.vec
├── ben-guj
│   ├── classes.npy
│   ├── coef.npy
│   ├── intercept_final.npy
│   ├── intercept_init.npy
│   ├── intercept_trans.npy
│   └── sparse.vec
so I have .npy and .vec files to be included in the project.
In my setup.py I'm trying to explicitly include this folder models via the include_package_data directive like:
setup(
setup_requires=['pbr'],
pbr=True,
packages=find_packages(),
include_package_data=True,
package_data={'models': ['*.npy','*.vec']},
ext_modules=cythonize(extensions)
)
and in the setup.cfg I have
[files]
packages =
indictrans
but running python setup.py install does not copy the models folder to the installation folder /usr/local/lib/python2.7/dist-packages/indictrans/.
If I print the it is the output of the find_packages I get
['indictrans', 'indictrans.tests', 'indictrans.libindic_', 'indictrans._utils', 'indictrans._decode', 'indictrans.polyglot_tokenizer', 'indictrans.models', 'indictrans.trunk', 'indictrans.libindic_.utils', 'indictrans.libindic_.soundex', 'indictrans.libindic_.utils.tests', 'indictrans.libindic_.soundex.utils', 'indictrans.libindic_.soundex.tests', 'indictrans.libindic_.soundex.utils.tests', 'indictrans.polyglot_tokenizer.tests', 'indictrans.trunk.tests']
so I will assume that indictrans/models would be included, but it is not.

Add include_package_data=True to your setup-function (you already did that).
Create a file MANIFEST.in in the same directory as setup.py
MANIFEST.in can look as follows:
include indictrans/models/ben-eng/*
include indictrans/models/ben-guj/*
You don't need setup.cfg for doing this.
Source: This great writeup of python packaging
EDIT about recursive-include:
According to the documentation this should also work:
recursive-include indictrans/models *.npy *.vec

include_package_data=True requires MANIFEST.in.
To include data for the module indictrans.models you have to provide the full name:
package_data={'indictrans.models': ['*.npy','*.vec']},

Skip over levels of a directory when importing python package

I've got a directory structure like:
Folder_in_PYTHONPATH/
├── Package1
│   ├── __init__.py
│   ├── src
│   │   ├── Class1.py
│   │   ├── Class2.py
│   │   └── __init__.py
│   └── test
│   └── testfile.py
├── Package2
│   ├── __init__.py
│   ├── src
│   │   ├── Class1.py
│   │   ├── Class2.py
│   │   └── __init__.py
│   └── test
│   ├── test1.py
│   └── test2.py
.
.
.
When I import things from this folder, I need to always type
import Package1.src.Class1
Is there any way to set up my __init__.py so that I can just type
import Package1.Class1
instead?

Add them into your packages' __init__.py files so they look like:
from src import Class1
from src import Class2
Have a look at the docs
I would recommend putting the *.py files in the top level folder of their package to get the import Package_1.Class1 behaviour you are after. The unit tests can stay in their own folder to keep them separate.

Importing a Python module in a package returns a different module from the same package

When I run an import in my development environment, I seem to get the wrong module.
How do I make sense of this?
$ python -c "import breathe.renderer.rst.doxygen.compound as com; print com"
<module 'breathe.parser.doxygen.compound' from 'breathe/parser/doxygen/compound.pyc'>
The layout of the breathe directory is:
breathe
├── directives.py
├── finder
│   ├── doxygen
│   │   ├── base.py
│   │   ├── compound.py
│   │   ├── index.py
│   │   └── __init__.py
│   └── __init__.py
├── __init__.py
├── nodes.py
├── parser
│   ├── doxygen
│   │   ├── compound.py
│   │   ├── compoundsuper.py
│   │   ├── index.py
│   │   ├── indexsuper.py
│   │   ├── __init__.py
│   ├── __init__.py
├── process.py
├── renderer
│   ├── __init__.py
│   └── rst
│   ├── doxygen
│   │   ├── base.py
│   │   ├── compound.py
│   │   ├── domain.py
│   │   ├── filter.py
│   │   ├── index.py
│   │   ├── __init__.py
│   │   └── target.py
│   ├── __init__.py
└── transforms.py

Check your breathe/renderer/rst/__init__.py file. I suspect that you have something like
from breathe.parser import doxygen
What happens is that when a module is imported, the module's __init__.py is executed. Then the next submodule is imported and it's __init__.py is run.
So, in the above situation, you'll get breathe.renderer.rst importing fine, but when running
breathe.renderer.rst.__init__, the breathe.parser.doxygen module is bound to the local name doxygen overriding the doxygen submodule. So you should see something like this:
>>> from breathe.renderer import rst
>>> print(rst.doxygen)
<module 'breathe.parser.doxygen' from 'breathe/parser/doxygen/__init__.pyc'>
Possible solutions:
Use absolute imports in __init__.py
Keep as much code as possible out of __init__.py
Use import inside the functions it's needed rather than globally. (This can also resolve circular import problems)
Note that there is nothing that prohibits code in __init__.py, and several modules in the stdlib do it too. However, because of exactly this sort of issue it is generally considered a good idea to avoid doing it too much.