We Keep Coding

Correct way to enable import of all submodule in python package

I'm making a python package using setuptools, and I'm having trouble making all nested folders in my source code available for import after installing the package. The directory I'm working in has a structcure like illustrated below.
├── setup.py
└── src
└── foo
├── a
│ ├── aa
│ │ └── aafile.py
│ └── afile.py
├── b
│ └── bfile.py
└── __init__.py
Currently, I can't import submodules, such as from foo.a import aa or from foo.a.aa import some_method, unless I explicitly pass the names of the submodules to setuptools. That is, setup.py needs to contain something like
from setuptools import setup
setup(
version="0.0.1",
name="foo",
py_modules=["foo", "foo.a", "foo.a.a", "foo.b"],
package_dir={"": "src"},
packages=["foo", "foo.a", "foo.a.a", "foo.b"],
include_package_data=True,
# more arguments go here
)
This makes organizing the code pretty cumbersome. Is there a simple way to just allows users of the package to install any submodule contained in src/foo?

You'll want setuptools.find_packages() – though all in all, you might want to consider tossing setup.py altogether in favor of a PEP 517 style build with no arbitrary Python but just pyproject.toml (and possibly setup.cfg).
from setuptools import setup, find_packages
setup(
version="0.0.1",
name="foo",
package_dir={"": "src"},
packages=find_packages(where='src'),
include_package_data=True,
)

Every package/subpackage must contain a (at least empty) __init__.py file to be considered so.
If you want to the whole package&subpackages tree to be imported with just one import foo consider filling your __init__.py files with the import of the relative subpackages.
# src/foo/__init__.py
import foo.a
import foo.b
# src/foo/a/__init__.py
import foo.a.aa
# src/foo/b/__init__.py
import foo.b.bb
Otherwise leave the __init__.py files empty and the user will need to manually load the subpackag/submodule he wants.

Building Python Package with source in different directory

When building a python package where the source tree looks like this:
src -\
+- module -\
<stuff>
+- setup.py
is pretty clear.
Is it possible to build a package where module source doesn't reside in the same location as the setup.py? For more specific use case the code for the module is either partially or full autogenerated in a location other then src
E.g.
src -\
+- setup.py
generated -\
module -\
<module code>

You can control the directory where packages reside by using the package_dir argument to setup(...)
and while it does appear to build a proper source distribution when package_dir is a relative path starting with .., it appears that pip will refuse to install it -- I'd suggest instead nesting your generated code inside that src directory instead and then using package_dir to select that.
Here's an example which moves all modules inside a generated subdir:
setup(
name='mypkg',
package_dir={'': 'generated'},
packages=find_packages('generated'),
)
Using a setup like:
$ tree .
.
├── generated
│   ├── mod1
│   │   └── __init__.py
│   └── mod2
│   └── __init__.py
└── setup.py
This would make the following succeed after install: import mod1; import mod2
If you wanted to make those modules available under a different prefix, you would do:
setup(
name='mypkg',
package_dir={'hello': 'generated'},
packages=[f'hello.{mod}' for mod in find_packages('generated')],
)
This would make import hello.mod1; import hello.mod2 succeed after installation

You can use relative paths in package lookup configuration. Examples:
all distributed sources are in generated
from setuptools import setup, find_packages
setup(
...
package_dir={'': '../generated'},
packages=find_packages(where='../generated'),
)
selected packages should be included from generated
In this example, only packages spam and eggs from generated will be included:
import pathlib
from setuptools import setup, find_packages
setup(
name='so',
package_dir={'spam': '../generated/spam', 'eggs': '../generated/eggs'},
packages=find_packages(where='../generated'), # or just ['spam', 'eggs']
)
Or implement a dynamic lookup like e.g.
package_dir={p.name: p.resolve() for p in pathlib.Path('..', 'generated').iterdir()}
better implementation by resolving all paths relative to the setup.py file
Resolving all paths relative to the setup.py script allows you to run the script from any other directory than src, e.g. you can run python src/setup.py bdist_wheel etc. You may or may not need it, depending on your use case. Nevertheless, the recipe is as usual: resolve all paths to __file__, e.g.
import pathlib
from setuptools import setup, find_packages
src_base = pathlib.Path(__file__, '..', '..', 'generated').resolve()
setup(
...
package_dir={'': str(src_base)},
packages=find_packages(where=src_base),
)

Packaging stub files

Let's say I have very simple package with a following structure:
.
├── foo
│   ├── bar
│   │   └── __init__.py
│   └── __init__.py
└── setup.py
Content of the files:
setup.py:
from distutils.core import setup
setup(
name='foobar',
version='',
packages=['foo', 'foo.bar'],
url='',
license='Apache License 2.0',
author='foobar',
author_email='',
description=''
)
foo/bar/__init__.py:
def foobar(x):
return x
The remaining files are empty.
I install the package using pip:
cd foobar
pip install .
and can confirm it is installed correctly.
Now I want to create a separate package with stub files:
.
├── foo
│   ├── bar
│   │   └── __init__.pyi
│   └── __init__.pyi
└── setup.py
Content of the files:
setup.py:
from distutils.core import setup
import sys
import pathlib
setup(
name='foobar_annot',
version='',
packages=['foo', 'foo.bar'],
url='',
license='Apache License 2.0',
author='foobar',
author_email='',
description='',
data_files=[
(
'shared/typehints/python{}.{}/foo/bar'.format(*sys.version_info[:2]),
["foo/bar/__init__.pyi"]
),
],
)
foo.bar.__init__.pyi:
def foobar(x: int) -> int: ...
I can install this package, see that it creates anaconda3/shared/typehints/python3.5/foo/bar/__init__.pyi in my Anaconda root, but it doesn't look like it is recognized by PyCharm (I get no warnings). When I place pyi file in the main package everything works OK.
I would be grateful for any hints how to make this work:
I've been trying to make some sense from PEP 484 - Storing and distributing stub files but to no avail. Even pathlib part seem to offend my version of distutils
PY-18597 and https://github.com/python/mypy/issues/1190#issuecomment-188526651 seem to be related but somehow I cannot connect the dots.
I tried putting stubs in the .PyCharmX.X/config/python-skeletons but it didn't help.'
Some things that work, but don't resolve the problem:
Putting stub files in the current project and marking as sources.
Adding stub package root to the interpreter path (at least in some simple cases).
So the questions: How to create a minimal, distributable package with Python stubs, which will be recognized by existing tools. Based on the experiments I suspect one of two problems:
I misunderstood the structure which should be created by the package in the shared/typehints/pythonX.Y - if this is true, how should I define data_files?
PyCharm doesn't consider these files at all (this seem to be contradicted by some comments in the linked issue).
It suppose to work just fine, but I made some configure mistake and looking for external problem which doesn't exist.
Are there any established procedures to troubleshoot problems like this?

Problem is that you didn't include the foo/__init__.pyi file in your stub distribution. Even though it's empty, it makes foo a stub files package, and enables search for foo.bar.
You can modify the data_files in your setup.py to include both
data_files=[
(
'shared/typehints/python{}.{}/foo/bar'.format(*sys.version_info[:2]),
["foo/bar/__init__.pyi"]
),
(
'shared/typehints/python{}.{}/foo'.format(*sys.version_info[:2]),
["foo/__init__.pyi"]
),
],

How do I distribute all contents of root directory to a directory with that name

I have a project named myproj structured like
/myproj
__init__.py
module1.py
module2.py
setup.py
my setup.py looks like this
from distutils.core import setup
setup(name='myproj',
version='0.1',
description='Does projecty stuff',
author='Me',
author_email='me#domain.com',
packages=[''])
But this places module1.py and module2.py in the install directory.
How do I specify setup such that the directory /myproj and all of it's contents are dropped into the install directory?

In your myproj root directory for this project, you want to move module1.py and module2.py into a directory named myproj under that, and if you wish to maintain Python < 3.3 compatibility, add a __init__.py into there.
├── myproj
│   ├── __init__.py
│   ├── module1.py
│   └── module2.py
└── setup.py
You may also consider using setuptools instead of just distutils. setuptools provide a lot more helper methods and additional attributes that make setting up this file a lot easier. This is the bare minimum setup.py I would construct for the above project:
from setuptools import setup, find_packages
setup(name='myproj',
version='0.1',
description="My project",
author='me',
author_email='me#example.com',
packages=find_packages(),
)
Running the installation you should see lines like this:
copying build/lib.linux-x86_64-2.7/myproj/__init__.py -> build/bdist.linux-x86_64/egg/myproj
copying build/lib.linux-x86_64-2.7/myproj/module1.py -> build/bdist.linux-x86_64/egg/myproj
copying build/lib.linux-x86_64-2.7/myproj/module2.py -> build/bdist.linux-x86_64/egg/myproj
This signifies that the setup script has picked up the required source files. Run the python interpreter (preferably outside this project directory) to ensure that those modules can be imported (not due to relative import).
On the other hand, if you wish to provide those modules at the root level, you definitely need to declare py_modules explicitly.
Finally, the Python Packaging User Guide is a good resource for more specific questions anyone may have about building distributable python packages.

How to make python setuptools find top level modules

I have a package with a structure that would (simplified) look like:
mypackage/
__init__.py
setup.py
module1.py
module2.py
mysubpackage/
__init__.py
mysubmodule1.py
mysubmodule2.py
I'm using a configuration for setup.py like this:
from setuptools import setup, find_packages
setup(
name = "mypackage",
version = "0.1",
author = "Foo",
author_email = "foo#gmail.com",
description = ("My description"),
packages=find_packages(),
)
The default where argument for find_packages() is '.', but it doesn't include my top-level modules (module1.py nor module2.py). However, all child submodules and subpackages are added when running python setup.py build.
How could I get top-level Python modules added too, without moving setup.py one level higher?

Thank you all for your responses.
Finally, I added a directory (not Python package) containing mypackage and the setup.py module. The structure now looks as follows:
myapp/
setup.py
mypackage/
__init__.py
module1.py
module2.py
mysubpackage/
__init__.py
mysubmodule1.py
mysubmodule2.py
Now using find_packages() works as expected. Thanks!