Let's say I have very simple package with a following structure:
.
├── foo
│ ├── bar
│ │ └── __init__.py
│ └── __init__.py
└── setup.py
Content of the files:
setup.py:
from distutils.core import setup
setup(
name='foobar',
version='',
packages=['foo', 'foo.bar'],
url='',
license='Apache License 2.0',
author='foobar',
author_email='',
description=''
)
foo/bar/__init__.py:
def foobar(x):
return x
The remaining files are empty.
I install the package using pip:
cd foobar
pip install .
and can confirm it is installed correctly.
Now I want to create a separate package with stub files:
.
├── foo
│ ├── bar
│ │ └── __init__.pyi
│ └── __init__.pyi
└── setup.py
Content of the files:
setup.py:
from distutils.core import setup
import sys
import pathlib
setup(
name='foobar_annot',
version='',
packages=['foo', 'foo.bar'],
url='',
license='Apache License 2.0',
author='foobar',
author_email='',
description='',
data_files=[
(
'shared/typehints/python{}.{}/foo/bar'.format(*sys.version_info[:2]),
["foo/bar/__init__.pyi"]
),
],
)
foo.bar.__init__.pyi:
def foobar(x: int) -> int: ...
I can install this package, see that it creates anaconda3/shared/typehints/python3.5/foo/bar/__init__.pyi in my Anaconda root, but it doesn't look like it is recognized by PyCharm (I get no warnings). When I place pyi file in the main package everything works OK.
I would be grateful for any hints how to make this work:
I've been trying to make some sense from PEP 484 - Storing and distributing stub files but to no avail. Even pathlib part seem to offend my version of distutils
PY-18597 and https://github.com/python/mypy/issues/1190#issuecomment-188526651 seem to be related but somehow I cannot connect the dots.
I tried putting stubs in the .PyCharmX.X/config/python-skeletons but it didn't help.'
Some things that work, but don't resolve the problem:
Putting stub files in the current project and marking as sources.
Adding stub package root to the interpreter path (at least in some simple cases).
So the questions: How to create a minimal, distributable package with Python stubs, which will be recognized by existing tools. Based on the experiments I suspect one of two problems:
I misunderstood the structure which should be created by the package in the shared/typehints/pythonX.Y - if this is true, how should I define data_files?
PyCharm doesn't consider these files at all (this seem to be contradicted by some comments in the linked issue).
It suppose to work just fine, but I made some configure mistake and looking for external problem which doesn't exist.
Are there any established procedures to troubleshoot problems like this?
Problem is that you didn't include the foo/__init__.pyi file in your stub distribution. Even though it's empty, it makes foo a stub files package, and enables search for foo.bar.
You can modify the data_files in your setup.py to include both
data_files=[
(
'shared/typehints/python{}.{}/foo/bar'.format(*sys.version_info[:2]),
["foo/bar/__init__.pyi"]
),
(
'shared/typehints/python{}.{}/foo'.format(*sys.version_info[:2]),
["foo/__init__.pyi"]
),
],
Related
I'm making a python package using setuptools, and I'm having trouble making all nested folders in my source code available for import after installing the package. The directory I'm working in has a structcure like illustrated below.
├── setup.py
└── src
└── foo
├── a
│ ├── aa
│ │ └── aafile.py
│ └── afile.py
├── b
│ └── bfile.py
└── __init__.py
Currently, I can't import submodules, such as from foo.a import aa or from foo.a.aa import some_method, unless I explicitly pass the names of the submodules to setuptools. That is, setup.py needs to contain something like
from setuptools import setup
setup(
version="0.0.1",
name="foo",
py_modules=["foo", "foo.a", "foo.a.a", "foo.b"],
package_dir={"": "src"},
packages=["foo", "foo.a", "foo.a.a", "foo.b"],
include_package_data=True,
# more arguments go here
)
This makes organizing the code pretty cumbersome. Is there a simple way to just allows users of the package to install any submodule contained in src/foo?
You'll want setuptools.find_packages() – though all in all, you might want to consider tossing setup.py altogether in favor of a PEP 517 style build with no arbitrary Python but just pyproject.toml (and possibly setup.cfg).
from setuptools import setup, find_packages
setup(
version="0.0.1",
name="foo",
package_dir={"": "src"},
packages=find_packages(where='src'),
include_package_data=True,
)
Every package/subpackage must contain a (at least empty) __init__.py file to be considered so.
If you want to the whole package&subpackages tree to be imported with just one import foo consider filling your __init__.py files with the import of the relative subpackages.
# src/foo/__init__.py
import foo.a
import foo.b
# src/foo/a/__init__.py
import foo.a.aa
# src/foo/b/__init__.py
import foo.b.bb
Otherwise leave the __init__.py files empty and the user will need to manually load the subpackag/submodule he wants.
I'm testing a trivial installable package with Python setuptools. I have read a handful of tutorials and official documentation, so I thought I had a good background. It seems I was wrong.
The minimal package I'm playing with has the following structure:
myapp
├── config
│ └── text.txt
├── MANIFEST.in
├── myapp
│ ├── __init__.py
│ ├── lib.py
│ └── __main__.py
└── setup.py
With the following contents:
__init__.py (this is just to make the package usable as a library when imported):
import myapp.lib
__main__.py (I want to make this application basically runnable from the terminal with -m or better, with an entry point):
import sys
import myapp
import myapp.lib
def main():
print("main function called")
print(f"current path: {sys.path}")
myapp.lib.funct()
if __name__ == "__main__":
main()
myapp/lib.py (this is the module with the logic, and it relies on external data):
from pathlib import Path
textfile = Path(__file__).parent.parent / 'config/text.txt'
def funct():
print("hello from function in lib")
with open(textfile, 'r') as f:
print(f.read())
config/text.txt (this is the data):
some text constant as part of the module
To package this module, I have crafted the following setup.py
setup(
name="myapp",
version="1.0.0",
description="A simple test",
author="me",
url='www.me.com',
author_email="me#email.com",
classifiers=[
"Programming Language :: Python :: 3",
],
packages=["myapp"],
include_package_data=True,
entry_points={"console_scripts": ["myapp=myapp.__main__:main"]},
)
As well as a MANIFEST.in file:
include config/*
OK, these are the ingredients. Now, I create a distribution with python setup.py sdist. This creates a file dist/myapp-1.0.0.tar.gz, which does include the data file (along with plenty of metadata I don't fully understand):
> tar tf myapp-1.0.0.tar.gz
myapp-1.0.0/
myapp-1.0.0/MANIFEST.in
myapp-1.0.0/PKG-INFO
myapp-1.0.0/config/
myapp-1.0.0/config/text.txt
myapp-1.0.0/myapp/
myapp-1.0.0/myapp/__init__.py
myapp-1.0.0/myapp/__main__.py
myapp-1.0.0/myapp/lib.py
myapp-1.0.0/myapp.egg-info/
myapp-1.0.0/myapp.egg-info/PKG-INFO
myapp-1.0.0/myapp.egg-info/SOURCES.txt
myapp-1.0.0/myapp.egg-info/dependency_links.txt
myapp-1.0.0/myapp.egg-info/entry_points.txt
myapp-1.0.0/myapp.egg-info/top_level.txt
myapp-1.0.0/setup.cfg
myapp-1.0.0/setup.py
Now, the problem comes when I try to install the package with pip install myapp-1.0.0.tar.gz. It does not copy the config/text.txt anywhere. In my site-packages folder, I see the directories myapp and myapp-1.0.0.dist-info, but there is no text file inside any of them.
There are various things I don't understand:
is the definition of textfile in myapp/lib.py correct/sensible? I mean, I find it cumbersome and almost like I'm hardcoding this, having to use a module on purpose to build a simple path to this data file. I feel I'm overdoing this, but I have not found this part of the packing step very well documented. It's surprising, as this must be a very common problem.
given the the data file IS in the tar file. Why did pip not copy it to site-packages?
what is this myapp-1.0.0.dist-info folder in my site-packages?
I'm struggling to figure out how to copy the wrapper generated by swig at the same level than the swig shared library. Consider this tree structure:
│ .gitignore
│ setup.py
│
├───hello
├───src
│ hello.c
│ hello.h
│ hello.i
│
└───test
test_hello.py
and this setup.py:
import os
import sys
from setuptools import setup, find_packages, Extension
from setuptools.command.build_py import build_py as _build_py
class build_py(_build_py):
def run(self):
self.run_command("build_ext")
return super().run()
setup(
name='hello_world',
version='0.1',
cmdclass={'build_py': build_py},
packages=["hello"],
ext_modules=[
Extension(
'hello._hello',
[
'src/hello.i',
'src/hello.c'
],
include_dirs=[
"src",
],
depends=[
'src/hello.h'
],
)
],
py_modules=[
"hello"
],
)
When I do pip install . I'll get this content on site-packages:
>tree /f d:\virtual_envs\py364_32\Lib\site-packages\hello
D:\VIRTUAL_ENVS\PY364_32\LIB\SITE-PACKAGES\HELLO
_hello.cp36-win32.pyd
>tree /f d:\virtual_envs\py364_32\Lib\site-packages\hello_world-0.1.dist-info
D:\VIRTUAL_ENVS\PY364_32\LIB\SITE-PACKAGES\HELLO_WORLD-0.1.DIST-INFO
INSTALLER
METADATA
RECORD
top_level.txt
WHEEL
As you can see hello.py (the file generated by swig) hasn't been copied in site-packages.
Thing is, I've already tried many answers from the below similars posts:
setup.py: run build_ext before anything else
python distutils not include the SWIG generated module
Unfortunately, the question still remains unsolved.
QUESTION: How can I fix my current setup.py so the swig wrapper will be copied at the same level than the .pyd file?
setuptools cannot do this the way you want: it will look for py_modules only where setup.py is located. The easiest way IMHO is keeping the SWIG modules where you want them in the namespace/directory structure: rename src to hello, and add hello/__init__.py (may be empty or simply include everything from hello.hello), leaving you with this tree:
$ tree .
.
├── hello
│ ├── __init__.py
│ ├── _hello.cpython-37m-darwin.so
│ ├── hello.c
│ ├── hello.h
│ ├── hello.i
│ ├── hello.py
│ └── hello_wrap.c
└── setup.py
Remove py_modules from setup.py. The "hello" in the package list will make setuptools pick up the whole package, and include __init__.py and the generated hello.py:
import os
import sys
from setuptools import setup, find_packages, Extension
from setuptools.command.build_py import build_py as _build_py
class build_py(_build_py):
def run(self):
self.run_command("build_ext")
return super().run()
setup(
name='hello_world',
version='0.1',
cmdclass={'build_py': build_py},
packages = ["hello"],
ext_modules=[
Extension(
'hello._hello',
[
'hello/hello.i',
'hello/hello.c'
],
include_dirs=[
"hello",
],
depends=[
'hello/hello.h'
],
)
],
)
This way, also .egg-linking the package works (python setup.py develop), so you can link the package under development into a venv or so. This is also the reason for the way setuptools (and distutils) works: the dev sandbox should be structured in a way that allows running the code directly from it, without moving modules around.
The SWIG-generated hello.py and the generated extension _hello will then live under hello:
>>> from hello import hello, _hello
>>> print(hello)
<module 'hello.hello' from '~/so56562132/hello/hello.py'>
>>> print(_hello)
<module 'hello._hello' from '~/so56562132/hello/_hello.cpython-37m-darwin.so'>
(as you can see from the extension filename, I am on a Mac right now, but this works exactly the same under Windows)
Also, beyond packaging, there's more useful information about SWIG and Python namespaces and packages in the SWIG manual: http://swig.org/Doc4.0/Python.html#Python_nn72
I have a project named myproj structured like
/myproj
__init__.py
module1.py
module2.py
setup.py
my setup.py looks like this
from distutils.core import setup
setup(name='myproj',
version='0.1',
description='Does projecty stuff',
author='Me',
author_email='me#domain.com',
packages=[''])
But this places module1.py and module2.py in the install directory.
How do I specify setup such that the directory /myproj and all of it's contents are dropped into the install directory?
In your myproj root directory for this project, you want to move module1.py and module2.py into a directory named myproj under that, and if you wish to maintain Python < 3.3 compatibility, add a __init__.py into there.
├── myproj
│ ├── __init__.py
│ ├── module1.py
│ └── module2.py
└── setup.py
You may also consider using setuptools instead of just distutils. setuptools provide a lot more helper methods and additional attributes that make setting up this file a lot easier. This is the bare minimum setup.py I would construct for the above project:
from setuptools import setup, find_packages
setup(name='myproj',
version='0.1',
description="My project",
author='me',
author_email='me#example.com',
packages=find_packages(),
)
Running the installation you should see lines like this:
copying build/lib.linux-x86_64-2.7/myproj/__init__.py -> build/bdist.linux-x86_64/egg/myproj
copying build/lib.linux-x86_64-2.7/myproj/module1.py -> build/bdist.linux-x86_64/egg/myproj
copying build/lib.linux-x86_64-2.7/myproj/module2.py -> build/bdist.linux-x86_64/egg/myproj
This signifies that the setup script has picked up the required source files. Run the python interpreter (preferably outside this project directory) to ensure that those modules can be imported (not due to relative import).
On the other hand, if you wish to provide those modules at the root level, you definitely need to declare py_modules explicitly.
Finally, the Python Packaging User Guide is a good resource for more specific questions anyone may have about building distributable python packages.
For a directory structure like the following, I haven't been able to make xy an importable package.
xy
├── __init__.py
├── z
│ ├── __init__.py
│ └── stuff.py
└── setup.py
If the setup.py were a directory up, I could use
from setuptools import setup
setup(name='xy',
packages=['xy'])
but short of that, no combination of package_dir and packages has let me import xy, only import z. Unfortunately, moving the setup.py a directory up isn't really an option due to an excessive number of hard-coded paths.
See the following answer for ideas how to use package_dir and packages to help with such projects: https://stackoverflow.com/a/58429242/11138259
In short for this case here:
#!/usr/bin/env python3
import setuptools
setuptools.setup(
# ...
packages=['xy', 'xy.z'],
#packages=setuptools.find_packages('..') # only if parent directory is otherwise empty
package_dir={
'xy': '.',
},
)