We Keep Coding

Build Cython packages into wheels but the package is installed at wrong directory

I use Cython.Distutils to create my Cython package, a wheel file is created.
But installing the wheel, I found my built package is installed directly under site-package director. e.g., venv3/lib/python3.9/site-packages
I wish to install the package under the package name like venv3/lib/python3.9/site-packages/my_package
My command: python setup.py bdist_wheel
setup.py:
from Cython.Distutils import build_ext
from setuptools import setup
from Cython.Build import cythonize
from pathlib import Path
this_dir = Path(__file__).parent
setup(
name='MY PKG',
cmdclass={'build_ext': build_ext},
ext_modules=cythonize("src/*.py"),
install_requires=[
'numpy>=1.19.2',
'Cython>=0.29.28'
],
version="0.1.1")
Is there a way that I can copy the installed *.so files under a directory?
Such that I won't have to move the *.so files like
mkdir my_package && mv *.so my_package/

check if current python code is running from package

is it possible to detect if current python code is running from package?
if yes - is it possible to get package metadata (name, version, description)?
package is created with this kind of setup.py
import os
from setuptools import setup, find_packages
setup(
name='my-pack-name',
description='my description ' + os.getenv('GIT_COMMIT', '*')[:7],
version=os.getenv('BUILD_VERSION', '0.0.0dev'),
packages=find_packages(),
)
build: python3 setup.py bdist_wheel -d ./artifact
install on target: pip3 install "my-pack-name-x.x.x.whl" --upgrade
now from my_pack_name/app.py that was inside my-pack-name-x.x.x.whl i want to detect that i'm running from installed package
and if so then get package metadata defined during setup.py execution

For Python >=3.8
https://docs.python.org/es/3.10/library/importlib.metadata.html
You can get the metadata for a package by:
from importlib.metadata import metadata
md = metadata("your package name")
author = md["Author"]
# etc ...
For Python <3.8
This is just an idea (not tested).
What about having the package metadata in a different module, and try relative import it in the app.py module?
# metadata.py
name='my-pack-name',
description='my description ' + os.getenv('GIT_COMMIT', '*')[:7],
version=os.getenv('BUILD_VERSION', '0.0.0dev')
In setup.py you could reuse that:
# setup.py
import os
from setuptools import setup, find_packages
import .metadata as md
setup(
name=md.name,
description=md.description + os.getenv('GIT_COMMIT', '*')[:7],
version=md.version,
packages=find_packages(),
)
And the in app.py
def get_metadata():
try:
import .metadata as md
except ImportError:
return None
else:
# return metadata here
That way if get_metadata returns None, you were not able to import the module, so you're not executing the app.py in your package, otherwise, you are in your package and as a bonus you got your metadata.

For the test if python code is running from an installed package (vs. from a development location), I use
if 'site-packages' in __file__:
...
I don't know if that's a good approach; seems to work thus far.

How do I build multiple wheel files from a single setup.py?

In my project, I have a single setup.py file that builds multiple modules using the following namespace pattern:
from setuptools import setup
setup(name="testmoduleserver",
packages=["testmodule.server","testmodule.shared"],
namespace_packages=["testmodule"])
setup(name="testmoduleclient",
packages=["testmodule.client","testmodule.shared"],
namespace_packages=["testmodule"])
I am trying to build wheel files for both packages. However, when I do:
python -m pip wheel .
It only ever builds the package for one of the definitions.
Why does only one package get built?

You cannot call setuptools.setup() more than once in your setup.py, even if you want to create several packages out of one codebase.
Instead you need to separate everything out into separate namespace packages, and have one setup.py for each (they all can reside in one Git repository!):
testmodule/
testmodule-client/
setup.py
testmodule/
client/
__init__.py
testmodule-server/
setup.py
testmodule/
server/
__init__.py
testmodule-shared/
setup.py
testmodule/
shared/
__init__.py
And each setup.py contains something along the lines
from setuptools import setup
setup(
name='testmodule-client',
packages=['testmodule.client'],
install_requires=['testmodule-shared'],
...
)
and
from setuptools import setup
setup(
name='testmodule-server',
packages=['testmodule.server'],
install_requires=['testmodule-shared'],
...
)
and
from setuptools import setup
setup(
name='testmodule-shared',
packages=['testmodule.shared'],
...
)
To build all three wheels you then run
pip wheel testmodule-client
pip wheel testmodule-server
pip wheel testmodule-shared

Install a package using pip does not create all my files

Hello Denizens of the Exchange of Stackness,
I have a library that I'm trying to distribute. I have created a setup.py and run
python setup.py sdist
I see that it creates a .tar.gz file under the dist/ directory, which has all my files and folders that I want in it. However, when I install it on a Windows 8 box (running Python 3.6.5rc1), I don't get any files- only a kivydnd-0.5.dist-info directory. When I install it on Linux (running Fedora 26, Python 2.7), I do see the package's files but I don't see the examples directory.
Can you tell me what I'm doing wrong?
The setup.py is here:
from setuptools import setup, find_packages
setup(
name='kivydnd',
version='0.5',
description='Kivy Drag-n-Drop for Widgets',
url='https://github.com/GreyGnome/KivyDnD',
author='GreyGnome',
author_email='myemail#example.com',
license='Apache License 2.0',
#packages=find_packages('kivydnd'),
packages=['kivydnd'],
zip_safe=False,
scripts=[
'examples/dndexample1.py',
'examples/dndexample2.py',
'examples/dndexample3.py',
'examples/dndexample_copy_draggable.py',
'examples/dndexample_drop_groups.py',
'examples/dndexample_relative_layout.py',
'examples/example_base_classes.py',
'examples/example_base_classes.pyc',
]
)
In my development directory, I perform:
python setup.py sdist
The resulting .tar.gz looks like this; this will also reflect the structure of the directory where I'm doing my development:
kivydnd-0.5/
kivydnd-0.5/setup.py
kivydnd-0.5/PKG-INFO
kivydnd-0.5/examples/
kivydnd-0.5/examples/example_base_classes.pyc
kivydnd-0.5/examples/dndexample1.py
kivydnd-0.5/examples/dndexample_copy_draggable.py
kivydnd-0.5/examples/dndexample3.py
kivydnd-0.5/examples/dndexample_relative_layout.py
kivydnd-0.5/examples/dndexample_drop_groups.py
kivydnd-0.5/examples/dndexample2.py
kivydnd-0.5/examples/example_base_classes.py
kivydnd-0.5/README.md
kivydnd-0.5/RELEASE_NOTES.md
kivydnd-0.5/LICENSE
kivydnd-0.5/kivydnd.egg-info/
kivydnd-0.5/kivydnd.egg-info/top_level.txt
kivydnd-0.5/kivydnd.egg-info/PKG-INFO
kivydnd-0.5/kivydnd.egg-info/not-zip-safe
kivydnd-0.5/kivydnd.egg-info/SOURCES.txt
kivydnd-0.5/kivydnd.egg-info/dependency_links.txt
kivydnd-0.5/setup.cfg
kivydnd-0.5/MANIFEST.in
kivydnd-0.5/kivydnd/
kivydnd-0.5/kivydnd/dnd_storage_singletons.py
kivydnd-0.5/kivydnd/debug_print.py
kivydnd-0.5/kivydnd/__init__.py
kivydnd-0.5/kivydnd/dropdestination.py
kivydnd-0.5/kivydnd/dragndropwidget.py
Here is what happens on Windows 8:
F:\>pip install kivydnd-0.5.tar.gz
Processing f:\kivydnd-0.5.tar.gz
Building wheels for collected packages: kivydnd
Running setup.py bdist_wheel for kivydnd ... done
Stored in directory: C:\Users\schwager\AppData\Local\pip\Cache\wheels\9a\11\cd
\68bfb0d34c7b73ec7e25c6f9c40c5926377747b5951ac2e6ab
Successfully built kivydnd
Installing collected packages: kivydnd
Successfully installed kivydnd-0.5
` c:\users\schwager\python\Lib\site-packages\kivydnd-0.5.dist-info\` I have:
DESCRIPTION.rst
INSTALLER
METADATA
metadata.json
RECORD
top_level.txt
WHEEL
Here is what happens on Linux:
pip install --target=/home/schwager/lib/python kivydnd-0.5.tar.gz
Processing ./kivydnd-0.5.tar.gz
Installing collected packages: kivydnd
Running setup.py install for kivydnd ... done
Successfully installed kivydnd-0.5
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
 $ ls /home/schwager/lib/python
kivydnd kivydnd-0.5-py2.7.egg-info
 $ ls -R /home/schwager/lib/python
/home/schwager/lib/python:
kivydnd kivydnd-0.5-py2.7.egg-info
/home/schwager/lib/python/kivydnd:
debug_print.py dnd_storage_singletons.py dragndropwidget.py dropdestination.py __init__.py
debug_print.pyc dnd_storage_singletons.pyc dragndropwidget.pyc dropdestination.pyc __init__.pyc
/home/schwager/lib/python/kivydnd-0.5-py2.7.egg-info:
dependency_links.txt installed-files.txt not-zip-safe PKG-INFO SOURCES.txt top_level.txt

I appears that my setup.py should look like this. The package will get installed under Python's site-packages directory, the examples under <path-to-share>/kivydnd-examples.
from setuptools import setup, find_packages
from codecs import open
from os import path
with open(path.join('.', 'README.md'), encoding='utf-8') as f:
long_description = f.read()
setup(
name='kivydnd',
version='0.5.0',
description='Kivy Drag-n-Drop for Widgets',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://github.com/GreyGnome/KivyDnD',
author='GreyGnome',
author_email='myemail#example.com',
license='Apache License 2.0',
keywords='kivy drag-n-drop',
packages=find_packages(exclude=[]),
data_files=[('share/kivydnd-examples',
[
'examples/dndexample1.py',
'examples/dndexample2.py',
'examples/dndexample3.py',
'examples/dndexample_copy_draggable.py',
'examples/dndexample_drop_groups.py',
'examples/dndexample_relative_layout.py',
'examples/example_base_classes.py',
'examples/example_base_classes.pyc',
]
)],
)

Marking Cython as a Build Dependency?

There is a Python package with a setup.py that reads thusly:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
name = 'fastahack',
ext_modules=[
Extension("fastahack.cfastahack",
sources=["fastahack/cfastahack.pyx", "lib/Fasta.cpp", "lib/split.cpp"],
libraries=["stdc++"],
include_dirs=["lib/"],
language="c++"),
],
package_data = {'lib': ['*.pyx', "*.c", "*.h", "README.rst"]},
package_dir = {"fastahack": "fastahack"},
cmdclass = {'build_ext': build_ext},
packages = ['fastahack', 'fastahack.tests'],
author = "Brent Pedersen",
author_email="bpederse#gmail.com",
#test_suite='nose.collector'
)
This setup.py can't be imported if Cython is not installed. As far as I know, importing setup.py is how tools like pip figure out the dependencies of a package. I want to set up this package so that it could be uploaded to PyPI, with the fact that it depends on Cython noted, so that Cython will be downloaded and installed when you try to "pip install fastahack", or when you try to "pip install" directly from the Git repository.
How would I package this module so that it installs correctly from the Internet when Cython is not installed? Always using the latest version of Cython would be a plus.

You can specify Cython as a build dependency using PEP-518 project specification.
In the file pyproject.toml (in the same directory as setup.py) insert:
[build-system]
requires = ["setuptools", "wheel", "Cython"]
Cython will then be installed before building your package.
Note that (currently) you need to pass --no-use-pep517 to pip install if you are installing your package locally as editable (ie with --editable or -e) setuptools v64 supports editable installs with pyproject.toml builds

My standard template for setup.py:
have_cython = False
try:
from Cython.Distutils import build_ext as _build_ext
have_cython = True
except ImportError:
from distutils.command.build_ext import build_ext as _build_ext
if have_cython:
foo = Extension('foo', ['src/foo.pyx'])
else:
foo = Extension('foo', ['src/foo.c'])
setup (
...
ext_modules=[foo],
cmdclass={'build_ext': build_ext}
And don't forget to provide extention .c files with package - that will allow users to build module without installing cython.

Use a try and except for the Cython import and modify your setup based on whether or not your import succeeds. Look at the setup.py of Pandas for an example