We Keep Coding

Using pypa's build on a python project leads to a generic "none-any.whl" wheel, but the package has OS-specific binaries (cython)

I am trying to build a package for distribution which has cython code that I would like to compile into binaries before uploading to PyPI. To do this I am using pypa's build,
python -m build
in the project's root directory. This cythonizes the code and generates the binaries for my system then creates the sdist and wheel in the dist directory. However, the wheel is named "--py3-none-any.whl". When I unzip the .whl I do find the appropriate binaries stored,
(e.g., cycode.cp39-win_amd64.pyd). The problem is I plan to run this in a GitHub workflow where binaries are built for multiple python versions and operating systems. That workflow works fine but overwrites (or causes a duplicate version error) when uploading to PyPI since all of the wheels from the various OS share the same name. Then if I install from PyPI on another OS I get "module can't be found" errors since the binaries for that OS are not there and, since it was a wheel, the installation did not re-compile the cython files.
I am working with 64-bit Windows, MacOS, and Ubuntu. Python versions 3.8-3.10. And a small set of other packages which are listed below.
Does anyone see what I am doing wrong here? Thanks!
Simplified Package
Tests\
Project\
__init__.py
pycode.py
cymod\
__init__.py
_cycode.pyx
_build.py
pyproject.toml
pyproject.toml
[project]
name='Project'
version = '0.1.0'
description = 'My Project'
authors = ...
requires-python = ...
dependencies = ...
[build-system]
requires = [
'setuptools>=64.0.0',
'numpy>=1.22',
'cython>=0.29.30',
'wheel>=0.38'
]
build-backend = "setuptools.build_meta"
[tool.setuptools]
py-modules = ["_build"]
include-package-data = true
packages = ["Project",
"Project.cymod"]
[tool.setuptools.cmdclass]
build_py = "_build._build_cy"
_build.py
import os
from setuptools.extension import Extension
from setuptools.command.build_py import build_py as _build_py
class _build_cy(_build_py):
def run(self):
self.run_command("build_ext")
return super().run()
def initialize_options(self):
super().initialize_options()
import numpy as np
from Cython.Build import cythonize
print('!-- Cythonizing')
if self.distribution.ext_modules == None:
self.distribution.ext_modules = []
# Add to ext_modules list
self.distribution.ext_modules.append(
Extension(
'Project.cymod.cycode',
sources=[os.path.join('Project', 'cymod', '_cycode.pyx')],
include_dirs=[os.path.join('Project', 'cymod'), np.get_include()]
)
)
# Add cythonize ext_modules
self.distribution.ext_modules = cythonize(
self.distribution.ext_modules,
compiler_directives={'language_level': "3"},
include_path=['.', np.get_include()]
)
print('!-- Finished Cythonizing')

Install .desktop file with setuptools and pyproject.toml

I have a GUI Python app that I'm trying to distribute a desktop entry with. Normally, one would write a setup.py with setuptools that has this in it:
from setuptools import setup
setup(
name = 'myapp',
version = '0.0.1',
packages = ['myapp'],
data_files = [
('share/applications', ['myapp.desktop']),
],
)
This is deprecated, however, and my goal is to use only pyproject.toml in my repo with no setup.py or setup.cfg needed. I have been unable to find any information on how I would go about doing this.

How to add/install python libraries to my github project?

I'm building my first project on GitHub and my python src code uses an open-source, 3rd-party library that I have installed on my computer. However, I heard it is best practice to create a dep (dependencies) folder to store any additional libraries I would need. How do I actually install the libraries in the dep folder and use them from there instead of my main computer?

You have to create a requirements.txt file with each package on a separate line. e.g.
pandas==0.24.2
You also might want to add a setup.py to your python package. In the setup you have to use "install_requires" argument. Although install_requires will not install packages when installing your package but will let the user know which packages are needed. The user can refer to the requirements.txt to see the requirements.
You can check it here: https://packaging.python.org/discussions/install-requires-vs-requirements/
The following is an example of setup.py file:
from distutils.core import setup
from setuptools import find_packages
setup(
name='foobar',
version='0.0',
packages=find_packages(),
url='',
license='',
author='foo bar',
author_email='foobar#gmail.com',
description='A package for ...'
install_requires=['A','B']
)

Never heard about installing additional libraries in a dependencies folder.
Create a setup python file in your root folder if you don't already have it, in there you can define what packages (libraries as you call them) your project needs. This is a simple setup file for example:
from setuptools import setup, find_packages
setup(
name = "yourpackage",
version = "1.2.0",
description = "Simple description",
packages = find_packages(),
install_requires = ['matplotlib'] # Example of external package
)
When installing a package that has this setup file it automatically also install every requirement in your VENV. And if you're using pycharm then it also warns you if there's a requirement that's not installed.

Why does setuptools not understand git+https URLs?

According to Dependency section in the setuptools manual git repository URLs can be specified in the dependency_links argument to setup with git+URL. Yet,
cd /tmp
mkdir py-test
cd py-test
touch __init__.py
and creation of a setup.py file with
from setuptools import setup, find_packages
from pkg_resources import parse_version
setup(
name = "py-test",
version = "1.0",
packages = ["."],
dependency_links = [
"git+https://github.com/wxWidgets/wxPython.git"
],
install_requires = ["wxPython"],
)
causes the error Download error on git+https://github.com/wxWidgets/wxPython.git: unknown url type: git+https -- Some packages may not be found! when I run python setup.py build && sudo setup.py install.
The installation of the package python-setuptools-git doesn't help.
I'm using setuptools 18.2 with python 2.7 on Ubuntu 15.04.

From the setuptools docs:
In the case of a VCS checkout, you should also append #egg=project-version in order to identify for what package that checkout should be used
So the fix is just to append the #egg=wxPython fragment onto the end:
dependency_links = [
"git+https://github.com/wxWidgets/wxPython.git#egg=wxPython"
]

Marking Cython as a Build Dependency?

There is a Python package with a setup.py that reads thusly:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
name = 'fastahack',
ext_modules=[
Extension("fastahack.cfastahack",
sources=["fastahack/cfastahack.pyx", "lib/Fasta.cpp", "lib/split.cpp"],
libraries=["stdc++"],
include_dirs=["lib/"],
language="c++"),
],
package_data = {'lib': ['*.pyx', "*.c", "*.h", "README.rst"]},
package_dir = {"fastahack": "fastahack"},
cmdclass = {'build_ext': build_ext},
packages = ['fastahack', 'fastahack.tests'],
author = "Brent Pedersen",
author_email="bpederse#gmail.com",
#test_suite='nose.collector'
)
This setup.py can't be imported if Cython is not installed. As far as I know, importing setup.py is how tools like pip figure out the dependencies of a package. I want to set up this package so that it could be uploaded to PyPI, with the fact that it depends on Cython noted, so that Cython will be downloaded and installed when you try to "pip install fastahack", or when you try to "pip install" directly from the Git repository.
How would I package this module so that it installs correctly from the Internet when Cython is not installed? Always using the latest version of Cython would be a plus.

You can specify Cython as a build dependency using PEP-518 project specification.
In the file pyproject.toml (in the same directory as setup.py) insert:
[build-system]
requires = ["setuptools", "wheel", "Cython"]
Cython will then be installed before building your package.
Note that (currently) you need to pass --no-use-pep517 to pip install if you are installing your package locally as editable (ie with --editable or -e) setuptools v64 supports editable installs with pyproject.toml builds

My standard template for setup.py:
have_cython = False
try:
from Cython.Distutils import build_ext as _build_ext
have_cython = True
except ImportError:
from distutils.command.build_ext import build_ext as _build_ext
if have_cython:
foo = Extension('foo', ['src/foo.pyx'])
else:
foo = Extension('foo', ['src/foo.c'])
setup (
...
ext_modules=[foo],
cmdclass={'build_ext': build_ext}
And don't forget to provide extention .c files with package - that will allow users to build module without installing cython.

Use a try and except for the Cython import and modify your setup based on whether or not your import succeeds. Look at the setup.py of Pandas for an example