We Keep Coding

Cythonize installs .so files to wrong location

I'd like to ask a question about how to configure setup.py with Cython, setuptools extensions etc. I'm trying cythonize a submodule with Cython.Distutils's build_ext. But the issue isn't Cython.Distutils's Extentsion module, (if it exists) because it isn't loaded. Only build_ext. So I create a setuptools.extension Extension in a list, and then cythonize the list of Extension objects. There's only one Extension in the list, an its as follows.
Extension("distance", ["kmerdb/distance.pyx"], include_dirs=[np.get_include()])
I've tried different methods of installation, from python setup.py install to pip install -e . to generating the wheel file and installing the wheel. I couldn't find anything that worked...
Okay, so I know very little about the process and that's probably why I am getting hung up but I've searched the whole site, no luck. So here goes.
I'm running the following shell script to install my package locally, and it works fine. The problem is the installation doesn't move the .so and the .py file into the kmerdb module I'm trying to export. Any suggestions? Thanks
>python setup.py sdist bdist_wheel
>/bin/auditwheel repair --plat manylinux2014_x86_64 dist/kmerdb-*linux_x86_64.whl
>mv wheelhouse/* dist
>rm dist/*linux_x86_64.whl
>pip install dist/kmerdb-*-manylinux2014_x86_64.whl
>ls ~/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.6.5-py3.10-linux-x86_64.egg/
distance.cpython-310-x86_64-linux-gnu.so distance.py kmerdb ...
Again, the files distance.cpython-310-x86_64-linux-gno.so is not moved into the kmerdb module, my Python packages, which I'm trying to install locally and configure for .whl upload to PyPI.
Python 3.10.1 (main, Jan 1 2022, 21:28:19) [GCC 11.1.0] on linux
Cython==0.29.26
setup.py
Extension("distance", ["kmerdb/distance.pyx"], include_dirs=[np.get_include()], define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],),
setup(
name=NAME,
version=VERSION,
description=DESCRIPTION,
long_description=long_description,
long_description_content_type='text/markdown',
author=AUTHOR,
author_email=EMAIL,
python_requires=REQUIRES_PYTHON,
url=URL,
download_url=CURRENT_RELEASE,
keywords = ["k-mer", "kmer", "k-merdb", "kmerdb", "kdb"],
classifiers=[
"Development Status :: 1 - Planning",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
"Topic :: Scientific/Engineering :: Bio-Informatics",
"Topic :: Software Development :: Libraries :: Python Modules",
],
packages=find_packages(exclude=["tests", "*.tests", "*.tests.*", "tests.*"]),
package_dir={'kmerdb': 'kmerdb'},
package_data={'kmerdb': ['CITATION']},
# If your package is a single module, use this instead of 'packages':
#py_modules=['kmerdb'],
#scripts=['bin/kmerdb', 'bin/kmerdb_report.R'],
entry_points={
'console_scripts': ['kmerdb=kmerdb:cli'],
},
install_requires=REQUIRED,#['Cython==0.29.21', 'numpy==1.18.1'],
extras_require=EXTRAS,
include_package_data=True,
license='GPLv3+',
test_suite='test',
# tests_require=['mamba', 'expect'],
ext_modules=cythonize(extensions),
library_dirs=["."],
zip_safe=False,
)

I'm going to self answer here. The issue stemmed from an improperly specified Extension.
Extension("kmerdb.distance", ["kmerdb/distance.pyx"], include_dirs=[np.get_include()])
All I had to do was include the module name for the fully specified submodule hierarchy. Fixed it!

Use different version of python package in the same environment

I need to have different versions of elasticseach-dsl installed in the same virtualenv. I would like to be able to import them using some sort of alias:
import elasticsearch_dsl1
import elasticsearch_dsl2
import elasticsearch_dsl5
I was thinking to create packages called elasricsearch_dsl* and in their setup.py add install requirements for the proper version of elasticsearch_dsl but when I install the packages it results in a conflict because all my packages require different versions of the same library.
I have no clue on how to proceed and if this is possible.
Thanks a lot to everyone

Short answer:
It is not possible.

I was able to find a solution to my problem.
I downloaded the source code of the elasticsearch-dsl libraries of different versions
Then changed the import statements to use elasticsearch1, elasticsearch2 etc.libraries instead of elasticsearch v1.x and v2.x
# example `elasticsearch-dsl` v5.4.0
from elasticsearch import Elasticsearch # --> from elasticsearch5 import Elasticsearch
Modified the setup.py to change the dependencies from elasticsearch to elasticsearch1, elasticsearch2 and so on for each version of the elasticsearch-dsl
I've been very lucky because of the existence of elasticsearch1, elasticsearch2... otherwise I would have to repeat the same procedure for them. And more lucky for the compatibility of dependencies of the various versions of the library.
I'm not proud of this hack but it worked.
setup.py
An example of a setup.py file for elasticsearch-dls==7.3.0, then elasticsearch-dsl7
from os.path import join, dirname
from setuptools import setup, find_packages
VERSION = (7, 3, 0)
__version__ = VERSION
__versionstr__ = ".".join(map(str, VERSION))
f = open(join(dirname(__file__), "README"))
long_description = f.read().strip()
f.close()
install_requires = [
"six",
"python-dateutil",
"elasticsearch7", # before "elasticsearch>=7.0.0,<8.0.0" <---
# ipaddress is included in stdlib since python 3.3
'ipaddress; python_version<"3.3"',
]
tests_require = [
"mock",
"pytest>=3.0.0",
"pytest-cov",
"pytest-mock<3.0.0",
"pytz",
"coverage<5.0.0",
]
setup(
name="elasticsearch-dsl7", # <---
description="Python client for Elasticsearch",
license="Apache-2.0",
url="https://github.com/elasticsearch/elasticsearch-dsl-py",
long_description=long_description,
version=__versionstr__,
author="Honza Král",
author_email="honza.kral#gmail.com",
maintainer="Seth Michael Larson",
maintainer_email="seth.larson#elastic.co",
packages=find_packages(where=".", exclude=("test_elasticsearch_dsl*",)),
python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
classifiers=[
"Development Status :: 4 - Beta",
"License :: OSI Approved :: Apache Software License",
"Intended Audience :: Developers",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
],
install_requires=install_requires,
test_suite="test_elasticsearch_dsl.run_tests.run_all",
tests_require=tests_require,
extras_require={"develop": tests_require + ["sphinx", "sphinx_rtd_theme"]},
)

setup.py script doesn't identify module

I have the following tree:
myPackage
|_myPackage
| |_mainScript.py
| |_f1.py
| |_f2.py
| |_Rscript.R
| |_ __init__.py
|_setup.py
|_MANIFEST.in
|_README.md
My aim is to create a setup.py file to install myPackage and to make it callable directly using the mypackage command from terminal.
the setup block from the setup.py code looks like this:
setup(
name=NAME,
version="1.0.0",
long_description=ldesc,
long_description_content_type="text/markdown",
author=AUTHOR,
author_email=EMAIL,
python_requires=REQUIRES_PYTHON,
url=URL,
packages=["myPackage"],
install_requires=REQUIRED,
package_data={"": ["*.R"]},
include_package_data=True,
entry_points={
"console_scripts": [
"mypackage=myPackage:main",
],
},
#include_package_data=True,
classifiers=[
"License :: OSI Approved :: GNU Lesser Public License v3 or Later",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.6",
"Programming Language :: R",
"Programming Language :: R :: 3",
"Programming Language :: R :: 3.6",
"Development Status :: 3 - Alpha",
"Operating System :: OS Independant",
],
)
Installation works, but I get the errorModuelNotFoundError: No module nmed "myPackage".
I also tried to use:
package_dir = {"": "myPackage"},
packages = find_packages()
scripts=["myPackage/mainScript.py.py"]
However, it returns the same error.
I would appreciate some help on this. I'm using Python3.6.0 and Ubuntu.
Thank you!

How about
"console_scripts": ["mypackage=myPackage.mainScript:main"],
assuming that's where def main(): lives?

My uploaded package is not searchable in pypi, neither in pypi web UI nor PIP3

I created a package, named furigana . This package page can be browsed by a user not logined to pypi. However, if I type "furigana" in the homepage of pypi, it can not find my package "furigana".
I also used pip3 search to search it, and it also find nothing.
I ran below commands to create archive and upload it by twine:
python3 setup.py test
python3 setup.py sdist
twine upload dist/furigana-0.0.7.tar.gz
The output message of twine is:
Uploading distributions to https://upload.pypi.org/legacy/
Uploading furigana-0.0.7.tar.gz
My ~/.pypirc is
[distutils]
index-servers =
pypi
[pypi]
repository: https://upload.pypi.org/legacy/
username:<my_username>
password:<my_password>
My setup.py is
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from setuptools import setup
from os import path
here = path.abspath(path.dirname(__file__))
# Get the long description from the relevant file
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
long_description = f.read()
setup(name='furigana',
version='0.0.7',
description='''convert Kanji in Japanese into Kanji attached with Hiragana (Furigana(振り仮名))
For example, "澱んだ街角" => "澱(よど)んだ街角(まちかど)" ''',
long_description = long_description,
author='Miki.Liu',
author_email='mikimotoh#gmail.com',
url='https://github.com/MikimotoH/furigana',
packages=['furigana'],
classifiers=[
# How mature is this project? Common values are
# 3 - Alpha
# 4 - Beta
# 5 - Production/Stable
'Development Status :: 3 - Alpha',
'Environment :: Console',
'Intended Audience :: Developers',
'Intended Audience :: Science/Research',
'License :: OSI Approved :: MIT License',
'Programming Language :: Python :: 3',
'Topic :: Software Development :: Libraries :: Python Modules',
'Topic :: Text Processing :: Linguistic',
],
keywords='Japanese Language Processing',
)
Cannot search package in PyPI with pip doesn't resolve my problem, the bitbucket repository is already removed.

Flake8 failed to load plugin "N8" on custom Formatter

I want to build a custom formatter for class and function names.
According to this doc it says that the naming convention falls under the N8** warning code.
After following this tutorial with the help of a sublink this is the resultant code
setup.py
from __future__ import with_statement
import setuptools
requires = [
"flake8 > 3.0.0",
]
setuptools.setup(
name="flake8_example",
license="MIT",
version="0.1.0",
description="our extension to flake8",
author="Me",
author_email="example#example.com",
url="https://gitlab.com/me/flake8_example",
packages=[
"flake8_example",
],
install_requires=requires,
entry_points={
'flake8.extension': [
'N8 = flake8_example:Example',
],
},
classifiers=[
"Framework :: Flake8",
"Environment :: Console",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 3",
"Topic :: Software Development :: Libraries :: Python Modules",
"Topic :: Software Development :: Quality Assurance",
],
)
flake8_example.py
from flake8.formatting import base
class Example(base.BaseFormatter):
"""Flake8's example formatter."""
def format(self, error):
return 'Example formatter: {0!r}'.format(error)
I setup it up by running pip install --editable .
Then to test it out I ran flake8 --format=example main.py
It throws this error:
flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "N8" due to 'module' object has no attribute 'Example'.

So if you're trying to write a formatter you need to read the documentation a bit more closely. In the registering section of the documentation it says:
Flake8 presently looks at three groups:
flake8.extension
flake8.listen
flake8.report
If your plugin is one that adds checks to Flake8, you will use
flake8.extension. If your plugin automatically fixes errors in code,
you will use flake8.listen. Finally, if your plugin performs extra
report handling (formatting, filtering, etc.) it will use
flake8.report.
(Emphasis mine.)
That means your setup.py should look like this:
entry_points = {
'flake8.report': [
'example = flake8_example:Example',
],
}
If your setup.py properly installs your package then running
flake8 --format=example ...
Should work just fine. The exception you're seeing, however, is due to the module either not having a class named Example. You should investigate whether packages will pick up a single file module or if you need to restructure your plugin so that it looks like:
flake8_example/
__init__.py
...
As that may be why your setup.py is not working appropriately.