We Keep Coding

Cythonize installs .so files to wrong location

I'd like to ask a question about how to configure setup.py with Cython, setuptools extensions etc. I'm trying cythonize a submodule with Cython.Distutils's build_ext. But the issue isn't Cython.Distutils's Extentsion module, (if it exists) because it isn't loaded. Only build_ext. So I create a setuptools.extension Extension in a list, and then cythonize the list of Extension objects. There's only one Extension in the list, an its as follows.
Extension("distance", ["kmerdb/distance.pyx"], include_dirs=[np.get_include()])
I've tried different methods of installation, from python setup.py install to pip install -e . to generating the wheel file and installing the wheel. I couldn't find anything that worked...
Okay, so I know very little about the process and that's probably why I am getting hung up but I've searched the whole site, no luck. So here goes.
I'm running the following shell script to install my package locally, and it works fine. The problem is the installation doesn't move the .so and the .py file into the kmerdb module I'm trying to export. Any suggestions? Thanks
>python setup.py sdist bdist_wheel
>/bin/auditwheel repair --plat manylinux2014_x86_64 dist/kmerdb-*linux_x86_64.whl
>mv wheelhouse/* dist
>rm dist/*linux_x86_64.whl
>pip install dist/kmerdb-*-manylinux2014_x86_64.whl
>ls ~/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.6.5-py3.10-linux-x86_64.egg/
distance.cpython-310-x86_64-linux-gnu.so distance.py kmerdb ...
Again, the files distance.cpython-310-x86_64-linux-gno.so is not moved into the kmerdb module, my Python packages, which I'm trying to install locally and configure for .whl upload to PyPI.
Python 3.10.1 (main, Jan 1 2022, 21:28:19) [GCC 11.1.0] on linux
Cython==0.29.26
setup.py
Extension("distance", ["kmerdb/distance.pyx"], include_dirs=[np.get_include()], define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],),
setup(
name=NAME,
version=VERSION,
description=DESCRIPTION,
long_description=long_description,
long_description_content_type='text/markdown',
author=AUTHOR,
author_email=EMAIL,
python_requires=REQUIRES_PYTHON,
url=URL,
download_url=CURRENT_RELEASE,
keywords = ["k-mer", "kmer", "k-merdb", "kmerdb", "kdb"],
classifiers=[
"Development Status :: 1 - Planning",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
"Topic :: Scientific/Engineering :: Bio-Informatics",
"Topic :: Software Development :: Libraries :: Python Modules",
],
packages=find_packages(exclude=["tests", "*.tests", "*.tests.*", "tests.*"]),
package_dir={'kmerdb': 'kmerdb'},
package_data={'kmerdb': ['CITATION']},
# If your package is a single module, use this instead of 'packages':
#py_modules=['kmerdb'],
#scripts=['bin/kmerdb', 'bin/kmerdb_report.R'],
entry_points={
'console_scripts': ['kmerdb=kmerdb:cli'],
},
install_requires=REQUIRED,#['Cython==0.29.21', 'numpy==1.18.1'],
extras_require=EXTRAS,
include_package_data=True,
license='GPLv3+',
test_suite='test',
# tests_require=['mamba', 'expect'],
ext_modules=cythonize(extensions),
library_dirs=["."],
zip_safe=False,
)

I'm going to self answer here. The issue stemmed from an improperly specified Extension.
Extension("kmerdb.distance", ["kmerdb/distance.pyx"], include_dirs=[np.get_include()])
All I had to do was include the module name for the fully specified submodule hierarchy. Fixed it!

Use different version of python package in the same environment

I need to have different versions of elasticseach-dsl installed in the same virtualenv. I would like to be able to import them using some sort of alias:
import elasticsearch_dsl1
import elasticsearch_dsl2
import elasticsearch_dsl5
I was thinking to create packages called elasricsearch_dsl* and in their setup.py add install requirements for the proper version of elasticsearch_dsl but when I install the packages it results in a conflict because all my packages require different versions of the same library.
I have no clue on how to proceed and if this is possible.
Thanks a lot to everyone

Short answer:
It is not possible.

I was able to find a solution to my problem.
I downloaded the source code of the elasticsearch-dsl libraries of different versions
Then changed the import statements to use elasticsearch1, elasticsearch2 etc.libraries instead of elasticsearch v1.x and v2.x
# example `elasticsearch-dsl` v5.4.0
from elasticsearch import Elasticsearch # --> from elasticsearch5 import Elasticsearch
Modified the setup.py to change the dependencies from elasticsearch to elasticsearch1, elasticsearch2 and so on for each version of the elasticsearch-dsl
I've been very lucky because of the existence of elasticsearch1, elasticsearch2... otherwise I would have to repeat the same procedure for them. And more lucky for the compatibility of dependencies of the various versions of the library.
I'm not proud of this hack but it worked.
setup.py
An example of a setup.py file for elasticsearch-dls==7.3.0, then elasticsearch-dsl7
from os.path import join, dirname
from setuptools import setup, find_packages
VERSION = (7, 3, 0)
__version__ = VERSION
__versionstr__ = ".".join(map(str, VERSION))
f = open(join(dirname(__file__), "README"))
long_description = f.read().strip()
f.close()
install_requires = [
"six",
"python-dateutil",
"elasticsearch7", # before "elasticsearch>=7.0.0,<8.0.0" <---
# ipaddress is included in stdlib since python 3.3
'ipaddress; python_version<"3.3"',
]
tests_require = [
"mock",
"pytest>=3.0.0",
"pytest-cov",
"pytest-mock<3.0.0",
"pytz",
"coverage<5.0.0",
]
setup(
name="elasticsearch-dsl7", # <---
description="Python client for Elasticsearch",
license="Apache-2.0",
url="https://github.com/elasticsearch/elasticsearch-dsl-py",
long_description=long_description,
version=__versionstr__,
author="Honza Král",
author_email="honza.kral#gmail.com",
maintainer="Seth Michael Larson",
maintainer_email="seth.larson#elastic.co",
packages=find_packages(where=".", exclude=("test_elasticsearch_dsl*",)),
python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
classifiers=[
"Development Status :: 4 - Beta",
"License :: OSI Approved :: Apache Software License",
"Intended Audience :: Developers",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
],
install_requires=install_requires,
test_suite="test_elasticsearch_dsl.run_tests.run_all",
tests_require=tests_require,
extras_require={"develop": tests_require + ["sphinx", "sphinx_rtd_theme"]},
)

Python, making a wheel file

I am trying to make wheel file and I want to add git repo in dependencies but I the following message appeared: error in me-pkg setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+ssh://g'".
The question is; How can I put git repo in dependencies or make this wheel file depend on another wheel file
setup.py :
import os
from setuptools import find_packages, setup
# allow setup.py to be run from any path
os.chdir(os.path.normpath(os.path.join(os.path.abspath(__file__), os.pardir)))
setup(
name='me-pkg',
version='0.1',
packages=find_packages(),
include_package_data=True,
license='BSD License', # example license
description='A simple Django app to conduct Web-based polls.',
url='https://www.example.com/',
author='xxxxx',
author_email='xxxxxxx#gmail.com',
classifiers=[
'Environment :: Web Environment',
'Framework :: Django',
'Framework :: Django :: X.Y', # replace "X.Y" as appropriate
'Intended Audience :: Developers',
'License :: OSI Approved :: BSD License', # example license
'Operating System :: OS Independent',
'Programming Language :: Python',
# Replace these appropriately if you are stuck on Python 2.
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Topic :: Internet :: WWW/HTTP',
'Topic :: Internet :: WWW/HTTP :: Dynamic Content',
],
install_requires=['git+https://github.com/xxx/xxx.git/stable/1.10.x'],
)
note : Dependency links not working

Pip can't find newly submitted package

I submitted a python3 enabled version of pynliner to pip yesterday but can't seem to install it locally.
https://pypi.python.org/pypi/pynliner3/
pip install pynliner3
Updating cache with response from "https://pypi.python.org/simple/pynliner3/"
Caching b/c date exists and max-age > 0
Analyzing links from page https://pypi.python.org/simple/pynliner3/
Could not find a version that satisfies the requirement pynliner3 (from versions: )
I see similar questions about this on Stackoverflow and the response always seem to be a workaround. Example "point pip directly to the tarball or repo".
However, I want to pose a different question here.
How do I get this to work as is?
Is this just a caching thing on pypi.python.org and I have to wait?
If so, how long does it take?
Sorry for dumb questions. Just trying to figure out how pip works. Thanks for your help.
And for reference here's my setup.py (clone from the orignal py2 repo)
from setuptools import setup
setup(name='pynliner3',
version='0.6',
description='Python CSS-to-inline-styles conversion tool for HTML using'
' BeautifulSoup and cssutils',
author='Tanner Netterville',
author_email='tannern#gmail.com',
install_requires=[
'BeautifulSoup4 >= 4.4.1',
'cssutils >=0.9.7',
],
tests_require=[
'mock'
],
test_suite='tests',
packages=['pynliner'],
license='MIT',
classifiers=[
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'License :: OSI Approved :: MIT License',
'Topic :: Text Processing :: Markup :: HTML',
'Programming Language :: Python',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5'
])

No distributions at all found, even though I recently published the package

Hello fellow programmers,
So lately I've been working on a project at my work that we want to open source. It's the package called django-push-notifications-manager (yes, what a unusual long name I know).
So I already did the registration and the uploading of the package, but for some kind of reason pip install django-push-notifications-manager will not work and will give the error:
Downloading/unpacking django-push-notifications-manager
Could not find any downloads that satisfy the requirement django-push-notifications-manager
Cleaning up...
No distributions at all found for django-push-notifications-manager
Storing debug log for failure in /Users/pauloostenrijk/.pip/pip.log
So I haven't been able to fix it by removing and replacing the package, deleting it, making a new version. None of them worked.
I think you guys would like to know what my setup.py was, so hereby:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import sys
import push_notifications
def get_packages(package):
"""
Return root package and all sub-packages.
"""
return [dirpath
for dirpath, dirnames, filenames in os.walk(package)
if os.path.exists(os.path.join(dirpath, '__init__.py'))]
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
version = push_notifications.__version__
if sys.argv[-1] == 'publish':
os.system('python setup.py sdist upload')
print("You probably want to also tag the version now:")
print(" git tag -a %s -m 'version %s'" % (version, version))
print(" git push --tags")
sys.exit()
readme = open('README.rst').read()
setup(
name='django-push-notifications-manager',
version=version,
description="""A plug and play package to handle push devices and push notifications for services such as ZeroPush and Urban Airship""",
long_description=readme,
author='Paul Oostenrijk',
author_email='paul#glemma.nl',
url='https://github.com/glemmaPaul/django-push-notifications-manager',
packages=get_packages('push_notifications'),
include_package_data=True,
install_requires=[
'django>=1.5.1',
'requests>=2.5.1'
],
license="BSD",
zip_safe=False,
keywords='django-push-notifications-manager',
classifiers=[
'Development Status :: 2 - Pre-Alpha',
'Framework :: Django',
'Intended Audience :: Developers',
'License :: OSI Approved :: BSD License',
'Natural Language :: English',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
],
)
Hope any of you guys could help me out! Thanks in advance :)

So after spending a long time of trying and figuring out I finally found the problem.
Don't look strange to me, but apparently PyPi has problems with the name django-push-notifications-manager, I think there can be 2 causes:
The length of the name
That having 2 dashes in the name is the maximum
I hope nobody will ever get this problem, it was head cracking!
Thanks for your time!