We Keep Coding

Cythonize installs .so files to wrong location

I'd like to ask a question about how to configure setup.py with Cython, setuptools extensions etc. I'm trying cythonize a submodule with Cython.Distutils's build_ext. But the issue isn't Cython.Distutils's Extentsion module, (if it exists) because it isn't loaded. Only build_ext. So I create a setuptools.extension Extension in a list, and then cythonize the list of Extension objects. There's only one Extension in the list, an its as follows.
Extension("distance", ["kmerdb/distance.pyx"], include_dirs=[np.get_include()])
I've tried different methods of installation, from python setup.py install to pip install -e . to generating the wheel file and installing the wheel. I couldn't find anything that worked...
Okay, so I know very little about the process and that's probably why I am getting hung up but I've searched the whole site, no luck. So here goes.
I'm running the following shell script to install my package locally, and it works fine. The problem is the installation doesn't move the .so and the .py file into the kmerdb module I'm trying to export. Any suggestions? Thanks
>python setup.py sdist bdist_wheel
>/bin/auditwheel repair --plat manylinux2014_x86_64 dist/kmerdb-*linux_x86_64.whl
>mv wheelhouse/* dist
>rm dist/*linux_x86_64.whl
>pip install dist/kmerdb-*-manylinux2014_x86_64.whl
>ls ~/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.6.5-py3.10-linux-x86_64.egg/
distance.cpython-310-x86_64-linux-gnu.so distance.py kmerdb ...
Again, the files distance.cpython-310-x86_64-linux-gno.so is not moved into the kmerdb module, my Python packages, which I'm trying to install locally and configure for .whl upload to PyPI.
Python 3.10.1 (main, Jan 1 2022, 21:28:19) [GCC 11.1.0] on linux
Cython==0.29.26
setup.py
Extension("distance", ["kmerdb/distance.pyx"], include_dirs=[np.get_include()], define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],),
setup(
name=NAME,
version=VERSION,
description=DESCRIPTION,
long_description=long_description,
long_description_content_type='text/markdown',
author=AUTHOR,
author_email=EMAIL,
python_requires=REQUIRES_PYTHON,
url=URL,
download_url=CURRENT_RELEASE,
keywords = ["k-mer", "kmer", "k-merdb", "kmerdb", "kdb"],
classifiers=[
"Development Status :: 1 - Planning",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
"Topic :: Scientific/Engineering :: Bio-Informatics",
"Topic :: Software Development :: Libraries :: Python Modules",
],
packages=find_packages(exclude=["tests", "*.tests", "*.tests.*", "tests.*"]),
package_dir={'kmerdb': 'kmerdb'},
package_data={'kmerdb': ['CITATION']},
# If your package is a single module, use this instead of 'packages':
#py_modules=['kmerdb'],
#scripts=['bin/kmerdb', 'bin/kmerdb_report.R'],
entry_points={
'console_scripts': ['kmerdb=kmerdb:cli'],
},
install_requires=REQUIRED,#['Cython==0.29.21', 'numpy==1.18.1'],
extras_require=EXTRAS,
include_package_data=True,
license='GPLv3+',
test_suite='test',
# tests_require=['mamba', 'expect'],
ext_modules=cythonize(extensions),
library_dirs=["."],
zip_safe=False,
)

I'm going to self answer here. The issue stemmed from an improperly specified Extension.
Extension("kmerdb.distance", ["kmerdb/distance.pyx"], include_dirs=[np.get_include()])
All I had to do was include the module name for the fully specified submodule hierarchy. Fixed it!

How to get rid of library root mark in PyCharm

I've started working on a Pyramid application, which requires setup.py usage, but as soon as I built the app, my app folder was marked as library root.
It is not convenient, because when I open a file, it is also opened under External Libraries unfolding it. This can be "fixed" by removing check on Always Select Opened File, but I like this feature, so I don't want to disable it.
I've also tried to tweak Project Structure in settings, but it didn't help.
How to get rid of this _library_root_ mark?
UPD. Contents of setup.py:
setup(
name='app',
version=0.1,
description='Blog with CMS',
classifiers=[
"Programming Language :: Python",
"Framework :: Pylons",
"Topic :: Internet :: WWW/HTTP",
"Topic :: Internet :: WWW/HTTP :: WSGI :: Application"
],
keywords="web services",
author='',
author_email='',
url='',
packages=find_packages(),
include_package_data=True,
zip_safe=False,
install_requires=['cornice', 'waitress'],
entry_points="""\
[paste.app_factory]
main=app:main
""",
paster_plugins=['pyramid']
)

The reason why it happens is PyCharm adds sd-blog/app to the interpreter paths once you pip install -e in order to be able to provide completion for app objects. Possible workaround:
remove app from the interpreter paths
mark sd-blog/app as a source root to restore the code insight broken in the step 1

Flake8 failed to load plugin "N8" on custom Formatter

I want to build a custom formatter for class and function names.
According to this doc it says that the naming convention falls under the N8** warning code.
After following this tutorial with the help of a sublink this is the resultant code
setup.py
from __future__ import with_statement
import setuptools
requires = [
"flake8 > 3.0.0",
]
setuptools.setup(
name="flake8_example",
license="MIT",
version="0.1.0",
description="our extension to flake8",
author="Me",
author_email="example#example.com",
url="https://gitlab.com/me/flake8_example",
packages=[
"flake8_example",
],
install_requires=requires,
entry_points={
'flake8.extension': [
'N8 = flake8_example:Example',
],
},
classifiers=[
"Framework :: Flake8",
"Environment :: Console",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 3",
"Topic :: Software Development :: Libraries :: Python Modules",
"Topic :: Software Development :: Quality Assurance",
],
)
flake8_example.py
from flake8.formatting import base
class Example(base.BaseFormatter):
"""Flake8's example formatter."""
def format(self, error):
return 'Example formatter: {0!r}'.format(error)
I setup it up by running pip install --editable .
Then to test it out I ran flake8 --format=example main.py
It throws this error:
flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "N8" due to 'module' object has no attribute 'Example'.

So if you're trying to write a formatter you need to read the documentation a bit more closely. In the registering section of the documentation it says:
Flake8 presently looks at three groups:
flake8.extension
flake8.listen
flake8.report
If your plugin is one that adds checks to Flake8, you will use
flake8.extension. If your plugin automatically fixes errors in code,
you will use flake8.listen. Finally, if your plugin performs extra
report handling (formatting, filtering, etc.) it will use
flake8.report.
(Emphasis mine.)
That means your setup.py should look like this:
entry_points = {
'flake8.report': [
'example = flake8_example:Example',
],
}
If your setup.py properly installs your package then running
flake8 --format=example ...
Should work just fine. The exception you're seeing, however, is due to the module either not having a class named Example. You should investigate whether packages will pick up a single file module or if you need to restructure your plugin so that it looks like:
flake8_example/
__init__.py
...
As that may be why your setup.py is not working appropriately.

What difference between "pip install" and "setup.py install" for static files

I have a problem with pip && setuptools. I have a simple project here: https://github.com/rmuslimov/rapidlog
If I make these commands:
mkvirtualenv rtests
pip install git+file:///%path_to_this_project%
rapidagent # This my app in this project
It works properly and creates templates and static files.
If I do:
mkvirtualenv rtests
cd %path_to_this_project%
python setup.py install
rapidagent
It cannot install my templates and static files data. So I can't start my application.
Here is the end of my setup.py file:
entry_points={
'console_scripts': [
'rapidagent = rapidlog.web.webagent:main'
],
},
include_package_data=True,
data_files=[('rapidlog/web/templates', ['rapidlog/web/templates/index.html']),
('rapidlog/web/static/css', glob('rapidlog/web/static/css/*')),
('rapidlog/web/static/images', glob('rapidlog/web/static/images/*')),
('rapidlog/web/static/js', glob('rapidlog/web/static/js/*')),
],
install_requires=[
'pika>=0.9.5',
'tornado>=2.3',
'wsgiref>=0.1.2',
],
classifiers=[
'License :: OSI Approved :: BSD License',
'Programming Language :: Python'
]
What special command does pip call ? What is the better way to solve this?

I find out that your os.path.join calls are relative to the current directory your python is called. So, the first thing is:
web/webagent.py: Change os.path.join('templates') and os.path.join('static') to be absolute
I just sent your a pull request with this change.
Update me with your next steps.

using Mysql and SqlAlchemy in Pyramid Framework

Pyramid Framework comes with a sample tutorial of sql alchemy that uses sqlite. The problem is that i want to use mysql so i change this
sqlalchemy.url = sqlite:///%(here)s/tutorial.db
Into this
sqlalchemy.url = mysql://root:22password#localhost/alchemy
when i try to run
../bin/pserve development.ini --reload
It gives me the following error
File "build/bdist.linux-i686/egg/sqlalchemy/connectors/mysqldb.py", line 52, in dbapi
ImportError: No module named MySQLdb
I understand that i should include the dependecies of my app in setup.py but i don't know what to include right now some help please my setup.py looks like this
import os
import sys
from setuptools import setup, find_packages
here = os.path.abspath(os.path.dirname(__file__))
README = open(os.path.join(here, 'README.txt')).read()
CHANGES = open(os.path.join(here, 'CHANGES.txt')).read()
requires = [
'pyramid',
'SQLAlchemy',
'transaction',
'pyramid_tm',
'pyramid_debugtoolbar',
'zope.sqlalchemy',
]
if sys.version_info[:3] < (2,5,0):
requires.append('pysqlite')
setup(name='tutorial',
version='0.0',
description='tutorial',
long_description=README + '\n\n' + CHANGES,
classifiers=[
"Programming Language :: Python",
"Framework :: Pylons",
"Topic :: Internet :: WWW/HTTP",
"Topic :: Internet :: WWW/HTTP :: WSGI :: Application",
],
author='',
author_email='',
url='',
keywords='web wsgi bfg pylons pyramid',
packages=find_packages(),
include_package_data=True,
zip_safe=False,
test_suite='tutorial',
install_requires = requires,
entry_points = """\
[paste.app_factory]
main = tutorial:main
[console_scripts]
populate_tutorial = tutorial.scripts.populate:main
""",
)

Try adding "MySQLdb" to the requires list. It was fine with sqlite3 as that comes with python (as of version 2.5), MySQLdb doesn't and needs to be installed separately.
UPDATE:
Try "mysql-python" in the requires list instead.

Okay i Solved my own question by following the directions on this page
http://www.saltycrane.com/blog/2010/02/install-mysqldb-virtualenv-ubuntu-karmic/

There is the huge list of Unofficial Windows Binaries for Python Extension Packages which are extremely useful for Windows users.
http://www.lfd.uci.edu/~gohlke/pythonlibs/

Solved in ubuntu environment
sudo apt-get build-dep python-mysqldb
source /bin/active && pip install MySQL-python