I'm trying to build a package that uses both python and cython modules. The problem I'm having deals with imports after building and installing where I'm not sure how to make files import from the .so file generated by the build process.
Before building my folder structure looks like this
root/
├── c_integrate.c
├── c_integrate.pyx
├── cython_builder.py
├── __init__.py
├── integrator_class.py
├── integrator_modules
│ ├── cython_integrator.py
│ ├── __init__.py
│ ├── integrator.py
│ ├── numba_integrator.py
│ ├── numpy_integrator.py
│ ├── quadratic_error.png
│ ├── report3.txt
│ ├── report4.txt
│ └── report5.txt
├── report6.txt
├── setup.py
└── test
├── __init__.py
└── test_integrator.py
Building with python3.5 setup.py build gives this new folder in root
root/build/
├── lib.linux-x86_64-3.5
│ ├── c_integrate.cpython-35m-x86_64-linux-gnu.so
│ ├── integrator_modules
│ │ ├── cython_integrator.py
│ │ ├── __init__.py
│ │ ├── integrator.py
│ │ ├── numba_integrator.py
│ │ └── numpy_integrator.py
│ └── test
│ ├── __init__.py
│ └── test_integrator.py
The setup.py file looks like this
from setuptools import setup, Extension, find_packages
import numpy
setup(
name = "integrator_package",
author = "foo",
packages = find_packages(),
ext_modules = [Extension("c_integrate", ["c_integrate.c"])],
include_dirs=[numpy.get_include()],
)
My question is then: how do I write import statements of the functions from the .so file into ìntegrator_class.py in root and cython_integrator and test_integrator located in the build directory. Appending to sys.path seems like a quick and dirty solution that I don't much like.
EDIT:
As pointed out in the comments I haven't installed the package. This is because I don't know what to write to import from the .so file
In no specific order:
The file setup.py is typically located below the root of a project. Example:
library_name/
__init__.py
file1.py
setup.py
README
Then, the build directory appears alongside the project's source and not in the project source.
To import the file c_integrate.cpython-35m-x86_64-linux-gnu.so in Python, just import "c_integrate". The rest of the naming is taken care of automatically as it is just the platform information. See PEP 3149
A valid module is one of
a directory with a modulename/__init__.py file
a file named modulename.py
a file named modulename.PLATFORMINFO.so
of course located in the Python path. So there is no need for a __init__.py file for a compiled Cython module.
For your situation, move the Cython code in the project directory and either do a relative import import .c_integrate or a full from integrator_modules import c_integrate where the latter only works when your package is installed.
A few of this information can be found in my blog post on Cython modules http://pdebuyl.be/blog/2017/cython-module.html
I believe that this should let you build a proper package, comment below if not.
EDIT: to complete the configuration (see comments below), the poster also
Fixed the module path in the setup.py file so that it is the full module name starting from the PYTHONPATH: Extension("integrator_package.integrator_modules.c_integrator", ["integrator_package/integrator_modules/c_integrator.c"] instead of Extension("c_integrate", ["c_integrate.c"])]
Cythonize the module, build it and use with a same Python interpreter.
Further comment: the setup.py file can cythonize the file as well. Include the .pyx file instead of the .c file as the source.
cythonize(Extension('integrator_package.integrator_modules.c_integrator',
["integrator_package/integrator_modules/c_integrator.pyx"],
include_dirs=[numpy.get_include()]))
Related
My Directory structure:
├── common
│ ├── common.py
│ └── __init__.py
├── project1
│ ├── __init__.py
│ └── scripts
│ ├── example_import.py
│ └── __init__.py
└── project2
├── __init__.py
└── scripts
└── __init__.py
I need to import common/common.py module in project1/scripts/example_import.py file
example_import.py:
import sys
sys.path.append("../common")
from common import Test
print("Module Not import error")
Error:
Traceback (most recent call last):
File "project1/scripts/example_import.py", line 3, in <module>
from common import Test
ImportError: No module named common
How to fix a issue?
Understanding how Python imports work is tricky in the beginning but makes sense once you understand how.
There are different way to fix your import issue. I would not recommend messing up with sys.path. Depending on where you are calling your script, you have multiple choice at hand.
Your Directory structure.
├── common
│ ├── common.py
│ └── __init__.py
├── project1
│ ├── __init__.py
│ └── scripts
│ ├── example_import.py
│ └── __init__.py
└── project2
├── __init__.py
└── scripts
└── __init__.py
On the root of directory
python project1/scripts/example_import.py
Will work, assuming that the imports in example_import.py looks like
from common.common import Test
If you want to use from common import Test, you need to add from common.common import Test in __init__.py file in common folder.
You can also use PYTHONPATH environment variable to tell Python where to look for the base module.
PYTHONPATH=/pathto/basefolder python folder/filesx.py
Another way is to create a setup.py on base and do development installation python -m pip install --editable . on environment you are working on.
Example of setup.py
#!/usr/bin/env python
from distutils.core import setup
setup(name='projectX',
version='1.0',
description='The World is not Enoug',
author='James Bond',
author_email='agent#007.net',
url='https://www.python.org/sigs/distutils-sig/',
packages=['common', 'project1', 'project2],
)
See Python Documentation for more setup.py options.
I'd like to figure out the cleanest and preferably self contained way to use my packages in scripts that are in a different directory to the package itself.
The example problem is as follows:
The modules in lib need to both be imported, and run as a script.
My project directory is as below and I'm having two issues:
in lib/api.py, I want to read in data_files/key.txt correctly when api.py is called or imported
in testing_script.py I want to import and use lib/get_data.py
I can't seem to find a clean way to do this, does this mean my project is structured in a non-pythonic way?
Thanks for the help.
my-project-git
├── LICENSE
├── README.md
├─── my_project
│ ├── data_files
│ │ ├── key.txt
│ │ ├── mappings.csv
│ ├── lib
│ │ ├── __init__.py
│ │ ├── api.py
│ │ └── get_data.py
│ └── test
│ ├── __init__.py
│ └── testing_script.py
├── requirements.txt
└── setup.py
As far as I know, there's isn't a pythonic way to structure your project.
This is what Kenneth Reitz recommended in 2013 and it's how I use it: https://www.kennethreitz.org/essays/repository-structure-and-python.
README.rst
LICENSE
setup.py
requirements.txt
sample/__init__.py
sample/core.py
sample/helpers.py
docs/conf.py
docs/index.rst
tests/test_basic.py
tests/test_advanced.py
Inside sample (my_project in your case) you can separate into categories as you like. E.g. Utils (common functions), Database (read, write), View (user commands), etc. It depends on your project.
As for calling the modules at the same level, you should define them in the __init__ file of the top hierarchy module which is sample in this case.
For example:
__init__ in _my_project
from sample.core import a_Class
from sample.core import a_function
from sample.core import anything
then from /test/test_basic.py you do:
from sample import a_Class
# or import sample
a = a_Class() # use the class from core.py
# or a = sample.a_Class()
Take a look at the sample module repository: https://github.com/navdeep-G/samplemod
I'm trying to build my first python package public available but I'm having some trouble with installing it on another machine, not sure what is wrong. My project is here.
After all the CI steps on the master branch, Travis publishes the latest version to the pypi. After that, we can install the package in any place:
pip install spin-clustering
But when I try to import it on my regular python it says that the module does not exist.
$ python -c "import spin"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'spin'
My package was originally called "spin" but the name was already taken on pypi, I changed it to "spin-clustering", but as scikit-learn is imported with "sklearn" I thought that would be possible to import my package as "spin". Not sure what I'm missing here.
This is my package structure:
├── LICENSE
├── Makefile
├── Pipfile
├── README.md
├── examples
│ ├── circle-example.ipynb
│ └── random-cluster-example.ipynb
├── setup.cfg
├── setup.py
└── spin
├── __init__.py
├── distances
│ ├── __init__.py
│ ├── distances.py
│ └── tests
│ └── __init__.py
├── neighborhood_spin.py
├── side_to_side_spin.py
├── tests
│ ├── __init__.py
│ ├── test_spin.py
│ └── test_utils.py
└── utils.py
And my setup.py
from setuptools import setup, find_packages
setup(name="spin-clustering",
maintainer="otaviocv",
maintainer_email="otaviocv.deluqui#gmail.com",
description="SPIN clustering method package.",
license="MIT",
version="0.0.3",
python_requires=">=3.6",
install_requires=[
'numpy>=1.16.4',
'matplotlib>=3.1.0'
]
)
In your setup.py, you also need to specify what packages will be installed. The simplest way is using the provided find_packages function, which will scan your folders and try to figure out what the packages are (in some slightly unusual cases, your project organization will make this not work right). Your code imports find_packages, but is not using it.
Since you have none listed, nothing is actually installed (except the requirements, if missing).
I am writing a Python 3 package, and I need to know how python determines which module is considered "main"? Is this something that is configured in setup.py, or some other file? Which setting in setup.py determines which module is called as __main__, like is it the entry_points attribute, or the packages attribute, or some other one?
Here is a simplified example of the package structure:
├── AUTHORS.rst
├── HISTORY.rst
├── LICENSE
├── MANIFEST.in
├── README.rst
├── requirements.txt
├── setup.cfg
├── setup.py
├── mypackage
│ ├── cli.py
│ ├── __init__.py
│ ├── mypackage.py
├── tests
│ └── test_mypackage.py
└── tox.in
So I just need to how how to set __main__ to be the mypackage.py module or the cli.py module. How do I actually set that setting?
The reason for setting "main" is because I am using someone else's pickle file, and that file needs to refer to classes that are in the __main__ module. So I need to make sure that I set the module in which the unpickling happens to be the "main" module, and also move the command line interface and other functions to that module.
This is my current project setup:
.
├── README.md
├── build
│ ├── bdist.macosx-10.8-intel
│ └── lib
├── dist
│ └── giordano-0.1-py2.7.egg
├── giordano.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ ├── not-zip-safe
│ └── top_level.txt
├── requirements.txt
├── setup.py
├── src
│ ├── giordano
│ └── spider
├── test.txt
└── venv
├── bin
├── include
├── lib
└── share
And this is my setup file:
from setuptools import setup
setup(name='giordano',
version='0.1',
packages=['giordano'],
package_dir={'giordano': 'src/giordano'},
zip_safe=False)
When I do python setup.py install, I am able to import giordano in my code without problems.
However, when I am doing python setup.py develop, this is the console output:
[venv] fixSetup$ python setup.py develop
running develop
running egg_info
writing giordano.egg-info/PKG-INFO
writing top-level names to giordano.egg-info/top_level.txt
writing dependency_links to giordano.egg-info/dependency_links.txt
reading manifest file 'giordano.egg-info/SOURCES.txt'
writing manifest file 'giordano.egg-info/SOURCES.txt'
running build_ext
Creating /Users/blah/Dropbox/projects/Giordano/venv/lib/python2.7/site-packages/giordano.egg-link (link to .)
Removing giordano 0.1 from easy-install.pth file
Adding giordano 0.1 to easy-install.pth file
Installed /Users/blah/Dropbox/projects/Giordano
Processing dependencies for giordano==0.1
Finished processing dependencies for giordano==0.1
I noticed that the egg is linked to . as opposed to src/giordano. I can no longer import giordano in my code.
Any ideas why develop is not respecting package_dir?
Try with 'giordano': 'src'. distutils/distribute looks for the module or package name in the directory you specify; in the code you pasted, the value is one directory too deep.