I am writing a Python 3 package, and I need to know how python determines which module is considered "main"? Is this something that is configured in setup.py, or some other file? Which setting in setup.py determines which module is called as __main__, like is it the entry_points attribute, or the packages attribute, or some other one?
Here is a simplified example of the package structure:
├── AUTHORS.rst
├── HISTORY.rst
├── LICENSE
├── MANIFEST.in
├── README.rst
├── requirements.txt
├── setup.cfg
├── setup.py
├── mypackage
│ ├── cli.py
│ ├── __init__.py
│ ├── mypackage.py
├── tests
│ └── test_mypackage.py
└── tox.in
So I just need to how how to set __main__ to be the mypackage.py module or the cli.py module. How do I actually set that setting?
The reason for setting "main" is because I am using someone else's pickle file, and that file needs to refer to classes that are in the __main__ module. So I need to make sure that I set the module in which the unpickling happens to be the "main" module, and also move the command line interface and other functions to that module.
Related
I'm experimenting with packaging some Pythons projects and have followed this guide. The anonymized file tree can be seen below. The toml file is a barebones one from the tutorial modified appropriately. Building and uploading works well. So far so good.
.
├── LICENSE
├── pyproject.toml
├── README.md
├── src
│ └── mymodule
│ ├── __init__.py
│ └── main.py
└── tests
My next intended step is to package an older smaller well behaving project which includes a test suite written with unittest. Simplified structure below.
.
├── mymodule
│ ├── submoduleA
│ │ ├── __init__.py
│ │ └ foo.py
│ ├── submoduleB
│ │ ├── __init__.py
│ │ └ bar.py
│ ├── baz.py
│ └── __init__.py
└── tests
├── test_submoduleA.py
└── test_submoduleB.py
This is where my progress grinds to a halt.
There are many different ways to skin a cat but none directly involves unittest as far as I can tell. I have opted to go ahead by using tox to call the former.
Similarly when I have a look at different Python project repos the structure under tests seem to differ a bit.
End intent/wish: Convert said older project to a packagable one, editing the tests as little as possible, and using the tests for testing while developing and to do basic tests on the target device later.
Questions:
What is the purpose of the tests folder? Eg to run tests while developing files in src, to test the built package and/or to verify a package works once installed?
Is it possible to use the pyproject.toml file with unittest?
My Directory structure:
├── common
│ ├── common.py
│ └── __init__.py
├── project1
│ ├── __init__.py
│ └── scripts
│ ├── example_import.py
│ └── __init__.py
└── project2
├── __init__.py
└── scripts
└── __init__.py
I need to import common/common.py module in project1/scripts/example_import.py file
example_import.py:
import sys
sys.path.append("../common")
from common import Test
print("Module Not import error")
Error:
Traceback (most recent call last):
File "project1/scripts/example_import.py", line 3, in <module>
from common import Test
ImportError: No module named common
How to fix a issue?
Understanding how Python imports work is tricky in the beginning but makes sense once you understand how.
There are different way to fix your import issue. I would not recommend messing up with sys.path. Depending on where you are calling your script, you have multiple choice at hand.
Your Directory structure.
├── common
│ ├── common.py
│ └── __init__.py
├── project1
│ ├── __init__.py
│ └── scripts
│ ├── example_import.py
│ └── __init__.py
└── project2
├── __init__.py
└── scripts
└── __init__.py
On the root of directory
python project1/scripts/example_import.py
Will work, assuming that the imports in example_import.py looks like
from common.common import Test
If you want to use from common import Test, you need to add from common.common import Test in __init__.py file in common folder.
You can also use PYTHONPATH environment variable to tell Python where to look for the base module.
PYTHONPATH=/pathto/basefolder python folder/filesx.py
Another way is to create a setup.py on base and do development installation python -m pip install --editable . on environment you are working on.
Example of setup.py
#!/usr/bin/env python
from distutils.core import setup
setup(name='projectX',
version='1.0',
description='The World is not Enoug',
author='James Bond',
author_email='agent#007.net',
url='https://www.python.org/sigs/distutils-sig/',
packages=['common', 'project1', 'project2],
)
See Python Documentation for more setup.py options.
I'd like to figure out the cleanest and preferably self contained way to use my packages in scripts that are in a different directory to the package itself.
The example problem is as follows:
The modules in lib need to both be imported, and run as a script.
My project directory is as below and I'm having two issues:
in lib/api.py, I want to read in data_files/key.txt correctly when api.py is called or imported
in testing_script.py I want to import and use lib/get_data.py
I can't seem to find a clean way to do this, does this mean my project is structured in a non-pythonic way?
Thanks for the help.
my-project-git
├── LICENSE
├── README.md
├─── my_project
│ ├── data_files
│ │ ├── key.txt
│ │ ├── mappings.csv
│ ├── lib
│ │ ├── __init__.py
│ │ ├── api.py
│ │ └── get_data.py
│ └── test
│ ├── __init__.py
│ └── testing_script.py
├── requirements.txt
└── setup.py
As far as I know, there's isn't a pythonic way to structure your project.
This is what Kenneth Reitz recommended in 2013 and it's how I use it: https://www.kennethreitz.org/essays/repository-structure-and-python.
README.rst
LICENSE
setup.py
requirements.txt
sample/__init__.py
sample/core.py
sample/helpers.py
docs/conf.py
docs/index.rst
tests/test_basic.py
tests/test_advanced.py
Inside sample (my_project in your case) you can separate into categories as you like. E.g. Utils (common functions), Database (read, write), View (user commands), etc. It depends on your project.
As for calling the modules at the same level, you should define them in the __init__ file of the top hierarchy module which is sample in this case.
For example:
__init__ in _my_project
from sample.core import a_Class
from sample.core import a_function
from sample.core import anything
then from /test/test_basic.py you do:
from sample import a_Class
# or import sample
a = a_Class() # use the class from core.py
# or a = sample.a_Class()
Take a look at the sample module repository: https://github.com/navdeep-G/samplemod
I'm trying to build a package that uses both python and cython modules. The problem I'm having deals with imports after building and installing where I'm not sure how to make files import from the .so file generated by the build process.
Before building my folder structure looks like this
root/
├── c_integrate.c
├── c_integrate.pyx
├── cython_builder.py
├── __init__.py
├── integrator_class.py
├── integrator_modules
│ ├── cython_integrator.py
│ ├── __init__.py
│ ├── integrator.py
│ ├── numba_integrator.py
│ ├── numpy_integrator.py
│ ├── quadratic_error.png
│ ├── report3.txt
│ ├── report4.txt
│ └── report5.txt
├── report6.txt
├── setup.py
└── test
├── __init__.py
└── test_integrator.py
Building with python3.5 setup.py build gives this new folder in root
root/build/
├── lib.linux-x86_64-3.5
│ ├── c_integrate.cpython-35m-x86_64-linux-gnu.so
│ ├── integrator_modules
│ │ ├── cython_integrator.py
│ │ ├── __init__.py
│ │ ├── integrator.py
│ │ ├── numba_integrator.py
│ │ └── numpy_integrator.py
│ └── test
│ ├── __init__.py
│ └── test_integrator.py
The setup.py file looks like this
from setuptools import setup, Extension, find_packages
import numpy
setup(
name = "integrator_package",
author = "foo",
packages = find_packages(),
ext_modules = [Extension("c_integrate", ["c_integrate.c"])],
include_dirs=[numpy.get_include()],
)
My question is then: how do I write import statements of the functions from the .so file into ìntegrator_class.py in root and cython_integrator and test_integrator located in the build directory. Appending to sys.path seems like a quick and dirty solution that I don't much like.
EDIT:
As pointed out in the comments I haven't installed the package. This is because I don't know what to write to import from the .so file
In no specific order:
The file setup.py is typically located below the root of a project. Example:
library_name/
__init__.py
file1.py
setup.py
README
Then, the build directory appears alongside the project's source and not in the project source.
To import the file c_integrate.cpython-35m-x86_64-linux-gnu.so in Python, just import "c_integrate". The rest of the naming is taken care of automatically as it is just the platform information. See PEP 3149
A valid module is one of
a directory with a modulename/__init__.py file
a file named modulename.py
a file named modulename.PLATFORMINFO.so
of course located in the Python path. So there is no need for a __init__.py file for a compiled Cython module.
For your situation, move the Cython code in the project directory and either do a relative import import .c_integrate or a full from integrator_modules import c_integrate where the latter only works when your package is installed.
A few of this information can be found in my blog post on Cython modules http://pdebuyl.be/blog/2017/cython-module.html
I believe that this should let you build a proper package, comment below if not.
EDIT: to complete the configuration (see comments below), the poster also
Fixed the module path in the setup.py file so that it is the full module name starting from the PYTHONPATH: Extension("integrator_package.integrator_modules.c_integrator", ["integrator_package/integrator_modules/c_integrator.c"] instead of Extension("c_integrate", ["c_integrate.c"])]
Cythonize the module, build it and use with a same Python interpreter.
Further comment: the setup.py file can cythonize the file as well. Include the .pyx file instead of the .c file as the source.
cythonize(Extension('integrator_package.integrator_modules.c_integrator',
["integrator_package/integrator_modules/c_integrator.pyx"],
include_dirs=[numpy.get_include()]))
I have the following directory structure:
├── DynamicProgramming
│ ├── 0-1_kp_problem.py
│ ├── b.py
│ ├── largest_contigous_subarray.py
│ ├── longest_common_substring.py
│ ├── min_change_for_given_money.py
│ ├── optimal_matrix_chain.py
│ ├── Readme.md
│ └── wis.py
├── helper
│ ├── a.py
│ └── __init__.py
└── Readme.md
The helper directory contains the library functions which will be used all over the code. How can I import the helper package from the scripts inside DynamicProgramming without adding it to the path?
Edit=>
I cannot move helper directory inside dynamicProgramming because there can be more than one directories using it.
You could use something like:
from ..helper import a
See python docs on packages.
If you run your code from project root folder, you are likely to succeed with import helper or import helper.a. If not, you would have to add current directory to PYTHONPATH:
$ export PYTHONPATH="."
better use project setup.py
Instead of playing with PYTHONPATH (what can be tricky business sometime), you shall create your project as python package.
You add setup.py into your project root, specify attributes of that package and build it from it.
setup.py can define multiple packages at once, but generally it is more often
using only one. For this purpose it would be better moving the helper package
into DynamicProgramming structure and import it from there.
Search for setup.py python packaging tutorials, it requires some study, but it will pay back.