I am trying to package a Python 3.6 project which additionally requires compiling CPP code. My test setup.py file currently looks like this:
print("Hello world from setup.py")
from distutils.command.build import build
from setuptools import setup
class TestBuild(build):
def run(self):
print("Hello world from TestBuild")
setup(
name="Test",
cmdclass={
'build': TestBuild,
},
)
The idea here is to place the CPP-compiling code inside TestBuild. When I run python setup.py build it seems to work well:
> python setup.py build
Hello world from setup.py
running build
Hello world from TestBuild
However, running python setup.py install does not seem to run the build step at all:
> python setup.py install
Hello world from setup.py
running install
running bdist_egg
running egg_info
writing Test.egg-info/PKG-INFO
writing top-level names to Test.egg-info/top_level.txt
writing dependency_links to Test.egg-info/dependency_links.txt
reading manifest file 'Test.egg-info/SOURCES.txt'
writing manifest file 'Test.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/Test-0.0.0-py3.5.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing Test-0.0.0-py3.5.egg
My questions:
Does the install command automatically invokes the build command?
If it does, what did I do wrong? If it does not, how can I ensure the build command runs?
Where can I find detailed description of the install process? I tried to dig inside the setuptools documentation and did not find anything like this.
Does the install command automatically invokes the build command?
Yes but it doesn't run python setup.py build — it calls the
corresponding code right inside one process hence you never see the
second print. If you really need to print it you have to override setuptools commands with custom classes; create such classes for build, bdist_egg and install commands.
Where can I find detailed description of the install process? I tried to dig inside the setuptools documentation and did not find anything like this.
I'm afraid distutils and setuptools source code is the only authoritative reference.
bdist_egg doesn't run build directly; it runs egg_info, build_clib and install_lib. install_lib calls distutils.build which runs build_py and build_ext. So it seems you have to override build_py, not build.
Related
I am developing using Python 3.7.
I'm trying to write a small Python package and install it into a virtual environment for testing. The library is structured like this:
my_package/
src/
my_package/
__init__.py
utilities.py
some_data_files/
setup.py
README.md
...
When I create virtual environment and try to install:
(env) C:\Users\path\to\my_package python setup.py install
The output is:
running install
running bdist_egg
running egg_info
writing my_package.egg-info\PKG-INFO
writing dependency_links to my_package.egg-info\dependency_links.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
warning: install_lib: 'build\lib' does not exist -- no Python modules to install
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\PKG-INFO -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\SOURCES.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\dependency_links.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\top_level.txt -> build\bdist.win-amd64\egg\EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist\my_package-0.0.1-py3.7.egg' and adding 'build\bdist.win-amd64\egg' to it
removing 'build\bdist.win-amd64\egg' (and everything under it)
Processing my_package-0.0.1-py3.7.egg
Copying my_package-0.0.1-py3.7.egg to c:\users\jon\development\my_package_env\env\lib\site-p
ackages
Adding my_package 0.0.1 to easy-install.pth file
Installed c:\users\jon\development\my_package_env\env\lib\site-packages\my_package-0.0.1-py3
.7.egg
Processing dependencies for my_package==0.0.1
Finished processing dependencies for my_package==0.0.1
There is an egg file in site-packages but no directory containing any of the files from the my_package directory. This is my first time writing a Python package for distribution, and I'm using the Python documentation as a guide, but I'm not ready to upload anywhere and would like to test locally. Can someone let me know what steps I must take to be able to test my package locally in a virtual environment?
Create a source distribution and a built distribution of your project, with these commands:
(env) C:\Users\jon\development\my_package_env>python3 setup.py sdist
(env) C:\Users\jon\development\my_package_env>python3 setup.py bdist_wheel
Note the two newly created archives in the dist/ directory
Try to install these distributions in fresh virtual environments:
C:\Users\jon\development\my_package_env>python3 -m venv sdist-env
C:\Users\jon\development\my_package_env>sdist-env\Scripts\activate.bat
(sdist-env) C:\Users\jon\development\my_package_env>pip install dist\my_package-0.0.1.tar.gz
(sdist-env) C:\Users\jon\development\my_package_env>deactivate
C:\Users\jon\development\my_package_env>python3 -m venv wheel-env
C:\Users\jon\development\my_package_env>wheel-env\Scripts\activate.bat
(wheel-env) C:\Users\jon\development\my_package_env>pip install dist\my_package-0.0.1-py3-none-any.whl
Once you're confident with this, you might want to look at tox to automate the testing of your project in multiple virtual environments.
I am trying to build a pulp distributor plugin which will execute a bash script containing arbitrary code so that I may trigger actions after an RPM repo is published.
These plugins are generally created using distutils. However, when I attempt to install my module, I receive the error:
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Typically, this means that the working directory is incorrect or __init.py__ is missing. In my case however, I am attempting to install from the correct working directory and I did create __init.py__ files (see the repo here).
I suspect that I am running into a pathing issue having something to do with my code being in a subdirectory so far removed from setup.py. What am I doing wrong? Why won't my module install?
When you face errors like this, one of the first things to check is what packages are actually being added to your distribution when you build it. In your case the package list is empty, but should contain at least the pulp_hook package:
$ python -c "from setuptools import find_packages; print(find_packages())"
[]
So why does setuptools not recognize pulp_hook as a regular package? Look at its structure: you have added file named __init.py__, but its name should be __init__.py. Once you rename the files, the pulp_hook and its subdirectories become regular packages:
$ python -c "from setuptools import find_packages; print(find_packages())"
['pulp_hook', 'pulp_hook.plugins', 'pulp_hook.plugins.distributors']
Now build/lib will be created because now distutils finds at least one package to install:
$ python setup.py install_lib
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/pulp_hook
copying pulp_hook/__init__.py -> build/lib/pulp_hook
creating build/lib/pulp_hook/plugins
copying pulp_hook/plugins/__init__.py -> build/lib/pulp_hook/plugins
creating build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/distributionhook.py -> build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/__init__.py -> build/lib/pulp_hook/plugins/distributors
I have a python project in which I have to execute this command every now and then.
python setup.py build_clib
followed by
python setup.py build_ext --inplace
I want this to happen with just a single line code of python setup.py build but that is not possible because I want my .so file inplace.
Is there any possible way of doing this?
You need to add these lines to your setup.cfg file.
[build_ext]
inplace=1
Short summary
I can upload a package to pyPI, and install it with pip, but Python is not seeing it when I try to import it (it says there is no module with the relevant name). I am using Python 2.7 in Windows 7 (Anaconda distribution, using iPython interpreter).
Detailed summary
I am learning how to upload a package (mymathFoo) at the cheese shop (pyPI), and trying to make sure I can install it once it is uploaded. When I manually paste the package folder into my Lib/site-packages directory, it imports and works fine from my interpreter.
What I have done:
0. Write package
It's a silly little package with an add and subtract module (e.g., add(1,2)). The directory is structured as follows:
\mymathFoo
__init__.py
add.py #adds two numbers
subtract.py #subtract two numbers
setup.py #shown below
README.txt #simple description of package
The __init__.py file is as follows:
from add import add
from subtract import subtract
1. Register the package at pyPI
Namely, from the command line, I enter:
python setup.py register
Which returns:
running register
running check
Registering mymathFoo to http://pypi.python.org/pypi
Server response (200): OK
Note my setup.py file is:
from distutils.core import setup
setup(name="mymathFoo",
version="0.6.2",
url="http://mymathfoo.net",
maintainer='Math Foo',
maintainer_email='mathfoo#math.org',
py_modules=['add','subtract'],
description="Silly little math turd to add/subtract.")
Note if I change it to py_modules='mymathFoo' I get the same error as below.
2. Upload the package
At the command line, I entered:
python setup.py sdist bdist_wininst upload
And the response was:
running sdist
running check
warning: sdist: manifest template 'MANIFEST.in' does not exist (using default fi
le list)
writing manifest file 'MANIFEST'
creating mymathFoo-0.6.3
copying files to mymathFoo-0.6.3...
copying README.txt -> mymathFoo-0.6.3
copying add.py -> mymathFoo-0.6.3
copying setup.py -> mymathFoo-0.6.3
copying subtract.py -> mymathFoo-0.6.3
creating dist
creating 'dist\mymathFoo-0.6.3.zip' and adding 'mymathFoo-0.6.3' to it
adding 'mymathFoo-0.6.3\add.py'
adding 'mymathFoo-0.6.3\PKG-INFO'
adding 'mymathFoo-0.6.3\README.txt'
adding 'mymathFoo-0.6.3\setup.py'
adding 'mymathFoo-0.6.3\subtract.py'
removing 'mymathFoo-0.6.3' (and everything under it)
running bdist_wininst
running build
running build_py
creating build
creating build\lib
copying add.py -> build\lib
copying subtract.py -> build\lib
installing to build\bdist.win-amd64\wininst
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\wininst
creating build\bdist.win-amd64\wininst\PURELIB
copying build\lib\add.py -> build\bdist.win-amd64\wininst\PURELIB
copying build\lib\subtract.py -> build\bdist.win-amd64\wininst\PURELIB
running install_egg_info
Writing build\bdist.win-amd64\wininst\PURELIB\mymathFoo-0.6.3-py2.7.egg-info
creating 'c:\users\eric\appdata\local\temp\tmp65xxe2.zip' and adding '.' to it
adding 'PURELIB\add.py'
adding 'PURELIB\mymathFoo-0.6.3-py2.7.egg-info'
adding 'PURELIB\subtract.py'
removing 'build\bdist.win-amd64\wininst' (and everything under it)
running upload
Submitting dist\mymathFoo-0.6.3.zip to http://pypi.python.org/pypi
Server response (200): OK
Submitting dist\mymathFoo-0.6.3.win-amd64.exe to http://pypi.python.org/pypi
Server response (200): OK
Things seem to have worked. So far so good.
3. Install this package locally using pip.
Then I go to install this package with pip at the command line:
pip install mymathFoo
To which I get:
Downloading/unpacking mymathFoo
Downloading mymathFoo-0.6.3.zip
Running setup.py (path:c:\users\eric\appdata\local\temp\pip_build_Eric\mymathF
oo\setup.py) egg_info for package mymathFoo
Installing collected packages: mymathFoo
Running setup.py install for mymathFoo
Successfully installed mymathFoo
Cleaning up...
Running the above causes the following directory to be copied into my Lib/site-packages folder:
mymathFoo-0.6.3-py2.7.egg-info
4. Import the package (not)
This is where I run into the problem. I open my iPython interpreter (using Anaconda distribution of Python, Windows 7):
import mymathFoo
I get:
Traceback (most recent call last):
File "<ipython-input-7-b7486b6a0225>", line 1, in <module>
import mymathFoo
ImportError: No module named mymathFoo
What am I missing? Why is my poor little module invisible?
Update
Note if I collapse all the files into the root directory (which I ultimately do not want to do), the errors go away. Unfortunately I will often want directories in my root directory, and nothing I have tried based on comments has fixed this.
I'm still looking for an answer, the discussion here looks promising:
http://www.scotttorborg.com/python-packaging/index.html#
I will work through that and post any solutions I find.
Discussion of related, but not the same, issues
"ImportError: No module named httplib2" even after installation
how to install package with pypi in python 2.7?
Note
This is based on some work I am doing with the book Python 101, by Michael Driscoll (it is in draft form right now).
Your package does install, just not the way you intended it to:
$ pip install mymathFoo
Downloading/unpacking mymathFoo
Using download cache from /Users/stu/tmp/pip-cache/http%3A%2F%2Fpypi.counsyl.com%2Froot%2Fpypi%2F%2Bf%2F64ef17a9135f05820c2d96050ce4315d%2FmymathFoo-0.6.3.zip
Running setup.py egg_info for package mymathFoo
Installing collected packages: mymathFoo
Running setup.py install for mymathFoo
Successfully installed mymathFoo
Cleaning up...
$ python
Type "help", "copyright", "credits" or "license" for more information.
>>> import add
>>> import subtract
>>>
Your setup.py says to install two modules, "add" and "subtract", and not to install a package named mymathFoo. Put add.py and subtract.py in a folder named 'mymathFoo', and set py_modules=['mymathFoo'] in setup.py.
Lastly, you can test your packaging without hitting pypi. Just run python setup.py install, and try to import you package.
it works for me:
>>> import mymathFoo
>>> dir(mymathFoo)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
though your module exports nothing. It looks like you changed your package since #Stu-Gla's answer, as you removed the add.py and subtract.py from sources.
By the way, you do not need to register a dummy package to pypi in order to test it, you can also use pip to install a package locally:
pip install /path/to/sources # path where the setup.py is
Below is my setup.py. I don't use anything from setuptools in my code and my project has no external dependencies
#!/usr/bin/env python
from distutils.core import setup
setup(name='dots',
...
packages=['dots','dots.configs','dots.management','dots.utils','dots.test'],
scripts=['dots/dots.py']
)
When I run python setup.py install, I get the following
running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /Users/kevinlin/.virtualenvs/p-dots/bin/dots.py to 755
running install_egg_info <- why?
Removing /Users/kevinlin/.virtualenvs/p-dots/lib/python2.7/site-packages/dots-0.1-py2.7.egg-info
Writing /Users/kevinlin/.virtualenvs/p-dots/lib/python2.7/site-packages/dots-0.1-py2.7.egg-info
(p-dots)Kevins-MacBook-Pro-2% python setup.py install
running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /Users/kevinlin/.virtualenvs/p-dots/bin/dots.py to 755
running install_egg_info
Removing /Users/kevinlin/.virtualenvs/p-dots/lib/python2.7/site-packages/dots-0.1-py2.7.egg-info
Writing /Users/kevinlin/.virtualenvs/p-dots/lib/python2.7/site-packages/dots-0.1-py2.7.egg-info
I notice that an .egg-info file is installed
ls -dl /Users/kevinlin/.virtualenvs/p-dots/lib/python2.7/site-packages/dots*
drwxr-xr-x 10 kevinlin staff 340 May 4 11:36 /Users/kevinlin/.virtualenvs/p-dots/lib/python2.7/site-packages/dots
-rw-r--r-- 1 kevinlin staff 205 May 4 11:36 /Users/kevinlin/.virtualenvs/p-dots/lib/python2.7/site-packages/dots-0.1-py2.7.egg-info
Why?
There was discussion about adding setuptools to the stdlib for Python 2.5, and then about adding only pkg_resource (one part of setuptools). Various reasons made that not happen. The core devs recognized that setuptools was an important third-party tool, and accepted a change to make projects installed with a pure-distutils setup.py write this egg-info file, to enable setuptools and tools built on top of it to know that these projects are already installed.
Changeset: http://hg.python.org/cpython/rev/93344da76acd
Ticket: http://bugs.python.org/issue1459476
(I haven’t checked the dates of the 2.5 discussions about setuptools, so my first sentences may be wrong. The rationale is still valid, see the ticket.)