I am trying to build a pulp distributor plugin which will execute a bash script containing arbitrary code so that I may trigger actions after an RPM repo is published.
These plugins are generally created using distutils. However, when I attempt to install my module, I receive the error:
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Typically, this means that the working directory is incorrect or __init.py__ is missing. In my case however, I am attempting to install from the correct working directory and I did create __init.py__ files (see the repo here).
I suspect that I am running into a pathing issue having something to do with my code being in a subdirectory so far removed from setup.py. What am I doing wrong? Why won't my module install?
When you face errors like this, one of the first things to check is what packages are actually being added to your distribution when you build it. In your case the package list is empty, but should contain at least the pulp_hook package:
$ python -c "from setuptools import find_packages; print(find_packages())"
[]
So why does setuptools not recognize pulp_hook as a regular package? Look at its structure: you have added file named __init.py__, but its name should be __init__.py. Once you rename the files, the pulp_hook and its subdirectories become regular packages:
$ python -c "from setuptools import find_packages; print(find_packages())"
['pulp_hook', 'pulp_hook.plugins', 'pulp_hook.plugins.distributors']
Now build/lib will be created because now distutils finds at least one package to install:
$ python setup.py install_lib
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/pulp_hook
copying pulp_hook/__init__.py -> build/lib/pulp_hook
creating build/lib/pulp_hook/plugins
copying pulp_hook/plugins/__init__.py -> build/lib/pulp_hook/plugins
creating build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/distributionhook.py -> build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/__init__.py -> build/lib/pulp_hook/plugins/distributors
Related
I am developing using Python 3.7.
I'm trying to write a small Python package and install it into a virtual environment for testing. The library is structured like this:
my_package/
src/
my_package/
__init__.py
utilities.py
some_data_files/
setup.py
README.md
...
When I create virtual environment and try to install:
(env) C:\Users\path\to\my_package python setup.py install
The output is:
running install
running bdist_egg
running egg_info
writing my_package.egg-info\PKG-INFO
writing dependency_links to my_package.egg-info\dependency_links.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
warning: install_lib: 'build\lib' does not exist -- no Python modules to install
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\PKG-INFO -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\SOURCES.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\dependency_links.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\top_level.txt -> build\bdist.win-amd64\egg\EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist\my_package-0.0.1-py3.7.egg' and adding 'build\bdist.win-amd64\egg' to it
removing 'build\bdist.win-amd64\egg' (and everything under it)
Processing my_package-0.0.1-py3.7.egg
Copying my_package-0.0.1-py3.7.egg to c:\users\jon\development\my_package_env\env\lib\site-p
ackages
Adding my_package 0.0.1 to easy-install.pth file
Installed c:\users\jon\development\my_package_env\env\lib\site-packages\my_package-0.0.1-py3
.7.egg
Processing dependencies for my_package==0.0.1
Finished processing dependencies for my_package==0.0.1
There is an egg file in site-packages but no directory containing any of the files from the my_package directory. This is my first time writing a Python package for distribution, and I'm using the Python documentation as a guide, but I'm not ready to upload anywhere and would like to test locally. Can someone let me know what steps I must take to be able to test my package locally in a virtual environment?
Create a source distribution and a built distribution of your project, with these commands:
(env) C:\Users\jon\development\my_package_env>python3 setup.py sdist
(env) C:\Users\jon\development\my_package_env>python3 setup.py bdist_wheel
Note the two newly created archives in the dist/ directory
Try to install these distributions in fresh virtual environments:
C:\Users\jon\development\my_package_env>python3 -m venv sdist-env
C:\Users\jon\development\my_package_env>sdist-env\Scripts\activate.bat
(sdist-env) C:\Users\jon\development\my_package_env>pip install dist\my_package-0.0.1.tar.gz
(sdist-env) C:\Users\jon\development\my_package_env>deactivate
C:\Users\jon\development\my_package_env>python3 -m venv wheel-env
C:\Users\jon\development\my_package_env>wheel-env\Scripts\activate.bat
(wheel-env) C:\Users\jon\development\my_package_env>pip install dist\my_package-0.0.1-py3-none-any.whl
Once you're confident with this, you might want to look at tox to automate the testing of your project in multiple virtual environments.
I am trying to figure how to install a Python package (ldap3) using a .egg file (fyi, the version of python-ldap3 we need does not have a .whl file) on a system that does not have easy_install available.
I found from an earlier thread (How can I install a .egg Python package on Windows (attempt using easy_install not working)) that I might be able to:
Untar the python-ldap3 tar.gz
Change dir to the untarred dir
run "python setup.py install"
So I tried the above, and it does seem to deploy/install the python-ldap3, but the installation hangs because it looks like it is trying to download something:
# python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to ldap3.egg-info/requires.txt
writing ldap3.egg-info/PKG-INFO
writing top-level names to ldap3.egg-info/top_level.txt
writing dependency_links to ldap3.egg-info/dependency_links.txt
reading manifest file 'ldap3.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'ldap3.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/ldap3
copying ./ldap3/__init__.py -> build/lib/ldap3
copying ./ldap3/version.py -> build/lib/ldap3
creating build/lib/ldap3/abstract
copying ./ldap3/abstract/cursor.py -> build/lib/ldap3/abstract
.
.
.
Installed /usr/lib/python2.6/site-packages/ldap3-2.5.1-py2.6.egg
Processing dependencies for ldap3==2.5.1
Searching for pyasn1>=0.1.8
Reading https://pypi.python.org/simple/pyasn1/
The "python setup.py install" hangs at that last line above ("Reading...."), but if I kill the process then check the python-ldap3 version, it looks like it installed all right:
# pip freeze | grep ldap3
ldap3==2.5.1
Also, FYI, our Python version is 2.6, and the pyasn1 is:
#pip freeze | grep pyasn1
pyasn1==0.4.3
I am not that familiar with Python and Python packaging, so I was wondering:
Is it all right to do the ldap3 2.5.1 installation per above (using
the 'python setup.py install', and then just killing the script)?
If not, what would be a safer/better way to get the ldap3 2.5.1 installed?
Thanks in advance!
Jim
EDIT: I forgot to mention that I think that the reason the script hung at the point it did was that we have to do this installation on an "isolated network", with no access to the external Internet, so we actually prefer/need to do the ldap3 installation without any external access.
EDIT 2: I guess that one other thing that is puzzling is why is the "python setup.py install" trying to get "pyasn1>=0.1.8" when "0.4.3" is already installed, since, from the release history, https://pypi.org/project/pyasn1/#history, "0.1.8" is OLDER than "0.4.3"?
EDIT 3: TO be clear, FYI, the package I am working with it the "ldap3" package, from https://github.com/cannatag/ldap3.
I am trying to package a Python 3.6 project which additionally requires compiling CPP code. My test setup.py file currently looks like this:
print("Hello world from setup.py")
from distutils.command.build import build
from setuptools import setup
class TestBuild(build):
def run(self):
print("Hello world from TestBuild")
setup(
name="Test",
cmdclass={
'build': TestBuild,
},
)
The idea here is to place the CPP-compiling code inside TestBuild. When I run python setup.py build it seems to work well:
> python setup.py build
Hello world from setup.py
running build
Hello world from TestBuild
However, running python setup.py install does not seem to run the build step at all:
> python setup.py install
Hello world from setup.py
running install
running bdist_egg
running egg_info
writing Test.egg-info/PKG-INFO
writing top-level names to Test.egg-info/top_level.txt
writing dependency_links to Test.egg-info/dependency_links.txt
reading manifest file 'Test.egg-info/SOURCES.txt'
writing manifest file 'Test.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/Test-0.0.0-py3.5.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing Test-0.0.0-py3.5.egg
My questions:
Does the install command automatically invokes the build command?
If it does, what did I do wrong? If it does not, how can I ensure the build command runs?
Where can I find detailed description of the install process? I tried to dig inside the setuptools documentation and did not find anything like this.
Does the install command automatically invokes the build command?
Yes but it doesn't run python setup.py build — it calls the
corresponding code right inside one process hence you never see the
second print. If you really need to print it you have to override setuptools commands with custom classes; create such classes for build, bdist_egg and install commands.
Where can I find detailed description of the install process? I tried to dig inside the setuptools documentation and did not find anything like this.
I'm afraid distutils and setuptools source code is the only authoritative reference.
bdist_egg doesn't run build directly; it runs egg_info, build_clib and install_lib. install_lib calls distutils.build which runs build_py and build_ext. So it seems you have to override build_py, not build.
I've got a folder called Packages in my home folder. In this folder I want to develop some packages. To test them I tried to install the package with
python setup.py install develop
while inside the environment, of course. This gives me the following output
develop
running install
running bdist_egg
running egg_info
writing BACnetScapy.egg-info/PKG-INFO
writing top-level names to BACnetScapy.egg-info/top_level.txt
writing dependency_links to BACnetScapy.egg-info/dependency_links.txt
reading manifest file 'BACnetScapy.egg-info/SOURCES.txt'
writing manifest file 'BACnetScapy.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetLayers.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetTags.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/__init__.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetFunctions.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetConstants.py -> build/bdist.linux-x86_64/egg/BACnetScapy
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetLayers.py to BACnetLayers.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetTags.py to BACnetTags.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetFunctions.py to BACnetFunctions.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetConstants.py to BACnetConstants.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/BACnetScapy-1.0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing BACnetScapy-1.0-py2.7.egg
Copying BACnetScapy-1.0-py2.7.egg to /home/lk/virtualenvs/FuzzingEnv/lib/python2.7/site-packages
Adding BACnetScapy 1.0 to easy-install.pth file
Installed /home/lk/virtualenvs/FuzzingEnv/lib/python2.7/site-packages/BACnetScapy-1.0-py2.7.egg
Processing dependencies for BACnetScapy==1.0
Finished processing dependencies for BACnetScapy==1.0
running develop
Checking .pth file support in build/bdist.linux-x86_64/egg
/home/lk/virtualenvs/FuzzingEnv/bin/python -E -c pass
TEST FAILED: build/bdist.linux-x86_64/egg does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
build/bdist.linux-x86_64/egg
and your PYTHONPATH environment variable currently contains:
''
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations
Please make the appropriate changes for your system and try again.
So the problem seems to be that the build folder which is automatically created in the package folder is not on the PYTHONPATH - but why should it be? And how should I add it to the PYTHONPATH? I tried to add this folder with
add2virtualenv /home/lk/Packages/BACnetScapy/build
but that did not work, either.
What am I doing wrong?
This is my setup.py
from setuptools import setup
setup(
name="BACnetScapy",
author="lk",
version="1.0",
packages=["BACnetScapy"],
include_package_data=True
)
The folder looks like that
/BACnetScapy
|
|-/BACnetScapy
| |- some.py
| |- modules.py
| |- __init__.py
| |-/data # a folder containing some data the package needs
|
|-setup.py
The command should be python setup.py develop, not python setup.py install develop
Short summary
I can upload a package to pyPI, and install it with pip, but Python is not seeing it when I try to import it (it says there is no module with the relevant name). I am using Python 2.7 in Windows 7 (Anaconda distribution, using iPython interpreter).
Detailed summary
I am learning how to upload a package (mymathFoo) at the cheese shop (pyPI), and trying to make sure I can install it once it is uploaded. When I manually paste the package folder into my Lib/site-packages directory, it imports and works fine from my interpreter.
What I have done:
0. Write package
It's a silly little package with an add and subtract module (e.g., add(1,2)). The directory is structured as follows:
\mymathFoo
__init__.py
add.py #adds two numbers
subtract.py #subtract two numbers
setup.py #shown below
README.txt #simple description of package
The __init__.py file is as follows:
from add import add
from subtract import subtract
1. Register the package at pyPI
Namely, from the command line, I enter:
python setup.py register
Which returns:
running register
running check
Registering mymathFoo to http://pypi.python.org/pypi
Server response (200): OK
Note my setup.py file is:
from distutils.core import setup
setup(name="mymathFoo",
version="0.6.2",
url="http://mymathfoo.net",
maintainer='Math Foo',
maintainer_email='mathfoo#math.org',
py_modules=['add','subtract'],
description="Silly little math turd to add/subtract.")
Note if I change it to py_modules='mymathFoo' I get the same error as below.
2. Upload the package
At the command line, I entered:
python setup.py sdist bdist_wininst upload
And the response was:
running sdist
running check
warning: sdist: manifest template 'MANIFEST.in' does not exist (using default fi
le list)
writing manifest file 'MANIFEST'
creating mymathFoo-0.6.3
copying files to mymathFoo-0.6.3...
copying README.txt -> mymathFoo-0.6.3
copying add.py -> mymathFoo-0.6.3
copying setup.py -> mymathFoo-0.6.3
copying subtract.py -> mymathFoo-0.6.3
creating dist
creating 'dist\mymathFoo-0.6.3.zip' and adding 'mymathFoo-0.6.3' to it
adding 'mymathFoo-0.6.3\add.py'
adding 'mymathFoo-0.6.3\PKG-INFO'
adding 'mymathFoo-0.6.3\README.txt'
adding 'mymathFoo-0.6.3\setup.py'
adding 'mymathFoo-0.6.3\subtract.py'
removing 'mymathFoo-0.6.3' (and everything under it)
running bdist_wininst
running build
running build_py
creating build
creating build\lib
copying add.py -> build\lib
copying subtract.py -> build\lib
installing to build\bdist.win-amd64\wininst
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\wininst
creating build\bdist.win-amd64\wininst\PURELIB
copying build\lib\add.py -> build\bdist.win-amd64\wininst\PURELIB
copying build\lib\subtract.py -> build\bdist.win-amd64\wininst\PURELIB
running install_egg_info
Writing build\bdist.win-amd64\wininst\PURELIB\mymathFoo-0.6.3-py2.7.egg-info
creating 'c:\users\eric\appdata\local\temp\tmp65xxe2.zip' and adding '.' to it
adding 'PURELIB\add.py'
adding 'PURELIB\mymathFoo-0.6.3-py2.7.egg-info'
adding 'PURELIB\subtract.py'
removing 'build\bdist.win-amd64\wininst' (and everything under it)
running upload
Submitting dist\mymathFoo-0.6.3.zip to http://pypi.python.org/pypi
Server response (200): OK
Submitting dist\mymathFoo-0.6.3.win-amd64.exe to http://pypi.python.org/pypi
Server response (200): OK
Things seem to have worked. So far so good.
3. Install this package locally using pip.
Then I go to install this package with pip at the command line:
pip install mymathFoo
To which I get:
Downloading/unpacking mymathFoo
Downloading mymathFoo-0.6.3.zip
Running setup.py (path:c:\users\eric\appdata\local\temp\pip_build_Eric\mymathF
oo\setup.py) egg_info for package mymathFoo
Installing collected packages: mymathFoo
Running setup.py install for mymathFoo
Successfully installed mymathFoo
Cleaning up...
Running the above causes the following directory to be copied into my Lib/site-packages folder:
mymathFoo-0.6.3-py2.7.egg-info
4. Import the package (not)
This is where I run into the problem. I open my iPython interpreter (using Anaconda distribution of Python, Windows 7):
import mymathFoo
I get:
Traceback (most recent call last):
File "<ipython-input-7-b7486b6a0225>", line 1, in <module>
import mymathFoo
ImportError: No module named mymathFoo
What am I missing? Why is my poor little module invisible?
Update
Note if I collapse all the files into the root directory (which I ultimately do not want to do), the errors go away. Unfortunately I will often want directories in my root directory, and nothing I have tried based on comments has fixed this.
I'm still looking for an answer, the discussion here looks promising:
http://www.scotttorborg.com/python-packaging/index.html#
I will work through that and post any solutions I find.
Discussion of related, but not the same, issues
"ImportError: No module named httplib2" even after installation
how to install package with pypi in python 2.7?
Note
This is based on some work I am doing with the book Python 101, by Michael Driscoll (it is in draft form right now).
Your package does install, just not the way you intended it to:
$ pip install mymathFoo
Downloading/unpacking mymathFoo
Using download cache from /Users/stu/tmp/pip-cache/http%3A%2F%2Fpypi.counsyl.com%2Froot%2Fpypi%2F%2Bf%2F64ef17a9135f05820c2d96050ce4315d%2FmymathFoo-0.6.3.zip
Running setup.py egg_info for package mymathFoo
Installing collected packages: mymathFoo
Running setup.py install for mymathFoo
Successfully installed mymathFoo
Cleaning up...
$ python
Type "help", "copyright", "credits" or "license" for more information.
>>> import add
>>> import subtract
>>>
Your setup.py says to install two modules, "add" and "subtract", and not to install a package named mymathFoo. Put add.py and subtract.py in a folder named 'mymathFoo', and set py_modules=['mymathFoo'] in setup.py.
Lastly, you can test your packaging without hitting pypi. Just run python setup.py install, and try to import you package.
it works for me:
>>> import mymathFoo
>>> dir(mymathFoo)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
though your module exports nothing. It looks like you changed your package since #Stu-Gla's answer, as you removed the add.py and subtract.py from sources.
By the way, you do not need to register a dummy package to pypi in order to test it, you can also use pip to install a package locally:
pip install /path/to/sources # path where the setup.py is