We Keep Coding

Python manually install package to virtual environment with setup.py

I am developing using Python 3.7.
I'm trying to write a small Python package and install it into a virtual environment for testing. The library is structured like this:
my_package/
src/
my_package/
__init__.py
utilities.py
some_data_files/
setup.py
README.md
...
When I create virtual environment and try to install:
(env) C:\Users\path\to\my_package python setup.py install
The output is:
running install
running bdist_egg
running egg_info
writing my_package.egg-info\PKG-INFO
writing dependency_links to my_package.egg-info\dependency_links.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
warning: install_lib: 'build\lib' does not exist -- no Python modules to install
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\PKG-INFO -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\SOURCES.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\dependency_links.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\top_level.txt -> build\bdist.win-amd64\egg\EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist\my_package-0.0.1-py3.7.egg' and adding 'build\bdist.win-amd64\egg' to it
removing 'build\bdist.win-amd64\egg' (and everything under it)
Processing my_package-0.0.1-py3.7.egg
Copying my_package-0.0.1-py3.7.egg to c:\users\jon\development\my_package_env\env\lib\site-p
ackages
Adding my_package 0.0.1 to easy-install.pth file
Installed c:\users\jon\development\my_package_env\env\lib\site-packages\my_package-0.0.1-py3
.7.egg
Processing dependencies for my_package==0.0.1
Finished processing dependencies for my_package==0.0.1
There is an egg file in site-packages but no directory containing any of the files from the my_package directory. This is my first time writing a Python package for distribution, and I'm using the Python documentation as a guide, but I'm not ready to upload anywhere and would like to test locally. Can someone let me know what steps I must take to be able to test my package locally in a virtual environment?

Create a source distribution and a built distribution of your project, with these commands:
(env) C:\Users\jon\development\my_package_env>python3 setup.py sdist
(env) C:\Users\jon\development\my_package_env>python3 setup.py bdist_wheel
Note the two newly created archives in the dist/ directory
Try to install these distributions in fresh virtual environments:
C:\Users\jon\development\my_package_env>python3 -m venv sdist-env
C:\Users\jon\development\my_package_env>sdist-env\Scripts\activate.bat
(sdist-env) C:\Users\jon\development\my_package_env>pip install dist\my_package-0.0.1.tar.gz
(sdist-env) C:\Users\jon\development\my_package_env>deactivate
C:\Users\jon\development\my_package_env>python3 -m venv wheel-env
C:\Users\jon\development\my_package_env>wheel-env\Scripts\activate.bat
(wheel-env) C:\Users\jon\development\my_package_env>pip install dist\my_package-0.0.1-py3-none-any.whl
Once you're confident with this, you might want to look at tox to automate the testing of your project in multiple virtual environments.

Why won't my python module install?

I am trying to build a pulp distributor plugin which will execute a bash script containing arbitrary code so that I may trigger actions after an RPM repo is published.
These plugins are generally created using distutils. However, when I attempt to install my module, I receive the error:
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Typically, this means that the working directory is incorrect or __init.py__ is missing. In my case however, I am attempting to install from the correct working directory and I did create __init.py__ files (see the repo here).
I suspect that I am running into a pathing issue having something to do with my code being in a subdirectory so far removed from setup.py. What am I doing wrong? Why won't my module install?

When you face errors like this, one of the first things to check is what packages are actually being added to your distribution when you build it. In your case the package list is empty, but should contain at least the pulp_hook package:
$ python -c "from setuptools import find_packages; print(find_packages())"
[]
So why does setuptools not recognize pulp_hook as a regular package? Look at its structure: you have added file named __init.py__, but its name should be __init__.py. Once you rename the files, the pulp_hook and its subdirectories become regular packages:
$ python -c "from setuptools import find_packages; print(find_packages())"
['pulp_hook', 'pulp_hook.plugins', 'pulp_hook.plugins.distributors']
Now build/lib will be created because now distutils finds at least one package to install:
$ python setup.py install_lib
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/pulp_hook
copying pulp_hook/__init__.py -> build/lib/pulp_hook
creating build/lib/pulp_hook/plugins
copying pulp_hook/plugins/__init__.py -> build/lib/pulp_hook/plugins
creating build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/distributionhook.py -> build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/__init__.py -> build/lib/pulp_hook/plugins/distributors

setup.py install develop with virtualenv - Bad install directory or PYTHONPATH

I've got a folder called Packages in my home folder. In this folder I want to develop some packages. To test them I tried to install the package with
python setup.py install develop
while inside the environment, of course. This gives me the following output
develop
running install
running bdist_egg
running egg_info
writing BACnetScapy.egg-info/PKG-INFO
writing top-level names to BACnetScapy.egg-info/top_level.txt
writing dependency_links to BACnetScapy.egg-info/dependency_links.txt
reading manifest file 'BACnetScapy.egg-info/SOURCES.txt'
writing manifest file 'BACnetScapy.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetLayers.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetTags.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/__init__.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetFunctions.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetConstants.py -> build/bdist.linux-x86_64/egg/BACnetScapy
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetLayers.py to BACnetLayers.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetTags.py to BACnetTags.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetFunctions.py to BACnetFunctions.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetConstants.py to BACnetConstants.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/BACnetScapy-1.0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing BACnetScapy-1.0-py2.7.egg
Copying BACnetScapy-1.0-py2.7.egg to /home/lk/virtualenvs/FuzzingEnv/lib/python2.7/site-packages
Adding BACnetScapy 1.0 to easy-install.pth file
Installed /home/lk/virtualenvs/FuzzingEnv/lib/python2.7/site-packages/BACnetScapy-1.0-py2.7.egg
Processing dependencies for BACnetScapy==1.0
Finished processing dependencies for BACnetScapy==1.0
running develop
Checking .pth file support in build/bdist.linux-x86_64/egg
/home/lk/virtualenvs/FuzzingEnv/bin/python -E -c pass
TEST FAILED: build/bdist.linux-x86_64/egg does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
build/bdist.linux-x86_64/egg
and your PYTHONPATH environment variable currently contains:
''
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations
Please make the appropriate changes for your system and try again.
So the problem seems to be that the build folder which is automatically created in the package folder is not on the PYTHONPATH - but why should it be? And how should I add it to the PYTHONPATH? I tried to add this folder with
add2virtualenv /home/lk/Packages/BACnetScapy/build
but that did not work, either.
What am I doing wrong?
This is my setup.py
from setuptools import setup
setup(
name="BACnetScapy",
author="lk",
version="1.0",
packages=["BACnetScapy"],
include_package_data=True
)
The folder looks like that
/BACnetScapy
|
|-/BACnetScapy
| |- some.py
| |- modules.py
| |- __init__.py
| |-/data # a folder containing some data the package needs
|
|-setup.py

The command should be python setup.py develop, not python setup.py install develop

python setup package in develop mode fails

I am attempting to setup a simple package in develop mode:
setup.py
from setuptools import setup, find_packages
setup(
name='rocflavors',
version='0.0.1',
packages=find_packages(exclude=['ez_setup','examples','tests']),
license='Creative Commons Attribution-Noncommercial-Share Alike license',
long_description=open('README').read(),
install_requires=[],
)
Attempt to setup: (works if I do python setup.py install)
(relux)roc-web5537:roc-flavors cmuench$ python setup.py install develop
running install
running bdist_egg
running egg_info
writing rocflavors.egg-info/PKG-INFO
writing top-level names to rocflavors.egg-info/top_level.txt
writing dependency_links to rocflavors.egg-info/dependency_links.txt
reading manifest file 'rocflavors.egg-info/SOURCES.txt'
writing manifest file 'rocflavors.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-intel/egg
running install_lib
running build_py
creating build/bdist.macosx-10.7-intel/egg
creating build/bdist.macosx-10.7-intel/egg/rocflavors
copying build/lib/rocflavors/__init__.py -> build/bdist.macosx-10.7-intel/egg/rocflavors
copying build/lib/rocflavors/models.py -> build/bdist.macosx-10.7-intel/egg/rocflavors
creating build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags
copying build/lib/rocflavors/templatetags/__init__.py -> build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags
copying build/lib/rocflavors/templatetags/rocflavors_extras.py -> build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags
copying build/lib/rocflavors/urls.py -> build/bdist.macosx-10.7-intel/egg/rocflavors
copying build/lib/rocflavors/views.py -> build/bdist.macosx-10.7-intel/egg/rocflavors
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/models.py to models.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags/rocflavors_extras.py to rocflavors_extras.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/urls.py to urls.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/views.py to views.pyc
creating build/bdist.macosx-10.7-intel/egg/EGG-INFO
copying rocflavors.egg-info/PKG-INFO -> build/bdist.macosx-10.7-intel/egg/EGG-INFO
copying rocflavors.egg-info/SOURCES.txt -> build/bdist.macosx-10.7-intel/egg/EGG-INFO
copying rocflavors.egg-info/dependency_links.txt -> build/bdist.macosx-10.7-intel/egg/EGG-INFO
copying rocflavors.egg-info/top_level.txt -> build/bdist.macosx-10.7-intel/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/rocflavors-0.0.1-py2.7.egg' and adding 'build/bdist.macosx-10.7-intel/egg' to it
removing 'build/bdist.macosx-10.7-intel/egg' (and everything under it)
Processing rocflavors-0.0.1-py2.7.egg
Removing /Users/cmuench/.virtualenvs/relux/lib/python2.7/site-packages/rocflavors-0.0.1-py2.7.egg
Copying rocflavors-0.0.1-py2.7.egg to /Users/cmuench/.virtualenvs/relux/lib/python2.7/site-packages
rocflavors 0.0.1 is already the active version in easy-install.pth
Installed /Users/cmuench/.virtualenvs/relux/lib/python2.7/site-packages/rocflavors-0.0.1-py2.7.egg
Processing dependencies for rocflavors==0.0.1
Finished processing dependencies for rocflavors==0.0.1
running develop
Checking .pth file support in build/bdist.macosx-10.7-intel/egg
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 2] No such file or directory: 'build/bdist.macosx-10.7-intel/egg/test-easy-install-35540.pth'
The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
build/bdist.macosx-10.7-intel/egg
This directory does not currently exist. Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).
I have 3 questions:
Why does it NOT work in develop mode.
What is the difference between develop mode and not using it?
Why does it need to be a package? If I change a .py file in the package how does it make it back into my package directory. I don't think I understand the point of packaging. Does it put compiled python code into the package when I make a change?

Make sure your virtualenv is activated. Instead of python setup.py install develop, try pip install -e .. Note the ., it means that you must be in the directory where setup.py is. If it fails, try with sudo.
Docs for pip: http://www.pip-installer.org/

Installing numpy as a dependency with setuptools

This might be a follow up question of this one.
I am using setuptools to install a package of mine. As a dependency I have listed numpy. I am using Python2.7 and when I do
python setup.py install
with this setup.py file:
from setuptools import setup
setup(name = "test_pack", install_requires = ["numpy"])
I end up with this error message:
ImportError: No module named numpy.distutils
What do I need to do in order to include numpy as a dependency and install it without having python-dev installed?
The complete output of python setup.py install:
running install
running bdist_egg
running egg_info
writing requirements to test_pack.egg-info/requires.txt
writing test_pack.egg-info/PKG-INFO
writing top-level names to test_pack.egg-info/top_level.txt
writing dependency_links to test_pack.egg-info/dependency_links.txt
reading manifest file 'test_pack.egg-info/SOURCES.txt'
writing manifest file 'test_pack.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/test_pack
copying build/lib/test_pack/__init__.py -> build/bdist.linux-x86_64/egg/test_pack
copying build/lib/test_pack/mod.py -> build/bdist.linux-x86_64/egg/test_pack
byte-compiling build/bdist.linux-x86_64/egg/test_pack/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/test_pack/mod.py to mod.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying test_pack.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying test_pack.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying test_pack.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying test_pack.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying test_pack.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/test_pack-0.0.0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing test_pack-0.0.0-py2.7.egg
Copying test_pack-0.0.0-py2.7.egg to /home/woltan/local/lib/python2.7/site-packages
Adding test-pack 0.0.0 to easy-install.pth file
Installed /home/woltan/local/lib/python2.7/site-packages/test_pack-0.0.0-py2.7.egg
Processing dependencies for test-pack==0.0.0
Searching for numpy
Reading http://pypi.python.org/simple/numpy/
Reading http://numpy.scipy.org
Reading http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103
Reading http://numeric.scipy.org
Best match: numpy 1.6.1
Downloading http://pypi.python.org/packages/source/n/numpy/numpy-1.6.1.zip#md5=462c22b8eb221c78ddd51de98fbb5979
Processing numpy-1.6.1.zip
Running numpy-1.6.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-AoFmdV/numpy-1.6.1/egg-dist-tmp-JH1j2R
non-existing path in 'numpy/distutils': 'site.cfg'
Could not locate executable g77
Found executable /opt/solstudio12.2/bin/f77
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
Found executable /opt/intel/Compiler/11.1/073/bin/intel64/ifort
Could not locate executable lf95
Could not locate executable pgf90
Could not locate executable pgf77
Found executable /opt/solstudio12.2/bin/f90
Found executable /opt/solstudio12.2/bin/f95
Could not locate executable fort
_configtest.c:1: warning: conflicting types for built-in function ‘exp’
_configtest.o: In function `main':
/tmp/easy_install-AoFmdV/numpy-1.6.1/_configtest.c:6: undefined reference to `exp'
collect2: ld returned 1 exit status
_configtest.c:1: warning: conflicting types for built-in function ‘exp’
_configtest.c:1:20: error: Python.h: No such file or directory
_configtest.o: In function `main':
/tmp/easy_install-AoFmdV/numpy-1.6.1/_configtest.c:6: undefined reference to `exp'
collect2: ld returned 1 exit status
_configtest.c:1:20: error: Python.h: No such file or directory

This is a known issue, tracked on numpy/numpy #2434.
I found a workaround for this: add numpy to setup_requires. Having it in both setup_requires and install_requires seems to work fine with the most recent version of setuptools.
So, your setup.py should look something like
setup(
# Your setup specific stuff here
setup_requires=["numpy"], # Just numpy here
install_requires=["numpy"], # Add any of your other dependencies here
)

Unless you have access to a binary distribution (pre-compiled/built) for numpy, you'll have to have the python headers available as it needs them to build numpy. This is why most package managers come with pre-compiled versions of these packages. For example you can apt-get install python-numpy, link that into your virtualenv, and when you try to install your program with install_requires=['numpy'] it should see that it's already installed.

To install numpy the setuptools will download the package and compile it from source. However, there are some prerequisites to compile numpy, you can check it here.
_configtest.c:1:20: error: Python.h: No such file or directory
this error indicates that at least you do not have python-dev package installed(if you are using ubuntu/debian).