I am developing using Python 3.7.
I'm trying to write a small Python package and install it into a virtual environment for testing. The library is structured like this:
my_package/
src/
my_package/
__init__.py
utilities.py
some_data_files/
setup.py
README.md
...
When I create virtual environment and try to install:
(env) C:\Users\path\to\my_package python setup.py install
The output is:
running install
running bdist_egg
running egg_info
writing my_package.egg-info\PKG-INFO
writing dependency_links to my_package.egg-info\dependency_links.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
warning: install_lib: 'build\lib' does not exist -- no Python modules to install
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\PKG-INFO -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\SOURCES.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\dependency_links.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying my_package.egg-info\top_level.txt -> build\bdist.win-amd64\egg\EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist\my_package-0.0.1-py3.7.egg' and adding 'build\bdist.win-amd64\egg' to it
removing 'build\bdist.win-amd64\egg' (and everything under it)
Processing my_package-0.0.1-py3.7.egg
Copying my_package-0.0.1-py3.7.egg to c:\users\jon\development\my_package_env\env\lib\site-p
ackages
Adding my_package 0.0.1 to easy-install.pth file
Installed c:\users\jon\development\my_package_env\env\lib\site-packages\my_package-0.0.1-py3
.7.egg
Processing dependencies for my_package==0.0.1
Finished processing dependencies for my_package==0.0.1
There is an egg file in site-packages but no directory containing any of the files from the my_package directory. This is my first time writing a Python package for distribution, and I'm using the Python documentation as a guide, but I'm not ready to upload anywhere and would like to test locally. Can someone let me know what steps I must take to be able to test my package locally in a virtual environment?
Create a source distribution and a built distribution of your project, with these commands:
(env) C:\Users\jon\development\my_package_env>python3 setup.py sdist
(env) C:\Users\jon\development\my_package_env>python3 setup.py bdist_wheel
Note the two newly created archives in the dist/ directory
Try to install these distributions in fresh virtual environments:
C:\Users\jon\development\my_package_env>python3 -m venv sdist-env
C:\Users\jon\development\my_package_env>sdist-env\Scripts\activate.bat
(sdist-env) C:\Users\jon\development\my_package_env>pip install dist\my_package-0.0.1.tar.gz
(sdist-env) C:\Users\jon\development\my_package_env>deactivate
C:\Users\jon\development\my_package_env>python3 -m venv wheel-env
C:\Users\jon\development\my_package_env>wheel-env\Scripts\activate.bat
(wheel-env) C:\Users\jon\development\my_package_env>pip install dist\my_package-0.0.1-py3-none-any.whl
Once you're confident with this, you might want to look at tox to automate the testing of your project in multiple virtual environments.
Related
I try to install a virtual environment from GitHub
first I made a virtual environment with Conda, then I install git in this environment. Next I cloned the project and cd to package package directory contains setup.py.
But when I try to install setup.py (pip install setup.py) I got an error:
Could not find suitable distribution for Requirement.parse('env_mltf.yml')
This is the output of my terminal:
running install
running bdist_egg
running egg_info
writing mltf.egg-info/PKG-INFO
/opt/anaconda3/envs/DL/lib/python3.9/site-packages/setuptools/dist.py:125: UserWarning: newlines not allowed and will break in the future
warnings.warn("newlines not allowed and will break in the future")
writing dependency_links to mltf.egg-info/dependency_links.txt
writing requirements to mltf.egg-info/requires.txt
writing top-level names to mltf.egg-info/top_level.txt
package init file './__init__.py' not found (or not a regular file)
reading manifest file 'mltf.egg-info/SOURCES.txt'
writing manifest file 'mltf.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.9-x86_64/egg
running install_lib
running build_py
creating build/bdist.macosx-10.9-x86_64/egg
creating build/bdist.macosx-10.9-x86_64/egg/gym_mltf
copying build/lib/gym_mltf/README.md -> build/bdist.macosx-10.9-x86_64/egg/gym_mltf
copying build/lib/gym_mltf/example.py -> build/bdist.macosx-10.9-x86_64/egg/gym_mltf
byte-compiling build/bdist.macosx-10.9-x86_64/egg/gym_mltf/example.py to example.cpython-39.pyc
creating build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying mltf.egg-info/PKG-INFO -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying mltf.egg-info/SOURCES.txt -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying mltf.egg-info/dependency_links.txt -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying mltf.egg-info/requires.txt -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying mltf.egg-info/top_level.txt -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/mltf-0.0.1-py3.9.egg' and adding 'build/bdist.macosx-10.9-x86_64/egg' to it
removing 'build/bdist.macosx-10.9-x86_64/egg' (and everything under it)
Processing mltf-0.0.1-py3.9.egg
Removing /opt/anaconda3/envs/DL/lib/python3.9/site-packages/mltf-0.0.1-py3.9.egg
Copying mltf-0.0.1-py3.9.egg to /opt/anaconda3/envs/DL/lib/python3.9/site-packages
mltf 0.0.1 is already the active version in easy-install.pth
Installed /opt/anaconda3/envs/DL/lib/python3.9/site-packages/mltf-0.0.1-py3.9.egg
Processing dependencies for mltf==0.0.1
Searching for env_mltf.yml
Reading https://pypi.org/simple/env_mltf.yml/
Reading https://pypi.org/simple/env-mltf.yml/
Couldn't find index page for 'env_mltf.yml' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
No local packages or working download links found for env_mltf.yml
error: Could not find suitable distribution for Requirement.parse('env_mltf.yml')
I am trying to package a Python 3.6 project which additionally requires compiling CPP code. My test setup.py file currently looks like this:
print("Hello world from setup.py")
from distutils.command.build import build
from setuptools import setup
class TestBuild(build):
def run(self):
print("Hello world from TestBuild")
setup(
name="Test",
cmdclass={
'build': TestBuild,
},
)
The idea here is to place the CPP-compiling code inside TestBuild. When I run python setup.py build it seems to work well:
> python setup.py build
Hello world from setup.py
running build
Hello world from TestBuild
However, running python setup.py install does not seem to run the build step at all:
> python setup.py install
Hello world from setup.py
running install
running bdist_egg
running egg_info
writing Test.egg-info/PKG-INFO
writing top-level names to Test.egg-info/top_level.txt
writing dependency_links to Test.egg-info/dependency_links.txt
reading manifest file 'Test.egg-info/SOURCES.txt'
writing manifest file 'Test.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/Test-0.0.0-py3.5.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing Test-0.0.0-py3.5.egg
My questions:
Does the install command automatically invokes the build command?
If it does, what did I do wrong? If it does not, how can I ensure the build command runs?
Where can I find detailed description of the install process? I tried to dig inside the setuptools documentation and did not find anything like this.
Does the install command automatically invokes the build command?
Yes but it doesn't run python setup.py build — it calls the
corresponding code right inside one process hence you never see the
second print. If you really need to print it you have to override setuptools commands with custom classes; create such classes for build, bdist_egg and install commands.
Where can I find detailed description of the install process? I tried to dig inside the setuptools documentation and did not find anything like this.
I'm afraid distutils and setuptools source code is the only authoritative reference.
bdist_egg doesn't run build directly; it runs egg_info, build_clib and install_lib. install_lib calls distutils.build which runs build_py and build_ext. So it seems you have to override build_py, not build.
I am trying to build a pulp distributor plugin which will execute a bash script containing arbitrary code so that I may trigger actions after an RPM repo is published.
These plugins are generally created using distutils. However, when I attempt to install my module, I receive the error:
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Typically, this means that the working directory is incorrect or __init.py__ is missing. In my case however, I am attempting to install from the correct working directory and I did create __init.py__ files (see the repo here).
I suspect that I am running into a pathing issue having something to do with my code being in a subdirectory so far removed from setup.py. What am I doing wrong? Why won't my module install?
When you face errors like this, one of the first things to check is what packages are actually being added to your distribution when you build it. In your case the package list is empty, but should contain at least the pulp_hook package:
$ python -c "from setuptools import find_packages; print(find_packages())"
[]
So why does setuptools not recognize pulp_hook as a regular package? Look at its structure: you have added file named __init.py__, but its name should be __init__.py. Once you rename the files, the pulp_hook and its subdirectories become regular packages:
$ python -c "from setuptools import find_packages; print(find_packages())"
['pulp_hook', 'pulp_hook.plugins', 'pulp_hook.plugins.distributors']
Now build/lib will be created because now distutils finds at least one package to install:
$ python setup.py install_lib
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/pulp_hook
copying pulp_hook/__init__.py -> build/lib/pulp_hook
creating build/lib/pulp_hook/plugins
copying pulp_hook/plugins/__init__.py -> build/lib/pulp_hook/plugins
creating build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/distributionhook.py -> build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/__init__.py -> build/lib/pulp_hook/plugins/distributors
I've got a folder called Packages in my home folder. In this folder I want to develop some packages. To test them I tried to install the package with
python setup.py install develop
while inside the environment, of course. This gives me the following output
develop
running install
running bdist_egg
running egg_info
writing BACnetScapy.egg-info/PKG-INFO
writing top-level names to BACnetScapy.egg-info/top_level.txt
writing dependency_links to BACnetScapy.egg-info/dependency_links.txt
reading manifest file 'BACnetScapy.egg-info/SOURCES.txt'
writing manifest file 'BACnetScapy.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetLayers.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetTags.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/__init__.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetFunctions.py -> build/bdist.linux-x86_64/egg/BACnetScapy
copying build/lib.linux-x86_64-2.7/BACnetScapy/BACnetConstants.py -> build/bdist.linux-x86_64/egg/BACnetScapy
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetLayers.py to BACnetLayers.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetTags.py to BACnetTags.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetFunctions.py to BACnetFunctions.pyc
byte-compiling build/bdist.linux-x86_64/egg/BACnetScapy/BACnetConstants.py to BACnetConstants.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BACnetScapy.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/BACnetScapy-1.0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing BACnetScapy-1.0-py2.7.egg
Copying BACnetScapy-1.0-py2.7.egg to /home/lk/virtualenvs/FuzzingEnv/lib/python2.7/site-packages
Adding BACnetScapy 1.0 to easy-install.pth file
Installed /home/lk/virtualenvs/FuzzingEnv/lib/python2.7/site-packages/BACnetScapy-1.0-py2.7.egg
Processing dependencies for BACnetScapy==1.0
Finished processing dependencies for BACnetScapy==1.0
running develop
Checking .pth file support in build/bdist.linux-x86_64/egg
/home/lk/virtualenvs/FuzzingEnv/bin/python -E -c pass
TEST FAILED: build/bdist.linux-x86_64/egg does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
build/bdist.linux-x86_64/egg
and your PYTHONPATH environment variable currently contains:
''
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations
Please make the appropriate changes for your system and try again.
So the problem seems to be that the build folder which is automatically created in the package folder is not on the PYTHONPATH - but why should it be? And how should I add it to the PYTHONPATH? I tried to add this folder with
add2virtualenv /home/lk/Packages/BACnetScapy/build
but that did not work, either.
What am I doing wrong?
This is my setup.py
from setuptools import setup
setup(
name="BACnetScapy",
author="lk",
version="1.0",
packages=["BACnetScapy"],
include_package_data=True
)
The folder looks like that
/BACnetScapy
|
|-/BACnetScapy
| |- some.py
| |- modules.py
| |- __init__.py
| |-/data # a folder containing some data the package needs
|
|-setup.py
The command should be python setup.py develop, not python setup.py install develop
I am attempting to setup a simple package in develop mode:
setup.py
from setuptools import setup, find_packages
setup(
name='rocflavors',
version='0.0.1',
packages=find_packages(exclude=['ez_setup','examples','tests']),
license='Creative Commons Attribution-Noncommercial-Share Alike license',
long_description=open('README').read(),
install_requires=[],
)
Attempt to setup: (works if I do python setup.py install)
(relux)roc-web5537:roc-flavors cmuench$ python setup.py install develop
running install
running bdist_egg
running egg_info
writing rocflavors.egg-info/PKG-INFO
writing top-level names to rocflavors.egg-info/top_level.txt
writing dependency_links to rocflavors.egg-info/dependency_links.txt
reading manifest file 'rocflavors.egg-info/SOURCES.txt'
writing manifest file 'rocflavors.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-intel/egg
running install_lib
running build_py
creating build/bdist.macosx-10.7-intel/egg
creating build/bdist.macosx-10.7-intel/egg/rocflavors
copying build/lib/rocflavors/__init__.py -> build/bdist.macosx-10.7-intel/egg/rocflavors
copying build/lib/rocflavors/models.py -> build/bdist.macosx-10.7-intel/egg/rocflavors
creating build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags
copying build/lib/rocflavors/templatetags/__init__.py -> build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags
copying build/lib/rocflavors/templatetags/rocflavors_extras.py -> build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags
copying build/lib/rocflavors/urls.py -> build/bdist.macosx-10.7-intel/egg/rocflavors
copying build/lib/rocflavors/views.py -> build/bdist.macosx-10.7-intel/egg/rocflavors
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/models.py to models.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/templatetags/rocflavors_extras.py to rocflavors_extras.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/urls.py to urls.pyc
byte-compiling build/bdist.macosx-10.7-intel/egg/rocflavors/views.py to views.pyc
creating build/bdist.macosx-10.7-intel/egg/EGG-INFO
copying rocflavors.egg-info/PKG-INFO -> build/bdist.macosx-10.7-intel/egg/EGG-INFO
copying rocflavors.egg-info/SOURCES.txt -> build/bdist.macosx-10.7-intel/egg/EGG-INFO
copying rocflavors.egg-info/dependency_links.txt -> build/bdist.macosx-10.7-intel/egg/EGG-INFO
copying rocflavors.egg-info/top_level.txt -> build/bdist.macosx-10.7-intel/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/rocflavors-0.0.1-py2.7.egg' and adding 'build/bdist.macosx-10.7-intel/egg' to it
removing 'build/bdist.macosx-10.7-intel/egg' (and everything under it)
Processing rocflavors-0.0.1-py2.7.egg
Removing /Users/cmuench/.virtualenvs/relux/lib/python2.7/site-packages/rocflavors-0.0.1-py2.7.egg
Copying rocflavors-0.0.1-py2.7.egg to /Users/cmuench/.virtualenvs/relux/lib/python2.7/site-packages
rocflavors 0.0.1 is already the active version in easy-install.pth
Installed /Users/cmuench/.virtualenvs/relux/lib/python2.7/site-packages/rocflavors-0.0.1-py2.7.egg
Processing dependencies for rocflavors==0.0.1
Finished processing dependencies for rocflavors==0.0.1
running develop
Checking .pth file support in build/bdist.macosx-10.7-intel/egg
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 2] No such file or directory: 'build/bdist.macosx-10.7-intel/egg/test-easy-install-35540.pth'
The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
build/bdist.macosx-10.7-intel/egg
This directory does not currently exist. Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).
I have 3 questions:
Why does it NOT work in develop mode.
What is the difference between develop mode and not using it?
Why does it need to be a package? If I change a .py file in the package how does it make it back into my package directory. I don't think I understand the point of packaging. Does it put compiled python code into the package when I make a change?
Make sure your virtualenv is activated. Instead of python setup.py install develop, try pip install -e .. Note the ., it means that you must be in the directory where setup.py is. If it fails, try with sudo.
Docs for pip: http://www.pip-installer.org/