I want to create a python package which will be cloned from its git repo when a build runs, so I will have the source inside the build agent. I would then like to run the python package as a command line tool, the package is called environment_manager.
Initially I thought I would follow a tutorial for creating a simple setup.py although this has proved to be a lot more difficult than I thought it would be and whenever I run python setup.py install --force I am not able to use my installed package, generally either module not found or the command is not recognised when I type it.
I have found that if I simply install with pip install . then I am actually able to use the tool from the command line and it works. I don't understand what the difference is, or why this only works when doing the pip install method.
Below is the setup.py file, I cannot see what is wrong with it:
from setuptools import setup, find_packages, find_namespace_packages
import pathlib
here = pathlib.Path(__file__).parent.resolve()
# Get the long description from the README file
long_description = (here / 'README.MD').read_text(encoding='utf-8')
setup(
name='environment_manager',
version='1.0.0',
package_dir={'': 'src'},
packages=find_namespace_packages(where='src', include='environment_manager.*'),
python_requires='>=3.8, <4',
install_requires=['boto3', 'botocore', 'pyyaml'],
extras_require={
'dev': ['pre-commit', 'black', 'pylint'],
'test': ['pytest', 'pytest-mock', 'coverage'],
},
entry_points={
'console_scripts': [
'environment-manager=environment_manager.environment_controller:main',
],
}
)
My project structure looks like:
environment_manager
/src
conf/
environment_manager/
environment_controller.py
config_parser.py
command.py
test/
unit_tests.py
I thought the correct way to install and run the tool from the command line was to use setup.py and setuptools but it seems like it is a lot easier and actually works if I just install it with pip.
Is installing it with pip over setup.py correct (as both ways the package appears when I type pip list) and are there any issues with my setup.py script? The script was taken from the pypa sample project and I removed most of what I didnt need.
setup.py is a python file, which usually tells you that the module/package you are about to install has been packaged and distributed with Distutils, which is the standard for distributing Python Modules. This allows you to easily install Python packages. Often it's enough to write: $ pip install .
In other words setup.py is a packaging file while pip is a package manager, therefore you should have setup.py file to be able to install with pip.
pip is a package manager which helps install, manage, and uninstall Python packages. It searches for them on PyPI, downloads them, and then runs their setup.py script.
Since you mentioned that you can run your binary executable after a pip install, but not a setup.py install, it is likely that each of them is installing the binary to separate locations.
One thing I would check is that you are using python and pip from the same version of Python, e.g:
% python --version
Python 3.8.6
% pip --version
pip 20.1.1 from /usr/lib/python3.8/site-packages/pip (python 3.8)
If these have different Python versions listed, they are likely installing to two separate directories - one in your PATH environment variable, and one which is not.
Next, I would check pip list -v after each install method, as this should list a Location header telling you where the package has been installed.
Related
Installing with pip, I can write the following requirements.txt file:
git+https://repo#branch#egg=foo&subdirectory=this/bar/foo
numpy
And successfully install the requirements file:
python3 -m pip install -r requirements.tx
However, I have co-located in the directory a setup.py script that lists:
setuptools.setup(
...
install_requires = get_lines('requirements.txt'),
...
)
And installing this submodule using pip involves pip running setup.py...which fails to handle the module link:
git+https://github.com/repo#branch#egg=foo&subdirectory=this/bar/foo
I can see a lot of ways around this, but it seems like there should be one non-ambiguous way to do this which changes as little as possible in the setup.py script.
Is there such a way?
You probably need to change the line in requirements.txt to something like:
foo # git+https://repo#branch#egg=foo&subdirectory=this/bar/foo
References:
https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers
https://www.python.org/dev/peps/pep-0440/#direct-references
Although I am not entirely sure it will work. There might be subtle differences between the notations accepted in requirements.txt files, pip directly and setuptools. In particular I do not know how well things like egg and subdirectory are supported.
Advices:
Avoid calling python setup.py install or python setup.py develop from now on, and make sure to call python -m pip install . or python -m pip install --editable . instead.
I do consider reading requirements.txt from within setup.py as a red flag (or at least yellow). The contents of install_requires of setuptools and requirements.txt usually serve different purposes.
I had to manually build a package and copy it to the site-packages directory. When I type pip list into a console it isn't listed, though I can use it in python scripts. How can I make pip aware of the package?
Installing it via pip is not an option.
You say "Installing it via pip is not an option.", but I'm assuming installing it via pip using a local copy still is. If so, the way to do that is to clone your library into a directory (say /my/lib/dir), where the root of the source for the root package appears below /my/lib/dir (ex: if the package you want to install is imported as import foo, then you should have /my/lib/dir/foo). If there is no file named setup.py in your copy of the code, then you need to create a simple one. Something like
# in a file called setup.py above the `foo` directory
from distutils.core import setup
setup(name='foo',
version='1.0',
packages=['foo'],
)
Finally, run pip install . from /my/lib/dir.
It's definitely a hack, but making pip aware of a package without installing it via pip is asking for a hack :-)
today I attempted to remove a file after my package (a python wheel) was installed via pip with the -t --target option.
Post-install script with Python setuptools
I am subclassing install in my setup.py like this:
class PostInstallCommand(install):
"""Post-installation for installation mode."""
def run(self):
install.run(self)
# here I am using
p = os.path.join(self.install_libbase,"myPackage/folder/removeThisPyc.pyc")
if os.path.isfile(p):
os.unlink(p)
#there is also self.install_platlib and
#self.install_purelib which seem to be used by pip distutil scheme
#Have not tested those yet
when running
python setup.py install
this works the file is removed upon install.
But through
pip install path-to-my-wheel.whl
this does not work and the file is still there.
pip install -t /target/dir path-to-my-wheel.whl
does not work either...
So question is, what is pip doing with distutils and or setuptools and how can make this work?
Another thing I noticed is that pip does not seem to be printing anything, I am printing in my setup.py in verbose mode?
Is there away to see the full output from python instead of the "pip" only stuff?
Reading educates:
http://pythonwheels.com/
2. Avoids arbitrary code execution for installation. (Avoids setup.py)
As I am using wheels and wheels wont execute the setup.py, my concept of doing this is rubbish.
https://github.com/pypa/packaging-problems/issues/64
I guess this is between deployment and installation, though I would obviously count my little change to installation...
Is there a way to avoid pyc file creation upon a pip install whl ?
I was wondering how the above "yum install package" & "python setup.py install" are used differently in CentOS? I used yum install ... all the time. However, when I try to do python setup.py install, I always get: this setup.py file couldn't be found even though its path shows up under echo $PATH, unless I try to use it in its current directory or use the absolute path.
When you type python setup.py install, your shell will check your $PATH for the python command, and run that. Then, python will be examining its arguments, which are setup.py install. It knows that it can be given the name of a script, so it looks for the file called setup.py so it can be run. Python doesn't use your $PATH to find scripts, though, so it should be a real path to a file. If you just give it the name setup.py it will only look in your current directory.
The source directory for a python module should not, ideally, be in your $PATH.
yum install is a command that will go to a package repository, download all the files needed to install something, and then put them in the right place. yum (and equivalents on other distributions, like apt for Debian systems) will also fetch and install any other packages you need, including any that aren't python modules.
Python has a package manager, too. You may also find using pip install modulename or pip install --user modulename (if you don't have administrative rights) easier than downloading and installing the module by hand. You can often get more recent versions of modules this way, as the ones provided by an operating system (through yum) tend to be older, more stable versions. Sometimes the module is not available through yum at all. pip can't install any extra packages that aren't python modules, though.
If you don't have pip already (it comes with Python3, but might need installing separately for Python2, depending on how it was set up), then you can install it by following the instructions here: https://pip.pypa.io/en/stable/installing/
I current have a setup.py file for an application I've written. Using setuptools I've easily been able to install pip-available requirements as such:
install_requires = [
'argparse',
'multiprocessing',
'requests',
'numpy',
'termcolor',
'prettytable'
]
The problem is that I also need to install MySQLdb, which is not installed via pip. When setting up locally, I had to download the tarball, uncompress, install, symlink, etc... To put it short, it was a PITA.
Is there anyway to automate this within my setup.py file? Rather than downloading the tarball and including it as a package? Even then, how would I run a setup.py within my own setup.py?
You can actually install MySQLdb via pip, but the package is named MySQL-python.
Now, your users will still need the package's C dependencies installed (libmysqlclient), but this is easily installed with a package manager. It would also be reasonably easy to provide a non-setup.py install script (e.g. a bash script) that installs the appropriate system dependencies (libmysqlclient) and calls your setup.py:
#!/bin/bash
apt-get install -y libmysqlclient-dev # Improve me! Check the platform first
python setup.py install
Just don't try to do too much in your setup.py. No one expects a setup.py script to install system packages, so you should refrain from doing that in yours.
Now, if requiring users to install a system package is too much (if they don't have root access, it can be), you should use a pure-Python MySQL client instead.
One such client is pymysql, which of course can be installed via pip.