I want to make a distributable package.
And my package depends on some OS package
Here what I want to install:
def install_libmagic():
if sys.platform == 'darwin':
subprocess.run(['brew', 'install', 'libmagic'])
elif sys.platform == 'linux':
subprocess.run(['apt-get', 'update'])
subprocess.run(['apt-get', 'install', '-y', 'libmagic1'])
else:
raise Exception(f'Unknown system: {sys.platform}, can not install libmagic')
I want this code to be executed only when smb call:
pip install mypacakge
I don't want it to be executed when I run: python setup.py bdist_wheel
How can I achieve this?
I tried this:
setup(
...
install_requires=install_libmagic(),
)
Also tried to override install command:
from setuptools.command.install import install
class MyInstall(install):
def run(self):
install_libmagic()
install.run(self)
setup(
...
cmdclass={'install': MyInstall}
)
But the function was executed on python setup.py bdist_wheel, which is not what I am trying to achieve.
I think you're mixing up the behaviors of built distributions (wheels) and source distributions.
If your goal is run some subprocesses at install time, then you can't do this with a built distribution. A built distribution executes no Python code at install time. It only executes setup.py at build time, which is why you're seeing your functions executed when you call python setup.py bdist_wheel.
On the other hand, a source distribution (python setup.py sdist) does execute the setup.py file at both build time and install time (roughly the same as python setup.py install) and would give you the behavior you're looking for.
However, as the comments have already mentioned, this is going to be very fragile and not very user-friendly or portable. What you're describing is really a distro/OS package that contains some Python module, and you'd probably be better off with that instead.
Related
Is there a way to make certain dependencies for a python package optional ? Or conditional on the installation of another dependency failing.
The two cases I have are:
I want to install dependency x AND y. However, in case either of the install fails, the package can probably work alright with just one of them, so installation should complete with a warning.
I want to install dependency x IF the installation of y failed.
You can have conditional dependencies, but not based on the success/failure of the installation of other dependencies.
You can have optional dependencies, but an install will still fail if an optional dependency fails to install.
The simplest way to do have dependencies be optional (i.e. won't fail the main installation if they fail to install) or conditional (based on the success/failure of other packages) is by adding a custom install command that manually shells out to pip to install the individual packages, and checks the result of each call.
In your setup.py:
import sys
import subprocess
from setuptools import setup
from setuptools.command.install import install
class MyInstall(install):
def run(self):
# Attempt to install optional dependencies
try:
subprocess.check_call([sys.executable, "-m", "pip", "install", "dependency-y"])
except subprocess.CalledProcessError:
subprocess.check_call([sys.executable, "-m", "pip", "install", "dependency-x"])
# Continue with regular installation
install.run(self)
setup(
...
cmdclass={
'install': MyInstall,
},
)
Note that this will only work if you publish a source distribution (.tar.gz or .zip). It will not work if you publish your package as a built distribution (.whl).
# setup.py
from setuptools import setup
from setuptools.command.install import install
from subprocess import check_call
class CustomInstall(install):
def run(self):
check_call("./build.sh")
install.run(self)
setup(
name='customlib',
packages=['customlib'],
version='0.0.1',
...
cmdclass={'install': CustomInstall}
)
build.sh contains a make & make install step which takes more than 10 minutes to finish.
Is there a PyPi way to "package" the output of build.sh to speed up the pip install process?
Use wheel. A wheel is a great standard format for passing around Python packages, and it can contain C code compiled for various architectures. PyPI supports uploading wheels for your project, and pip will download them when available.
Very useful docs can be found here: https://packaging.python.org/tutorials/distributing-packages/#packaging-your-project
I use pip with setuptools to install a package.
I want pip to copy some resource files to, say, /etc/my_package.
My setup.py looks like this:
setup(
...
data_files=[('/etc/my_package', ['config.yml'])]
)
When running pip install, the file ends up in
~/.local/lib/python3.5/site-packages/etc/my_package/config.yml
instead of /etc/my_package.
What am I doing wrong?
(pip version 9.0.1)
Short answer: use pip install --no-binary :all: to install your package.
I struggled with this for a while and eventually figured out that there is some weirdness/inconsistency in how data_files are handled between binary wheels and source distributions. Specifically, there is a bug with wheels that makes all paths in data_files relative to the install location (see https://github.com/pypa/wheel/issues/92 for an issue tracking this).
"Thats fine", you might say, "but I'm not using a wheel!". Not so fast! It turns out recent versions of pip (I am working with 9.0.1) will try to compile a wheel even from a source distribution. For example, if you have a package my_package you can see this doing something like
$ python setup.py sdist # create source tarball as dist/my_package.tar.gz
[...]
$ pip install dist/my_package.tar.gz # install the generated source
[...]
Building wheels for collected packages: my_package
Running setup.py bdist_wheel for my_package ... done
pip tries to be helpful and build a wheel to install from and cache for later. This means you will run into the above bug even though in theory you are not using bdist_wheel yourself. You can get around this by running python setup.py install directly from the package source folder. This avoids the building and caching of built wheels that pip will try to do but is majorly inconvenient when the package you want is already on PyPI somewhere. Fortunately pip offers an option to explicitly disable binaries.
$ pip install --no-binary :all: my_package
[...]
Skipping bdist_wheel for my_package, due to binaries being disabled for it.
Installing collected packages: my_package
Running setup.py install for my_package ... done
Successfully installed my_package-0.1.0
Using the --no-binary option prevents wheel building and lets us reference absolute paths in our data_files paths again. For the case where you are installing a lot of packages together and want to selectively disable wheels you can replace :all: with a comma separated list of packages.
it seems that data_files can't support absolute path, it will add sys.prefix before "/etc/my_package", if you want to put config.yml to ../site_packages/my_package, please try:
import os
import sys
from distutils.sysconfig import get_python_lib
relative_site_packages = get_python_lib().split(sys.prefix+os.sep)[1]
date_files_relative_path = os.path.join(relative_site_packages, "my_package")
setup(
...
data_files=[(date_files_relative_path, ['config.yml'])]
)
I ended up writing an init() function that installs the config file on first run instead of creating it during the installation:
def init():
try:
if not path.isdir(config_dir):
os.mkdir(cs_dir)
copyfile(pkg_resources.resource_filename(
__name__, "default_config.yml"), config_file)
print("INFO: config file created. ")
except IOError as ex:
print("ERROR: could not create config directory: " + str(ex)
if __name__ == "__main__":
init()
main()
I'm trying to add a post-install step into the setup.py. The installation works but the code inside run(self) is never executed.
Things I've tried (with no result):
Install it using both "pip install (-e) ." and "python setup.py (develop)" [and later uninstall it, reinstall it, delete .egg-info folder,...]
Variations using: do_egg_install, build, bdist_egg,...
Versions:
pip 8.1.2 (for Python 3.5)
setuptools-20.2.2
Toy example:
from setuptools import setup
from setuptools.command.install import install
class MyCommand(install):
def run(self):
print("Hello, developer, how are you? :)")
install.run(self)
if __name__ == '__main__':
setup(
cmdclass={
'install': MyCommand,
}
)
I'm using this code:
code = 'import setuptools;__file__={0!r};execfile(__file__)'.format(os.path.join(path, 'setup.py'))
args = ['install', '--single-version-externally-managed']
subprocess.check_call([sys.executable, '-c', code, args])
To execute a setup.py and install the package. The problem occurs when setup.py uses distutils instead of setuptools: --single-version-externally-managed is not recognized by distutils.
How can I force setup.py to use setuptools?
What you have written is basically what pip does. Based on the code you wrote, you will be using setuptools' setup function because you've imported from setuptools. Setuptools paves over Distutils' setup function in its __init__.py. Therefore, it doesn't mater if the setup.py script imports distutils or not. Setuptools will always win...
If for some reason you still have issues while running your command. Try compiling the file before execution. exec(compile(...)) rather than execfile(...)
In response to #jknair answer... I'd also discourage the use of ez_setup.py, because it's code duplication, has unexpected behavior and is often excluded during package distribution (which makes it hard for tools like pip to run the setup.py without an ImportError).