I'm using this code:
code = 'import setuptools;__file__={0!r};execfile(__file__)'.format(os.path.join(path, 'setup.py'))
args = ['install', '--single-version-externally-managed']
subprocess.check_call([sys.executable, '-c', code, args])
To execute a setup.py and install the package. The problem occurs when setup.py uses distutils instead of setuptools: --single-version-externally-managed is not recognized by distutils.
How can I force setup.py to use setuptools?
What you have written is basically what pip does. Based on the code you wrote, you will be using setuptools' setup function because you've imported from setuptools. Setuptools paves over Distutils' setup function in its __init__.py. Therefore, it doesn't mater if the setup.py script imports distutils or not. Setuptools will always win...
If for some reason you still have issues while running your command. Try compiling the file before execution. exec(compile(...)) rather than execfile(...)
In response to #jknair answer... I'd also discourage the use of ez_setup.py, because it's code duplication, has unexpected behavior and is often excluded during package distribution (which makes it hard for tools like pip to run the setup.py without an ImportError).
Related
I want to make a distributable package.
And my package depends on some OS package
Here what I want to install:
def install_libmagic():
if sys.platform == 'darwin':
subprocess.run(['brew', 'install', 'libmagic'])
elif sys.platform == 'linux':
subprocess.run(['apt-get', 'update'])
subprocess.run(['apt-get', 'install', '-y', 'libmagic1'])
else:
raise Exception(f'Unknown system: {sys.platform}, can not install libmagic')
I want this code to be executed only when smb call:
pip install mypacakge
I don't want it to be executed when I run: python setup.py bdist_wheel
How can I achieve this?
I tried this:
setup(
...
install_requires=install_libmagic(),
)
Also tried to override install command:
from setuptools.command.install import install
class MyInstall(install):
def run(self):
install_libmagic()
install.run(self)
setup(
...
cmdclass={'install': MyInstall}
)
But the function was executed on python setup.py bdist_wheel, which is not what I am trying to achieve.
I think you're mixing up the behaviors of built distributions (wheels) and source distributions.
If your goal is run some subprocesses at install time, then you can't do this with a built distribution. A built distribution executes no Python code at install time. It only executes setup.py at build time, which is why you're seeing your functions executed when you call python setup.py bdist_wheel.
On the other hand, a source distribution (python setup.py sdist) does execute the setup.py file at both build time and install time (roughly the same as python setup.py install) and would give you the behavior you're looking for.
However, as the comments have already mentioned, this is going to be very fragile and not very user-friendly or portable. What you're describing is really a distro/OS package that contains some Python module, and you'd probably be better off with that instead.
Question
Is there a way to build a wheel for a package while in a different repository such that the wheel has been built exactly as it would be if you built the wheel inside of the repository containing the package?
Example
Consider the following repo:
/repo-containing-your-package
|___ your_module/
|___ setup.py
Build method A
When I run python setup.py bdist_wheel from within repo-containing-your-package it builds the wheel as expected, including your_module. This means after I install pip install ./dist/your_module-#.#.#-py3-none-any.whl (which is successful), I can run python -m your_module.foo from the command line.
When the package is building, I get output that verifies that my module has been picked up by the wheel:
creating 'dist/your_module-#.#.#-py3-none-any.whl' and adding 'build/bar' to it
adding 'your_module/__init__.py'
etc...
Build method B
However, if I run python ../repo-containing-your-package/setup.py bdist_wheel from a repository that is a sibling to repo-containing-your-package, it does not build the wheel as expected, as it fails to include your_module. This means after I install pip install ./dist/your_module-#.#.#-py3-none-any.whl (which is successful), attempting python -m your_module.foo fails:
Error while finding module specification for 'your_module.foo' (ModuleNotFoundError: No module named 'your_module')
The fact that the module has not been properly installed with the package is confirmed by reviewing the build output, which does not include the adding 'your_module' output that method A includes.
Two solutions I know of:
change working directory in setup.py
If you can modify the setup script, you can change the working directory programmatically. Add an os.chdir call early enough in the setup script:
import os
from setuptools import setup
os.chdir(os.path.dirname(__file__))
setup(...)
You can also change the working directory with other means without having to modify the setup script, e.g. in bash:
$ pushd path/to/repo; python setup.py bdist_wheel; popd
Use pip wheel
pip has a subcommand wheel that builds a wheel from the given arg; this arg is usually the name of the package, but can be a directory containing the setup script. Pass -e in that case so the wheel has the correct name:
$ pip wheel -e path/to/repo
The importlib_resources backport for Python < 3.7 of the importlib.resources standard library module has the following section in the setup.cfg file:
[options]
python_requires = >=2.7,!=3.0,!=3.1,!=3.2,!=3.3
setup_requires =
setuptools
wheel
install_requires =
pathlib2; python_version < '3'
typing; python_version < '3.5'
packages = find:
Why does setup_requires include setuptools? This does not seem to make sense since:
the first line of the setup.py file imports setuptools, so by the time the setup function is called and reads the setup.cfg file that instructs to install setuptools it is already too late to install setuptools:
from setuptools import setup
setup()
setuptools is already installed on any fresh Python installation (well, only tested on Windows 10 and MacOS 10.15 with Python 3.8.0):
$ python -V
Python 3.8.0
$ pip list
Package Version
---------- -------
pip 19.2.3
setuptools 41.2.0
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
No, setuptools should not be included in setup_requires, according to PEP 518 (bold emphasis mine):
Setuptools tried to solve this with a setup_requires argument to its
setup() function [3]. This solution has a number of issues, such as:
No tooling (besides setuptools itself) can access this information without executing the setup.py, but setup.py can't be executed without having these items installed.
While setuptools itself will install anything listed in this, they won't be installed until during the execution of the setup() function, which means that the only way to actually use anything added here is through increasingly complex machinations that delay the import and usage of these modules until later on in the execution of the setup() function.
This cannot include setuptools itself nor can it include a replacement to setuptools, which means that projects such as numpy.distutils are largely incapable of utilizing it and projects cannot take advantage of newer setuptools features until their users naturally upgrade the version of setuptools to a newer one.
The items listed in setup_requires get implicitly installed whenever you execute the setup.py but one of the common ways that the setup.py is executed is via another tool, such as pip, who is already managing dependencies. This means that a command like pip install spam might end up having both pip and setuptools downloading and installing packages and end users needing to configure both tools (and for setuptools without being in control of the invocation) to change settings like which repository it installs from. It also means that users need to be aware of the discovery rules for both tools, as one may support different package formats or determine the latest version differently.
The accepted answer is mostly correct, but where PEP 518 says.
[The setup_requires mechanism] cannot include setuptools itself...
It's technically incorrect, and as importlib_resources demonstrates, it can actually include setuptools. The problem is that including setuptools in setup_requires serves mostly as documentation. It declares that setuptools is a build requirement (required to run setup.py), but it won't be capable of satisfying that requirement if it's not already satisfied.
But, the presence of setuptools in setup_requires is technically correct and does serve the purpose of declaring the requirement and asking setuptools to verify that the requirement is in fact installed (alongside other setup-time requirements).
It is, however, just a legacy artifact and doesn't provide that much value, and as can be seen in the question and answers, it does lead to confusion. The recommended, proper, approach is to use PEP 517 and 518 declarations and builders, but that part of the ecosystem hasn't matured yet, so setuptools vestiges will remain. Try not to let them bother you.
Why does setup_requires includes setuptools? This does not seem to make sense
Does not make sense at all. On the other hand it doesn't hamper anything so why not?
I've coauthored a Python package (let's call it "M") that has a number of dependent Python packages. The dependent packages are listed using install_requires in M's setup.py file. When users run python setup.py install for our package M, the process of installing one of those dependencies (let's call it "L") involves the compilation of an external C++ library if the user does not already have L installed. The compilation may generate a number of gcc warnings. I know the reasons for the warnings, and the issues are benign. I would like to suppress the printing of those warnings when a user runs setup.py install for our package M. At first, I thought that putting the following in setup.py for M would work,
setup (
...
extra_compile_args = ['-Wno-enum-conversion', '-Wno-strict-prototypes'],
...
)
but it turns out to do nothing – the warnings are still printed when the Python installation tools install the dependency L.
How can I suppress the warnings when users run python setup.py install for M?
Does anybody encounter this warning when executing python setup.py install of a PyPI package?
install_requires defines what the package requires. A lot of PyPI packages have this option. How can it be an "unknown distribution option"?
python setup.py uses distutils which doesn't support install_requires. setuptools does, also distribute (its successor), and pip (which uses either) do. But you actually have to use them. I.e. call setuptools through the easy_install command or pip install.
Another way is to import setup from setuptools in your setup.py, but this not standard and makes everybody wanting to use your package have to have setuptools installed.
This was the first result on my google search, but had no answer.
I found that upgrading setuptools resolved the issue for me (and pip for good measure)
pip install --upgrade pip
pip install --upgrade setuptools
Hope this helps the next person to find this link!
I'm on a Mac with python 2.7.11. I have been toying with creating extremely simple and straightforward projects, where my only requirement is that I can run python setup.py install, and have setup.py use the setup command, ideally from distutils. There are literally no other imports or code aside from the kwargs to setup() other than what I note here.
I get the error when the imports for my setup.py file are:
from distutils.core import setup
When I use this, I get warnings such as
/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'
warnings.warn(msg)
If I change the imports (and nothing else) to the following:
from distutils.core import setup
import setuptools # noqa
The warnings go away.
Note that I am not using setuptools, just importing it changes the behavior such that it no longer emits the warnings. For me, this is the cause of a truly baffling difference where some projects I'm using give those warnings, and some others do not.
Clearly, some form of monkey-patching is going on, and it is affected by whether or not that import is done. This probably isn't the situation for everyone researching this problem, but for the narrow environment in which I'm working, this is the answer I was looking for.
This is consistent with the other (community) comment, which says that distutils should monkeypatch setuptools, and that they had the problem when installing Ansible. Ansible appears to have tried to allow installs without having setuptools in the past, and then went back on that.
https://github.com/ansible/ansible/blob/devel/setup.py
A lot of stuff is up in the air... but if you're looking for a simple answer for a simple project, you should probably just import setuptools.
ATTENTION! ATTENTION! Imperfect answer ahead. To get the "latest memo" on the state of packaging in the Python universe, read this fairly detailed essay.
I have just ran into this problem when trying to build/install ansible. The problem seems to be that distutils really doesn't support install_requires. Setuptools should monkey-patch distutils on-the-fly, but it doesn't, probably because the last release of setuptools is 0.6c11 from 2009, whereas distutils is a core Python project.
So even after manually installing the setuptools-0.6c11-py2.7.egg running setup.py only picks up distutils dist.py, and not the one from site-packages/setuptools/.
Also the setuptools documentation hints to using ez_setup and not distutils.
However, setuptools is itself provided by distribute nowadays, and that flavor of setup() supports install_requires.
This is a warning from distutils, and is a sign that you do not have setuptools installed.
Installing it from http://pypi.python.org/pypi/setuptools will remove the warning.
In conclusion:
distutils doesn't support install_requires or entry_points, setuptools does.
change from distutils.core import setup in setup.py to from setuptools import setup or refactor your setup.py to use only distutils features.
I came here because I hadn't realized entry_points was only a setuptools feature.
If you are here wanting to convert setuptools to distutils like me:
remove install_requires from setup.py and just use requirements.txt with pip
change entry_points to scripts (doc) and refactor any modules relying on entry_points to be full scripts with shebangs and an entry point.
sudo apt-get install python-dev # for python2.x installs
sudo apt-get install python3-dev # for python3.x installs
It will install any missing headers. It solved my issue
As far as I can tell, this is a bug in setuptools where it isn't removing the setuptools specific options before calling up to the base class in the standard library: https://bitbucket.org/pypa/setuptools/issue/29/avoid-userwarnings-emitted-when-calling
If you have an unconditional import setuptools in your setup.py (as you should if using the setuptools specific options), then the fact the script isn't failing with ImportError indicates that setuptools is properly installed.
You can silence the warning as follows:
python -W ignore::UserWarning:distutils.dist setup.py <any-other-args>
Only do this if you use the unconditional import that will fail completely if setuptools isn't installed :)
(I'm seeing this same behaviour in a checkout from the post-merger setuptools repo, which is why I'm confident it's a setuptools bug rather than a system config problem. I expect pre-merge distribute would have the same problem)
I've now seen this in legacy tools using Python2.7, where a build (like a Dockerfile) installs an unpinned dependancy, for example pytest. PyTest has dropped Python 2.7 support, so you may need to specify version < the new package release.
Or bite the bullet and convert that app to Python 3 if that is viable.
It works fine if you follow the official documentation:
import setuptools
setuptools.setup(...)
Source: https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py