I have a setup.py that looks like this:
from setuptools import setup
from subprocess import call
from setuptools.command.install import install
class MyInstall(install):
def run(self):
call(["pip install -r requirements.txt --no-clean"], shell=True)
install.run(self)
setup(
author='Attila Zseder',
version='0.1',
name='entity_extractor',
packages=['...'],
install_requires=['DAWG', 'mrjob', 'cchardet'],
package_dir={'': 'modules'},
scripts=['...'],
cmdclass={'install': MyInstall},
)
I need MyInstall because I want to install some libraries from github and I didn't want to use dependency_links option, because it's discouraged (for example here), so I can do this with requirements.txt.
When I install this package with pip, everything is working fine, but for some reasons I have to solve this in a way that it also works with pure python setup.py install. And it doesn't.
When overriding cmdclass in setup() with my own class, install_requires seems to be ignored. As soon as I comment out that line, those packages are being installed.
I know that install_requires is not supported for example in distutils (if I remember well), but it is in setuptools. And then cmdclass wouldn't have any effect on install_requires.
I googled this problem for hours, found a lot of kind of related answers on stackoverflow, but not for this particular problem.
With putting every needed package to requirements.txt, everything's working fine, but I would like to understand why this is happening. Thanks!
The same problem just happened to me. It somehow seems like something triggers setuptools to do an 'old-style install' with distutils, which indeed does not support install_requires.
You call install.run(self) which calls run(self) in setuptools/setuptools/command/install.py, line 51-74
https://bitbucket.org/pypa/setuptools/src/8e8c50925f18eafb7e66fe020aa91a85b9a4b122/setuptools/command/install.py?at=default
def run(self):
# Explicit request for old-style install? Just do it
if self.old_and_unmanageable or self.single_version_externally_managed:
return _install.run(self)
# Attempt to detect whether we were called from setup() or by another
# command. If we were called by setup(), our caller will be the
# 'run_command' method in 'distutils.dist', and *its* caller will be
# the 'run_commands' method. If we were called any other way, our
# immediate caller *might* be 'run_command', but it won't have been
# called by 'run_commands'. This is slightly kludgy, but seems to
# work.
#
caller = sys._getframe(2)
caller_module = caller.f_globals.get('__name__','')
caller_name = caller.f_code.co_name
if caller_module != 'distutils.dist' or caller_name!='run_commands':
# We weren't called from the command line or setup(), so we
# should run in backward-compatibility mode to support bdist_*
# commands.
_install.run(self)
else:
self.do_egg_install()
I'm not sure whether this behaviour is intended, but replacing
install.run(self)
with
install.do_egg_install()
should solve your problem. At least it works for me, but I would also appreciate a more detailed answer. Thanks!
According to https://stackoverflow.com/a/20196065 a more correct way to do this may be to override bdist_egg command.
You could try:
from setuptools.command.bdist_egg import bdist_egg as _bdist_egg
class bdist_egg(_bdist_egg):
def run(self):
call(["pip install -r requirements.txt --no-clean"], shell=True)
_bdist_egg.run(self)
...
setup(...
cmdclass={'bdist_egg': bdist_egg}, # override bdist_egg
)
It worked for me and install_requireis no more ignored. Nevertheless, I still don't understand why most people seem to override cmdclass install and do not complain about install_require being ignored.
I know this is an old question, but I ran into a similar problem. The solution I have found fixes this problem for me is very subtle: The install class you're setting in cmd_class must physically be named install. See this answer on a related issue.
Note that I use the class name install for my derived class because that is what python setup.py --help-commands will use.
You also should use self.execute(_func_name, (), msg="msg") in your post_install instead of calling the function directly
So implementing something like this should cause you to avoid the do_egg_install workaround implemented above by KEgg.
from setuptools.command.install import install as _install
...
def _post_install():
#code here
class install(_install):
def run(self):
_install.run(self)
self.execute(_post_install, (), msg="message here")
Related
I'm working on distributing a Python package. It depends on the library lupa. I want to run a post-install script that depends on lupa that initializes some data within the package after it's been installed. After looking at some answers around StackOverflow, my stripped setup.py essentially looks like this:
# setup.py
from distutils.core import setup
from setuptools.command.install import install
class PostInstallCommand(install):
def run(self):
# Do normal setup
install.run(self)
# Now do my setup
from mymodule.script import init
init()
setup(
# ...
install_requires = [
# ...
"lupa >= 1.10",
# ...
],
cmdclass = {
'install': PostInstallCommand
}
)
However, when emulating a fresh install/setup with tox on Python 3.10, I get this error:
File "C:\My\Computer\Temp\pip-req-build-pl0jria3\setup.py", line 26, in run
from mymodule.script import init
File "C:\My\Computer\Temp\pip-req-build-pl0jria3\mymodule\script.py", line 28, in <module>
import lupa
ModuleNotFoundError: No module named 'lupa'
I was under the impression that anything put into install_requires would be installed by the time setup() finished, but that appears not to be the case (also corroborated by this answer). Is there anything I can do to ensure that lupa is installed prior to mymodule.script.init(), or is that stage of the setup process entirely out of the user's hands?
After doing a good bit of research, it seems this kind of post-install script is somewhat against the core philosophy of setuptools, which means that a request like this is unlikely to be added, or at least added anytime soon.
Fortunately, this is somewhat of a blessing in disguise; my post-install script is actually an "update" console entry point that the user calls anytime they've added mods or updated any of the packages data. This script can be (and is supposed to be) called many times by the user, so by having it as part of the install process it helps introduce the purpose of the script to the user right from the start. This makes the slight annoyance on install tolerable, at least in my circumstance.
I have a distutils-style Python package which requires a specific, and quite large, dependency for its build step. Currently, this dependency is specified under the setup_requires argument to distutils.setup. Unfortunately, this means the dependency will be built for any execution of setup.py, including when running setup.py clean. This creates the rather ironic situation of the clean step sometimes causing large amount of code to be compiled.
As I said, this setup dependency is only required for the build step. Is there a way to encode this logic in setup.py so that all commands that do not invoke the build command are run without it?
You can always order the Distribution to fetch some packages explicitly, same way as they will be if you define them in setup_requires. Example with numpy dependency required for build command only:
from distutils.command.build import build as build_orig
from setuptools import setup, find_packages, Command, dist
class build(build_orig):
def run(self):
self.distribution.fetch_build_eggs(['numpy'])
# numpy becomes available after this line. Test it:
import numpy
print(numpy.__version__)
super().run()
setup(
name='spam',
packages=find_packages(),
cmdclass={'build': build,}
...
)
The dependencies are passed the same as they would be defined in setup_requires arg, so version specs are also ok:
self.distribution.fetch_build_eggs(['numpy>=1.13'])
Although I must note that fetching dependencies via setup_requires is usually much slower than installing them via pip (especially when you have some heavy dependencies that must be built from source first), so if you can be sure you will have pip available (or use python3.4 and newer), the approach suggested by phd in his answer will save you time. Fetching eggs via distribution may, however, come handy when building for old python versions or obscure python installations like the system python on MacOS.
if sys.argv[0] == 'build':
kw = {'setup_requires': [req1, req2, …]}
else:
kw = {}
setup(
…,
**kw
)
Another approach to try is override build command with a custom cmdclass:
from setuptools.command.build import build as _build
class build(_build):
def run(self):
subprocess.call(["pip", "install", req1, req2…])
_build.run(self)
setup(
…,
cmdclass={'build': build},
)
and avoid setup_requires at all.
Note: distutils is deprecated and the accepted answer has been updated to use setuptools
I'm trying to add a post-install task to Python distutils as described in How to extend distutils with a simple post install script?. The task is supposed to execute a Python script in the installed lib directory. This script generates additional Python modules the installed package requires.
My first attempt is as follows:
from distutils.core import setup
from distutils.command.install import install
class post_install(install):
def run(self):
install.run(self)
from subprocess import call
call(['python', 'scriptname.py'],
cwd=self.install_lib + 'packagename')
setup(
...
cmdclass={'install': post_install},
)
This approach works, but as far as I can tell has two deficiencies:
If the user has used a Python interpreter other than the one picked up from PATH, the post install script will be executed with a different interpreter which might cause a problem.
It's not safe against dry-run etc. which I might be able to remedy by wrapping it in a function and calling it with distutils.cmd.Command.execute.
How could I improve my solution? Is there a recommended way / best practice for doing this? I'd like to avoid pulling in another dependency if possible.
The way to address these deficiences is:
Get the full path to the Python interpreter executing setup.py from sys.executable.
Classes inheriting from setuptools.Command (such as setuptools.command.install.install which we use here) implement the execute method, which executes a given function in a "safe way" i.e. respecting the dry-run flag.
Note however that the --dry-run option is currently broken and does not work as intended anyway.
I ended up with the following solution:
import os, sys
from setuptools import setup
from setuptools.command.install import install as _install
def _post_install(dir):
from subprocess import call
call([sys.executable, 'scriptname.py'],
cwd=os.path.join(dir, 'packagename'))
class install(_install):
def run(self):
_install.run(self)
self.execute(_post_install, (self.install_lib,),
msg="Running post install task")
setup(
...
cmdclass={'install': install},
)
Note that I use the class name install for my derived class because that is what python setup.py --help-commands will use.
I think the easiest way to perform the post-install, and keep the requirements, is to decorate the call to setup(...):
from setup tools import setup
def _post_install(setup):
def _post_actions():
do_things()
_post_actions()
return setup
setup = _post_install(
setup(
name='NAME',
install_requires=['...
)
)
This will run setup() when declaring setup. Once done with the requirements installation, it will run the _post_install() function, which will run the inner function _post_actions().
The distutils module allows to include and install resource files together with Python modules. How to properly include them if resource files should be generated during a building process?
For example, the project is a web application which contains CoffeeScript sources that should be compiled into JavaScript and included in a Python package then. Is there a way to integrate this into a normal sdist/bdist process?
I spent a fair while figuring this out, the various suggestions out there are broken in various ways - they break installation of dependencies, or they don't work in pip, etc. Here's my solution:
in setup.py:
from setuptools import setup, find_packages
from setuptools.command.install import install
from distutils.command.install import install as _install
class install_(install):
# inject your own code into this func as you see fit
def run(self):
ret = None
if self.old_and_unmanageable or self.single_version_externally_managed:
ret = _install.run(self)
else:
caller = sys._getframe(2)
caller_module = caller.f_globals.get('__name__','')
caller_name = caller.f_code.co_name
if caller_module != 'distutils.dist' or caller_name!='run_commands':
_install.run(self)
else:
self.do_egg_install()
# This is just an example, a post-install hook
# It's a nice way to get at your installed module though
import site
site.addsitedir(self.install_lib)
sys.path.insert(0, self.install_lib)
from mymodule import install_hooks
install_hooks.post_install()
return ret
Then, in your call to the setup function, pass the arg:
cmdclass={'install': install_}
You could use the same idea for build as opposed to install, write yourself a decorator to make it easier, etc. This has been tested via pip, and direct 'python setup.py install' invocation.
The best way would be to write a custom build_coffeescript command and make it a subcommand of build. More details are given in other replies to similar/duplicate questions, for example this one:
https://stackoverflow.com/a/1321345/150999
I've got a Python package with its setup.py having dependencies declared via the usual way, in install_requires=[...]. One of the packages there, scikits.timeseries, has a setup.py expecting numpy to already be installed, thus, I'd like some way to have numpy installed first. For this case and in general, can the order of dependency installation be controlled? How?
Currently the order in which setup.py pulls down dependencies (as listed in the arg install_requires) seems practically random. Also, in the setup.py setup(...) I tried using the arg:
extras_require={'scikits.timeseries': ['numpy']}
...without success, the order of installing dependencies was unaffected.
I also tried setting up a pip requirements file, but there too, pip's order of installing dependencies didn't match the line-order of the requirements file, so no luck.
Another possibility would be to have a system call near the top of setup.py, to install numpy before the setup(...) call, but I hope there's a better way. Thanks in advance for any help.
If scikits.timeseries needs numpy, then it should declare it as a dependency. If it did, then pip would handle things for you (I'm pretty sure setuptools would, too, but I haven't used it in a long while). If you control scikits.timeseries, then you should fix it's dependency declarations.
Use setup_requires parameter, for instance to install numpy prior scipy put it into setup_requires and add __builtins__.__NUMPY_SETUP__ = False hook to get numpy installed correctly:
setup(
name='test',
version='0.1',
setup_requires=['numpy'],
install_requires=['scipy']
)
def run(self):
__builtins__.__NUMPY_SETUP__ = False
import numpy
Here's a solution which actually works. It's not an overly "pleasant" method to have to resort to, but "desperate times...".
Basically, you have to:
Override the setuptools "install command" class (plus the closely related analogs)
Execute pip from the script via command line statements for which you can enforce the order
The drawbacks to this are:
Pip must be installed. You can't just execute setup.py in an environment without that.
The console output of the initial "prerequisite" installs don't appear for some weird reason. (Perhaps I'll post an update here down the line fixing that...)
The code:
from setuptools import setup
# Override standard setuptools commands.
# Enforce the order of dependency installation.
#-------------------------------------------------
PREREQS = [ "ORDERED-INSTALL-PACKAGE" ]
from setuptools.command.install import install
from setuptools.command.develop import develop
from setuptools.command.egg_info import egg_info
def requires( packages ):
from os import system
from sys import executable as PYTHON_PATH
from pkg_resources import require
require( "pip" )
CMD_TMPLT = '"' + PYTHON_PATH + '" -m pip install %s'
for pkg in packages: system( CMD_TMPLT % (pkg,) )
class OrderedInstall( install ):
def run( self ):
requires( PREREQS )
install.run( self )
class OrderedDevelop( develop ):
def run( self ):
requires( PREREQS )
develop.run( self )
class OrderedEggInfo( egg_info ):
def run( self ):
requires( PREREQS )
egg_info.run( self )
CMD_CLASSES = {
"install" : OrderedInstall
, "develop" : OrderedDevelop
, "egg_info": OrderedEggInfo
}
#-------------------------------------------------
setup (
...
install_requires = [ "UNORDERED-INSTALL-PACKAGE" ],
cmdclass = CMD_CLASSES
)
You can add numpy to setup_requires section:
setup_requires=['numpy'],