I have a setup.py file that builds an extension. If I change one of the source files, distutils recognizes this and rebuilds the extension, showing all the compile / link commands.
However, if the only thing I change is setup.py (I'm fiddling trying to make library dependencies work), then it doesn't seem to rebuild (e.g., none of the compile/link commands show up). I've tested this by removing one of the source files in the line
sources = ['foo.c', 'bar.c' ...]
and when I pip install -e . or python setup.py install, it still creates a new file for the extension, but it must be a version cached somewhere, since it shouldn't compile.
How do I clear this cache? I have tried
python setup.py clean --all
or using the --ignore-installed and --no-cache-dir flags when doing pip install -e .
The only way I have found to make it rebuild is if I add garbage in a source file, triggering a rebuild and error, remove the garbage, and pip install -e . again...
Just delete under site-packages path any file related to it, you may find sometimes more than one version or some files packaged as zip files or run the following command python setup.py clean --all.
Recompile and install again.
But I will recommend to use python setup.py develop so you don't need to reinstall it with every change, you will be able to frequently edit your code and not have to re-install it again. python setup.py install is used to install typically a ready to use third-party packages.
Check here to better understand python packaging.
Summary:
python setup.py clean --all
python setup.py develop
I needed to run
python setup.py clean --all
python setup.py develop
Thanks to DhiaTN for getting me there.
Related
I would like to install some Python module, namely the MATLAB Engine. MATLAB is installed globally under /usr/local/sw/matlab, so I change to the corresponding directory and then run setup.py as explained in the linked documentation. However, I am not root, so I added the --user flag, as documented by the official Python Docs.
So this is what I get:
> python setup.py install --user
running install
running build
running build_py
creating build
error: could not create 'build': Permission denied
Apparently it tries to build the module in the current directory, where I have no write access. I can sidestep this by running
python setup.py build --build-base=/path/to/temp/build/dir
But then I don't know how to install that. Also I cannot combine those 2 approaches as install does not recognize the --build-base option.
I furthermore tried to copy the whole matlabroot/extern/engines/python directory somewhere else and run python setup.py install --user there, but this yields
error: The installation of MATLAB is corrupted. Please reinstall MATLAB or contact Technical Support for assistance.
I guess it does not find the matlabroot anymore.
So how do I build and install some source located somewhere under /usr... without being root?
You can use Virtualenv or venv to deal with this kind of issues.
Edit: As we went a bit deeper into issue itself, we managed to figure out the answer.
If you wish to use virtual environment: python setup.py build --build-base="/path/to/directory_you_can_write_to" install --prefix="/path/to/virtualenv/" --user
If you wish to accomplish what #Feodoran asked for: python setup.py build --build-base="/path/to/directory_you_can_write_to" install --user
Useful links:
https://de.mathworks.com/help/matlab/matlab_external/install-matlab-engine-api-for-python-in-nondefault-locations.html
https://python.org/3.2/install
today I attempted to remove a file after my package (a python wheel) was installed via pip with the -t --target option.
Post-install script with Python setuptools
I am subclassing install in my setup.py like this:
class PostInstallCommand(install):
"""Post-installation for installation mode."""
def run(self):
install.run(self)
# here I am using
p = os.path.join(self.install_libbase,"myPackage/folder/removeThisPyc.pyc")
if os.path.isfile(p):
os.unlink(p)
#there is also self.install_platlib and
#self.install_purelib which seem to be used by pip distutil scheme
#Have not tested those yet
when running
python setup.py install
this works the file is removed upon install.
But through
pip install path-to-my-wheel.whl
this does not work and the file is still there.
pip install -t /target/dir path-to-my-wheel.whl
does not work either...
So question is, what is pip doing with distutils and or setuptools and how can make this work?
Another thing I noticed is that pip does not seem to be printing anything, I am printing in my setup.py in verbose mode?
Is there away to see the full output from python instead of the "pip" only stuff?
Reading educates:
http://pythonwheels.com/
2. Avoids arbitrary code execution for installation. (Avoids setup.py)
As I am using wheels and wheels wont execute the setup.py, my concept of doing this is rubbish.
https://github.com/pypa/packaging-problems/issues/64
I guess this is between deployment and installation, though I would obviously count my little change to installation...
Is there a way to avoid pyc file creation upon a pip install whl ?
Two options in setup.py develop and install are confusing me. According to this site, using develop creates a special link to site-packages directory.
People have suggested that I use python setup.py install for a fresh installation and python setup.py develop after any changes have been made to the setup file.
Can anyone shed some light on the usage of these commands?
python setup.py install is used to install (typically third party) packages that you're not going to develop/modify/debug yourself.
For your own stuff, you want to first install your package and then be able to frequently edit the code without having to re-install the package every time — and that is exactly what python setup.py develop does: it installs the package (typically just a source folder) in a way that allows you to conveniently edit your code after it’s installed to the (virtual) environment, and have the changes take effect immediately.
Note: It is highly recommended to use pip install . (regular install) and pip install -e . (developer install) to install packages, as invoking setup.py directly will do the wrong things for many dependencies, such as pull prereleases and incompatible package versions, or make the package hard to uninstall with pip.
Update:
The develop counterpart for the latest python -m build approach is as follows (as per):
From the documentation. The develop will not install the package but it will create a .egg-link in the deployment directory back to the project source code directory.
So it's like installing but instead of copying to the site-packages it adds a symbolic link (the .egg-link acts as a multiplatform symbolic link).
That way you can edit the source code and see the changes directly without having to reinstall every time that you make a little change. This is useful when you are the developer of that project hence the name develop. If you are just installing someone else's package you should use install
Another thing that people may find useful when using the develop method is the --user option to install without sudo. Ex:
python setup.py develop --user
instead of
sudo python setup.py develop
Problem: the package I want to install is outdated on pip, and conda doesn't have it in the repo. So, when I install a python package from github using,
git clone package_url
cd package_name
python setup.py
should I DOWNLOAD the package from within the directory that is the directory in which conda or pip usually would install my package? For example, should I run git clone from within:
['/Users/home/anaconda/lib/python2.7/site-packages',
'/Users/home/anaconda/lib/site-python']
OR, can I just run git clone, from whatever directory I happen to be in.
The concern is that I download from git in something like, /Users/home/Downloads, and then, when I run the setup.py file, I would only install within the /Users/home/Downloads directory, and then when I import, I wouldn't be able to find the package.
Accepted answer: I can run the git clone command in terminal from within any directory. Then, I can change directory into the newly established directory for the package that I cloned, and run the setup.py script. Running the setup.py script should "automatically install [the package] within the site-packages of whatever python [is being] used when python [is invoked]". I hope this helps someone overly anxious about running setup.py files.
Run it from the folder containing setup.py.
Doing:
python setup.py install
Will install the package in the appropriate directory. The file already contains the logic that puts the package in the right installation directory, so you don't need to worry about how the package make its way to its installation directory.
It can be simpler to use pip for this package as well, by pointing pip directly at the URL:
pip install git+http://....git
The git+ in front of the URL is required.
You can even go a step further and install a specific branch:
pip install git+http://....git#branchname
You can run the setup.py file as you stated, and you follow it by install as follow:
python setup.py install
Usually, this would lead to installing the package you want to the python path.
I was wondering how the above "yum install package" & "python setup.py install" are used differently in CentOS? I used yum install ... all the time. However, when I try to do python setup.py install, I always get: this setup.py file couldn't be found even though its path shows up under echo $PATH, unless I try to use it in its current directory or use the absolute path.
When you type python setup.py install, your shell will check your $PATH for the python command, and run that. Then, python will be examining its arguments, which are setup.py install. It knows that it can be given the name of a script, so it looks for the file called setup.py so it can be run. Python doesn't use your $PATH to find scripts, though, so it should be a real path to a file. If you just give it the name setup.py it will only look in your current directory.
The source directory for a python module should not, ideally, be in your $PATH.
yum install is a command that will go to a package repository, download all the files needed to install something, and then put them in the right place. yum (and equivalents on other distributions, like apt for Debian systems) will also fetch and install any other packages you need, including any that aren't python modules.
Python has a package manager, too. You may also find using pip install modulename or pip install --user modulename (if you don't have administrative rights) easier than downloading and installing the module by hand. You can often get more recent versions of modules this way, as the ones provided by an operating system (through yum) tend to be older, more stable versions. Sometimes the module is not available through yum at all. pip can't install any extra packages that aren't python modules, though.
If you don't have pip already (it comes with Python3, but might need installing separately for Python2, depending on how it was set up), then you can install it by following the instructions here: https://pip.pypa.io/en/stable/installing/