I deploy my app with virtualenv --no-site-packages and pip install -r requirements.txt. There is feedgen module in requirements, that has site-packages/feedgen/ext directory with extensions. Among other, I deploy foobar.py extension for feedgen, and the answer is:
What is the best practice to deploy module's lib? In case of, e.g., Heroku, AWS,etc
Is it ok to use setup.py? Or Makefile? Or shell-script to copy file? Or what else?
PS I,m not ready to contribute to feedgen yet, as my code is still 'smells'. (I understand that so the problem would be solved by itself)
The most popular practice is to put a module on pypi and be able to install it with pip. The second way is cloning from github and running python setup.py install. The second way is usually used for installation from source and the first often provides compiled wheels that do not need compilation.
If you do not want to share the code you could use installation from files in pip and use private git repositories instead of github.
An example from pyarrow:
Usual installation from pip is written on this page:
pip install pyarrow
Installation from source. Here it is a Cython module so the author uses build_ext instead of install
git clone https://github.com/apache/arrow.git
...
cd arrow/python
...
python setup.py build_ext --inplace
Related
I would like to install some Python module, namely the MATLAB Engine. MATLAB is installed globally under /usr/local/sw/matlab, so I change to the corresponding directory and then run setup.py as explained in the linked documentation. However, I am not root, so I added the --user flag, as documented by the official Python Docs.
So this is what I get:
> python setup.py install --user
running install
running build
running build_py
creating build
error: could not create 'build': Permission denied
Apparently it tries to build the module in the current directory, where I have no write access. I can sidestep this by running
python setup.py build --build-base=/path/to/temp/build/dir
But then I don't know how to install that. Also I cannot combine those 2 approaches as install does not recognize the --build-base option.
I furthermore tried to copy the whole matlabroot/extern/engines/python directory somewhere else and run python setup.py install --user there, but this yields
error: The installation of MATLAB is corrupted. Please reinstall MATLAB or contact Technical Support for assistance.
I guess it does not find the matlabroot anymore.
So how do I build and install some source located somewhere under /usr... without being root?
You can use Virtualenv or venv to deal with this kind of issues.
Edit: As we went a bit deeper into issue itself, we managed to figure out the answer.
If you wish to use virtual environment: python setup.py build --build-base="/path/to/directory_you_can_write_to" install --prefix="/path/to/virtualenv/" --user
If you wish to accomplish what #Feodoran asked for: python setup.py build --build-base="/path/to/directory_you_can_write_to" install --user
Useful links:
https://de.mathworks.com/help/matlab/matlab_external/install-matlab-engine-api-for-python-in-nondefault-locations.html
https://python.org/3.2/install
Two options in setup.py develop and install are confusing me. According to this site, using develop creates a special link to site-packages directory.
People have suggested that I use python setup.py install for a fresh installation and python setup.py develop after any changes have been made to the setup file.
Can anyone shed some light on the usage of these commands?
python setup.py install is used to install (typically third party) packages that you're not going to develop/modify/debug yourself.
For your own stuff, you want to first install your package and then be able to frequently edit the code without having to re-install the package every time — and that is exactly what python setup.py develop does: it installs the package (typically just a source folder) in a way that allows you to conveniently edit your code after it’s installed to the (virtual) environment, and have the changes take effect immediately.
Note: It is highly recommended to use pip install . (regular install) and pip install -e . (developer install) to install packages, as invoking setup.py directly will do the wrong things for many dependencies, such as pull prereleases and incompatible package versions, or make the package hard to uninstall with pip.
Update:
The develop counterpart for the latest python -m build approach is as follows (as per):
From the documentation. The develop will not install the package but it will create a .egg-link in the deployment directory back to the project source code directory.
So it's like installing but instead of copying to the site-packages it adds a symbolic link (the .egg-link acts as a multiplatform symbolic link).
That way you can edit the source code and see the changes directly without having to reinstall every time that you make a little change. This is useful when you are the developer of that project hence the name develop. If you are just installing someone else's package you should use install
Another thing that people may find useful when using the develop method is the --user option to install without sudo. Ex:
python setup.py develop --user
instead of
sudo python setup.py develop
Problem: the package I want to install is outdated on pip, and conda doesn't have it in the repo. So, when I install a python package from github using,
git clone package_url
cd package_name
python setup.py
should I DOWNLOAD the package from within the directory that is the directory in which conda or pip usually would install my package? For example, should I run git clone from within:
['/Users/home/anaconda/lib/python2.7/site-packages',
'/Users/home/anaconda/lib/site-python']
OR, can I just run git clone, from whatever directory I happen to be in.
The concern is that I download from git in something like, /Users/home/Downloads, and then, when I run the setup.py file, I would only install within the /Users/home/Downloads directory, and then when I import, I wouldn't be able to find the package.
Accepted answer: I can run the git clone command in terminal from within any directory. Then, I can change directory into the newly established directory for the package that I cloned, and run the setup.py script. Running the setup.py script should "automatically install [the package] within the site-packages of whatever python [is being] used when python [is invoked]". I hope this helps someone overly anxious about running setup.py files.
Run it from the folder containing setup.py.
Doing:
python setup.py install
Will install the package in the appropriate directory. The file already contains the logic that puts the package in the right installation directory, so you don't need to worry about how the package make its way to its installation directory.
It can be simpler to use pip for this package as well, by pointing pip directly at the URL:
pip install git+http://....git
The git+ in front of the URL is required.
You can even go a step further and install a specific branch:
pip install git+http://....git#branchname
You can run the setup.py file as you stated, and you follow it by install as follow:
python setup.py install
Usually, this would lead to installing the package you want to the python path.
I have a Python script which uses open source third party libraries for geoprocessing (OGR and Shapely). My plan is to execute this script on a computer without having to install the required libraries.
I know that there are tools such as py2exe available for this purpose. However, compiling an executable file is not my first choice as I noticed that such files can get pretty large in size. Besides, I would like to use the code within another script. I would therefore like to create a portable python script which already includes the third party methods needed for executing.
Is it possible to include third party methods in a Python script in order to avoid the installation of third party libraries? And if not, what can I do instead, besides compiling an executable file? I work on Windows OS.
You can export your libraries using pip and embbed them into your application.
pip wheel requires the wheel package to be installed, which provides the "bdist_wheel" setuptools extension that it uses.
To build wheels for your requirements and all their dependencies to a local directory:
pip install wheel
pip freeze > requirements.txt
At this point check requirements.txt, clean it up, then you can download wheels in a local folder :
pip wheel --wheel-dir=/local/wheels -r requirements.txt
And then to install those requirements just using your local directory of wheels (and not from PyPI):
pip install --no-index --find-links=/local/wheels -r requirements.txt
Then though you'll need pip, though it's shipped with latest versions of python.
Check this : https://pip.readthedocs.io/en/latest/user_guide/#requirements-files
If your third party lib does not have any dependencies you can get the source files (.py) put into your project folder and use it as package by using import, else it has dependencies your project size grow more better create exe for your script.
I had a custom script programmed and it is using the authors own module that is hosted on Google code in a Mercurial repo. I understand how to clone the repo but this will just stick the source into a folder on my computer. Is there a proper way to add the module into my python install to make it available for my projects? (e.g. with modules hosted on pypi you can use virtualenv and pip to install).
Thanks
Dave O
In exactly the same way. Just pass the address of the repo to pip install, using the -e parameter:
pip install -e hg+http://code.google.com/path/to/repo
If the module isn't on pypi, clone the repository with Hg and see if there's a setup.py file. If there is, open a command prompt, cd to that directory, and run:
python setup.py install