I am still relatively new to python packaging, each time I think I find "the" solution, I am thrown another curve ball, here is my problem followed by what I've tried:
I have CentOS and Ubuntu systems with Python 2.7.3 installed that is partitioned from the net so I have to create an "all in one package"
The target system does NOT have setuptools, easy_install, pip, virtualenv installed (this is the problem I'm trying to solve here)
The requirements.txt (or setup.py install_dependencies) is fairly heavy (Flask, etc...) for the application (though really, this isn't the problem)
My packaging sophistication has progressed slowly:
For connected systems, I had a really nice process going with
packaging: python2.7 setup.py sdist
installation: create a virtualenv, untar the distribution, python setup.py install
For the disconnected system, I've tried a few things. Wheels seem to be appropriate but I can't get to the "final" installation that includes setuptools, easy_install, pip. I am new to wheels so perhaps I am missing something obvious.
I started with these references:
Python on Wheels, this was super helpful but I could not get my .sh scripts, test data, etc... installed so I am actually using a wheel/sdist hybrid right now
Wheel, the Docs, again, very helpful but I am stuck on "the final mile of a disconnected system"
I then figured out I could package virtualenv as a wheel :-) Yay
I then figured out I could package easy_install as a python program :-) Yay, but it depends on setuptools, boo, I can't find how to get these packaged / installed
Is there a reference around for bootstrapping a system that has Python, is disconnected, but does not have setuptools, pip, wheels, virtualenv? My list of things a person must do to install this simple agent is becoming just way too long :/ I suppose if I can finish the dependency chain there must be a way to latch in a custom script to setup.py to shrink the custom steps back down ...
Your process will likely vary according to what platform you are targeting, but in general, a typical way to get what you are trying to achieve is to download packages on an online machine, copy them over to the offline one, and then install them from a file rather than from a URL or repository).
A possible workflow for RPM-based distros may be:
Install python-pip through binary packages (use rpm or yum-downloadonly, to download the package on an online machine, then copy it over and install it on the offline one with rpm -i python-pip.<whatever-version-and-architecture-you-downloaded>).
On your online machine, use pip install --download <pkgname> to download the packages you need.
scp or rsync the packages to a given directory X onto your offline machine
Use pip install --find-links=<your-dir-here> <pkgname> to install packages on your offline machine.
If you have to replicate the process on many servers, I'd suggest you set up your own repositories behind a firewall. In case of pip, it is very easy, as it's just a matter of telling pip to use a directory as its own index:
$ pip install --no-index --find-links=file:///local/dir/ SomePackage
For RPM or DEB repos is a bit more complicated (but not rocket science!), but possibly also not that necessary, as you really only ought to install python-pip once.
The pip install --download option that #mac mentioned has been deprecated and removed. Instead the documentation states that the pip download method should be used instead. So the workflow should be:
Download the python package or installer using your online machine.
Install python using the offline method used by your package manager or the python installer for windows on the offline machine.
On the online machine use pip download -r requirements.txt where "requirments.txt" contains the packages you will be needing the proper format
Use pip install --find-links=<your-dir-here> <pkgname> to install packages on your offline machine.
Related
Two options in setup.py develop and install are confusing me. According to this site, using develop creates a special link to site-packages directory.
People have suggested that I use python setup.py install for a fresh installation and python setup.py develop after any changes have been made to the setup file.
Can anyone shed some light on the usage of these commands?
python setup.py install is used to install (typically third party) packages that you're not going to develop/modify/debug yourself.
For your own stuff, you want to first install your package and then be able to frequently edit the code without having to re-install the package every time — and that is exactly what python setup.py develop does: it installs the package (typically just a source folder) in a way that allows you to conveniently edit your code after it’s installed to the (virtual) environment, and have the changes take effect immediately.
Note: It is highly recommended to use pip install . (regular install) and pip install -e . (developer install) to install packages, as invoking setup.py directly will do the wrong things for many dependencies, such as pull prereleases and incompatible package versions, or make the package hard to uninstall with pip.
Update:
The develop counterpart for the latest python -m build approach is as follows (as per):
From the documentation. The develop will not install the package but it will create a .egg-link in the deployment directory back to the project source code directory.
So it's like installing but instead of copying to the site-packages it adds a symbolic link (the .egg-link acts as a multiplatform symbolic link).
That way you can edit the source code and see the changes directly without having to reinstall every time that you make a little change. This is useful when you are the developer of that project hence the name develop. If you are just installing someone else's package you should use install
Another thing that people may find useful when using the develop method is the --user option to install without sudo. Ex:
python setup.py develop --user
instead of
sudo python setup.py develop
I am following a tutorial on communicate TCP using twisted with Python. When I ran my code module it failed on the first line:
ImportError: No module named twisted.internet.protocol
So I guess there is no twisted install with my current of python 2.7.8. I am using Mac OS X 10.9.8.
So I tried to install twisted and version I found is tarball 14.0.0.
I install from terminal with command line:
sudo python setup.py install.
And everything look OK to me.
Now I ran the code module and I still have the same error. I guess I need to configure the new install twisted with python. I am looking for a guidance but still no luck yet.
When you ran:
sudo python setup.py install
You may well have screwed up your system.
setup.py install scribbles random files onto random pieces of your filesystem. Your OS maintainer has certain ideas about what files should exist in the "system" part of your system - loosely speaking, the part outside of the "home directory" area. When you run setup.py install using sudo you give it permission to write files of this "system" area. There's a reason normal users aren't allowed to write files there.
Perhaps some critical system service depends on a certain version of a Python library and this sudo python setup.py install command replaces the installed version (the version that your OS maintainer installed and shipped to you) with a different, incompatible version.
Or perhaps it's not a critical system service, maybe it's just one of the programs you use from time to time. Either way, it's now broken.
Never, never, never, never, never run that command. It doesn't matter what package you're thinking about installing: this is the wrong way to install it.
You should probably wipe your host and do a fresh install of your OS. Unfortunately this drastic solution is the most straightforward. setup.py install does not keep a record of what files it wrote or where. You can try to undo whatever damage it did, but figuring out what that damage is is complicated (beyond the scope of this answer) and time-consuming.
Then, install virtualenv and/or virtualenv wrapper. Then create a virtual environment, activate it, and install the desired Python packages into the environment.
virtualenv gives you isolated Python environments that don't interfere with each other or with your system Python. You can have as many of them as you want. They're cheap to create and if things go wrong you can easily destroy them and start over again.
Here's what the process looks like on Linux:
exarkun#top:/tmp$ virtualenv virtualenv-demo
New python executable in virtualenv-demo/bin/python
Installing setuptools, pip...done.
exarkun#top:/tmp$ . virtualenv-demo/bin/activate
(virtualenv-demo)exarkun#top:/tmp$ pip install twisted
Downloading/unpacking twisted
Using download cache from /home/exarkun/.pip/download-cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FT%2FTwisted%2FTwisted-14.0.0.tar.bz2
Running setup.py (path:/tmp/virtualenv-demo/build/twisted/setup.py) egg_info for package twisted
Downloading/unpacking zope.interface>=3.6.0 (from twisted)
.
.
.
Installing /tmp/virtualenv-demo/lib/python2.7/site-packages/zope.interface-4.1.1-py2.7-nspkg.pth
Successfully installed twisted zope.interface
Cleaning up...
(virtualenv-demo)exarkun#top:/tmp$
What's the best way to deploy a Python package to a Windows Server if some of the dependencies need to be compiled? Installing Visual Studio is out of the question and I'm reluctant to pass a 250MB file around everytime things need updating. pip install -r requirements.txt is the goal.
Any suggestions?
easy_install - allows installing from exe
easy_install does install from exe installers, if they are available. pip does not install from exe, so in case of C/C++ packages, easy_install is often the way to install.
making pip to install binaries
pip is supporting installation from wheel format.
In case you have a wheel on pypi or another index server you use, allow pip to use wheels (--use-wheels). Latest version of pip is using wheel format by default, older ones required setting --use-wheel switch.
building wheels
You can either build (compile) wheels, or you can convert eggs or exe installers to wheels.
For this, use wheel command.
Serving packages in wheel format
Some packages (e.g. lxml) do not provide wheels on pypi server. If you want to use them, you have to manage your own index. Easiest one is to use dedicated directory and pip configured with --find-links pointing there.
See SO answer to caching packages for more tips and details.
Very good alternative is devpi index server, which is easy to set up, has very good workflow and will integrate with your pip very well.
Using a compile-once, deploy-many-times strategy might help, with the use of Python's newest package distribution format, wheels. They are an already-compiled, binary format. So the requirements are just a development / build platform that's similar / running the same python as the deployment platform (the latter won't require a build chain anymore).
Install the latest pip, setuptools and wheel packages. You can then use the pip wheel command as you would pip install, except it will instead produce a folder called (by default) wheelhouse full of the necessary, already-compiled wheels for another python/pip to install what you required.
You can then pass those wheel filepaths directly to a pip install command (e.g on your deployment machine), or use the normal package name(s) with --no-index --find-links= flags to point to the folder location where those wheels are. This location can also be a simple http folder - for example.
I have an external package I want to install into my python virtualenv from a tar file.
What is the best way to install the package?
I've discovered 2 ways that can do it:
Extract the tar file, then run python setup.py install inside of the extracted directory.
pip install packagename.tar.gz from example # 7 in https://pip.pypa.io/en/stable/reference/pip_install/#examples
Is if there is any difference doing them in these 2 ways.
On the surface, both do the same thing: doing either python setup.py install or pip install <PACKAGE-NAME> will install your python package for you, with a minimum amount of fuss.
However, using pip offers some additional advantages that make it much nicer to use.
pip will automatically download all dependencies for a package for you. In contrast, if you use setup.py, you often have to manually search out and download dependencies, which is tedious and can become frustrating.
pip keeps track of various metadata that lets you easily uninstall and update packages with a single command: pip uninstall <PACKAGE-NAME> and pip install --upgrade <PACKAGE-NAME>. In contrast, if you install a package using setup.py, you have to manually delete and maintain a package by hand if you want to get rid of it, which could be potentially error-prone.
You no longer have to manually download your files. If you use setup.py, you have to visit the library's website, figure out where to download it, extract the file, run setup.py... In contrast, pip will automatically search the Python Package Index (PyPi) to see if the package exists there, and will automatically download, extract, and install the package for you. With a few exceptions, almost every single genuinely useful Python library can be found on PyPi.
pip will let you easily install wheels, which is the new standard of Python distribution. More info about wheels.
pip offers additional benefits that integrate well with using virtualenv, which is a program that lets you run multiple projects that require conflicting libraries and Python versions on your computer. More info.
pip is bundled by default with Python as of Python 2.7.9 on the Python 2.x series, and as of Python 3.4.0 on the Python 3.x series, making it even easier to use.
So basically, use pip. It only offers improvements over using python setup.py install.
If you're using an older version of Python, can't upgrade, and don't have pip installed, you can find more information about installing pip at the following links:
Official instructions on installing pip for all operating systems
Instructions on installing pip on Windows (including solutions to common problems)
Instructions on installing pip for Mac OX
pip, by itself, doesn't really require a tutorial. 90% of the time, the only command you really need is pip install <PACKAGE-NAME>. That said, if you're interested in learning more about the details of what exactly you can do with pip, see:
Quickstart guide
Official documentation.
It is also commonly recommended that you use pip and virtualenv together. If you're a beginner to Python, I personally think it'd be fine to start of with just using pip and install packages globally, but eventually I do think you should transition to using virtualenv as you tackle more serious projects.
If you'd like to learn more about using pip and virtualenv together, see:
Why you should be using pip and virtualenv
A non-magical introduction to Pip and Virtualenv for Python beginners
Virtual Environments
python setup.py install is the analog of make install: it’s a limited way to compile and copy files to destination directories. This doesn’t mean that it’s the best way to really install software on your system.
pip is a package manager, which can install, upgrade, list and uninstall packages, like familiar package managers including: dpkg, apt, yum, urpmi, ports etc. Under the hood, it will run python setup.py install, but with specific options to control how and where things end up installed.
In summary: use pip.
The question is about the preferred method to install a local tarball containing a python package, NOT about the advantage of uploading package to an indexing service like PyPi.
As lest I know some software distributor does not upload their package to PyPi, instead asking developers to download package from their website and install.
python setup.py install
This can work but not recommended. It's not necessary to unwrap the tarball file and go into it to run setup.py file.
pip install ../path/to/packagename.tar.gz
This is the way designed and preferred. Concise and align with PyPi-style packages.
More information about pip install can be found here: https://pip.readthedocs.io/en/stable/reference/pip_install/
I have a problem that comes from me following tutorials without really understanding what I'm doing. The root of the problem I think is the fact that I don't understand how the OS X filesystem works.
The problem is bigger than Python but it was when I started learning about Python that I realized how little I really understand. So in the beginning I started following tutorials which led me to use the easy_install command a lot and when a lot of tutorials recommended PIP I never got it running. So I have run a lot of commands and installed a lot of different packages.
As I have understood Lion comes with a python install. I have been using this a lot and from this I have installed various packages with easy_install. Is there any way to go back to default installation and begin from the very beginning? Is this something I want to do? If so why?
Is there any advantage of using a Python version I have installed with Homebrew? How can I see from where Python is run when I run the Python command?
When I do install something with either easy_install, homebrew, macports etc where do things actually end up?
Homebrew installs its software inside the /usr/local subdirectory on your Mac. OS X doesn't install anything there on its own; in fact, /usr/local is reserved for user-installed stuff. Since Homebrew never installs files outside /usr/local (and doesn't even have the ability to, unless you run brew using sudo - which is not recommended_) and OS X never installs files inside there, never the two shall mix.
easy_install and pip install files into system directories by default. That's why you have to run those commands with sudo to install packages with them.
I can't recommend virtualenv enough, regardless of which OS you're using. It installs a copy of Python, along with any packages or modules you want, inside a directory of your choosing. For example:
$ cd /tmp
$ virtualenv foo
New python executable in foo/bin/python
Installing setuptools............done.
Installing pip...............done.
$ cd foo
$ bin/pip install sqlalchemy
Downloading/unpacking sqlalchemy
Downloading SQLAlchemy-0.7.7.tar.gz (2.6Mb): 2.6Mb downloaded
Running setup.py egg_info for package sqlalchemy
[...]
Successfully installed sqlalchemy
Cleaning up...
[work, work, work]
[decide this was a bad idea]
$ cd /tmp; rm -rf foo
...and all traces of the project are now completely gone.
Use easy_install to install virtualenv into OS X itself - like you've done for those other packages - but then do all new development inside isolated directories that you can wipe clean at a moment's notice. This is pretty much the standard way of developing and deploying Python applications these days.
The advantage of using a Python installed via a package manager like Homebrew or MacPorts would be that this provides a simple way of removing the Python installation and reinstalling it. Also, you can install a more recent version than the one Mac OS X provides.