I'm packaging up a minimal Ubuntu distro to fit in a 4GB disk image, for use on a VPS. This image is a (C++) webapp which (among other things) writes and runs simple Python scripts to handle conversions between csv and xls files, with csvkit and XlsxWriter doing the heavy lifting. My entire Python knowledge is unfortunately limited to writing and running these scripts.
Problem: I install pip in the image to handle the download and install of csvkit and XlsxWriter. This creates a huge amount of cruft, including what seems to be a C++ development environment, just to install what I imagine (presumably incorrectly) is simply Python source code. I can't really afford this in a 4GB distribution.
Is there a lightweight alternative to using pip to do this? Can I just copy over a handful of files from the dev machine, for example? I suppose one alternative is simply to uninstall pip after use, but I'd rather keep the disk image clean if possible (if nothing else, it will compress better).
XlsxWriter doesn't have any dependencies so it can be installed from source using python setup.py install. See the XlsxWriter installation docs.
However csvkit has a number of dependencies and may prove difficult to install from source. Which is why it is best to stick with pip since it handles that for you, which I'm sure you know.
Maybe try to install pip via get-pip which should give the smallest possible bootstrapping installation:
$ curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ python get-pip.py
If you are using python3.4 or newer you might harness ensurepip from standard library. It allows installing pip if it was not installed alongside with python, after doing
python -m ensurepip
you should be able to use pip as if it was installed together with python.
Related
I have developed a tool that my team can use after running the setup.py script. The tool requires this library: https://github.com/c2nes/javalang
How can I make my python setup script install this library on their computer regardless of what OS they are on. They can't run my tool without that library (Some people are on windows, mac, and linux.)
pip can install projects on Github as a dependency too!
All you need to do is, in your requirements.txt, add a line like following:
..
git+https://github.com/c2nes/javalang.git
then install the dependency using:
$ pip install -r requirements.txt
What you are looking for exists on PyPI. Instead of git+https://.. line above, just say: javalang. Oh and BTW, unless they are running old versions of Python, they should already have pip installed. If they don't use your operating systems package manager or get-pip.py as you said.
I am still relatively new to python packaging, each time I think I find "the" solution, I am thrown another curve ball, here is my problem followed by what I've tried:
I have CentOS and Ubuntu systems with Python 2.7.3 installed that is partitioned from the net so I have to create an "all in one package"
The target system does NOT have setuptools, easy_install, pip, virtualenv installed (this is the problem I'm trying to solve here)
The requirements.txt (or setup.py install_dependencies) is fairly heavy (Flask, etc...) for the application (though really, this isn't the problem)
My packaging sophistication has progressed slowly:
For connected systems, I had a really nice process going with
packaging: python2.7 setup.py sdist
installation: create a virtualenv, untar the distribution, python setup.py install
For the disconnected system, I've tried a few things. Wheels seem to be appropriate but I can't get to the "final" installation that includes setuptools, easy_install, pip. I am new to wheels so perhaps I am missing something obvious.
I started with these references:
Python on Wheels, this was super helpful but I could not get my .sh scripts, test data, etc... installed so I am actually using a wheel/sdist hybrid right now
Wheel, the Docs, again, very helpful but I am stuck on "the final mile of a disconnected system"
I then figured out I could package virtualenv as a wheel :-) Yay
I then figured out I could package easy_install as a python program :-) Yay, but it depends on setuptools, boo, I can't find how to get these packaged / installed
Is there a reference around for bootstrapping a system that has Python, is disconnected, but does not have setuptools, pip, wheels, virtualenv? My list of things a person must do to install this simple agent is becoming just way too long :/ I suppose if I can finish the dependency chain there must be a way to latch in a custom script to setup.py to shrink the custom steps back down ...
Your process will likely vary according to what platform you are targeting, but in general, a typical way to get what you are trying to achieve is to download packages on an online machine, copy them over to the offline one, and then install them from a file rather than from a URL or repository).
A possible workflow for RPM-based distros may be:
Install python-pip through binary packages (use rpm or yum-downloadonly, to download the package on an online machine, then copy it over and install it on the offline one with rpm -i python-pip.<whatever-version-and-architecture-you-downloaded>).
On your online machine, use pip install --download <pkgname> to download the packages you need.
scp or rsync the packages to a given directory X onto your offline machine
Use pip install --find-links=<your-dir-here> <pkgname> to install packages on your offline machine.
If you have to replicate the process on many servers, I'd suggest you set up your own repositories behind a firewall. In case of pip, it is very easy, as it's just a matter of telling pip to use a directory as its own index:
$ pip install --no-index --find-links=file:///local/dir/ SomePackage
For RPM or DEB repos is a bit more complicated (but not rocket science!), but possibly also not that necessary, as you really only ought to install python-pip once.
The pip install --download option that #mac mentioned has been deprecated and removed. Instead the documentation states that the pip download method should be used instead. So the workflow should be:
Download the python package or installer using your online machine.
Install python using the offline method used by your package manager or the python installer for windows on the offline machine.
On the online machine use pip download -r requirements.txt where "requirments.txt" contains the packages you will be needing the proper format
Use pip install --find-links=<your-dir-here> <pkgname> to install packages on your offline machine.
I'm having trouble installing scipy via the binaries provided at http://sourceforge.net/projects/scipy/files/scipy/
Double clicking on the mpkg file after mounting the dmg installer gives the following error:
"scipy 0.13.0 can't be installed on this disk. scipy requires System Python 2.7 to install"
However, I already have python 2.7 and numpy installed. The python 2.7 came default with OSX Lion, so I assume it is System Python. With other python modules, one normally can download the binary then run
python setup.py install
Is there a way to cd through the mpkg file and locate a setup.py? Any advice install via this dmg installer?
I know there are other ways to manage python modules, like port and brew. However, I already installed a bunch of packages through setup.py, and I couldn't figure out how to get port to recognize those packages (for example, it will try to reinstall python and numpy via port)
Thanks!
If you have Mavericks and XCode 5, then you'll have to install Command Line Tools manually from the Apple Developer Site. I found this helpful post
You've got a few misconceptions here.
With other python modules, one normally can download the binary then run python setup.py install
No, that's what you do with source packages.
Is there a way to cd through the mpkg file and locate a setup.py?
No. What's inside an mpkg are pkg files. Which are filled with xar archives filled with cpio archives. Inside there is the built version of SciPy—that is, the files that setup.py would have copied to your site-packages if you'd run it—not the source package.
But you can download the source package yourself.
Or, better, let pip (or easy_install, but pip is better) download and run the setup.py for you.
Any advice install via this dmg installer?
If it won't work, my advice would be to not use it, and instead install with pip.
This blog post explains it, but I'll give you the details relevant to you below.
I know there are other ways to manage python modules, like port and brew. However, I already installed a bunch of packages through setup.py, and I couldn't figure out how to get port to recognize those packages.
You can't. MacPorts will not touch your system Python; it builds its own separate Python 2.7, with a completely independent site-packages directory and everything else. You would have to reinstall everything for this second Python 2.7. And deal with the confusion of having two Python 2.7 installations on the same machine.
Don't do that unless you absolutely have to.
In fact, if you want to use Homebrew for anything (and you do, see below), uninstall MacPorts, unless you really need it for something.
So, here are the steps:
Uninstall MacPorts.
I assume you already have Xcode and its Command Line Tools.
I assume you already have Homebrew.
Install a Fortran compiler with brew install gfortran.
Lion's Python 2.7 comes with easy_install, but not pip. So sudo easy_install pip to fix that. While you're at it, I'd suggest sudo easy_install readline, because you'll want that for ipython, and it won't work right with pip.
Apple's pre-installed NumPy has to be upgraded, and rebuilt with Fortran support, to make SciPy work. Fix that with sudo pip install --upgrade --force-reinstall numpy.
If you want ipython, pandas, etc. sudo pip install each of them as well.
In case you're considering upgrading soon, the exact same steps worked for me with OS X 10.9.0, except for some extra work to get the Xcode 5 command line tools set up.
I have an external package I want to install into my python virtualenv from a tar file.
What is the best way to install the package?
I've discovered 2 ways that can do it:
Extract the tar file, then run python setup.py install inside of the extracted directory.
pip install packagename.tar.gz from example # 7 in https://pip.pypa.io/en/stable/reference/pip_install/#examples
Is if there is any difference doing them in these 2 ways.
On the surface, both do the same thing: doing either python setup.py install or pip install <PACKAGE-NAME> will install your python package for you, with a minimum amount of fuss.
However, using pip offers some additional advantages that make it much nicer to use.
pip will automatically download all dependencies for a package for you. In contrast, if you use setup.py, you often have to manually search out and download dependencies, which is tedious and can become frustrating.
pip keeps track of various metadata that lets you easily uninstall and update packages with a single command: pip uninstall <PACKAGE-NAME> and pip install --upgrade <PACKAGE-NAME>. In contrast, if you install a package using setup.py, you have to manually delete and maintain a package by hand if you want to get rid of it, which could be potentially error-prone.
You no longer have to manually download your files. If you use setup.py, you have to visit the library's website, figure out where to download it, extract the file, run setup.py... In contrast, pip will automatically search the Python Package Index (PyPi) to see if the package exists there, and will automatically download, extract, and install the package for you. With a few exceptions, almost every single genuinely useful Python library can be found on PyPi.
pip will let you easily install wheels, which is the new standard of Python distribution. More info about wheels.
pip offers additional benefits that integrate well with using virtualenv, which is a program that lets you run multiple projects that require conflicting libraries and Python versions on your computer. More info.
pip is bundled by default with Python as of Python 2.7.9 on the Python 2.x series, and as of Python 3.4.0 on the Python 3.x series, making it even easier to use.
So basically, use pip. It only offers improvements over using python setup.py install.
If you're using an older version of Python, can't upgrade, and don't have pip installed, you can find more information about installing pip at the following links:
Official instructions on installing pip for all operating systems
Instructions on installing pip on Windows (including solutions to common problems)
Instructions on installing pip for Mac OX
pip, by itself, doesn't really require a tutorial. 90% of the time, the only command you really need is pip install <PACKAGE-NAME>. That said, if you're interested in learning more about the details of what exactly you can do with pip, see:
Quickstart guide
Official documentation.
It is also commonly recommended that you use pip and virtualenv together. If you're a beginner to Python, I personally think it'd be fine to start of with just using pip and install packages globally, but eventually I do think you should transition to using virtualenv as you tackle more serious projects.
If you'd like to learn more about using pip and virtualenv together, see:
Why you should be using pip and virtualenv
A non-magical introduction to Pip and Virtualenv for Python beginners
Virtual Environments
python setup.py install is the analog of make install: it’s a limited way to compile and copy files to destination directories. This doesn’t mean that it’s the best way to really install software on your system.
pip is a package manager, which can install, upgrade, list and uninstall packages, like familiar package managers including: dpkg, apt, yum, urpmi, ports etc. Under the hood, it will run python setup.py install, but with specific options to control how and where things end up installed.
In summary: use pip.
The question is about the preferred method to install a local tarball containing a python package, NOT about the advantage of uploading package to an indexing service like PyPi.
As lest I know some software distributor does not upload their package to PyPi, instead asking developers to download package from their website and install.
python setup.py install
This can work but not recommended. It's not necessary to unwrap the tarball file and go into it to run setup.py file.
pip install ../path/to/packagename.tar.gz
This is the way designed and preferred. Concise and align with PyPi-style packages.
More information about pip install can be found here: https://pip.readthedocs.io/en/stable/reference/pip_install/
I want to create a native Mac OS X package installer, the creation of the package is not really the problem, the real deal is the dependencies that have to get installed; I need to install Python, NumPy and Matplotlib (only if the required versions are not already installed).
I have heard really bad things about the Package Maker app, I've been reading a little and have already even found a nice tutorial although it is quite outdated. As a reference, here's Apple's reference guide.
I imagine I would have to use the uncompiled source provided from each of these three projects.
It would really help me to see the PackageMaker file that is used to create the official Python installer, if such file is available somewhere, please point me to it.
Anyway:
What would be the best way to do this? Is using a PackageMaker silly for this purpose? Any other literature that would help me?
Extra:
What would be the easiest way to test my installers?
I'm assuming that you want to install the packages that you mentioned because you are developing a Python application. Have you looked at PyInstaller? It "converts (packages) Python programs into stand-alone executables, under Windows, Linux, Mac OS X, Solaris and AIX", so you don't have to worry about what's installed on the target system.
And if you use PyInstaller, the "extra" would be easy. Simply copy the resulting executable to any other machine and test it out by executing it.
Something like /tmp/install.sh:
cd ~
curl -C - -O http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.7.tar.gz
tar -xzf virtualenv-1.7.tar.gz
cd ./virtualenv-1.7
python setup.py install
cd ~
rm virtualenv-1.7.tar.gz
rm -rd ./virtualenv-1.7
virtualenv myenvfolder
source myenvfolder/bin/activate
easy_install pip
pip install NumPy
pip install Matplotlib
And then:
chmod +x /tmp/install.sh;
/tmp/install.sh
Maybe you can use macports binary-packages or binary-archives?
and maybe fabric or puppet.
Puppet on OSX.
Macports is as simple as apt-get to use and takes care of all dependencies. By default macports installs to /opt/local so installs don't interfere with apple installs. Default is to compile from source. Some packages are big and have a lot of dependencies so compiling takes a lot of time and all the recourse on the machine. If you make a binary-archive you only have to compile ones pr machine arcithecture/osx-version. Then you only need to install macports and sett up a share with binary-archives. With fabric or puppet you can automate builds and distribution.
Then if you in the near future find out that you need pytable or numexpr it is as simple as: sudo port install py26-tables
and if other people need it to you can make binary-archive of it and put it on the share.
To install Python's latest version you can go here, and you can install NumPy and matplotlib directly in your terminal with these commands:
pip3 install matplotlib
pip3 install numpy