We Keep Coding

Can't install Proj 8.0.0 for cartopy linux

I am trying to install Cartopy on Ubuntu and need to install proj v8.0.0 binaries for Cartopy. However when I try to apt-get install proj-bin I can only get proj v6.3.1. How do I install the latest (or at least v8.0.0) proj for cartopy?

I'm answering my own question here partly to help others with this problem, and partly as an archive for myself so I know how to fix this issue if I come across it again. I spent quite a while trying to figure it out, and wrote detailed instructions, so see below:
Installing cartopy is a huge pain, and I've found using conda to be a very bad idea (it has bricked itself and python along with it multiple times for me)
THIS INSTALLATION IS FOR LINUX.
Step 0. Update apt:
apt update
Step 1. Install GEOS:
Run the following command to install GEOS:
apt-get install libgeos-dev
In case that doesn't do it, install all files with this:
apt-get install libgeos-dev libgeos++-dev libgeos-3.8.0 libgeos-c1v5 libgeos-doc
Step 2. Install proj dependencies:
Install cmake:
apt install cmake
Install sqlite3:
apt install sqlite3
Install curl devlopment package:
apt install curl && apt-get install libcurl4-openssl-dev
Step 3. Install Proj
Trying apt-get just in case it works:
Unfortunately, cartopy requires proj v8.0.0 as a minimum, but if you install proj using apt you can only install proj v6.3.1
Just for reference in case anything changes, this is the command to install proj from apt:
apt-get install proj-bin
I'm fairly sure this is all you need, but in case it's not, this command will install the remaining proj files:
apt-get install proj-bin libproj-dev proj-data
To remove the above installation, run:
apt-get remove proj-bin
or:
apt-get remove proj-bin libproj-dev proj-data
Building Proj from source
So if the above commands don't work (it's not working as of 2022/4/8), then follow the below instructions to install proj from source:
Go to your install folder and download proj-9.0.0 (or any version with proj-x.x.x.tar.gz):
wget https://download.osgeo.org/proj/proj-9.0.0.tar.gz
Extract the tar.gz file:
tar -xf proj-9.0.0.tar.gz
cd into the folder:
cd proj-9.0.0
Make a build folder and cd into it:
mkdir build && cd build
Run (this may take a while):
cmake ..
cmake --build .
cmake --build . --target install
Run to make sure everything installed correctly:
ctest
The test command failed on one test for me (19 - nkg), but otherwise was fine.
You should find the required files in the ./bin directory
Finally:
Move binaries to the /bin directory:
cp ./bin/* /bin
As per Justino, you may also need to move the libraries:
cp ./lib/* /lib
Now after all this, you can finally install cartopy with pip:
pip install cartopy
After doing this, my cartopy still wasn't working. I went home to work on this next week, came back, and all of a sudden it was working so maybe try restarting

The libraries should be copied manually
sudo cp ./lib/* /lib
This works for me

Accessing clipboard on Travis-CI

I am trying to run an (integration?) test on my application, to verify that it actually copies the expected string to the clipboard with pyperclip.
This part is working on my development machine (Windows 10); but fails on travis-ci, where I get the following in my travis job log.
self = <pyperclip.init_no_clipboard.<locals>.ClipboardUnavailable object at 0x7ff0cd743588>
args = ('7 809823 102890 string 291',), kwargs = {}
def __call__(self, *args, **kwargs):
> raise PyperclipException(EXCEPT_MSG)
E pyperclip.PyperclipException:
E Pyperclip could not find a copy/paste mechanism for your system.
E For more information, please visit https://pyperclip.readthedocs.io/en/latest/introduction.html#not-implemented-error
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/pyperclip/__init__.py:301: PyperclipException
According the to the pyperclip documentation, this occurs on Linux when there is no copy/paste mechanism. The solution is to install one of the following (quoting pyperclip docs):
sudo apt-get install xsel to install the xsel utility.
sudo apt-get install xclip to install the xclip utility.
pip install gtk to install the gtk Python module.
pip install PyQt4 to install the PyQt4 Python module.
So in my .travis.yml file, I have
before_install:
- sudo apt-get install xclip
I've also tried xsel, with the same results.
Since the system on travis is Ubuntu 16.04.6, I tried adding sudo apt-get install python3-pyperclip to the before_install key, with the same result.
I was not able to install either gtk or PyQt4 by adding them to the install key in .travis.yml.
install:
- pip install -r requirements_dev.txt
- pip install PyQt4
# or
- pip install gtk
As these both result in the following error:
Could not find a version that satisfies the requirement gtk (from versions: )
No matching distribution found for gtk
The command "pip install gtk" failed and exited with 1 during .
By this point, my before_install looks like this:
- sudo apt-get install xclip
- sudo apt-get install xsel
- sudo apt-get install python3-pyperclip
- sudo apt-get install gtk2.0
This seems like overkill (and still does not work); but I am currently out of ideas on how to make that test pass. Any pointers would be greatly appreciated.
Thanks

xclip requires an X server to be running, while Travis machines run in headless mode. You need to run the command in a virtual framebuffer; install xvfb in addition to xclip and use xvfb-run pytest instead of pytest. Full config example:
language: python
addons:
apt:
packages:
- xclip
- xvfb
python:
- "3.7"
# setup installation
install:
- pip install -r requirements_dev.txt
script: xvfb-run pytest
Here's an example build on Travis. Notice that I used addons to declare dependencies that should be installed with APT; your solution with installing them explicitly in a before_install section is perfectly valid too, just a matter of taste.

How to update pip3 to its latest version in Ubuntu 18.04?

I get this error when running
$ pip3 install -U pip
Requirement already up-to-date: pip in ./dlenv/lib/python3.6/site-packages (10.0.1)
launchpadlib 1.10.6 requires testresources, which is not installed.
I have searched in apt and testresources seems to be installed already.
apt search testresources
Sorting... Done
Full Text Search... Done
python-testresources/bionic,bionic 2.0.0-2 all
PyUnit extension for managing expensive test fixtures - Python 2.x
python3-testresources/bionic,bionic 2.0.0-2 all
PyUnit extension for managing expensive test fixtures - Python 3.
I've gone through this github issue, which was not clear with a solution.

try this,
sudo apt install python3-testresources

Suggest installing pip by the PyPA guide on Ubuntu 18.04, since the pip version is too old if we install it by sudo apt install python3-pip.
Following work for me:
# curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
# python get-pip.py
BTW, the default location of pip is /usr/local/bin/pip, in case can't find the PATH.

Try this
sudo apt-get remove python-pip python-dev - This will remove the pip and python
then install the required version of python and pip

Installing Python Libraries in Built-from-source Python 2.7.11

I have built python from scratch and it now sits in a folder in my home directory. I am unsure of how to install libraries for this particular version/location.
Ubuntu comes with 2.7.6 and I need something in particular from 2.7.11.
Thanks!

First of all get setuptools
apt-get update
apt-get install python-setuptools
apt-get install python-pip
If you plan to use git and other tools install
apt-get install git
apt-get install python-dev
Once pip and setuptools are in place you can get any modules with either pip
or easy_install

Get setuptools from https://pypi.python.org/pypi/setuptools
run
wget https://bootstrap.pypa.io/ez_setup.py -O - | sudo python
now you got setuptools and ez_setup.py - locate this file and run
python ez_setup.py
Now you have easy_install, so you can either use it or install pip:
easy_install pip
At that point you should be able to use pip.

scipy ImportError on travis-ci

I'm setting up Travis-CI for the first time. I install scipy in what I believe is the standard way:
language: python
python:
- "2.7"
# command to install dependencies
before_install:
- sudo apt-get -qq update
- sudo apt-get -qq install python-numpy python-scipy python-opencv
- sudo apt-get -qq install libhdf5-serial-dev hdf5-tools
install:
- "pip install numexpr"
- "pip install cython"
- "pip install -r requirements.txt --use-mirrors"
# command to run tests
script: nosetests
Everything builds. But when the nosetests begin, I get
ImportError: No module named scipy.ndimage
Update: Here is a more direct demonstration of the problem.
$ sudo apt-get install python-numpy python-scipy python-opencv
$ python -c 'import scipy'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named scipy
The command "python -c 'import scipy'" failed and exited with 1 during install.
I tried installing scipy using pip also. I tried installing gfortran first. Here is one example of a failed build. Any suggestions?
Another Update: Travis has since added official documentation on using conda with Travis. See ostrokach's answer.

I found two ways around this difficulty:
As #unutbu suggested, build your own virtual environment and install everything using pip inside that environment. I got the build to pass, but installing scipy from source this way is very slow.
Following the approach used by the pandas project in this .travis.yml file and the shell scripts that it calls, force travis to use system-wide site-packages, and install numpy and scipy using apt-get. This is much faster. The key lines are
virtualenv:
system_site_packages: true
in travis.yml before the before_install group, followed by these shell commands
SITE_PKG_DIR=$VIRTUAL_ENV/lib/python$TRAVIS_PYTHON_VERSION/site-packages
rm -f $VIRTUAL_ENV/lib/python$TRAVIS_PYTHON_VERSION/no-global-site-packages.txt
and then finally
apt-get install python-numpy
apt-get install python-scipy
which will be found when nosetests tries to import them.
Update
I now prefer a conda-based build, which is faster than either of the strategies above. Here is one example on a project I maintain.

This is covered in the official conda documentation: Using conda with Travis CI.
The .travis.yml file
The following shows how to modify the .travis.yml file to use Miniconda for a project that supports Python 2.6, 2.7, 3.3, and 3.4.
NOTE: Please see the Travis CI website for information about the basic configuration for Travis.
language: python
python:
# We don't actually use the Travis Python, but this keeps it organized.
- "2.6"
- "2.7"
- "3.3"
- "3.4"
install:
- sudo apt-get update
# We do this conditionally because it saves us some downloading if the
# version is the same.
- if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
else
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
fi
- bash miniconda.sh -b -p $HOME/miniconda
- export PATH="$HOME/miniconda/bin:$PATH"
- hash -r
- conda config --set always_yes yes --set changeps1 no
- conda update -q conda
# Useful for debugging any issues with conda
- conda info -a
# Replace dep1 dep2 ... with your dependencies
- conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION dep1 dep2 ...
- source activate test-environment
- python setup.py install
script:
# Your test script goes here

I found this approach to work:
http://danielnouri.org/notes/2012/11/23/use-apt-get-to-install-python-dependencies-for-travis-ci/
Add these lines to your Travis configuration to use a virtualenv with --system-site-packages:
virtualenv:
system_site_packages: true
You can thus install Python packages via apt-get in the before_install section, and use them in your virtualenv:
before_install:
- sudo apt-get install -qq python-numpy python-scipy
A real-world use of this approach can be found in nolearn.

As Dan Allan pointed out in his update, he now prefers a conda-based build. Here is a gist courtesy Dan Blanchard giving a full .travis.yml file example that will pre-install scipy on the test machine:
language: python
python:
- 2.7
- 3.3
notifications:
email: false
# Setup anaconda
before_install:
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
- chmod +x miniconda.sh
- ./miniconda.sh -b
- export PATH=/home/travis/miniconda/bin:$PATH
- conda update --yes conda
# The next couple lines fix a crash with multiprocessing on Travis and are not specific to using Miniconda
- sudo rm -rf /dev/shm
- sudo ln -s /run/shm /dev/shm
# Install packages
install:
- conda install --yes python=$TRAVIS_PYTHON_VERSION atlas numpy scipy matplotlib nose dateutil pandas statsmodels
# Coverage packages are on my binstar channel
- conda install --yes -c dan_blanchard python-coveralls nose-cov
- python setup.py install
# Run test
script:
- nosetests --with-cov --cov YOUR_PACKAGE_NAME_HERE --cov-config .coveragerc --logging-level=INFO
# Calculate coverage
after_success:
- coveralls --config_file .coveragerc