I'm trying to install cfgrib (https://pypi.org/project/cfgrib/) on databricks.
Cfgrib depends only on the c-library eccodes.
I installed both eccodes and cfgrib through the libraries page of databricks.
However, when I do "import cfgrib", it keeps on saying:
"Cannot find the ecCodes library"
On cfgrib package page, they say the best way to install it is via conda with:
$ conda install -c conda-forge cfgrib
Months ago with an older runtime version of the cluster (that I don't have anymore) I managed to install it throgh conda writing direcly on the databricks notebook: "%sh conda install -c conda-forge eccodes"
However when I try to do the same now it keeps running for like 1 hour and doesn't install it.
How can I install it?
Is there a way to install conda libraries on the cluster itself and not in the notebook?
Thank you very much,
(my first question, any feedback is welcomed :)
I have been using the following conda & python verison:
conda version : 4.6.14
conda-build version : 3.17.8
python version : 3.7.3.final.0
I installed simpletransformers in the following manner:
conda create -n simpletransformers python pandas tqdm
conda activate simpletransformers
conda install pytorch cpuonly -c pytorch
conda install -c anaconda scipy
conda install -c anaconda scikit-learn
pip install transformers
pip install seqeval
pip install tensorboardx
pip install simpletransformers
After doing so, I've been trying to import the classification model without much luck:
import simpletransformers
I get the following error:
ModuleNotFoundError: No module named 'simpletransformers'
Can someone point out where I'm going wrong? I'm using PyCharm as my IDE.
The setup docs work for me on Mac and Ubuntu using Anaconda:
Install Anaconda or Miniconda
Create a new virtual python 3.7 environment and install pandas and tqdm
conda create -n simplet python=3.7 pandas tqdm
conda activate simplet
PyTorch
3 a. GPU (use_cuda=True in your model): conda install pytorch cudatoolkit=10.1 -c pytorch
3 b. CPU (use_cuda=False in your model): conda install pytorch cpuonly -c pytorch
If you want to use fp16 training on an NVIDIA GPU install apex (don't use pip)
Install simpletransformers.
pip install simpletransformers
Download the .whl file from "https://pypi.org/project/simpletransformers/#files"
Open command prompt
type pip install "path/simpletransformers-0.13.2-py3-none-any.whl" and hit enter
Check whether the package gets installed.
Note that simpletransformers requires Python '>=3.6'
Whenever I have a package that is not available via Anaconda Cloud, i.e., I have to install from PyPI or GitHub, then I create a YAML environment definition for it. This follows the best practices enumerated in "Using Pip in a Conda Environment".
The advantage of a YAML is that it allows Conda to solve for everything at once and it lets one treat envs as immutable objects (i.e., if you need to alter the env, edit the YAML and recreate). This helps avoid the mess that inevitably seems to result from running a series of conda install, pip install, or conda update commands.
For me this is a multi-stage process, but it has been a reliable workflow for me:
Workflow for Mixed Conda-PyPI Environments
Look at the setup.py or requirements.txt of the non-Conda package. Here it is for simpletransformers.
For each requirement, check Anaconda Cloud (or conda search) to see if it is available as a Conda package.
If it is available, add it to the YAML file as a (non-pip) dependency. This ensures that everything that can come from Conda does.
Also, keep track of the channels that these packages come from. Note, I will not use a private channel I am unfamiliar with. In this case pytorch, conda-forge, and defaults (i.e., anaconda) suffice.
Include the packages that are PyPI-only under the pip section of the YAML, including the main package of interest (i.e., simpletransformers). Technically, you don't need to include the other dependencies, since pip will pull them in automatically, but I like to keep them explicit so that if I ever update the YAML I might check again if someone ported the PyPI packages to Conda Forge.
Create the env using the YAML
conda env create -n st_env -f simpletransformers.yaml
Check to see if any additional packages were implicitly pulled in as dependencies from PyPI, but were actually available through Conda. Edit the YAML to put these in the Conda dependencies section. In this case, keras is apparently also needed.
Remove the env and recreate using the updated maximally Conda version.
Most important: never change the env except through editing the YAML.
YAML for SimpleTransformers Environment
simpletransformers.yaml
name: st_env
channels:
- pytorch
- conda-forge
- defaults
dependencies:
- python=3.7
- pandas
- tqdm
- cpuonly
- pytorch
- transformers
- scipy
- scikit-learn
- requests
- tensorboardx
- keras
- pip
- pip:
- seqeval
- simpletransformers
Install with
conda env create -n st_env -f simpletransformers.yaml
If you have pip installed in your environment, just do hit a pip install simpletransformers in your terminal or If you're using jupyter notebook/colab, etc. then paste !pip install simpletransformers in your first cell and run it.
Then import simpletransformers
import simpletransformers
This error raised while installing geopandas. I've looking for its solution on the web, but none of them really explain what happened and how to solve it..
This is the full error:
Collecting geopandas
Using cached https://files.pythonhosted.org/packages/24/11/d77c157c16909bd77557d00798b05a5b6615ed60acb5900fbe6a65d35e93/geopandas-0.4.0-py2.py3-none-any.whl
Requirement already satisfied: shapely in c:\users\alvaro\anaconda3\envs\tfdeeplearning\lib\site-packages (from geopandas) (1.6.4.post2)
Requirement already satisfied: pandas in c:\users\alvaro\anaconda3\envs\tfdeeplearning\lib\site-packages (from geopandas) (0.20.3)
Collecting fiona (from geopandas)
Using cached https://files.pythonhosted.org/packages/3a/16/84960540e9fce61d767fd2f0f1d95f4c63e99ab5d8fddc308e8b51b059b8/Fiona-1.8.4.tar.gz
Complete output from command python setup.py egg_info:
A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\Alvaro\AppData\Local\Temp\pip-install-oxgkjg8l\fiona\
pip install wheel
pip install pipwin
pipwin install numpy
pipwin install pandas
pipwin install shapely
pipwin install gdal
pipwin install fiona
pipwin install pyproj
pipwin install six
pipwin install rtree
pipwin install geopandas
here are the source links:
http://geopandas.org/install.html#installation
https://pip.pypa.io/en/latest/user_guide/#installing-from-wheels
https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy
If you still have problems, consider uninstalling the above (pip uninstall) and reinstalling.
I solved this problem by running the following commands:
pip install pipwin
pipwin install gdal
pipwin install fiona
pip install geopandas
Works successfully on Windows.
Geospatial Data Abstraction Library (GDAL) is a library designed for vector geospatial data formats. It's a prerequisite for installing Fiona, the Python API for OGR (which doesn't really stand for anything), which is in turn a prerequisite for Geopandas. On UNIX-like systems the gdal-config script tells Fiona stuff about your particular gdal installation.
It seems that your gdal-config is not in one of the usual places on your PATH, so Fiona was unable to find it.
If you're using Anaconda, best is to remove gdal with conda remove gdal and then do a fresh conda install geopandas.
As a general rule, if you're using Conda you should never use pip to install something inside it unless you're absolutely sure conda offers no support for it. (Many package can be found on conda by specifying the right channel - -c argument.) And specifically in the case of geopandas, the maintainers recommend using conda over pip, since pip requires you to install the dependencies correctly.
I had a lot of issues myself installing geopandas, mostly showing error when downloading fiona and gdal. I did every step above and did a conda install geopandas but failed. The only thing worked for me is to install fiona and gdal wheel separately.
go to the link by Christoph: gohlke:https://www.lfd.uci.edu/~gohlke/pythonlibs/#fiona
You can search for fiona and gdal wheel files. Make sure you choose the file as per your python version, if it is 3.7 then there would be cp37.
Download the file
go to command prompt, put cd and then pip install , install GDAL wheel file, then fiona, then just do pip install geopandas.
This solution worked for me.
To install gdal, I followed the following steps:
downloaded the version that satisfies my computer (64 bit) from
https://www.lfd.uci.edu/~gohlke/pythonlibs/ . The file was GDAL-3.1.4-cp37-cp37m-win_amd64.whl
Put the file in a folder on the desktop.
From cmd, i moved to that directory and executed python -m pip install GDAL-3.1.4-cp37-cp37m-win_amd64.whl
This is followed by installing fiona the same way: python -m pip install Fiona-1.8.18-cp37-cp37m-win_amd64.whl
For shapely, i executed conda install -c conda-forge shapely
After that, i was able to install keplergl as usual: pip install keplergl
install descartes: conda install -c conda-forge descartes (or python -m pip install descartes).
In this way, i didn't have to play around with the 'Environmental Variables' as this may affect other programs
Cheers..
Installing geopandas
Geopandas has very complex multi-language dependencies, some of which need to be built with consistent compiler versions across packages. Because of this, the geopandas docs recommend installing using conda in a new environment using conda-forge only. Here are some general best practices to keep in mind:
conda is the recommended installation method. You can install geopandas from pip or source, but it's going to be a bumpy ride and it's not recommended. If you're installing conda for the first time, I recommend you start with miniconda (or better yet miniforge, a conda-forge-first miniconda variant), not anaconda, to keep your base env lean.
When using conda, you should not mix and match conda channels.
When installing geopandas, try creating a fresh environment rather than installing into your base environment. If you have anaconda installed, it comes with a large number of packages from the "defaults" channel installed in your base environment. I recommend deleting anaconda and installing miniconda, then installing into a new environment.
Try to create a new environment with everything you plan to use all at once rather than iteratively modifying the environment. In other words, if you want to use geopandas with scikit_learn, folium, and rasterio, install them together with a single conda create command
As a last resort, delete your conda installation and re-install miniconda. Desperate times call for desperate measures, and this usually resolves gnarly installation nightmares.
To create a fresh conda environment in which you install all necessary dependencies at the same time, using the conda-forge channel:
conda create -n my-geopandas-env -c conda-forge geopandas [all other packages you need]
For example, I might set up an environment with something along the lines of...
conda create -n my-geopandas-env -c conda-forge python=3.9 \
ipython ipykernel geopandas scipy seaborn fiona matplotlib cartopy
Bundling your installations into a single environment creation step like this reduces the chance of packages falling out of sync. To speed this process up, you could first install mamba or mambaforge, a faster drop-in replacement for conda, into your base environment and then run the above commands with mamba instead of conda.
Generally, it's best to avoid installing much of anything in your base environment (cross-environment system utilities like mamba are some of the few exceptions). If you already have a complex base environment (maybe you started with anaconda rather than miniconda) this may be the time to delete your entire conda installation and start from scratch (I know that's terrifying... sorry! but it'll save you heartache in the future). mamba is great for speeding this process up.
Connecting your editor to the conda environment
Once you have installed all of the packages you need, activate your environment with conda activate my-geopandas-env. See the conda guide to managing environments for more info.
Jupyter/ipython
Some editors/IDEs including jupyter require additional packages - jupyter requires that ipython and ipykernel be installed in order to load the environment within the notebook or editor - that's why I included ipykernel in my list above. See the ipykernel docs for more info.
Other IDES
To link this environment to an IDE such as VSCODE, spider, etc., find the location of this python version with conda run -n my-geopandas-env which python then point your editor to this python executable. Check the docs of your specific editor to get more targeted info about how to set up a conda environment for use with your editor:
Spider: FAQ on using an existing environment and Spider wiki guide to working with packages and environments
VSCode: Using python environments in vscode
PyCharm: Configure a conda virtual environment
I don't have conda installed, then using just pip I followed these steps:
Download GDAL and Fiona wheels directly on:
GDAL: https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal
FIONA: https://www.lfd.uci.edu/~gohlke/pythonlibs/#fiona
Then:
pip install <gdal.whl>
pip install <fiona.whl>
In my case I did pip install GDAL-3.4.1-cp38-cp38-win_amd64.whl and Fiona-1.8.21-cp38-cp38-win_amd64.whl. Where cp38 stands for python 3.8.
After that you are able to install geopandas with pip as well.
pip install geo pandas
For me, the only solution was to install the ready binaries from here
https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal
Then just install locally
pip install GDAL-3.1.4-cp38-cp38-win_amd64.whl
One way in which I could install geopandas was through the Anaconda Navigator. Get into the environment and install the package 'geopandas'. After that I could import the geopandas package in spyder
I will add
!pip install descartes
to #JDOaktown list.
I started with pip install geopandas and got the error, but later tried with conda install --channel conda-forge geopandas and the error disappeared.
Successfully installed in RHEL 7.8.
It automatically downloaded the required packages. This might be helpful
Installing collected packages: certifi, pyproj, shapely, attrs, click, click-plugins, munch, cligj, fiona, geopandas
Successfully installed attrs-20.3.0 certifi-2020.11.8 click-7.1.2 click-plugins-1.1.1 cligj-0.7.0 fiona-1.8.17 geopandas-0.8.1 munch-2.5.0 pyproj-3.0.0.post1 shapely-1.7.1
If you want to install GDAL, Geopandas, Shapely, Fiona etc in a windows Virtual Environment download .whl files for all of them and first install GDAL using
pip install gdal-.whl
Following this command edit the activate.bat file in you venv\Scripts folder and add
GDAL_CONFIG = \venv\Lib\site-packages\osgeo
Then you can install rest using pip install
I started off with a clean environment gdal_test in Conda environments, but made the mistake of using the old activate gdal_test instead of conda activate gdal_test. This made Conda Environment resolving take forever, which is why I resolved to other methods at first.
Takeaway: let conda handle it, with a proper new environment.
I ran into this problem not with anaconda/windows, but with python:3.6 Docker image. Google search always led me to this question, so I think I will share how I resolve my issue in case others also end up here.
Based on here, you need to install system relevant packages in the Dockerfile before running pip install geopandas or pip install requirements.txt:
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
libatlas-base-dev \
libgdal-dev \
gfortran
The following worked on macOS:
brew install gdal --HEAD
Verify the installation by running gdal-config --version
Following that pip installation as normal worked without a problem.
I want to install Cartopy on Windows, which has some dependencies according to http://scitools.org.uk/cartopy/docs/latest/installing.html#installing.
When using pip install cartopy in the cmd prompt, it gives an error where it wants me to install GEOS 3.3.3 and Proj4 4.9.0.
I've downloaded geos-3.6.2.tar.bz2, but I cannot figure out how to install it. I've extracted the files, used cd to the right directory.
What can I do to install it correctly?
Is there an easy way to install Cartopy?
Do you have Anaconda?
try:
conda install -c scitools cartopy
When you are installing through pip then better install using Binary file .Whl extension file
You can find all the package binary file below.
Especially in case of Cartopy installation through pip, install cartopy using binary wheel file.
https://www.lfd.uci.edu/~gohlke/pythonlibs/
If you are on Windows and you have installed Anaconda, on your search, type "Anaconda prompt" and then type this command
conda install -c conda-forge cartopy
I tried conda install -c conda-forge cartopy
and tried installing it through Anaconda navigator 1.9.12 and I scoured multitude of ways and tried them all and failed. Only this
conda install -c scitools cartopy
works.
I first ran this conda update -n base -c defaults condo following instructions provided among responses here:Updating Anaconda fails: Environment Not Writable Error
I am trying to install Tabula for Python, as it seems it is the way of extracting tables from PDFs. However I am unable to install it.
I am using Anaconda and have followed the step on Tabula's Anaconda page (https://anaconda.org/auto/tabula) to attempt to install it:
conda install -c auto tabula
But I just get an error message:
link here
As far as I'm aware, I have added the "auto" channel so it should be able to install it. But I guess I must be missing something. Any help much appreciated!
You could just try:
conda install -c conda-forge tabula-py
It works fine for me.
Since you are using Windows and in the link you provide I just see Linux-64 and Linux-32 I think that installing Tabula with Conda can return errors.
Activate your Conda environment and install Tabula using pip:
pip3 install tabula-py
Note
As pointed out in a comment by chezou the conda-forge way of installing Tabula seems not the best way to go if you want to keep it updated:
Conda package is supported by someone else and it seems not maintained
well.
As of Feb 24th, 2019, conda version is v1.1.1 while the latest
pypi package is 1.3.1.
I would recommend installing via pip.
Conda sources are limited to packages available in the channels you have set up. You will need to either:
Set up a channel in Conda that contains tabula.(I tried this with other pacages but couldn't figure out a working method.)
https://conda.io/docs/user-guide/tasks/manage-environments.html
Install tabula into your Anaconda environment from source.
https://docs.python.org/2/install/
use pip to install tabula in Conda environment.(you will need to install pip first)
https://github.com/ContinuumIO/anaconda-issues/issues/1429
Then:
If tabula is in pipy, this will probably work.
https://anaconda.org/conda-forge/tabula-py
To install this package with conda run one of the following:
> conda install -c conda-forge tabula-py conda install -c
> conda-forge/label/cf201901 tabula-py