I have an environment created using miniconda with python 3.6.8, called basepy_3_6_8.
I want to save the environment snapshot to a file and then recreate it later on another machine:
There are different commands to capture the environment snapshot, with slightly different outputs. Which of these can I use to guarantee that the exact environment used by the user is recreated in the target?
I was hoping pip freeze > requirements.txt and pip install -r requirements.txt would work independent of the source environment, but I noticed that pip freeze from within a conda environment does not capture the python version.
Here is the code to create the conda environment, and output of different commands:
$ conda create -n myenv python=3.6.8
$ conda activate myenv
(myenv)$ pip freeze
astroid==2.1.0
autopep8==1.4.3
certifi==2018.11.29
colorama==0.4.1
isort==4.3.4
lazy-object-proxy==1.3.1
mccabe==0.6.1
pycodestyle==2.4.0
pylint==2.2.2
six==1.12.0
typed-ast==1.1.1
wincertstore==0.2
wrapt==1.11.0
(myenv)$ pip list
Package Version
----------------- ----------
astroid 2.1.0
autopep8 1.4.3
certifi 2018.11.29
colorama 0.4.1
isort 4.3.4
lazy-object-proxy 1.3.1
mccabe 0.6.1
pip 18.1
pycodestyle 2.4.0
pylint 2.2.2
setuptools 40.6.3
six 1.12.0
typed-ast 1.1.1
wheel 0.32.3
wincertstore 0.2
wrapt 1.11.0
(myenv)$ conda list
# packages in environment at C:\Users\alias\AppData\Local\Continuum\miniconda3\envs\myenv:
#
# Name Version Build Channel
certifi 2018.11.29 py36_0
pip 18.1 py36_0
python 3.6.8 h9f7ef89_0
setuptools 40.6.3 py36_0
sqlite 3.26.0 he774522_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.15.26706 h3a45250_0
wheel 0.32.3 py36_0
wincertstore 0.2 py36h7fe50ca_0
(myenv)$ conda list --export
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: win-64
certifi=2018.11.29=py36_0
pip=18.1=py36_0
python=3.6.8=h9f7ef89_0
setuptools=40.6.3=py36_0
sqlite=3.26.0=he774522_0
vc=14.1=h0510ff6_4
vs2015_runtime=14.15.26706=h3a45250_0
wheel=0.32.3=py36_0
wincertstore=0.2=py36h7fe50ca_0
I am eventually interested in a general tool that can capture the current environment of a specified type (conda, virtualenv, venv, global python environment) so as to install it uniformly on another machine. What is the best approach for this?
I've never used conda, but I'd try to use two different tools to manage the python version and your project dependencies.
To install a specific python version, I'd use pyenv: https://github.com/pyenv/pyenv.
pyenv also has a plugin to manage virtualenvs (https://github.com/pyenv/pyenv-virtualenv) that should support Anaconda and Miniconda: https://github.com/pyenv/pyenv-virtualenv#anaconda-and-miniconda
To manage your dependencies (packages you install in your virtual env), you have a few alternatives:
Pip freeze: it doesn't automatically guarantee reproducibility though, because it doesn't have a lock file to pinpoint the exact dependency tree
Poetry: https://github.com/sdispater/poetry (supports a lock file)
Pipenv: https://github.com/pypa/pipenv (supports a lock file)
Hope this is helpful.
Related
I do use anaconda on an ubuntu 20.04 and since a week or so I have a problem with processes like
/home/csi/anaconda3/bin/python /home/csi/anaconda3/bin/conda list -f --json spyder
creating a huge load between 40 und 160. The number of processes is about 110.
I have been using 4 environments created by ananconda-navigator all of them are Python 3-Projects, none of them is called spyder. One is called spyder5.
I am using the activation of conda python with the delivered .bashrc-snippet.
:~$ conda list conda
# packages in environment at /home/csi/anaconda3:
#
# Name Version Build Channel
_anaconda_depends 2020.07 py38_0
anaconda custom py38_1
anaconda-client 1.9.0 py38h06a4308_0
anaconda-navigator 2.1.2 py38h06a4308_0
anaconda-project 0.10.2 pyhd3eb1b0_0
conda 4.11.0 py38h06a4308_0
conda-build 3.20.5 py38_1
conda-content-trust 0.1.1 pyhd3eb1b0_0
conda-env 2.6.0 1
conda-pack 0.6.0 pyhd3eb1b0_0
conda-package-handling 1.7.3 py38h27cfd23_1
conda-repo-cli 1.0.4 pyhd3eb1b0_0
conda-token 0.3.0 pyhd3eb1b0_0
conda-verify 3.4.2 py_1
The process starts automatically again after a while. So to work without that huge load I created a bash script trying to kill all those processes every ten seconds.
I did not try to uninstall anaconda3 and recreate all Environments. I will do this when I am upgrading the Ubuntu-Version in May anyway.
But does anyone have had an issue alike? Or does anyone know howto deactivate the periodically start of conda list? This is really annoying.
If you need more information I would be happy to provide them for you!
So after a while an update of conda fixed this issue. However, this was strange behaviour.
I did reinstall spyder via pip3 command of the anaconda installation. After that I updated anaconda and conda via conda command.
:~$ pip3 install --force-reinstall spyder
:~$ conda update anaconda
:~$ conda update conda
This fixed the issue of "conda list" processes but not the issue of the one process creating a huge load. However now the system is usable again.
This question already has answers here:
How can I install packages using pip according to the requirements.txt file from a local directory?
(19 answers)
Closed 6 months ago.
I have to install python packages from requirements files that's provided to me. However, when I use pip install -r requirements.txt command I get an error saying ERROR: Invalid requirement (from line 3 in requirements.txt. And when I comment the third line the error just continues to be there for the next lines. What does that mean and how can I install packages from the file?
Here's how the file contents look like:
# Name Version Build Channel
alabaster 0.7.12 py36_0
altgraph 0.17 pypi_0 pypi
appdirs 1.4.4 py_0
argh 0.26.2 py36_0
astroid 2.4.2 py36_0
async_generator 1.10 py36h28b3542_0
atomicwrites 1.4.0 py_0
attrs 20.3.0 pyhd3eb1b0_0
auto-py-to-exe 2.7.11 pypi_0 pypi
autopep8 1.5.4 py_0
babel 2.9.0 pyhd3eb1b0_0
backcall 0.2.0 py_0
bcrypt 3.2.0 py36he774522_0
black 19.10b0 py_0
bleach 3.2.2 pyhd3eb1b0_0
bottle 0.12.19 pypi_0 pypi
... So on
I am using new environment in Anaconda with python version 3.6.12.
First, freeze all of your pip packages in the requirements.txt file using the command
pip freeze > requirements.txt
This should create the requirements.txt file in the correct format. Then try installing using the command
pip install -r requirements.txt
Make sure you're in the same folder as the file when running this command.
If you get some path name instead of the version number in the requirements.txt file, use this pip command to work around it.
pip list --format=freeze > requirements.txt
Change your requirements.txt content as below and try pip install -r requirements.txt again.
alabaster==0.7.12
altgraph==0.17
appdirs== 1.4.4
argh==0.26.2
astroid== 2.4.2
async_generator==1.10
atomicwrites==1.4.0
attrs==20.3.0
auto-py-to-exe==2.7.11
autopep8==1.5.4
babel==2.9.0
backcall==0.2.0
bcrypt==3.2.0
black==19.10b0
bleach==3.2.2
bottle==0.12.19
If you use Anaconda for environment management you most likely created requirements.txt file via:
conda list --explicit > requirements.txt
To recreate the environment with all your listed packages use:
conda env create --file requirements.txt
See CONDA CHEAT SHEET.
When I try:
% conda update pandas
Collecting package metadata (current_repodata.json): done
Solving environment: |
Updating pandas is constricted by
anaconda -> requires pandas==1.0.5=py38h959d312_0
If you are sure you want an update of your package either try `conda update --all` or install a specific version of the package you want using `conda install <pkg>=<version>`
done
# All requested packages already installed.
This answer on stack overflow said to try:
conda install -c conda-forge pandas==1.1.0
but that just hangs:
% conda install -c conda-forge pandas==1.1.0
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: |
As you can see below, I can install the old 1.0.5 version of pandas using
conda install -c anaconda pandas
so I guess my environment seems to be functioning ok...I just can't install v 1.1.0 of Pandas from conda-forge
% conda install -c anaconda pandas
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /opt/anaconda3
added / updated specs:
- pandas
The following packages will be downloaded:
package | build
---------------------------|-----------------
ca-certificates-2020.6.24 | 0 132 KB anaconda
certifi-2020.6.20 | py38_0 159 KB anaconda
conda-4.8.5 | py38_0 3.1 MB anaconda
openssl-1.1.1g | h1de35cc_0 3.4 MB anaconda
pandas-1.0.5 | py38h959d312_0 9.8 MB anaconda
------------------------------------------------------------
Total: 16.6 MB
The following packages will be SUPERSEDED by a higher-priority channel:
ca-certificates pkgs/main --> anaconda
certifi pkgs/main --> anaconda
conda pkgs/main --> anaconda
openssl pkgs/main --> anaconda
pandas pkgs/main --> anaconda
Proceed ([y]/n)? y
Downloading and Extracting Packages
openssl-1.1.1g | 3.4 MB | ################################################################################################################################################################################################ | 100%
pandas-1.0.5 | 9.8 MB | ################################################################################################################################################################################################ | 100%
certifi-2020.6.20 | 159 KB | ################################################################################################################################################################################################ | 100%
conda-4.8.5 | 3.1 MB | ################################################################################################################################################################################################ | 100%
ca-certificates-2020 | 132 KB | ################################################################################################################################################################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
I tried Anaconda Navigator but when I select "pandas" to be upgraded and click on "Apply" a window appears saying the following packages will be modified, but the window is empty. The Apply button in that window is disabled so I don't think it is doing anything:
I was able to upgrade to pandas 1.1.3 via pip:
% pip install pandas --upgrade
Collecting pandas
Downloading pandas-1.1.3-cp38-cp38-macosx_10_9_x86_64.whl (10.1 MB)
|████████████████████████████████| 10.1 MB 2.1 MB/s
Requirement already satisfied, skipping upgrade: python-dateutil>=2.7.3 in /opt/anaconda3/lib/python3.8/site-packages (from pandas) (2.8.1)
Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /opt/anaconda3/lib/python3.8/site-packages (from pandas) (2020.1)
Requirement already satisfied, skipping upgrade: numpy>=1.15.4 in /opt/anaconda3/lib/python3.8/site-packages (from pandas) (1.18.5)
Requirement already satisfied, skipping upgrade: six>=1.5 in /opt/anaconda3/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)
Installing collected packages: pandas
Attempting uninstall: pandas
Found existing installation: pandas 1.0.5
Uninstalling pandas-1.0.5:
Successfully uninstalled pandas-1.0.5
Successfully installed pandas-1.1.3
I guess this is ok, not sure if my anaconda environment will now have lost its integrity in some way. I guess my question still stands, regarding the way to upgrade via anaconda/conda, or perhaps there is no difference and it is fine to mix anaconda/conda and pip commands. I really don't know.
First, some general points regarding conda-forge and pip:
It is not recommended to install packages from conda-forge to the base environment
It is recommended to avoid mixing conda and pip if possible
This is because chances are high that you will end up with an environment in an inconsistent state without an ability to install or remove packages at all. If this happens to the conda's base environment then the only way is often to reinstall Anaconda completely.
The best practice in general is to use conda virtual environments, especially if you want/have to use conda-forge and pip. Then, if your environment gets into an inconsistent state you can just delete it and start anew.
In you case, it could look something like this:
Create a new conda environment pandas_project with pandas 1.1.0 installed from the official conda channel: conda create -n pandas_project pandas==1.1.0
Switch to the newly created environment conda activate pandas_project
Install as many packages as you can using conda and the official channel.
Setup conda-forge (for this environement only (!), note the --env flag): conda config --env --add channels conda-forge and conda config --env --set channel_priority strict
Install from conda-forge packages that are not available in the official channel
Save the state of the environment with conda list --explicit > pandas_project_env.txt.
Use pip to install packages that are not available neither from the official nor from the conda-forge channel
If the environment ever ends up in an inconsistent state, remove it conda env remove --name pandas_project and recreate it. Packages from the official and the conda-forge channels can be reinstalled quickly conda install --file pandas_project_env.txt
Anaconda comes with a whole bunch of packages pre-installed. As such, they of course have interdependencies that sometimes also restrict that not the newest version of some package can be used. So in your case, you can see that when trying
conda update pandas
it gave you
Updating pandas is constricted by
anaconda -> requires pandas==1.0.5=py38h959d312_0
Basically telling you that the pre-installed anaconda package bundle requires that pandas is at version 1.0.5 to function properly
When you do
conda install -c conda-forge pandas=1.1.0
then conda tries to disentangle all the requirements that led to 1.0.5 being installed previously and tries to find a way to get the version you required working. Since the list of packages that are pre-installed in your base enviroment is long (check conda list), this takes a long time (what you described as hanging) and will probably fail eventually.
I was able to upgrade to pandas 1.1.3 via pip:
This is because essentially, pip will "not care" about all the interdependencies of libraries that where pre-installed with anaconda
not sure if my anaconda environment will now have lost its integrity
As a summary:
conda is convinced that at the current state 1.0.5 is the version needed for all packages to work properly
with pip install you forcefully installed a different version
so yes, in principle you have now a state of inconsisten package dependencies caused by using pip install to upgrade a package that was previously managed by conda, which is something you should never do, see the anaconda website for details.
To avoid complications with pre-installed packages in your base env, you can instead create a new environment and freely install your required version in there:
conda create -n <env-Name> pandas=1.1
conda activate <env-Name>
This creates a virtual environment where only pandas is installed. Then you can conda install other packages as needed
I'm relatively new to Jupyter Notebook and have been struggling with python versions with Jupyter Notebook.
I installed seaborn but import error occurred saying no seaborn package found. It shows on upper right corner of Jupyter "Python 3" but it returned Python 2.7 when I run !python --version. Also when I run print(sys.path), the result is below.
['', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/mysql-0.0.1-py3.5.egg', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python35.zip', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/plat-darwin', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages', '/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/IPython/extensions', '/Users/Cynthia/.ipython']
My guess is that my python kernel isn't pointing correctly to python3 although notebook shows it's python3. Could someone pls help me solve this? It would be helpful if there could be code to run in Jupyter cell. Thank you!
When you run python --version, It won't spit python 3.5.x, because python refers to python2 unless you aliased python as python3. So it makes sense that you see python 2.7.x when you run python --version.
As for py2 when you run conda env list, they are env names you set. They are just names, not python versions.
What needs to be done I think is to find out where your jupyterlab is installed, which I think in (base) environment. In your base environment, run conda list, where you will see a list like this:
(base) ➜ test conda list
# packages in environment at /Users/gwanghyeongim/.pyenv/versions/miniconda3-latest:
#
# Name Version Build Channel
brotlipy 0.7.0 py38haf1e3a3_1000
ca-certificates 2020.6.24 0
certifi 2020.6.20 py38_0
cffi 1.14.1 py38hed5b41f_0
chardet 3.0.4 py38_1003
conda 4.8.4 py38_0
conda-package-handling 1.6.1 py38h1de35cc_0
cryptography 2.9.2 py38ha12b0ac_0
idna 2.10 py_0
libcxx 10.0.0 1
libedit 3.1.20191231 h1de35cc_1
libffi 3.3 hb1e8313_2
ncurses 6.2 h0a44026_1
openssl 1.1.1g h1de35cc_0
pip 20.2.2 py38_0
pycosat 0.6.3 py38h1de35cc_1
pycparser 2.20 py_2
pyopenssl 19.1.0 py_1
pysocks 1.7.1 py38_1
python 3.8.3 h26836e1_1
python.app 2 py38_10
readline 8.0 h1de35cc_0
requests 2.24.0 py_0
#and so on...
See if you see jupyterlab in the list. If so, your jupyter notebook is in (base) environment.
Now the most likely scenario is you installed seaborn in py2 environment. That means you dind't install seaborn in your base environment. Install it by running conda install seaborn or pip install seaborn.
If something didn't work so far, try runnning conda upgrade --all -y to upgrade packages. It might be from collision between deprecated packages.
P.S
My suggestion is you create a separate environment and run packages on it.
Run conda create -n your_env_name to do so(replace your_env_name to the name you want set)
Activate by running conda activate the_env_you_just_created
If 2 doesn't work somehow, make sure you run conda init your_shell, where your_shell can be found by running echo $SHELL, where the last word after / is your shell.
Make sure you see (your_env_name) at the first part of command prompt. If so, your env is activated. Now install packages on here and do your project, rather than on base environment.
Since you mentioned you use conda you can do something like the following. From your terminal:
conda create -n sb python=3
conda activate sb
conda config --env --add channels conda-forge
conda install -y pandas matplotlib numpy scipy seaborn jupyterlab # some default packages
jupyter lab
Whenever you want to use this conda environment again you have to do
conda activate sb
before you can run jupyter lab.
Note, if you didn't changed the default, you should see your terminal prompt changing when activating an environment, i.e. the name of the environment comes before your prompt. In our case here (sb) <prompt>.
To solve the issue with your current conda environment, more information is needed.
Using the latest miniconda2 I created a conda environment & then did all my conda package installs. Then when I finished I realized I had FORGOT TO ACTIVATE the environment (I'm dizzy) with the command
source activate myenv
so all my conda installs went into the local/system wide environment. I simply did
conda uninstall package-name
for all the packages but I'm certain not all the dependencies were removed. For example, conda install of matplotlib showed
The following NEW packages will be INSTALLED:
cycler: 0.10.0-py36_0
dbus: 1.10.20-0
expat: 2.1.0-0
fontconfig: 2.12.1-3
freetype: 2.5.5-2
glib: 2.50.2-1
gst-plugins-base: 1.8.0-0
gstreamer: 1.8.0-0
icu: 54.1-0
jpeg: 9b-0
libffi: 3.2.1-1
libgcc: 5.2.0-0
libiconv: 1.14-0
libpng: 1.6.30-1
libxcb: 1.12-1
libxml2: 2.9.4-0
matplotlib: 2.0.2-np113py36_0
pcre: 8.39-1
pyparsing: 2.2.0-py36_0
pyqt: 5.6.0-py36_2
qt: 5.6.2-5
sip: 4.18-py36_0
which conda shows
/home/cargo/miniconda2/bin/conda
First I created the environment
conda create -n myenv python=3
Then I did all the conda installs
conda install numpy
conda install pandas
.
.
conda install statsmodels
Then I realized I failed to activate the environment b/c the terminal prompt was
cargo#cargo-VirtualBox:~$ INSTEAD OF (myenv)cargo#cargo-VirtualBox:~$
The terminal prompt is what got my attention before I even tried to do anything else; I'm 110% certain I didn't activate the environment as I can't find the command in the bash history. Next I did uninstalls for all the conda packages as stated above. I guess it's good now that I no longer see the python3 packages in the local/root environment but I can't explain/understand why?
conda list
# packages in environment at /home/cargo/miniconda2:
#
asn1crypto 0.22.0 py27_0
beautifulsoup4 4.6.0 <pip>
cairo 1.14.8 0
cffi 1.10.0 py27_0
.
.
wheel 0.29.0 py27_0
yaml 0.1.6 0
zlib 1.2.8 3
cargo#cargo-VirtualBox:~$