I have an environment.yml file, which contains the necessary packages required to build a Docker container. Though, one of the packages I am using is installed locally (--offline flag is passed).
Shell file, which installs necessary packages looks like this right now:
conda env update -n base -f ./environment.yml
conda install --offline <LOCAL_PACKAGE.tar.bz2>
Is it possible to somehow add this package inside my environment.yml file?
Related
I want to export an environment AND its packages, using conda 4.10.
Reading the conda docs, it suggests exporting environments using conda env export > environment.yml. However, I am not sure if it is my problem (and if so, what the solution is), but there is no package information.
name: guest
channels:
- defaults
prefix: C:\Anaconda3\envs\guest
After some googling, I learnt to export packages using conda list --export > requirements.txt. This time, there is no environment information.
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: win-64
ca-certificates=2021.5.30=h5b45459_0
certifi=2021.5.30=py38haa244fe_0
...
How do I export both into one file and use it? Or should I just export two files, create an environment first, and install the packages?
On a side note, how do I make my packages match requirements.txt, that is, to remove extra packages, install missing ones, and update/ downgrade to the specific version? Is there a command for this, or should I delete the whole environment and start from scratch?
How can I replicate the python environment setup of a windows machine onto another computer and be able to run very specific scripts successfully.
We have scripts that were written and run in python 3.6.5 in the anaconda environment, we want to be able to run these scripts on a new Windows 10 computer.
The scripts also connect to a local database on the computer (Postgres).
Since you are using anaconda environment, i assume that you have been using the virtualenv for the project you mentioned. It is actually easy to replicate with the following codes:
# list all virtualenvs in your anaconda folder
$ conda info –envs # this will list all virtualenvs created by you, you can then choose the specific virtualenv here.
# to activate the virtualenv of your interest
$ conda activate [virtualenv_name]
# export all packages used in the specific virtualenv (conda activated)
$ pip freeze > requirements.txt # save the output file as requirements.txt
# set up a new conda virtualenv in current or separate machine and install with the requirements.txt
$ conda create --name <env_name> python=3.6.5 --file requirements.txt
# Please note that occasionally you may need to check requirements.txt if there is any abnormal list of packages. The format should be in either [package==version] or [package].
OR you can create the entire virtualenv directly.
# copy exactly same virtualenv on separate machine
# export all packages used in the specific virtualenv (conda activated), including current python version and virtualenv name
$ conda env export > environment.yml # save the output file as environment.yml
# set up a new conda virtualenv in current or separate machine and install with the requirements.txt
$ conda env create -f environment.yml # using Conda; to modify “name” in the environment.yml file if to set up own same anaconda/machine
I have been using Anaconda(4.3.23) on my GuestOS ubuntu 14.04 which is installed on Vmware on HostOS windows 8.1. I have setup an environment in anaconda and have installed many libraries, some of which were very hectic to install (not straight forward pip installs). few libraries had inner dependencies and had to be build together and from their git source.
Problem
I am going to use Cloud based VM (Azure GPU instance) to use GPU. but I don't want to get into the hectic installation again as i don't want to waste money on the time it will take me to install all the packages and libraries again
Is there any way to transfer/copy my existing env (which has everything already installed) to the Cloud VM?
From the very end of this documentation page:
Save packages for future use:
conda list --export > package-list.txt
Reinstall packages from an export file:
conda create -n myenv --file package-list.txt
If conda list --export failes like this ...
Executing conda list --export > package-list.txt creates a file which looks like this:
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: win-64
_tflow_1100_select=0.0.1=gpu
absl-py=0.5.0=py_0
astor=0.7.1=py_0
...
But creating a new environment by executing conda create -n myenv --file package-list.txt gives me this error:
Solving environment: ...working... failed
PackagesNotFoundError: The following packages are not available from current channels:
- markdown==2.6.11=py_0
...
... then try to use conda env export
According to this discussion execute the following command on your source machine:
source activate yourEnvironment
conda env export --no-builds > file.txt
On the target machine execute:
conda env create --file /path/to/file.txt
The file generated by conda env export looks a bit different, but it contains pip packages as well:
name: yourEnvironment
channels:
- conda-forge
- defaults
dependencies:
- absl-py=0.5.0
...
- pip:
- astroid==2.0.4
...
## You can try below approach to move all the package from one machine to other :
## Note : Machine that packages are being moved should be same and python version also should be same
$ pip install conda-pack
# To package an environment:
## Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env
## Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz
## Pack environment located at an explicit path into my_env.tar.gz
$ conda pack -p /explicit/path/to/my_env
# After following above approach, you will end up with a tar.gz file. Now to install package from this zip file follow below approach.
## To install the environment:
## Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env
## Use Python without activating or fixing the prefixes. Most Python
## libraries will work fine, but things that require prefix cleanups
## will fail.
$ ./my_env/bin/python
## Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate
## Run Python from in the environment
(my_env) $ python
## Cleanup prefixes from in the active environment.
## Note that this command can also be run without activating the environment
## as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack
You can probably get away with copying the whole Anaconda installation to your cloud instance.
According to github thread execute the following command on your source machine:
https://github.com/conda/conda/issues/3847
source activate yourEnvironment
conda env export --no-builds > environment.yml
On the target machine execute:
conda env create -f environment.yml
The file generated by conda env export looks a bit different, but it contains pip packages as well:
name: yourEnvironment
channels:
conda-forge
defaults
dependencies:
absl-py=0.5.0
...
pip:
astroid==2.0.4
...
I found the answer from this
you can export your Anaconda environment using:
conda env export > environment.yml
In order to recreate it on another machine using:
conda env create -f environment.yml
You can modify the environment.yml as required because some of the python libraries may be obsolete or due to version conflict in future releases.
The conda docs at http://conda.pydata.org/docs/using/envs.html explain how to share environments with other people.
However, the docs tell us this is not cross platform:
NOTE: These explicit spec files are not usually cross platform, and
therefore have a comment at the top such as # platform: osx-64 showing the
platform where they were created. This platform is the one where this spec
file is known to work. On other platforms, the packages specified might not
be available or dependencies might be missing for some of the key packages
already in the spec.
NOTE: Conda does not check architecture or dependencies when installing
from an explicit specification file. To ensure the packages work correctly,
be sure that the file was created from a working environment and that it is
used on the same architecture, operating system and platform, such as linux-
64 or osx-64.
Is there a good method to share and recreate a conda environment in one platform (e.g. CentOS) in another platform (e.g. Windows)?
This answer is given with the assumption that you would like to make sure that
the same versions of the packages that you generally care about are on
different platforms and that you don't care about the exact same versions of
all packages in the entire dependency tree. If you are trying to install the
exact same version of all packages in your entire dependency tree that has a
high likelihood of failure since some conda packages have different
dependencies for osx/win/linux. For example, the recipe for
otrobopt
will install different packages on Win vs. osx/linux, so the environment list
would be different.
Recommendation: manually create an environment.yaml file and specify or pin
only the dependencies that you care about. Let the conda solver do the rest.
Probably worth noting is that conda-env (the tool that you use to manage conda
environments) explicitly recommends that you "Always create your
environment.yml file by hand."
Then you would just do conda env create --file environment.yml
Have a look at the readme for
conda-env.
They can be quite simple:
name: basic_analysis
dependencies:
- numpy
- pandas
Or more complex where you pin dependencies and specify anaconda.org channels to
install from:
name: stats-web
channels:
- javascript
dependencies:
- python=3.4 # or 2.7 if you are feeling nostalgic
- bokeh=0.9.2
- numpy=1.9
- nodejs=0.10
- flask
- pip:
- Flask-Testing
In either case, you can create an environment with conda env create --file environment.yaml.
NOTE: You may need to use .* as a version suffix if you're using an older version of conda.
Whilst it is possible to create your environment.yml file by hand, you can ensure that your environment works across platforms by using the conda env export --from-history flag.
This will only include packages that you’ve explicitly asked for, as opposed to including every package in your environment.
For example, if you create an environment and install a package conda install python=3.8 numpy, it will install numerous other dependencies as well as python and numpy.
If you then run conda env export > environment.yml, your environment.yml file will include all the additional dependencies conda automatically installed for you.
On the other hand, running conda env export --from-history will just create environment.yml with python=3.8 and numpy and thus will work across platforms.
Answer adapted from the docs.
For those interested in a solution to maintain a single environment file that can be used in Linux, macOS, and Windows, please check the conda-devenv tool at https://github.com/ESSS/conda-devenv.
conda-env export should be used used to export your complete environment to file named my_env.yml.
Check the working solution on getting only prefix on OS X instead of complete dependency including pip.
Step 1: deactivate from the environment if activated. else it will create yml file with only prefix.
Step 2: run below command to export
conda-env export -n my_env > my_env.yml
it will export every required dependency, channel and pip install in a yml file which is importable to share with others.
Step 3: run below command to import
conda-env create -n my_env -f= my_env.yml
it will create the exact environment as is on sharing fellow machine.
An aspect missing from the other answers is that the question asker mentions "spec files" and not "environment.yml" file. These are different.
Spec file
A spec file specifies the exact package URL's and is used to recreate identical environments (on the same platform).
It looks like this:
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-64
#EXPLICIT
https://repo.anaconda.com/pkgs/free/osx-64/mkl-11.3.3-0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/numpy-1.11.1-py35_0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/openssl-1.0.2h-1.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/pip-8.1.2-py35_0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/python-3.5.2-0.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/readline-6.2-2.tar.bz2
https://repo.anaconda.com/pkgs/free/osx-64/setuptools-25.1.6-py35_0.tar.bz2
It can be obtained with conda list --explicit from the conda environment of interest.
To create a new environment with it one would use the conda create command:
conda create --name <env_name> --file <spec file path>
environment.yml
The environment.yml file is described well in this answer.
It can be obtained with the following commands from the conda environment of interest:
conda env export to get all packages in the current environment
conda env export --from-history to only get the packages explicitly installed (i.e. not automatically added depenedencies)
This question is quite old and conda has developed in the meantime. Perhaps the original meaning of spec file was equal to environment.yml files, but for completeness I am adding this answer.
None of these solutions worked out for me, the problem the OP has raised is about platform dependent suffixes added in the dependency making it impossible to be used it in a cross-platform way.
Turns out the solution for this is to export the environment with an additional option called no-builds
Suppose you want to export your environment from MacOS to Debian.
1.) Invoke conda env export --no-builds > env_macos.yml
2.) Invoke cp env_macos.yml env_debian.yml
3.) Move env_debian.yml to your Debian host
4.) conda env create -f env_debian.yml
Soon after you do 4, again there may be same package resolving related issues for certain packages, just remove those entries alone and invoke 4 again. Things will work.
Reference
I can create a new conda environment, with program biopython with this:
conda create --name snowflakes biopython
What if I do not want to install any program? It seems I can not do that:
» conda create --name tryout
Error: too few arguments, must supply command line package specs or --file
You can specify one or more default packages to install when creating
an environment. Doing so allows you to call conda create without
explicitly providing any package names.
To set the provided packages, call conda config like this:
conda config --add create_default_packages PACKAGE_NAME
You can give a package name of just "python" to get a base, empty install.
conda create --name myenv python
conda create --name myenv python=3.4
If you've created a create_default_packages block in your .condarc file, #joelion's answer will install those packages. If you don't want those, use the --no-default-packages flag. For example:
conda create --name myenv python --no-default-packages
This is how to create a truly empty (light) conda_env with 0 packages:
conda create --name myenv --no-default-packages
it will take a few seconds to create and finish.
To create an environment that is absolutely empty, without python and/or any other default package, just make a new folder in envs directory in your Anaconda installation (Anaconda3 in this example):.
~\Anaconda3\envs>mkdir empy_env
The first time that you activate this environment a directory named Scripts in Windows, bin in Linux, with a few batch files are created. At the time of this post this works for Anaconda version 4.3.30 both in Windows and Linux.
I have noticed that #cel has suggested the same thing in the first comment under the question, but obviously it hasn't got the attention it deserves!
For Conda 2020.11 Linux, the following command will create an empty environment.
conda create --name your-env-name