When I create conda environments for my projects where I use pytorch,
torch and torchvision packages take along time to install due to slow connection in my region (takes hours sometimes).
Therefore to start working on my projects quickly I don't create a new env, I just use the packages in the base env. I know this will get hairy soon.
That's why I want to know if there is a way to make a new created env inherit specific packages from base env without re-installing.
ps: I understand that conda leverages hard links but I don't understand how to use this in this case. I appreciate your help.
Cloning
The simplest way to use only already installed packages in a new environment is to clone an existing environment (conda create --clone foo --name bar). Generally, I don't recommend cloning the base environment since it includes Conda and other infrastructure that is only needed in base.
At a workflow-level, it might be advantageous to consider creating some template environments which you can clone for different projects.
YAML Definitions
However, OP mentions only wanting specific packages. I would still create a new env for this, but start with an existing env using an exported YAML.
conda env export -n foo > bar.yaml
Edit the bar.yaml to remove whatever packages that you don't want (again, if foo == base, remove conda), then create the new environment with
conda env create -f bar.yaml --name bar
This will ensure that exactly the packages from the previous environment are used.
Overall, if you using cloning and recreating from YAML files (which include build specifications), then Conda will minimize downloading as well as physical disk usage.
Related
I have to connect to a server where my user has access to one small partition from /home/users/user_name where I have a quota of limited space and a bigger partition into /big_partition/users/user
After I am logging into that server I will arrive at /home/users/user_name at the bigging. After that, I am doing the following steps.
cd /big_partition/users/user
create conda --prefix=envs python=3.6
on the 4th line, it says Package plan for installation in environment /big_partition/users/user/envs: which is ok.
press y, and not I am getting the following message.
OSError: [Errno 122] Disk quota exceeded: '/home/users/user_name/.conda/envs/.pkgs/python-3.6.2-0/lib/python3.6/unittest/result.py'
Can anyone help me to understand how can I move the .conda folder from /home/users/user_name to /big_partition/users/user at the moment when I am creating this environment?
Configure Environment and Package Default Locations
I'd guess that, despite your efforts to put your environments on the large partition, there is still a default user-level package cache and that is filling up the home partition. At minimum, set up a new package cache and a default environments directory on the large partition:
# create a new pkgs_dirs (wherever, doesn't have to be hidden)
mkdir -p /big_partition/users/user/.conda/pkgs
# add it to Conda as your default
conda config --add pkgs_dirs /big_partition/users/user/.conda/pkgs
# create a new envs_dirs (again wherever)
mkdir -p /big_partition/users/user/.conda/envs
# add it to Conda as your default
conda config --add envs_dirs /big_partition/users/user/.conda/envs
Now you don't have to fuss around with using the --prefix flag any more - your named environments (conda create -n foo) will by default be created inside this directory and you can activate by name instead of directory (conda activate foo).
Transferring Previous Environments and Package Cache
Unfortunately, there's not a great way to move Conda environments across filesystems without destroying the hardlinks. Instead, you'll need to recreate your environments. Since you may or may not want to bother with this, I'm only going to outline it. I can elaborate if needed.
Archive environments. Use conda env export -n foo > foo.yaml (One per environment.)
Move package cache. Copy contents of old package cache (/home/users/user_name/.conda/envs/.pkgs/) to new package cache.
Recreate environments. Use conda env create -n foo -f foo.yaml.
Again, you could just skip this altogether. This is mainly if you want to be very thorough about transferring and not having to redownload stuff for environments you already created.
After this you can delete some the stuff under the old ~/.conda/envs/pkgs folder.
I found the solution. All I need to do is to export CONDA_ENVS_PATH with the path where I want to be the .conda
export CONDA_ENVS_PATH=.
Is a cloned Conda environment similar to a Python Virtual Environment?
conda create --clone arcgispro-py3 --name arcgispro-py3_clone
Or are there any benefits to create a Visual Environment for this cloned environment?
I think I understand what you're asking. "virtual environment" when it comes to python usually refers to python environments created using virtualenv specifically. You could consider conda environments "virtual environments" as well, but that just gets confusing to refer to them that way and people don't do that. Say "conda environment".
Conda is its own environment and package manager, it can be used in place of virtualenv in a lot of cases. As with most things with overlapping (and largely incompatible) use, there are pros and cons to using one over the other. Since I don't know your use case, I don't know which might be right for you.
You shouldn't create a virtual environment from your conda environment. That would probably just cause a mess down the road. Either use a system python to create a virtualenv or use conda to create a conda environment, I don't recommend mixing them.
I have created an environment using conda and set up tensorflow along with a few other packages. How do I clone this environment such that it is a direct replica of the environment itself?
I plan to clone the environment (call it base) twice: Once for testing packages and again for production. In total I would have the base environment, the test version, and production version. The goal being I can test out packages in "test" and if they work, add them to production. If they really do work as expected, add them to "base". With that being said, I am not savy with the command line and get tripped up easily with naming conventions. Any explicit answers would be very helpful.
Hi Jack i think the best way would be to use this command:
conda create --name cloned_env --clone original_env
cloned_env --> is the new conda environment
original_env --> is the new conda environment we are cloning
Another smart way to do this would be by creating a requirements.txt that contains all the packages with their version from the environment you want to clone.
And then use that file to install the packages in the new enviroment.
I have been working with base environment in anaconda till now and had installed all the required python modules/libraries. When I tried to install tensorflow-cpu, I came to know that I have to create another environment to install it from official documentation of anaconda. Now I have created the new environment named 'tf' to install tensorflow-cpu. Since I can activate only one environment at a time, I don't want want to install all the python modules/libraries again to this new tf environment as it will consume space in memory.
Is there any way, I can use all the modules of base environment to tf environment or vice versa?
Please help in this context!
It's definitely not a good idea too mix envs. They were invented to separate dependencies for independent projects. Trying to mix them is a "wrong" way of using envs.
Also, you don't have to create a new env when installing tf as the docs say, just install it where you want it to be, everything will work just fine. The docs recommend you to create one because it's not a good idea to install it into base env.
You have an env named tf now. Activate it and install all dependencies you need, then use it in your project.
Do not pollute your base env with packages, this one is not supposed to be used for development, rather for conda itself. Create a new env when you start a new project
The typical command to export a Anaconda environment to a YAML file is:
conda env export --name my_env > myenv.yml
However, one huge issue is the readbility of this file as it includes hard specifications for all of the libraries and all of their dependencies. Is there a way for Anaconda to export a list of the optimally smallest subset of commands that would subsume these dependencies to make the YAML more readable? For example, if all you installed in a conda environment was pip and scipy, is there a way for Anaconda to realize that the file should just read:
name: my_env
channels:
- defaults
dependencies:
- scipy=1.3.1
- pip=19.2.3
That way, the anaconda environment will still have the exact same specification, if not an improved on (if an upstream bug is fixed) and anyone who looks at the yml file will understand what is "required" to run the code, in the sense that if they did want to/couldn't use the conda environment they would know what packages they needed to install?
Options from the Conda CLI
This is sort of what the --from-history flag is for, but not exactly. Instead of including exact build info for each package, it will include only what are called explicit specifications, i.e., the specifications that a user has explicitly requested via the CLI (e.g., conda install scipy=1.3.1). Have a try:
conda env export --from-history --name my_env > myenv.yml
This will only include versions if the user originally included versions during installation. Hence, creating a new environment is very likely not going to use the exact same versions and builds. On the other hand, if the user originally included additional constraints beyond version and build they will also be included (e.g., a channel specification conda install conda-forge::numpy will lead to conda-forge::numpy).
Another option worth noting is the --no-builds flag, which will export every package in the YAML, but leave out the build specifiers. These flags work in a mutually exclusive manner.
conda-minify
If this is not sufficient, then there is an external utility called conda-minify that offers some functionality to export an environment that is minimized based on a dependency tree rather than through the user's explicit specifications.
Have a look at pipreqs. It creates a requirements.txt file only based on the imports that you are explicitely doing inside your project (and you even have a --no-pin option to ignore the version numbers). You can later use this file to create a conda environemnt via conda install --file requirements.txt.
However, if you're aiming for an evironments.yml file you have to create it manually. But that's just copy and paste from the clean requirements.txt. You only have to separate conda from "pip-only" installs.