I have created an environment using conda and set up tensorflow along with a few other packages. How do I clone this environment such that it is a direct replica of the environment itself?
I plan to clone the environment (call it base) twice: Once for testing packages and again for production. In total I would have the base environment, the test version, and production version. The goal being I can test out packages in "test" and if they work, add them to production. If they really do work as expected, add them to "base". With that being said, I am not savy with the command line and get tripped up easily with naming conventions. Any explicit answers would be very helpful.
Hi Jack i think the best way would be to use this command:
conda create --name cloned_env --clone original_env
cloned_env --> is the new conda environment
original_env --> is the new conda environment we are cloning
Another smart way to do this would be by creating a requirements.txt that contains all the packages with their version from the environment you want to clone.
And then use that file to install the packages in the new enviroment.
Related
Is a cloned Conda environment similar to a Python Virtual Environment?
conda create --clone arcgispro-py3 --name arcgispro-py3_clone
Or are there any benefits to create a Visual Environment for this cloned environment?
I think I understand what you're asking. "virtual environment" when it comes to python usually refers to python environments created using virtualenv specifically. You could consider conda environments "virtual environments" as well, but that just gets confusing to refer to them that way and people don't do that. Say "conda environment".
Conda is its own environment and package manager, it can be used in place of virtualenv in a lot of cases. As with most things with overlapping (and largely incompatible) use, there are pros and cons to using one over the other. Since I don't know your use case, I don't know which might be right for you.
You shouldn't create a virtual environment from your conda environment. That would probably just cause a mess down the road. Either use a system python to create a virtualenv or use conda to create a conda environment, I don't recommend mixing them.
I have been working with base environment in anaconda till now and had installed all the required python modules/libraries. When I tried to install tensorflow-cpu, I came to know that I have to create another environment to install it from official documentation of anaconda. Now I have created the new environment named 'tf' to install tensorflow-cpu. Since I can activate only one environment at a time, I don't want want to install all the python modules/libraries again to this new tf environment as it will consume space in memory.
Is there any way, I can use all the modules of base environment to tf environment or vice versa?
Please help in this context!
It's definitely not a good idea too mix envs. They were invented to separate dependencies for independent projects. Trying to mix them is a "wrong" way of using envs.
Also, you don't have to create a new env when installing tf as the docs say, just install it where you want it to be, everything will work just fine. The docs recommend you to create one because it's not a good idea to install it into base env.
You have an env named tf now. Activate it and install all dependencies you need, then use it in your project.
Do not pollute your base env with packages, this one is not supposed to be used for development, rather for conda itself. Create a new env when you start a new project
When I create conda environments for my projects where I use pytorch,
torch and torchvision packages take along time to install due to slow connection in my region (takes hours sometimes).
Therefore to start working on my projects quickly I don't create a new env, I just use the packages in the base env. I know this will get hairy soon.
That's why I want to know if there is a way to make a new created env inherit specific packages from base env without re-installing.
ps: I understand that conda leverages hard links but I don't understand how to use this in this case. I appreciate your help.
Cloning
The simplest way to use only already installed packages in a new environment is to clone an existing environment (conda create --clone foo --name bar). Generally, I don't recommend cloning the base environment since it includes Conda and other infrastructure that is only needed in base.
At a workflow-level, it might be advantageous to consider creating some template environments which you can clone for different projects.
YAML Definitions
However, OP mentions only wanting specific packages. I would still create a new env for this, but start with an existing env using an exported YAML.
conda env export -n foo > bar.yaml
Edit the bar.yaml to remove whatever packages that you don't want (again, if foo == base, remove conda), then create the new environment with
conda env create -f bar.yaml --name bar
This will ensure that exactly the packages from the previous environment are used.
Overall, if you using cloning and recreating from YAML files (which include build specifications), then Conda will minimize downloading as well as physical disk usage.
I am trying to create a "clean" Python virtual environment using conda:
conda create -n somename python=3.7 --no-default-packages
But somehow I have access to all the packages installed in base inside this environment. pip list returns all the Python packages, and I can import those packages in Python.
However, the actual environment's "site-packages" folder does not contain those extra Python modules as the base folder does.
So what should I do to create an independent/separate virtual environment? I am using Windows10.
I had PYTHONPATH/HOME explicitly specified in path, after deleting now it works good.
It sounds silly, but make sure that you are actually activating the new environment. Also make sure to check that which python and which pip refer to the new environment, I've had problems before where tmux makes conda activations silently fail.
I would also check your PYTHONPATH variable
echo $PYTHONPATH
just in case you inherit dist-packages (check your ~/.profile and ~/.bashrc)
I would like to create a conda environment on a machine that has no network connection. What I've done so far is:
On a machine that is connected to the internet:
conda create -n python3 python=3.4 anaconda
Conda archived all of the relevant packages into \Anaconda\pkgs. I put these into a separate folder and moved it to the machine with no network connection. The folder has the path PATHTO\Anaconda_py3\win-64
I tried
conda create -n python=3.4 anaconda --offline --channel PATHTO\Anaconda_py3
This gives the error message
Fetching package metadata:
Error: No packages found in current win-64 channels matching: anaconda
You can search for this package on Binstar with
binstar search -t conda anaconda
What am I doing wrong? How do I tell conda to create an environment based on the packages in this directory?
You could try cloning root which is the base env.
conda create -n yourenvname --clone root
Short answer: copy the whole environment from another machine with the same OS.
Why
Dependency. A package depends on other packages. When you install a package online, the package manager conda analyzes the package dependencies and install all the required packages for you.
The dependency is especially heavy in anaconda. Cause anaconda is a meta package depends on another 160+ packages.
Meta packages,are packages do not contain actual softwares and simply depend on other packages to be installed.
It's totally absurd to download all these dependencies one by one and install them on the offline machine.
Detail Solution
Get conda installed on another machine with same OS. Install the packages you need in an isolated virtual environment.
# create a env named "myvenv", name it whatever you want
# and install the package into this env
conda create -n myvenv --copy anaconda
--copy is used to
Install all packages using copies instead of hard- or
soft-linking.
Find where the environments are stored with
conda info
The 1st value of key "envs directories" is the location. Go there and package the whole sub-folder named "myvenv" (the env name in previous step) into an archive.
Copy the archive to your offline machine. Check "envs directories" from conda info. And extract the environment from the archive into the env directory on the offline machine.
Done.
In addition to copying the pkgs folder, you need to index it, so that conda knows how to find the dependencies. See this ticket for more details and this script for an example of indexing the pkgs folder.
Using --unknown as #asmeurer suggests will only work if the package you're trying to install has no dependencies, otherwise you will get a "Could not find some dependencies" error.
Cloning is another option, but this will give you all root packages, which may not be what you want.
A lot of the answers here are not 100% related to the "when offline" part. They talk about the rest of OP's question, not reflected in question title.
If you came here because you need offline env creation on top of an existing Anaconda install you can try:
conda create --offline --name $NAME
You can find the --offline flag documented here
Have you tried without the --offline?
conda create -n anaconda python=3.4 --channel PATHTO\Anaconda_py3
This works for me if I am not connected to the Internet if I do have anaconda already on the machine but in another location. If you are connected to the Internet when you run this command you will probably get an error associated with not finding something on Binstar.
I'm not sure whether this contradicts the other answers or is the same but I followed the instructions in the conda documentation and set up a channel on the local file system.
Then it's a simple matter of moving new package files to the local directory, running conda index on the channel sub-folder (which should have a name like linux-64).
I also set the Anaconda config setting offline to True as described here but not sure if that was essential.
Hope that helps.
The pkgs directory is not a channel. The flag you are looking for is --unknown, which causes conda to include files in the pkgs directory even if they aren't found in one of the channels.
Here's what worked for me in Linux -
(a) Create a blank environment - Just create an empty directory under $CONDA_HOME/envs. Verify with - conda info --envs.
(b) Activate the new env - source activate
(c) Download the appropriate package (*.bz2) from https://anaconda.org/anaconda/repo on a machine with internet connection and move it to the isolated host.
(d) Install using local package - conda install . For example - conda install python-3.6.4-hc3d631a_1.tar.bz2, where python-3.6.4-hc3d631a_1.tar.bz2 exists in the current dir.
That's it. You can verify by the usual means (python -V, conda list -n ). All related packages can be installed in the same manner.
I found the simplest method to be as follows:
Run 'conda create --name name package' with no special switches
Copy the URL of the first package it tried (unsuccessfully) to download
Use the URL on a connected machine to fetch the tar.bz2
Copy the tar.bz2 to the offline machine's /home/user/anaconda3/pkgs
Deploy the tar.bz2 in place
Delete the now unneeded tar.bz2
Repeat until the 'conda create' command succeeds
Here's a solution that may help. It's not very pretty but it gets the job done. So i suppose you have a machine where you have a conda environment in which you've installed all the packages you need. I will refer to this as ENV1 You will have to go to this environment directory and locate it. It is usually found in \Anaconda3\envs. I suggest compressing the folder but you could just use it as is. Copy the desired environment folder into your offline machine's directory for anaconda environments. This first step should get your new environment to respond to commands like conda activate.
You will notice though that software like spyder and jupyter don't work anymore (probably because of path differences). My solution to this was to clone the base environment in the offline machine into a new environment that i will refer to as ENV2. What you need to do then is copy the contents of ENV2 into those of ENV1 and replace files.
This should overwrite the files related to spyder, jupyter.. and keep your imported packages intact.