Trying to use gym open-ai package (and somen other) I ran into some problems,
which structure I don't really understand.
As an example:
I tried to install gym in three different conda environments.
One way to do this is
pip install gym
Another is:
git clone https://github.com/openai/gym.git
cd gym
pip install -e .
A third would be:
pip3 install gym
In some environments I would use Python2, in other env. maybe Python 3.7
Even more possibilities for installation would be:
sudo pip install gym
(and even more permutations would be possible, if we would take into account,
if we activate an environment or don't activate any environment).
To me things get even more complicated, because I tried to install conda with
a not-administrator-user-account in Ubuntu, so that conda (or rather the user itself could not install any files in the /usr directory).
I began to test some of this possibilities and cases, because installation of some libaries
(e.g. keras-rl) seemed to need access to common ressources (/usr/ dir.), even if
installed in an local conda environment. But if so: would the installations in
different conda-environments interact?
And what, if one would install a package as local user in a conda environment and
afterward install a pip or pip3 as administrator. Would the admin-installation
overwrite (or overrule or interact) the environmental installation (or parts of it)?
While experimenting with the different possibilities (or more: while trying to
find a installations, which did not produce any errors like "gym not found" or
"attribute error ... " ) there did occur errors like:
Found existing installation: gym 0.15.4
Can't uninstall 'gym'. No files were found to uninstall.
after executing:
sudo pip3 install gym --force
So on this basis my questions specifically would be:
(1) Is there a best practice for establish good conda environments
(which don't tend to interact, especially if some packages
need sudo priviledges)?
And (2) if some environments interact with
general (sudo) ressources, how can they be resolved in a way,
that distinct environments can be tested and established beneath each other?
Annotation:
there was a similiar question:
conda environment pip is trying to install dependencies globally
some time ago, but the advice, not to use sudo, seems to be difficult to follow,
if some packages require access to global ressources.
So I would like to ask for a solution to interactions at bit more specifically.
you should not use sudo to install something in a conda environment. Most likely the used pip command is not stemming from the actual (activated?) environment, but the actual system-wide pip is used. Therefore you would need to use to use sudo to install to a system owned prefix.
You can check whether you are using the desired pip by invoking "which pip". The path should point to your environment. If it does not, you shall install pip inside your conda env.
I had the same problem before. I activated conda envirement and installed with pip3 locally since conda does not have support for it. Warning: Possible of wreckig some packs.
The conda envirement should ALLWAYS be activated before installing anything orelse it ends up as a global installation.
install a new conda envirement without using sudo. If it ask for sudo you need to remove the whole thing and clean up a bit. Its very easy to forget and NEVER use sudo !
You can try installing a newer version of python3.x (python 2 is getting history very soon anyways they said. Pip = python2, pip3 = python3. And to answer one of your new question if by installing globally will mess things up, not outside conda.
google pycharm and conda. there you can just use it to install 3 different types of envirements with python. Actually a darn good editor for python coding. The rest is more linux related when we talk about cleaning up PATHS etc.
I have no better to add! Hope you get it right.
Related
Right now I'm trying to install python (3.10) and all further installations on my new pc (windows 10) and so far everything is set up:
Python installed
Windows paths for "Python" & "Python\Scrips"
I am able to call the python and pip version and also install some packages. But after installing virtualenv and creating one the - at the moment - unfixable error appears: I am unable to install packages into the pip-path of the virtualenviroment itself. Whenever I'm trying to run any pip-command I'm getting the following error:
Unable to create process using 'C:\Users\ExampleUser\AppData\Local\Programs\Python\Python310\python.exe "C:\folder\env\Scripts\pip.exe" '
As you can see, it's always refering to the original python-path, but on the other hand it's refering to the pip-path of the virtualenv!? Don't know if it's helpful, but when typing in where python and where pip the paths inside the venv are the first one listed. I've also watched out for no blank spaces in my path...
Unfortunately no explanation out there could help me until now and I never faced this problem on my old machine - mostly the same, except some older version of python, pip and virtualenv.
Does anyone else has an idea what I am missing?
downloading Python 3 at the official website and installing it via express installation
Copy & Paste the standalone python into the /python folder and overwriting the python version
running python -m pip install --upgrade pip in cmd
Now pip and python 3 are installed in their latest version.
It's work for me
Could you use venv to create your virtual environment, instead of virtualenv (given that venv is the recommended way to create virtual environments for Python 3.3, and newer)?
If using venv is an option, this procedure may give you some idea on how to do it.
I have not done any Python development on Windows, but I think the basics would be:
python3 -m venv your-env-directory
your-env-directory\Scripts\activate.bat
If using venv is not an option, maybe you can try specifying the -v flag when running your virtualenv command to increase verbosity so you can further troubleshoot what's going on.
try upgrade pip version python -m pip install --upgrade pip
[ Sorry if this answer turns out to be more of a comment than an answer. I only have 21 reputation, so I cannot comment ]
When trying to install pip packages and run python files, is the CWD (Current Working Directory) C:\folder\env\Scripts? If so, try chaning your CWD to C:\folder. I had a similar problem and doing this fixed it.
You may need to look into a cygwin environment, and look into a chroot or jail environment to run the application without conflict.
Have you tried to use virtualenv-wrapper-win module.
It helps me a lot to manage virtual envs
Life is much easier using Anaconda 3 (it's definitely bloated compared to normal Python though), or use the minimal Miniconda (barebone install, basically just Python + a package manager). You can download it here: https://docs.conda.io/en/latest/miniconda.html#windows-installers
Then you can make a new virtual environment super easy:
conda create -n myenv pip
conda activate
If you have multiple environments you do: conda activate [environment_name]
Now you're in your new environment with pip installed. And you get drop down menus in the Windows menu to get to your new environment too, so there isn't any searching required. They just appear. Now if you want to link Jupyter Notebook or Spyder to the installation, it takes more steps since you need more packages. I used this guide which basically activates Jupyter first, then Spyder IDE. https://medium.com/#apremgeorge/using-conda-python-environments-with-spyder-ide-and-jupyter-notebooks-in-windows-4e0a905aaac5
Since you created the environment with pip added you can pip install whatever packages you need. I had to do this recently with OpenBLAS backed NumPy and SciPy (the defaults from pip, not from conda). Now Miniconda is the closest thing to basic Python installation, and comes with some nice tools to make your life easier. Hopefully this is helpful.
Since conda install and pip install in many cases do essentially the same thing, what would be the best option? Is there a case when someone should stick to pip install only? Symmetrical, is there a case when one should stick to conda install only? Is there a way to shoot in one's foot by using both conda and pip install in a single environment?
If both approaches are essentially the same and don't contradict each other there should be no reason to stick solely to one of them but not to the other.
Don't mix conda install and pip install within conda environment. Probably, decide to use conda or virtualenv+piponce and for all. And here is how you decide which one suits you best:
Conda installs various (not only python) conda-adopted packages within conda environment. It gets your environments right if you are into environments.
Pip installs python packages within Python environment (virtualenv is one of them). It gets your python packages installed right.
Safe way to use conda: don't rush for the latest stuff and stick to the available packages and you'll be fine.
Safe way to use pip+virtualenv: if you see a dependency issue or wish to remove and clean up after package - don't. Just burn the house, abandon your old environment and create a new one. One command line and 2-5 minutes later things gonna be nice and tidy again.
Pip is the best tool for installing Python packages among the two of them. Since pip packages normally come out first and only later are adopted for conda (by conda staff or contributors). Chances are, after updating or installing the latest version of Python some of the packages would only be available through pip. And the latest freshest versions of packages would only be available in pip. And mixing pip and conda packages together can be a nightmare (at least if you want to utilize conda's advantages).
Conda is the best when it comes to managing dependencies and replicating environments. When uninstalling a package conda can properly clean up after itself and has better control over conflicting dependency versions. Also, conda can export environment config and, if the planets are right at the moment and the new machine is not too different, replicate that environment somewhere else. Also, conda can have larger control over the environment and can, for example, have a different version of Python installed inside of it (virtualenv - only the Python available in the system). You can always create a conda package when you have no freedom of choosing what to use.
Some relevant facts:
Conda takes more space and time to setup
Conda might be better if you don't have admin rights on the system
Conda will help when you have no system Python
virtualenv+pip will free you up of knowing lots of details like that
Some outdated notions:
Conda used to be better for novice developers back in the day (2012ish). There is no usability gap anymore
Conda was linked to Continuum Analytics too much. Now Conda itself is open source, the packages - not so much.
Depends on the complexity of your environment really.
Using pip for a few simple packages should not generate any issues.
Using more pip installs raises the question "Why not use a pip venv then?".
If you're not doing anything major, you might be able to have a mix of pip and conda installs.
There is an extensive explanation why mixing them can be a bad idea here: Using Pip in a Conda Environment.
Edit:
I'm going to close this question as the reason its happening is different from my original assumption, and it's clearer to ask the question anew:
Pip installs packages in the wrong directory with virtualenv
The accepted answer doesn't directly answer the original question, but is a very useful overview.
Based on discussion below the issue is that even after
$ source ~/PycharmProjects/Practice/venv/bin/activate
$ pip install numpy
numpy is installed in /usr/local/lib/python2.7/site-packages
What could the reason for this be?
Original:
Using Python on OS X via Homebrew:
I've been trying much of the day to sort this out but either I get a must supply either home or prefix/exec-prefix -- not both error, or the package I try to install goes into totally the wrong place:
$ pip3 --version
pip 18.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
$ cd venv
$ pip3 install numpy
..... [snip with following error:]
"must supply either home or prefix/exec-prefix -- not both")
Using this hint
$ pip3 install numpy -t .
Then I get a new error,
`Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/.../pip-install-0fvveq3v/package/'
Searching around SO gives various possibilities involving pip install setuptools. but pip install throws the above error or installs in the wrong place. i.e. the solution involves something that's causing the error in the first place.
I tried to use the Python.org installer but it didn't install pip at all. (The custom installer showed the option checked but with size zero).
An introductory overview is available in this nice tutorial. Here is a good summary with more detail. But, if you renamed or moved the virtual env dir after its creation, it could break it. Create a new one from scratch: $ cd ~/PycharmProjects; python3 -mvenv newenv ; Activate: $ source newenv/bin/activate ; Install something: $ pip install colorama (same as pip3 install only if venv activated); Check: ls ~/PycharmProjects/newenv/lib/python3*/site-packages ; Deactivate: $ deactivate
Then you could try this solution for Pycharm: how to associate a virtual environment with a python project in pycharm. PyCharm indeed comes bundled with virtualenv which could have been customized, please look into Pycharm-specific resources: creating virtual environments and installing packages in Pycharm.
If you have installed PyPI's mainstream virtualenv, by default it will create new environments with the python interpreter that virtualenv was installed with. But it's possible to specify an alternate Python Interpreter upon a new env creation: $ virtualenv -p python3.7 newenvname
Regarding the error DistutilsOptionError: must supply either home or prefix - please check this and this for solutions. Homebrew'ed mappings between python and pip are described here. The normal pip install --user is disabled in homebrewed Python, but there are workarounds. MacOS system Python doesn't provide pip, but it can installed, reinstalled or upgraded for any specific python version manually. Original non-brewed installers are also available for all Python versions: https://www.python.org/downloads/mac-osx/
By default there's no pip.conf, but it can be created by hand to customize things. All possible pip.conf locations (per-user, per-venv, and global/system-wide, and how they override each other) are listed here. If someone faces an issue, they could use pip config list command to see their active configuration, or locate pip.conf and find it.
Finally, you may want to ensure you aren't using pip against macOS' system python. Shell commands such as $ brew info python, which pip, which pip3, pip3 -V, which python3 can help you see what you actually use. Since the macOS default $PATH used to be /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin, stock macOS binaries (including python) may take precedence over some homebrew'ed installations (including python). If so, a custom PATH could be exported via the ~/.bashrc if required.
Whenever I want to install a python package, I find the pip install <package> instructions on most of the sites / README.md documentation in github etc.
A colleague told me recently to try first a conda install <package> and if this fails (because the package is not available) to use the pip install process afterwards.
Is the try with the conda installation step really necessary / beneficial or can I just do the pip install directly?
It depends on your use case. Conda does more than pip does. Conda was developed after pip because the Conda folks did not think pip did enough. It aims to handle library dependencies outside of the realm of python such as C libraries, R packages, or really anything with a wheel. as well as handling the python packages themselves. This is important because these packages do not have the standard setup.py in their source code, so python will not install them into the site-packages directory which is useful for easy importing.
It is important to note that you can't use pip and conda interchangeably, as conda has a different packaging format.
To answer your question succinctly: If you use one, I would stick to it for the entirety of whatever you are doing, and not use conda "until it doesn't work for something" and then just switch to pip for installations that conda can't handle. That is a super good way to get into trouble you can't explain.
My advice: if you are sticking to python and python only, use pip. If you are considering outside libraries that have value to your project, conda is a good option.
*I hope this isn't a repeat. I've tried looking for clarity, but I'm having trouble.
I'm fairly new to Python and used Homebrew to avoid using the system Python on Mac. I guess I also installed anaconda at some point. (probably following some tutorial)
Now when I do which for the following, I get different bins:
which python: /Users/ryangoree/anaconda3/bin/python
which python2: /usr/local/bin/python2
which python3: /Users/ryangoree/anaconda3/bin/python3
which pip: /Users/ryangoree/anaconda3/bin/pip
So my issue is that when I'm using python2, there are modules that that I can't use. If I try to pip install them, It just tells me they're already installed since they are in the anaconda directory.
I don't know what I don't know right now, but I'm sure there is a better way to handle this. Can someone enlighten me or send me on the right path to developing with Python and managing the packages.
Thank you!
This stumped me for a while until I figured out one pain when using anaconda: just because you’re in a conda environment doesn’t mean that pip belongs to that environment. Instead you must run conda install pip for pip to be associated with that environment. Then every pip install will be tied to that environment.
You can check your PYTHONPATH to see the order in which the various python installations are interrogated.
Better still you should create each conda environment with its own python using:
conda create -n <envname> python=2.7 # python 2.7
conda create -n <envname> python=3.6 # python 3.6
which will automatically include pip for that environment.