We Keep Coding

Python in R - Error: could not find a Python environment for /usr/bin/python

I don't understand how R handles the Python environment and Python version and keep getting the error Error: could not find a Python environment for /usr/bin/python. I installed Miniconda and created a conda environment in the shell:
conda activate r-reticulate
Then, in R, I try to install keras (same problem with package tensorflow):
library(keras)
reticulate::use_condaenv()
install_keras(method = "conda", conda = reticulate::conda_binary())
... and get the following error:
Error: could not find a Python environment for /usr/bin/python
I tried to figure out what Python R should be using by
reticulate::py_config()
and get
python: /usr/bin/python
libpython: /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib
pythonhome: /System/Library/Frameworks/Python.framework/Versions/2.7:/System/Library/Frameworks/Python.framework/Versions/2.7
version: 2.7.16 (default, Jul 5 2020, 02:24:03) [GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.21) (-macos10.15-objc-
numpy: /Users/bestocke/Library/Python/2.7/lib/python/site-packages/numpy
numpy_version: 1.16.6
tensorflow: [NOT FOUND]
python versions found:
/usr/bin/python3
/usr/local/bin/python3
/usr/bin/python
I don't understand this. This seems to be using Python 2.7. When trying to figure out which Python is being used in the shell, I get:
> which python
/opt/miniconda3/envs/r-reticulate/bin/python
and
> ls -l /opt/miniconda3/envs/r-reticulate/bin/python
lrwxr-xr-x 1 username wheel 9 Aug 2 15:21 /opt/miniconda3/envs/r-reticulate/bin/python -> python3.6
Suggesting Python 3.6 should be used.
What am I getting wrong here?

Try to follow the guide at https://tensorflow.rstudio.com/installation/:
In your R-studio console :
install.packages("tensorflow")
library(tensorflow)
install_tensorflow()
If you have not installed Anaconda / Miniconda manually, then at step no. 3, a prompt will ask your permission to install Miniconda. If you already have conda installed, then :
Create new environment r-reticulate in conda : conda create -n r-reticulate
Install tensorflow from R-studio console with parameters : install_tensorflow(method = 'conda', envname = 'r-reticulate')
Load the reticulate package library(reticulate)
Activate the conda environment in R-studio use_condaenv('r-reticulate')
Load the tensorflow libray library(tensorflow)
Check if tensorflow is active tf$constant("Hellow Tensorflow")
References :
https://tensorflow.rstudio.com/installation/
https://rstudio.github.io/reticulate/

I installed using virtualenv, and I found that I have to specify the full path to the env by envname. It does not work by method="virtualenv", envname="r-reticulate"

I hope there is time to add information. I tried to do what Anugerah Erlaut said, but trying to install Keras GPU on R-studio Server, through WSL.
I knew the solution would work because I tried on another computer, but installing on Windows, and Keras CPU. After testing a while (and get frustrated), it seems that Rstudio does not have permissions to change the r-reticulate env.
So, I tried the solution running or "pure R", on bash command line, and it worked just fine!

Emacs gud-pdb with miniconda python environments

How does one get emacs to use the same PATH to the python and pdb executables as I get from the term when I have set a specific miniconda environment?
I.e. in emacs, when I run M-x pdb, I would like it to use the same executable as I do if I have done the following from the term:
$ source activate my_py3_env
$ pdb
I know currently this isn't what is happening. I switch to a python 3 env, which modifies my PATH appropriately, but when I run M-x pdb for a python script and print sys.version from within that script I get:
2.7.17 |Anaconda, Inc.| (default, Oct 21 2019, 19:04:46)
So it seems to be picking up the 'base' miniconda env which is still 2.7
I would half of expected it to pick up whats in /usr/bin but that doesn't seem to be the case, i.e. if I execute:
$ /usr/bin/python
I get
Python 2.7.15+ (default, Oct 7 2019, 17:39:04)
To summarise, is there a way to get emacs M-x pdb to 'follow' the conda environment I am currently in without me having to manually specify the location of the pdb executable for each environment?

You can find the conda or virtualenv python path with (swap in the analogous conda equivalent for starting the env):
source .py2james/bin/activate
and
which python in the terminal. Note the path to the python exe.
you can do: option+x pdb
then: /Users/janderson/code/python/awsomeapp/.py2james/bin/python -m pdb main.py and the emacs pdb debugger will start.

Tox not finding python3.6 even with the shim present. What is wrong with my pyenv setup?

Trying to breathe some life back into a django package that has fallen into a state of disrepair. They use tox for testing so I've setup pyenv on my MacBook. I've installed 3 versions of python as you can see below, and everything looks like it should work, but if it was I wouldn't be asking why it is not.
I've replaced my home directory with ~ to make it a bit easier to read.
pyenv was installed with brew install pyenv and the various versions of python were installed with pyenv install #.#.#
The shims exist:
$ echo $PATH
~/.pyenv/shims:~/.platformsh/bin:/usr/local/sbin:...
$ which python3.6
~/.pyenv/shims/python3.6
$ which python3.4
~/.pyenv/shims/python3.4
$ which python3.5
~/.pyenv/shims/python3.5
But executing them does not work as expected:
$ pyenv local 3.4.9 3.5.6 3.6.8
$ python3.4
Python 3.4.9 (default, Feb 12 2019, 10:33:47)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ python3.5
pyenv: python3.5: command not found
The `python3.5' command exists in these Python versions:
3.5.6
$ python3.6
pyenv: python3.6: command not found
The `python3.6' command exists in these Python versions:
3.6.8
And tox fails like this:
py34-1.11: commands succeeded
ERROR: py36-1.11: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.6: pyenv: python3.6: command not found\n\nThe `python3.6' command exists in these Python versions:\n 3.6.8\n\n", None)
ERROR: py36-2.0: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.6: pyenv: python3.6: command not found\n\nThe `python3.6' command exists in these Python versions:\n 3.6.8\n\n", None)
py36-latest: commands succeeded
docs: commands succeeded
But in the .tox folder you will find these VirtualEnvs that can be activated manually.
$ ls .tox
dist docs flake8 log py34-1.11 py36-1.11 py36-2.0 py36-latest
Because at some point it was working....
I do understand the mechanics of why it isn't working, what I don't understand is why pyenv is not setting up the environment correctly (Or maybe this is exactly how it is supposed to behave). Everything I read seems to indicate that python3.6 should launch a python3.6.8 interpreter
$ bash -x python3.6
+ set -e
+ '[' -n '' ']'
+ program=python3.6
+ [[ python3.6 = \p\y\t\h\o\n* ]]
+ export PYENV_ROOT=~/.pyenv
+ PYENV_ROOT=~/.pyenv
+ exec /usr/local/Cellar/pyenv/1.2.9/libexec/pyenv exec python3.6
pyenv: python3.6: command not found
The `python3.6' command exists in these Python versions:
3.6.8

pyenv by default picks the python "locally", that is it looks for the PYTHON_VERSION environment variable or a .python-version file.
Personally I find this setup a little cumbersome (needing to have these files littered around all projects, especially in projects which need multiple versions). Fortunately, you can make these "shims" function anywhere with a default version of python by using pyenv global #.#.#
In your case, to make the python3.6 shim execute 3.6.8 without needing to set up the .python-version files, you'd run pyenv global 3.6.8 -- you can run this multiple times for different python versions as well: pyenv global 3.6.8 3.5.6 ...
The reason that you're likely not having these resolve inside tox is tox clears the environment when executing, so the PYTHON_VERSION environment variable will not carry through. You can turn that off by setting passenv= in your tox.ini. For example:
[testenv]
passenv = PYTHON_VERSION

How to install libraries to python in zeppelin-spark2 in HDP

I am using HDP Version: 2.6.4
Can you provide a step by step instructions on how to install libraries to the following python directory under spark2 ?
The sc.version (spark version) returns
res0: String = 2.2.0.2.6.4.0-91
The spark2 interpreter name and value is as following
zeppelin.pyspark.python: /usr/local/Python-3.4.8/bin/python3.4
The python version and current libraries are
%spark2.pyspark
import pip
import sys
sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])
print("--")
print (sys.version)
print("--")
print(installed_packages_list)
--
3.4.8 (default, May 30 2018, 11:05:04)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)]
--
['pip==9.0.1', 'setuptools==28.8.0']
Update 1: using pip install [package name] actually leads to two problems
1) The HDP is pointing at python2.6 rather than python3.4.8
2) pip3 is not there for some reason
Therefore, I am thinking of installing miniconda and pointing Zeppelin there and installing all the packages in conda to prevent conflict between python 2.6 and 3.4.8

This was painful for us. The workaround that works is:
Install the python packages you need from the terminal using pip or pip3 accordingly.
By default the zeppelin.pyspark.python on the spark interpreter is set to: python. This python did not recognize the packages we had installed using the terminal. We had to update zeppelin.pyspark.python : /usr/bin/python (the path to the python command, you can get it using the command 'which python')
Now the interpreter and zeppelin notebooks were able to access all the packages we installed from the terminal.

You need to open your terminal and type pip and press the TAB key. The pip versions available on your sandbox shall be listed. Use pip3 to install the packages you require. The way to do so remains the same pip3 install "packageName". This would make the package available with the Python3 installation you wish to use in Zeppelin.

cannot locate pip.pm for installing python modules

I'm having trouble installing the regex module for python, and would appreciate any suggestion and help. When I used "pip install" on a Windows system, the following error shows up:
Can't locate pip.pm in #INC <#INC contains: C:/Perl64/site/lib C:/Perl64/lib .) at C:\pathA\pathB\perl\bin/pip line 5.
BEGIN failed--compilation aborted at C:\pathA\pathB\perl\bin/pip line 5.
All I did previously is to install pip, following this post: How do I install Python libraries?
I succeeded in the following steps:
python ez_setup.py
python get-pip.py
but not:
pip install setuptools --upgrade
that is when I got the errors.
I am not familiar with using python on Windows, but I need to do it this time. Seems like there is another pip installed for perl on this computer, but when I check the environment variables, I can't really see anything about pip.
For your information -- the python version is:
2.7.7 |Anaconda 2.0.1 (64-bit)| (default, Jun 11 2014, 10:40:02) [MSC v.1500 64 bit (AMD64)]
Thanks a lot in advance for your help!

Apparently Perl also has a tool named pip. Your PATH variable contains the directory for Perl's pip before the one containing Python's pip. You need to find the location of Python's pip and run it directly using an absolute path or edit the PATH variable for Windows so that the directory containing Python's pip is before the directory containing Perl's pip.

The answers is perfect. Just a addon: you can do that by
Right click on the start button
Go to system> Advanced system settings> Environment Variables
In system variables select path and then click on edit
Reorder paths so that C:\Python27 entry comes before the perl\bin
entry
Save your changes by clicking ok