We Keep Coding

How to run pipenv python as emacs Python shell

I use python-mode.el and have just discovered Pipenv. I can start Python within a project environment from a terminal by first changing to the project directory and entering "pipenv run python". But python-mode wants to execute a process, not a command string.
I tried creating a shell script like this:
#!/bin/bash
cd ~/myprojectdir
pipenv run python
but got this complaint
Warning: the environment variable LANG is not set!
We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
This variable is set in my terminal environment: how do i ensure it's set for pipenv? But the bigger question is, how do i run this pipenv virtual environment python inside emacs to get a buffer with a Python process?

pipenv.el helps here, by setting the appropriate variables and providing a porcelain around Pipenv inside Emacs.

Looks like i have to explicitly source my .bashrc file (or equivalent). Changing the contents of my runpipenv.sh script to this
#!/bin/bash
source ~/.bashrc
cd ~/myprojectdir
pipenv run python
(where my .bashrc file sets LANG) and then calling
(setq py-shell-name "runpipenv.sh")
in Emacs seems to work.

How does one enter a Python virtualenv when executing a bashscript?

If one defines which version of python to use in a bash script, it would be
export PYTHON = "/path/python/python-3.5.1/bin/python"
But for Python virtualenv's, one executes these commands in the command line
cd /path/pathto/virtualenv
source activate
cd another_directory
How does one "enter" a Python virtualenv in a bash script? What is the standard approach here?

We have to distinguish two cases here:
You want to use/call python (or python-based tools) in your bash script, but python or those tools should be taken from and run in a virtualenv
You want a script that, amongst other things, lets the shell from which you call it enter the virtualenv, so that you can interactively call python (or python-based tools) inside the virtualenv
Case 1: Using a virtualenv inside a script
How does one "enter" a Python virtualenv in a bash script?
Just like on the interactive bash command line:
source /path/to/the/virtual_env/bin/activate
What is the standard approach here?
The standard approach is not to enter the virtualenv in a bash script. Instead, call python and/or the python-based commands you want to use by their full path. To make this easier and less repetitive, you can use aliases and variables.
Case 2: Activating a virtualenv in an interactive bash session by calling a script
There already is such a script. It's called activate and it's located in the bin directory of the virtualenv. You have to source it rather than calling it like a normal command. Only then will it run in the same session instead of in a subshell, and thus only then can it make modifications to the session that won't be lost due to the subshell terminating at the end of the script.
So just do:
source /path/to/the/virtual_env/bin/activate
in your interactive shell session.
But what if you want to do more than the activate script does? You can put
source /path/to/the/virtual_env/bin/activate
into a shell script. But, due to the reason mentioned above, it won't have much effect when you call your script normally. Instead, source your script to use it from an interactive session.
Thus:
Content of my_activate.sh
#!/bin/bash
# Do something
# ...
# then
source /path/to/the/virtual_env/bin/activate
# Do more stuff
# ...
and in your interactive session
source my_activate.sh

I recommend using virtualenvwrapper. It provides some useful tools for managing your virtual environments.
pip install --user virtualenvwrapper
When you create the virtual environment, you specify which version of python should be used in the environment.
mkvirtualenv -p /usr/local/bin/python2.6 myproject.2.6
mkvirtualenv -p /usr/local/bin/python3.3 myproject.3.3
Then, "enter" the environment with the workon command.
workon myproject.2.6

Here are few steps to follow, one thing you can do is
export PYTHON = "/path/pathto/virtualenv/python"
Use this path in bashrc to use. Or you can do something like:-
vim ~/.bashrc
Go to end and set
alias python=/path/pathto/virtualenv/python
source ~/.bashrc

How can I temporary bypass python virtual environment from inside a bash script?

I do have a bash script that needs to install some Python packages on the system instead of an virtual environment which may or may not be activated when the script is executed.
This script is called by people that may already have a python virtual environment activated and I do want to be sure that for few commands I do not use it.
I tried to use the deactivate command but it seems that is not available, even if bash detects the virtual environment (presence of the VIRTUAL_ENV variable).
As a side note, I don't want to permanently disable the virtual environment. I just want to run few commands outside it. How can I do this?

If activating before starting the script
If you did the activate step in a parent shell, not in the shell instance running the script itself, then non-exported variables and functions are unavailable during its runtime.
To be entirely clear about definitions:
source my-virtualenv/bin/activate # this runs in the parent shell
./my-shell-script # the shell script itself is run in a child process created by
# fork()+execve(); does not inherit shell variables / functions, so
# deactivate WILL NOT WORK here.
(source my-shell-script) # creates a subshell with fork(), then directly invokes
# my-shell-script inside that subshell; this DOES inherit shell
# variables / functions, and deactivate WILL WORK here.
You have three options:
Export the deactivate function and its dependencies from the parent shell, before starting the script.
This is as given below, and looks something like:
source my-virtualenv/bin/activate
export VIRTUAL_ENV ${!_OLD_VIRTUAL_#}
export -f deactivate
./my-script-that-needs-to-be-able-to-deactivate
You could optionally define an activation function that does this for you, like so:
# put this in your .bashrc
activate() {
source "$1"/bin/activate && {
export VIRTUAL_ENV ${!_OLD_VIRTUAL_#}
export -f deactivate
}
}
# ...and then activate virtualenvs like so:
activate my-virtualenv
Make some guesses, within the script, about what the prior Python environment looked like.
This is less reliable, for obvious reasons; however, as virtualenv does not export the shell variables containing the original PYTHON_HOME, that information is simply unavailable to child-process shells; a guess is thus the best option available:
best_guess_deactivate() {
if [[ $VIRTUAL_ENV && $PATH =~ (^|:)"$VIRTUAL_ENV/bin"($|:) ]]; then
PATH=${PATH%":$VIRTUAL_ENV/bin"}
PATH=${PATH#"$VIRTUAL_ENV/bin:"}
PATH=${PATH//":$VIRTUAL_ENV/bin:"/}
unset PYTHONHOME VIRTUAL_ENV
fi
}
...used within a limited scope as:
run_python_code_in_virtualenv_here
(best_guess_deactivate; run_python_code_outside_virtualenv_here)
run_python_code_in_virtualenv_here
Run the script in a forked child of the shell that first sourced activate with no intervening exec() call
That is, instead of invoking your script as a regular subprocess, with:
# New shell instance, does not inherit non-exported (aka regular shell) variables
./my-shell-script
...source it into a forked copy of the current shell, as
# Forked copy of existing shell instance, does inherit variables
(source ./my-shell-script)
...or, if you trust it to hand back control to your interactive shell after execution without messing up state too much (which I don't advise), simply:
# Probably a bad idea
source ./my-shell-script
All of these approaches have some risk: Because they don't use an execve call, they don't honor any shebang line on the script, so if it's written specifically for ksh93, zsh, or another shell that differs from the one you're using interactively, they're likely to misbehave.
If activating within the script
The most likely scenario is that the shell where you're running deactivate isn't a direct fork()ed child (with no intervening exec-family call) of the one where activate was sourced, and thus has inherited neither functions or (non-exported) shell variables created by that script.
One means to avoid this is to export the deactivate function in the shell that sources the activate script, like so:
printf 'Pre-existing interpreter: '; type python
. venv-dir/bin/activate
printf 'Virtualenv interpreter: '; type python
# deactivate can be run in a subshell without issue, scoped to same
printf 'Deactivated-in-subshell interpreter: '
( deactivate && type python ) # this succeeds
# however, it CANNOT be run in a child shell not forked from the parent...
printf 'Deactivated-in-child-shell (w/o export): '
bash -c 'deactivate && type python' # this fails
# ...unless the function is exported with the variables it depends on!
export -f deactivate
export _OLD_VIRTUAL_PATH _OLD_VIRTUAL_PYTHONHOME _OLD_VIRTUAL_PS1 VIRTUAL_ENV
# ...after which it then succeeds in the child.
printf 'Deactivated-in-child-shell (w/ export): '
bash -c 'deactivate && type python'
My output from the above follows:
Pre-existing interpreter: python is /usr/bin/python
Virtualenv interpreter: python is /Users/chaduffy/test.venv/bin/python
Deactivated-in-subshell interpreter: python is /usr/bin/python
Deactivated-in-child-shell (w/o export): bash: deactivate: command not found
Deactivated-in-child-shell (w/ export): python is /usr/bin/python
Assuming you've fixed that, let's run once more through using a subshell to scope deactivation to make it temporary:
. venv-dir/activate
this-runs-in-venv
# minor performance optimization: exec the last item in the subshell to balance out
# ...the performance cost of creating that subshell in the first place.
(deactivate; exec this-runs-without-venv)
this-runs-in-venv

You could always reference the global Python directly:
/usr/bin/python2.7 -E my_python_command
If you're concerned about such a path being unreliable, you could either:
Configure Python in a unique, safe, static location upon installation, and reference that
Invoke a subshell that is not inside the virtual environment
Use an alternate name for your virtualenv Python executable so that it doesn't conflict on the path. ie. the virtualenv Python would be python-virtual, and python would still lead to the global installation.
Then it would be something like:
python -E my_command # global
python-virtual my_command # virtual

how to deactivate virtualenv from a bash script

I'm a novice in shell scripting, but I want to make a bash script for activate/deactivate a virtual enviroment using virtualenv.
Then I want to use this script like a service in Ubuntu copying it inside /etc/init.d folder.
In my script, I have a variable like this:
VENV=/opt/odoo/odoo_server/venv_oddo/bin
This variable represents the bin path in my virtual enviroment.
Inside the script, I can activate the virtual enviroment with this statement:
. ${VENV}/activate
This is possible because activate is a file inside bin directory in the virtual enviroment.
But I don't know the statement to use in my script to deactivate my virtual enviroment.
I can't do this: . ${VENV}/deactivate
The problem is that doesn't exist a file named deactivate, but deactivated is a function inside the bin/activate file in the virtual enviroment.

Just deactivate. It will work in the script as well as in command line, as long as you're using bash.
Edit: also in most cases it is a better idea to spell full python path in your scripts and services. It is stateless, more portable and works pretty much everywhere. So instead of doing
. $VENV/bin/activate
/path/to/my/script.py --parameters
it is usually preferable to do
$VENV/bin/python /path/to/my/script --parameters
Trust me, it will save you debugging time)

It'll be hard to make a service like that useful.
. ${VENV}/activate # note the dot
or
source ${VENV}/activate
will source the activate script, i.e. run its contents as if they were part of the shell or script where you source them. virtualenvironment's activate is designed for this usage. In contrast, just executing the script normally with
${VENV}/activate # note: NO dot and NO 'source' command
will run its content in a subshell and won't have any useful effect.
However, your service script will already run in a subshell of its own. So except for any python commands you run as part of the service start process, it won't have any effect.
On the plus side, you won't even have to care about de-activating the environment, unless you want to run even more python stuff in the service start process, but outside of your virtualenv.

The deactivate "command" provided by virtualenvwrapper is actually a shell function, likewise so for workon. If you have a virtual env active, you can list the names of these functions with typeset -F.
In order to use them in a script, they need to be defined there, because shell functions do not propagate to child shells.
To define these functions, source the virtualenvwrapper.sh script in the shell script where you intend to invoke these functions, e.g.:
source $(which virtualenvwrapper.sh)
That allows you to invoke these functions in your shell script like you would do in the shell:
deactivate
Update: What I described works for the other functions provided by virtualenvwrapper (e.g. workon). I incorrectly assumed it would work also for deactivate, but that one is a more complicated case, because it is a function that will be defined only in the shell where workon or activate was run.

copy deactivate code in ${VENV}/activate.
paste your ~/.bashrc
deactivate() {
# reset old environment variables
if [ -n "$_OLD_VIRTUAL_PATH" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if [ -n "$_OLD_VIRTUAL_PYTHONHOME" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "$BASH" -o -n "$ZSH_VERSION" ] ; then
hash -r
fi
if [ -n "$_OLD_VIRTUAL_PS1" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi
unset VIRTUAL_ENV
if [ ! "$1" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}
run command.
$ $VENV/activate
$ deactivate
I have selectively used without problems python 2.7 and python 3.5 in this way.
I want to know the reason for the negative evaluation.

Somehow deactivate also can't be found in my case (usually I work under far2l inside bash). I use the solution:
unset VIRTUAL_ENV & deactivate
After that pip -V is showing path in .local.

If you only need to programatically disable / change virtualenv, you can use a shell function instead of a shell script. For example, put at the end of your ~/.bashrc or ~/.bash_aliases (if you have it set up) or ~/.zshrc or ~/.zsh_aliases (if you use zsh):
function ch() {
# change this to your directory
cd ~/git-things/my_other_py_project
# this works, as a shell function won't spawn a subshell as the script would
deactivate
# re-source to change the virtualenv (my use case; change to fit yours)
source .venv-myotherpyproject/bin/activate
}
Restart the shell or re-source the file you changed with source ~/.zsh_aliases and use the command ch to execute the function.

how to use python to do a series of commands in terminal

I put the following in my python_go.py
import os
os.system("cd some_dir") # This is the directory storing an existing virtual environment
os.system(". activate") #because I want to activate the virtual environment
os.system("cd another_dir") #this the directory I can start my work
I hope I can run the python_go.py, it can do the work mentioned above.
But when I run it, seems it can only do the first step, the rest of it, e.g. . activate seems not working.
Can someone tell me how to do it? Thank you!!

Most probably you don't have to change to some_dir to source activate so saving these lines
. some_dir/activate
cd another_dir
as, let's say go.sh and doing
. go.sh
has the same effect

If you're relying to os.system(". activate") to work if it's in the directory some_dir that won't work because the current directory won't persist across calls to os.system().
You're going to be better off calling a shell script that aggregates all three of the commands you want to do and execute that once from the python script.
Otherwise you want to set up the environment for the parent python process using os.chdir() before calling os.system on the activate call. Also, the os.system(". activate") call won't do what you want because the "dot space" notation will load information into a shell that's going to go away when the os.system call finishes.
Edited (to your followup comment):
Your shell script should look like this (do_activate.sh):
cd some_dir
. activate
cd another_dir
and the python code like this:
os.system("db_activate.sh").
Keep in mind that whatever environment variables were saved by ". activate" won't persist after the os.system call.

Your code does nothing. os.system() starts a new shell for each command i.e., all os.system() calls have no positive effect: cd and . activate may have effect only on the current shell (and possibly its children).
If all you want is to activate a virtualenv in the current shell then you should use a shell command:
$ . some_dir/activate && cd another_dir
Note: the command has effect only on the current (running) shell (and its descendants).
virtualenvwrapper provides several hooks that allows to execute commands before/after activating a virtualenv e.g., you could put cd another_dir into $VIRTUAL_ENV/bin/postactivate then it is enough to run:
$ workon <virtualenv-name>
to activate virtualenv-name virtualenv and run all the hooks (cd another_dir in this case).

You probably want to install virtualenvwrapper which does what you want:
workon envname will source the file and activate the virtualenv.
You can then do setvirtualenvproject in the desired directory and you'll automatically go to the directory where the project is located. You only need to do this command once as it'll then happen automatically from then on.