If one defines which version of python to use in a bash script, it would be
export PYTHON = "/path/python/python-3.5.1/bin/python"
But for Python virtualenv's, one executes these commands in the command line
cd /path/pathto/virtualenv
source activate
cd another_directory
How does one "enter" a Python virtualenv in a bash script? What is the standard approach here?
We have to distinguish two cases here:
You want to use/call python (or python-based tools) in your bash script, but python or those tools should be taken from and run in a virtualenv
You want a script that, amongst other things, lets the shell from which you call it enter the virtualenv, so that you can interactively call python (or python-based tools) inside the virtualenv
Case 1: Using a virtualenv inside a script
How does one "enter" a Python virtualenv in a bash script?
Just like on the interactive bash command line:
source /path/to/the/virtual_env/bin/activate
What is the standard approach here?
The standard approach is not to enter the virtualenv in a bash script. Instead, call python and/or the python-based commands you want to use by their full path. To make this easier and less repetitive, you can use aliases and variables.
Case 2: Activating a virtualenv in an interactive bash session by calling a script
There already is such a script. It's called activate and it's located in the bin directory of the virtualenv. You have to source it rather than calling it like a normal command. Only then will it run in the same session instead of in a subshell, and thus only then can it make modifications to the session that won't be lost due to the subshell terminating at the end of the script.
So just do:
source /path/to/the/virtual_env/bin/activate
in your interactive shell session.
But what if you want to do more than the activate script does? You can put
source /path/to/the/virtual_env/bin/activate
into a shell script. But, due to the reason mentioned above, it won't have much effect when you call your script normally. Instead, source your script to use it from an interactive session.
Thus:
Content of my_activate.sh
#!/bin/bash
# Do something
# ...
# then
source /path/to/the/virtual_env/bin/activate
# Do more stuff
# ...
and in your interactive session
source my_activate.sh
I recommend using virtualenvwrapper. It provides some useful tools for managing your virtual environments.
pip install --user virtualenvwrapper
When you create the virtual environment, you specify which version of python should be used in the environment.
mkvirtualenv -p /usr/local/bin/python2.6 myproject.2.6
mkvirtualenv -p /usr/local/bin/python3.3 myproject.3.3
Then, "enter" the environment with the workon command.
workon myproject.2.6
Here are few steps to follow, one thing you can do is
export PYTHON = "/path/pathto/virtualenv/python"
Use this path in bashrc to use. Or you can do something like:-
vim ~/.bashrc
Go to end and set
alias python=/path/pathto/virtualenv/python
source ~/.bashrc
Related
I have a written a makefile to use to run Python code. The idea is it will allow me to run my linter, pytest, and run the code all in one command. Importantly, it will also spin up a python environment to use.
An excerpt of it looks like:
VENV := venv
all: venv
$(VENV)/bin/activate: requirements.txt
python3 -m venv $(VENV)
./$(VENV)/bin/pip install -r requirements.txt
# venv is a shortcut target
venv: $(VENV)/bin/activate
run: venv
flake8 --exclude=venv
pytest --ignore=venv
mypy advent_of_code/dayone.py
python advent_of_code/dayone.py
The rest of the code is found here: https://github.com/andrewblance/advent_of_code_2020
However, when I run my code I do not think it is using this environment. I think it still uses my default python environment. When I run this code I can see the version of Pytest and mypy that is being used depends on what is installed on the default python environment.
Have I done something wrong that means when I run flake & etc that I don't use the new environment? How can I change it so it only uses the environment specified in the makefile, and how do I "turn off" the environment once I am done with it?
Short: you can't.
Longer: every process has its own environment. Its environment is inherited from the process that started it. But, it's impossible for a child process to modify or change the environment of its parent.
Every command is a process. So the make program is a process, and each command line that make invokes is a process. So when you run the command python3 -m venv $(VENV) that starts a shell process, which runs python3 -m venv ... which is another process. Then whatever change to the environment python3 made is lost when python3 exits, and whatever change to the environment was made in the shell that started python3 is lost when the shell exits, and then other shells are started with other commands, and when all the commands are done make will exit and any changes made to its environment are lost when you get back to your shell prompt.
At the top of my makefile I have this line:
SHELL := /bin/sh
which is needed for most of the commands. However, I would like to also have a make command to activate my virtual env, which is on a different path.
Here is the code that I wrote for it:
activate:
source ~/.envs/$(APP)/bin/activate; \
The problem with this is, that this just prints out what is written here, and it doesn't get executed. I read that it might have something todo with only bash knowing about source, but I can't figure out how to temporarily switch modes within the activate command.
How would I have to write this method, so that it activates my virtualenv?
It does get executed.
Virtualenv works by modifying your current process's environment (that's why you have to "source" it). However, one process cannot modify the environment of the other process. So, to run your recipe make invokes a shell and passes it your virtualenv command, it works, then the shell exits, and your virtualenv is gone.
In short, there's no easy way to do this in a makefile. The simplest thing to do is create a script that first sources the virtualenv then runs make, and run that instead of running make.
Create a file called "make-venv" like this:
#!/bin/bash
source ./.venv/bin/activate
$2
Then add this to the first line of your Makefile
SHELL=./make-venv
Now, make-venv activates virtualenv before every command runs. Probably inefficient, but functional.
You can do it by using set, which allows you to set or unset values of shell options and positional parameters:
set -a && . venv/bin/activate && set +a
I use python-mode.el and have just discovered Pipenv. I can start Python within a project environment from a terminal by first changing to the project directory and entering "pipenv run python". But python-mode wants to execute a process, not a command string.
I tried creating a shell script like this:
#!/bin/bash
cd ~/myprojectdir
pipenv run python
but got this complaint
Warning: the environment variable LANG is not set!
We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
This variable is set in my terminal environment: how do i ensure it's set for pipenv? But the bigger question is, how do i run this pipenv virtual environment python inside emacs to get a buffer with a Python process?
pipenv.el helps here, by setting the appropriate variables and providing a porcelain around Pipenv inside Emacs.
Looks like i have to explicitly source my .bashrc file (or equivalent). Changing the contents of my runpipenv.sh script to this
#!/bin/bash
source ~/.bashrc
cd ~/myprojectdir
pipenv run python
(where my .bashrc file sets LANG) and then calling
(setq py-shell-name "runpipenv.sh")
in Emacs seems to work.
I put the following in my python_go.py
import os
os.system("cd some_dir") # This is the directory storing an existing virtual environment
os.system(". activate") #because I want to activate the virtual environment
os.system("cd another_dir") #this the directory I can start my work
I hope I can run the python_go.py, it can do the work mentioned above.
But when I run it, seems it can only do the first step, the rest of it, e.g. . activate seems not working.
Can someone tell me how to do it? Thank you!!
Most probably you don't have to change to some_dir to source activate so saving these lines
. some_dir/activate
cd another_dir
as, let's say go.sh and doing
. go.sh
has the same effect
If you're relying to os.system(". activate") to work if it's in the directory some_dir that won't work because the current directory won't persist across calls to os.system().
You're going to be better off calling a shell script that aggregates all three of the commands you want to do and execute that once from the python script.
Otherwise you want to set up the environment for the parent python process using os.chdir() before calling os.system on the activate call. Also, the os.system(". activate") call won't do what you want because the "dot space" notation will load information into a shell that's going to go away when the os.system call finishes.
Edited (to your followup comment):
Your shell script should look like this (do_activate.sh):
cd some_dir
. activate
cd another_dir
and the python code like this:
os.system("db_activate.sh").
Keep in mind that whatever environment variables were saved by ". activate" won't persist after the os.system call.
Your code does nothing. os.system() starts a new shell for each command i.e., all os.system() calls have no positive effect: cd and . activate may have effect only on the current shell (and possibly its children).
If all you want is to activate a virtualenv in the current shell then you should use a shell command:
$ . some_dir/activate && cd another_dir
Note: the command has effect only on the current (running) shell (and its descendants).
virtualenvwrapper provides several hooks that allows to execute commands before/after activating a virtualenv e.g., you could put cd another_dir into $VIRTUAL_ENV/bin/postactivate then it is enough to run:
$ workon <virtualenv-name>
to activate virtualenv-name virtualenv and run all the hooks (cd another_dir in this case).
You probably want to install virtualenvwrapper which does what you want:
workon envname will source the file and activate the virtualenv.
You can then do setvirtualenvproject in the desired directory and you'll automatically go to the directory where the project is located. You only need to do this command once as it'll then happen automatically from then on.
I've seen a lot of scripts for daemonizing a python script in linux, but not much information about how to use them. Could anyone guide me on this?
I currently have a lengthy python script that listens on a socket for an incoming message, if it's the correct format accepts it and then stores it into the database. The script itself just opens the socket and then listens on a while true (which does the job!) and does all the work in there.
To daemonize it, would I have to modify my current script or call it from a separate script? I've seen examples of both but got neither to work.
Also, I'm using virtualenv which might the root of my problems, any hints on using this with daemonized scripts?
Create a shell-script that activates the virtual environment, and runs your Python script in the background.
Also, there should by a python module in the virtual environment that you can import and activate the environment from too. I don't have virtualenv working at the moment, so I can not check where it is, but search for activate (or something similar) in the virtual environment and you should find it.
Edit: Added a minimal Debian init.d script
The absolute minimal script needed to start a daemon when the computer boots, is this:
#!/bin/sh
/path/to/program &
The & makes the program run in the background, so it wont stop the rest of the boot process.
For a more complete script, copy /etc/init.d/skeleton and edit the new file. The important part to edit is the block at the beginning (between ### BEGIN INIT INFO and ### END INIT INFO, which is used by the update-rc.d program), and the NAME, DAEMON and DAEMON_ARGS variables. Hopefully that should be all that's needed for making a startup-script.
Activate the script as this:
sudo update-rc.d <name of script> defaults
sudo update-rc.d <name of script> enable
And to start it:
sudo update-rc.d <name of script> start
The <name of script> is just the name, not the full path.
script
export PYTHONPATH=.:/home/ubuntu/.local/lib/python2.7/site-packages/:/home/ubuntu/python/lib/python2.7/site-packages/
exec start-stop-daemon --start --chuid ubuntu --exec /home/ubuntu/python_envs/MyProj/bin/python /home/ubuntu/www/MyProj/MyProj.py -- --config-file-dir=/home/ubuntu/www/MyProj/config/ >> /home/ubuntu/startup.log 2>&1 &
end script
When you need to run an application in a python virtualenv, you can either 'activate' the virtualenv, or use that environment's unique python command.
As per the website "If you directly run a script or the python interpreter from the virtualenv's bin/ directory (e.g. path/to/env/bin/pip or /path/to/env/bin/python script.py) there's no need for activation." - http://pypi.python.org/pypi/virtualenv
I also have some python modules that were compiled from source. Those need to be in the PYTHONPATH environment variable. That could be part of your virtualenv activation, done with virtualwrapper, or explicitly called (as I do below.)
Calling the program from an UPSTART job works as well. My example is above.
On an Ubuntu 10.10 instance on Amazon EC2, I had better luck with the start-stop-daemon command.
I also struggled with some of the other upstart 'stanzas.' I am calling a python application with a specific virtualenv and some parameters to my executed program.