We Keep Coding

How to run some configuration commands to allow for a Makefile to run properly in a data processing project?

I am using a Makefile to run relatively small data science processing projects (typically involving running around 10-15 python scripts).
I want to have my project folder set up properly at the beginning of the makefile, which does certain things that will allow another user to replicate the process including:
create a substitution drive to avoid long file paths with spaces that are out of my control
create folders to house output data
create and setup a new python environment
I have tried to do this in the following manner:
config :
subst A: "B:\Network Drive\with-lots\Of Spaces (and other issues)"
mkdir output-data
conda create -n new_env
conda activate new_env
pip install requirements.txt
output-data/file1.csv : script_one.py Q:/inputfile1.csv
python $^ $#
...
The main issue here is that running this more than once will give an error causing make to stop, since the path substitution will have been done, the directory will have already been created, and the environment is already set up.
Is there a better approach that would allow the config phony target to be run only once, or at least have it run every command that it needs to run without shutting down due to an error?
Or is there a better approach for having a replicable environment set up for someone else to get it going on their own system, i.e. outside the Makefile?

You could create a marker empty file to indicate that the configuration step has already been done:
.config.done:
subst A: "B:\Network Drive\with-lots\Of Spaces (and other issues)"
mkdir output-data
conda create -n new_env
conda activate new_env
pip install requirements.txt
touch $#
output-data/file1.csv: .config.done script_one.py Q:/inputfile1.csv
python $^ $#
...
But a better solution would be to have a way to know if each configuration step has been done or not by looking at a result file or directory. mkdir output-data, for instance, is easy: if output-data exists we know it has been done, so you could add output-data as an order-only prerequisite of the next step (order-only because it is a directory and you care only about its existence, not its last modification time). This way if one step fails the corresponding file or directory is not be created, the complete make run fails and after fixing what needs to be you can restart the configuration from where it stopped.
If you do not have a resulting file or directory to look at for some steps you can use the same empty file trick as above.
You could then describe your configuration with a much finer grain:
.PHONY: config
config: .requirements.installed
.requirements.installed: .new_env.activated
pip install requirements.txt
touch $#
.new_env.activated: .new_env.created
conda activate new_env
touch $#
.new_env.created: | output-data
conda create -n new_env
touch $#
output-data: .subst-A.done
mkdir $#
.subst-A.done:
subst A: "B:\Network Drive\with-lots\Of Spaces (and other issues)"
touch $#
output-data/file1.csv: config script_one.py Q:/inputfile1.csv
python $^ $#
By removing some dependencies you can even parallelize the configuration (if you run make with the -j option):
.PHONY: config
config: .subst-A.done .new_env.activated .requirements.installed | output-data
.requirements.installed:
pip install requirements.txt
touch $#
.new_env.activated: .new_env.created
conda activate new_env
touch $#
.new_env.created:
conda create -n new_env
touch $#
output-data:
mkdir $#
.subst-A.done:
subst A: "B:\Network Drive\with-lots\Of Spaces (and other issues)"
touch $#
output-data/file1.csv: config script_one.py Q:/inputfile1.csv
python $^ $#

Activate Anaconda Python environment from makefile

I want to use a makefile to build my project's environment using a makefile and anaconda/miniconda, so I should be able to clone the repo and simply run make myproject
myproject: build
build:
#printf "\nBuilding Python Environment\n"
#conda env create --quiet --force --file environment.yml
#source /home/vagrant/miniconda/bin/activate myproject
If I try this, however, I get the following error
make: source: Command not found
make: *** [source] Error 127
I have searched for a solution, but [this question/answer(How to source a script in a Makefile?) suggests that I cannot use source from within a makefile.
This answer, however, proposes a solution (and received several upvotes) but this doesn't work for me either
( \
source /home/vagrant/miniconda/bin/activate myproject; \
)
/bin/sh: 2: source: not found
make: *** [source] Error 127
I also tried moving the source activate step to a separate bash script, and executing that script from the makefile. That doesn't work, and I assume for the a similar reason, i.e. I am running source from within a shell.
I should add that if I run source activate myproject from the terminal, it works correctly.

I had a similar problem; I wanted to create, or update, a conda environment from a Makefile to be sure my own scripts could use the python from that conda environment.
By default make uses sh to execute commands, and sh doesn't know source (also see this SO answer). I simply set the SHELL to bash and ended up with (relevant part only):
SHELL=/bin/bash
CONDAROOT = /my/path/to/miniconda2
.
.
install: sometarget
source $(CONDAROOT)/bin/activate && conda env create -p conda -f environment.yml && source deactivate
Hope it helps

You should use this, it's functional for me at moment.
report.ipynb : merged.ipynb
( bash -c "source ${HOME}/anaconda3/bin/activate py27; which -a python; \
jupyter nbconvert \
--to notebook \
--ExecutePreprocessor.kernel_name=python2 \
--ExecutePreprocessor.timeout=3000 \
--execute merged.ipynb \
--output=$< $<" )

I had the same problem. Essentially the only solution is stated by 9000. I have a setup shell script inside which I setup the conda environment (source activate python2), then I call the make command. I experimented with setting up the environment from inside Makefile and no success.
I have this line in my makefile:
installpy :
./setuppython2.sh && python setup.py install
The error messages is:
make
./setuppython2.sh && python setup.py install
running install
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/test-easy-install-29183.write-test'
Essentially, I was able to set up my conda environment to use my local conda that I have write access. But this is not picked up by the make process. I don't understand why the environment set up in my shell script using 'source' is not visible in the make process; the source command is supposed to change the current shell. I just want to share this so that other people don't wast time trying to do this. I know autotoools has a way of working with python. But the make program is probably limited in this respect.
My current solution is a shell script:
cat py2make.sh
#!/bin/sh
# the prefix should be change to the target
# of installation or pwd of the build system
PREFIX=/some/path
CONDA_HOME=$PREFIX/anaconda3
PATH=$CONDA_HOME/bin:$PATH
unset PYTHONPATH
export PREFIX CONDA_HOME PATH
source activate python2
make
This seems to work well for me.
There were a solution for similar situation but it does not seems to work for me:
My modified Makefile segment:
installpy :
( source activate python2; python setup.py install )
Error message after invoking make:
make
( source activate python2; python setup.py install )
/bin/sh: line 0: source: activate: file not found
make: *** [installpy] Error 1
Not sure where am I wrong. If anyone has a better solution please share it.

how do you install requirements to arbitrary virtualenv in python scripts?

I am trying to install requirements for each project in a list automatically into its own virtualenv. I have gotten to the point of making the virtualenv correctly, but I cannot get it to activate and install requirements into only that virtualenv:
#!/usr/bin/env python
import subprocess, sys, time, os
HOMEPATH = os.path.expanduser('~')
CWD = os.getcwd()
d = {'cwd': ''}
if len(sys.argv) == 2:
projects = sys.argv[1:]
def call_sp(command, **arg_list):
p = subprocess.Popen(command, shell=True, **arg_list)
p.communicate()
def my_makedirs(path):
if not path.startswith('/home/cchilders'):
path = os.path.join(HOMEPATH, path)
try: os.makedirs(path)
except: pass
for project in projects:
path = os.path.join(CWD, project)
my_makedirs(path)
git_string = 'git clone git#bitbucket.org:codyc54321/{}.git {}'.format(project, d['cwd'])
call_sp(git_string)
d = {'executable': 'bash'}
call_sp("""source /usr/local/bin/virtualenvwrapper.sh && mkvirtualenv --no-site-packages {}""".format(project), **d)
# call_sp("""source /usr/local/bin/virtualenvwrapper.sh && workon {}""".format(project), **d)
# below, the dot (.) means the same as 'source'. the dot doesn't error, calling source does
call_sp('. /home/cchilders/.virtualenvs/{}/bin/activate'.format(project))
d = {'cwd': path}
call_sp("pip install -r requirements.txt", **d)
It works up to
call_sp("""source /usr/local/bin/virtualenvwrapper.sh && mkvirtualenv --no-site-packages {}""".format(project), **d)
but when the script ends, I am not active in the venv and the venv does not have any packages from requirements. Both efforts to source the venv (the one commented out and live) both fail.
The answer that helped me get the mkvirtualenv to work is subprocess.Popen: mkvirtualenv not found.
I also noticed I have a need to do more than just pip install, in one case I need to run 'python setup.py mycommand' which automates setup for each project. How can run commands as if a virtualenv is activated and also install dependencies to arbitrary venvs in a python script?
The only way I've found around this is turning the virtualenv on by hand, then calling my python script by hand. I was surprised, turning it on by bash worked, but calling the python script bombed (maybe because it's a different process than the bash one)
Thank you

This is because each call_sp call creates a new shell, so after the first call to call_sp ends all the settings created by sourcing of virtualenvwrapper are gone. You have to combine all your commands into the single call_sp chain. Otherwise you can just start shell using 'Popen' and feed commands to it using communicate.
If you go with the later you need to be careful with synchronizing and detecting when installation of requirements ends. Pip can take a long time downloading and installing packages with complex dependencies.

This is the way I have done this kind of bootstrapping for virtual environments. Let the script take care of it's own env and just run the script. Running this app.py will setup its VE and modules if missing.
./requirements.txt file
flask
./app.py script
#!/bin/bash
""":"
VENV=$(realpath -s $(dirname $0)/ve)
PYTHON=$VENV/bin/python
if [ ! -f "$PYTHON" ]; then
echo "installing env app"
python3 -m venv $VENV
${VENV}/bin/pip install -r $(dirname $0)/requirements.txt
fi
exec $PYTHON $0 $#
"""
import flask
print("I am Python with flask", flask)
No matter what dir we are in, app.py bootstrapps though the bash script header, installing a ve if python does not exist, running pip, and whatever else you need. Then exec $PYTHON $0 $# is a slick way to swap out bash process for the python process keeping the same pid.
When python takes over, it skips over the bash part because that script is in triple quotes string. So the first line python executes is import flask (well it discards the bash script string 1st). Another cool thing is the pid of the bash process is the same as the pid of the python process. So any daemon utility that babysits this will still see the pid it started.
The last trick in this is that bash needs one extra quote to balance its string """:" at the top. Python does not care about that extra quote
I hope you see the pattern. To upgrade modules in requirements.txt, just rm the ve and run the app again. Simple.

autoenv executes even in subfolder

I uses autoenv for automatic virtualenv activate. Python project's top folder has .env file with following contents
source venv/bin/activate
This command executed whenever cd to any sub folder of the project. Then throws
-bash: venv/bin/activate: No such file or directory
It failed because it is trying to execute activate relative to sub folder. Why it executes even in subfolder? How to resolve the issue?

Had this issue today. The current answer doesn't address the fact that the environment is activated every time you cd into a subfolder or back to the root folder. Solved it with the following .env script:
venv=venv
currentvenv=""
if [[ $VIRTUAL_ENV != "" ]]
then
# Strip out the path and just leave the env name
currentvenv="${VIRTUAL_ENV##*/}"
fi
if [[ "$currentvenv" != "$venv" ]]
then
echo "Switching to environment: $venv"
workon $venv
#else
# echo "Already on environment $venv"
fi
Replace venv with the name of your environment. You can uncomment the else block to see that it doesn't try to activate the environment every time, given that the desired environment is already activated.
Note: If you're not using virtualenvwrapper then you should replace the workon command with whatever command you're using to activate your virtual environment. I do recommend using virtualenvwrapper though.

In your workspace root, a .env containing:
test (command -v deactivate) && deactivate
and in each of your relevant project folders:
workon venv_of_project
As this person points out, it means that cding around in a project will turn the workspace on and off, but at least it is simple and very clear what is going on.

How to activate virtualenv in Linux?

I have been searching and tried various alternatives without success and spent several days on it now - driving me mad.
Running on Red Hat Linux with Python 2.5.2
Began using most recent Virtualenv but could not activate it, I found somewhere suggesting needed earlier version so I have used Virtualenv 1.6.4 as that should work with Python 2.6.
It seems to install the virtual environment ok
[necrailk#server6 ~]$ python virtualenv-1.6.4/virtualenv.py virtual
New python executable in virtual/bin/python
Installing setuptools............done.
Installing pip...............done.
Environment looks ok
[necrailk#server6 ~]$ cd virtual
[necrailk#server6 ~/virtual]$ dir
bin include lib
Trying to activate
[necrailk#server6 ~/virtual]$ . bin/activate
/bin/.: Permission denied.
Checked chmod
[necrailk#server6 ~/virtual]$ cd bin
[necrailk#server6 bin]$ ls -l
total 3160
-rw-r--r-- 1 necrailk biz12 2130 Jan 30 11:38 activate
-rw-r--r-- 1 necrailk biz12 1050 Jan 30 11:38 activate.csh
-rw-r--r-- 1 necrailk biz12 2869 Jan 30 11:38 activate.fish
-rw-r--r-
Problem, so I changed it
[necrailk#server6 bin]$ ls -l
total 3160
-rwxr--r-- 1 necrailk biz12 2130 Jan 30 11:38 activate
-rw-r--r-- 1 necrailk biz12 1050 Jan 30 11:38 activate.csh
-rw-r--r-- 1 necrailk biz12 2869 Jan 30 11:38 activate.fish
-rw-r--r-- 1 necrailk biz12 1005 Jan 30 11:38 activate_this.py
-rwxr-xr-x 1 necrailk biz
Try activate again
[necrailk#server6 ~/virtual]$ . bin/activate
/bin/.: Permission denied.
Still no joy...

Here is my workflow after creating a folder and cd'ing into it:
$ virtualenv venv --distribute
New python executable in venv/bin/python
Installing distribute.........done.
Installing pip................done.
$ source venv/bin/activate
(venv)$ python

You forgot to do source bin/activate where source is a executable name.
Struck me first few times as well, easy to think that manual is telling "execute this from root of the environment folder".
No need to make activate executable via chmod.

You can do
source ./python_env/bin/activate
or just go to the directory
cd /python_env/bin/
and then
source ./activate
Good Luck.

Go to the project directory. In my case microblog is the flask project directory and under microblog directory there should be app and venv folders. then run the below command, This is one worked for me in Ubuntu.
source venv/bin/activate

Cd to the environment path, go to the bin folder.
At this point when you use ls command, you should see the "activate" file.
now type
source activate

$ mkdir <YOURPROJECT>
Create a new project
$ cd <YOURPROJECT>
Change directory to that project
$ virtualenv <NEWVIRTUALENV>
Creating new virtualenv
$ source <NEWVIRTUALENV>/bin/activate
Activating that new virtualenv

run this code it will get activated if you on a windows machine
source venv/Scripts/activate
run this code it will get activated if you on a linux/mac machine
. venv/bin/activate

The problem there is the /bin/. command. That's really weird, since . should always be a link to the directory it's in. (Honestly, unless . is a strange alias or function, I don't even see how it's possible.) It's also a little unusual that your shell doesn't have a . builtin for source.
One quick fix would be to just run the virtualenv in a different shell. (An obvious second advantage being that instead of having to deactivate you can just exit.)
/bin/bash --rcfile bin/activate
If your shell supports it, you may also have the nonstandard source command, which should do the same thing as ., but may not exist. (All said, you should try to figure out why your environment is strange or it will cause you pain again in the future.)
By the way, you didn't need to chmod +x those files. Files only need to be executable if you want to execute them directly. In this case you're trying to launch them from ., so they don't need it.

instead of ./activate
use source activate

For Windows You can perform as:
TO create the virtual env as: virtualenv envName –python=python.exe (if not create environment variable)
To activate the virtual env : > \path\to\envName\Scripts\activate
To deactivate the virtual env : > \path\to\env\Scripts\deactivate
It fine works on the new python version .

Windows 10
In Windows these directories are created :
To activate Virtual Environment in Windows 10.
down\scripts\activate
\scripts directory contain activate file.
Linux Ubuntu
In Ubuntu these directories are created :
To activate Virtual Environment in Linux Ubuntu.
source ./bin/activate
/bin directory contain activate file.
Virtual Environment copied from Windows to Linux Ubuntu vice versa
If Virtual environment folder copied from Windows to Linux Ubuntu then according to directories:
source ./down/Scripts/activate

I would recommend virtualenvwrapper as well. It works wonders for me and how I always have problems with activating. http://virtualenvwrapper.readthedocs.org/en/latest/

Create your own Python virtual environment called <Your Env _name >:.
I have given it VE.
git clone https://github.com/pypa/virtualenv.git
python virtualenv.py VE
To activate your new virtual environment, run (notice it's not ./ here):
. VE/bin/activate
Sample output (note prompt changed):
(VE)c34299#a200dblr$
Once your virtual environment is set, you can remove the Virtualenv repo.

On Mac, change shell to BASH (keep note that virtual env works only in bash shell )
[user#host tools]$. venv/bin/activate
.: Command not found.
[user#host tools]$source venv/bin/activate
Badly placed ()'s.
[user#host tools]$bash
bash-3.2$ source venv/bin/activate
(venv) bash-3.2$
Bingo , it worked. See prompt changed.
On Ubuntu:
user#local_host:~/tools$ source toolsenv/bin/activate
(toolsenv) user#local_host~/tools$
Note : prompt changed

I had trouble getting running source /bin/activate then I realized I was using tcsh as my terminal shell instead of bash. once I switched I was able to activate venv.

Probably a little late to post my answer here but still I'll post, it might benefit someone though,
I had faced the same problem,
The main reason being that I created the virtualenv as a "root" user
But later was trying to activate it using another user.
chmod won't work as you're not the owner of the file, hence the alternative is to use chown (to change the ownership)
For e.g. :
If you have your virtualenv created at /home/abc/ENV
Then CD to /home/abc
and run the command : chown -Rv [user-to-whom-you want-change-ownership] [folder/filename whose ownership needs to be changed]
In this example the commands would be : chown -Rv abc ENV
After the ownership is successfully changed you can simply run source /ENV/bin/./activate and your should be able to activate the virtualenv correctly.

1- open powershell and navigate to your application folder
2- enter your virtualenv folder ex : cd .\venv\Scripts\
3- active virtualenv by type .\activate