Using python subprocess with module load - python

I'm currently using Python 2.7 on a unix environment.
I need to run R scripts in my python scripts but I can't manage to make it work because my R module needs to be loaded before (using "module load")
Here's my python script :
import os
import subprocess as sp
os.system('module load R/3.2.3')
out = sp.check_output(['Rscript','test.R'], universal_newlines=True)
I keep having the same error : "[Errno 2] No such file or directory"
Any idea ?
I looked here and here but couldn't make it work.
Thank you for your help !

So what "module load" actually does is set some environment variables in the calling shell. So when you do this:
os.system('module load R/3.2.3')
Python creates a process, runs /bin/sh in it, and passes that command to the shell. The module environment variables are set in that shell. Then that shell exits--job done!
The environment variables do not--and cannot--propagate back to the Python process. So when you do this:
sp.check_output(['Rscript','test.R'])
It's totally irrelevant that you ran module load before.
So how can you fix this? Well, one possibility would be to explicitly specify the path to Rscript:
sp.check_output(['/your/full/path/to/Rscript','test.R'])
Another would be to combine your commands:
sp.check_output('module load R/3.2.3 && Rscript test.R', shell=True)
Finally, you could simply run module load before running your Python script in the first place. The environment variables it sets can propagate all the way to the R invocation within Python.
By the way, it is possible to invoke R directly from Python: http://rpy.sourceforge.net/rpy2/doc-dev/html/introduction.html

Related

command with subprocess.call() to run virtualenv doesn't work

I'm trying to run my virtualenv called env (created before) using call() from subprocess and it doesn't work. Command is fine, it's working when I type it directly in terminal.
python code:
import subprocess
subprocess.call("source env/bin/activate", shell=True)
I was trying also:
import os
os.system("source env/bin/activate")
any ideas why command is not performed or what should I use instead os.system() and subprocess.call()?
In both examples, your code launches a subprocess. That subprocess, which has its own environment, attempts to run the "source" command, and then exits. Since the subprocess has its own environment, this has no effect on the current process.
Assuming your end goal is to run some other command in the subprocess, you should just run it directly. You don't specifically need to activate the virtual environment.
subprocess.call(["./env/bin/pip", "list"])
Avoid using the shell=True option if at all possible; it can be quite dangerous if you're not extremely careful with it.
If you really need to set the environment variables that the activate script sets in this script, you need to manually set them in os.environ. You can read the activate script to see what they are. They usually won't matter, though.

Launching a python script in a new terminal from another python script

I'm using LXDE, I would like to Launch a python script in a new terminal from another python script.
I would like the new python script to be totally independent.
I have tried a lot of things...
Calling xterm (or x-terminal-emulator) directly from python with the subprocess.call or subprocess.Popen with or without shell=True argument, it didn't work. It gives me an error about display not being set.
I have also created a sh file which calls the other python script and tried to call it, same results.
Any way to do this?
This works fine for me:
blocking:
import os
os.system("xterm -e \"python christmaskittens.py\"")
non blocking:
import os
os.system("xterm -e \"python christmaskittens.py\" &")

python shell script execution that passes environment to the parent

So I know how subprocess works and use it a lot, but I've run into a strange issue. I need to execute an export of some environment variables. The reason is that some program (black-box) executes a program that seems like it runs in a subshell, so it doesn't have access to the environment variables but it has access to all my files.
I can't hard code the environment variables so I want to source or . the file that has the export commands in it. However, if I source or . that file in a subprocess, it won't make any difference to its parent process. In which case I either need some function besides subprocess that can execute shell commands without creating a subprocess, if that exists. Another issue is that a subprocess doesn't have the proper permissions to read the file.
And copying the environment variables via os isn't really possible either.
Does anything besides subprocess exist? Or is there some other kind of workaround?
IMHO the simplest solution consists in creating a new shell script (let's call it run_black_box.sh) which sources the setup script (let's assume it is named setup.sh) to initialize the environment and then calls the black_box program.
Here is a possible content of run_black_box.sh:
#/bin/bash
source setup.sh
black_box
The you can pass run_black_box.sh to subprocess for execution.

Create a process that runs in background using python

I want to write a python program that runs in the background.
I mean, like we install Python package. And later, we can run any script using python in front of the script name. This means that some python process is running in background which can take inputs and perform actions.
And in case of linux, you can call grep from anywhere. That means grep is also running in the background somehow.
I want to write something like that in python. When I call certain function with name and arguments at any time, it should perform the intended action without caring for the original code. But I am not able to find how to achieve that.
Can anyone please help me here?
Thanks in advance.
Clarification: the fact that you can run python or grep in a console just by typing their name, does not mean that they run in background. It means that there exist an executable file in some location, and this location is listed in the environment variable PATH.
For example, on my system I can run Python by typing python. The python executable is installed at /usr/local/bin/python, and has the execute permission bit on.
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
and yes, /usr/local/bin is contained in PATH.
You can do the same with python scripts:
ensure that the very first line of your script contains #!/usr/bin/python or #!/usr/bin/env python
give your script execute permissions: chmod a+x yourScript
either move your script to one of the directories contained in $PATH, or add the directory where your script is located to PATH: export PATH=$PATH:/home/you/scripts
Have a look at
http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
you can roll out your own daemon by inheriting the Daemon class and overriding run method
from daemon import Daemon
class run_daemon(Daemon):
def run(self):
import sys
run_daemon.execute_shell_command(sys.argv[1])
#staticmethod
def execute_shell_command(ShellCommand):
import subprocess
process = subprocess.Popen(ShellCommand, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.communicate()

Ansicon doesn't install from python but only from cmd

I'm trying to install ansicon on Windows 8.1. I extracted the files and got to the level that I need to call ansicon -i. When I type this in my cmd and run python scripts that works great but when I call t from python by os.system('ansicon -i') that doesn't work and seems like it doesn't have any influence on the cmd.
Why os.system('ansicon -i') doesn't work and what alternative method can I use from within python?
First off, it’s not the -i flag that really does the work. -i only tells it to add itself to AutoRun. The -p flag that -i implies is what really does the work: -p tells it to inject a DLL into the parent process, and therein lies the problem: when you use os.system, you spawn a shell, which then runs the command you give it. But then you have Python running cmd running ansicon, and ansicon will inject into cmd, and then cmd, having finished its work, will exit.
Rather than using os.system, use the subprocess module, e.g.:
subprocess.check_call(['ansicon', '-p'])
The subprocess module (unlike os.system) will execute the command directly without a shell in-between (unless you pass shell=True). Then Python will spawn ansicon, and ansicon will inject into Python, as desired.
That said, rather than having ansicon inject itself into Python, Python could probably just load the DLL itself, avoiding some hardship:
import sys
import math
import ctypes
bitness = 1 << round(math.log2(round(math.log2(sys.maxsize + 1))))
ctypes.WinDLL('ANSI{}.DLL'.format(bitness))

Categories