I'm trying to run vnc server, but in order to do it first I need to run 'module load vnc'.
If I call which module in loaded bash shell then the command in not found is the PATH but in the same time it's available. It looks like the command is built-in.
In other words it looks like I need to execute two commands at once module load vnc;vncserver :8080 -localhost and I'm writing script to start it from python.
I have tried different variants with subprocess.Popen like
subprocess.Popen('module load vnc;vncserver :8080 -localhost', shell=True)
which returns 127 exit code or command not found.
subprocess.Popen('module load vnc;vncserver :8080 -localhost', shell=False)
showing
File <path>/subprocess.py line 621, in \__init__
errread, errwrite)
OSError: [Errno 2] No such file or directory.
If I specify shell=True, it executes from /bin/sh but I need it from /bin/bash.
Specifying executable='/bin/bash' doesn't help as it loads new bash shell but it starts as string but not as process, i.e. I see in ps list exactly the same command I would like to start.
Would you please advise how to do start this command from subprocess module? Is it possible to have it started with shell=False?
Environment Modules usually just modifies a couple environment variables for you. It's usually possible to skip the module load whatever step altogether and just not depend on those modules. I recommend
subprocess.Popen(['/possibly/path/to/vncserver', ':8080', '-localhost'],
env={'WHATEVER': 'you', 'MAY': 'need'})
instead of loading the module at all.
If you do insist on using this basic method, then you want to start bash yourself with Popen(['bash',....
If you want to do it with shell=False, just split this into two Popen calls.
subprocess.check_call('module load vnc'.split())
subprocess.Popen('vncserver :8080 -localhost'.split())
You can call module from a Python script. The module command is provided by the environment-modules software, which also provides a python.py initialization script.
Evaluating this script in a Python script enables the module python function. If environment-modules is installed in /usr/share/Modules, you can find this script at /usr/share/Modules/init/python.py.
Following code enables module python function:
import os
exec(open('/usr/share/Modules/init/python.py').read())
Thereafter you can load your module and start your application:
module('load', 'vnc')
subprocess.Popen(['vncserver', ':8080', '-localhost'])
Related
I am creating a script to run shell commands for simulation purposes using a web app. I want to run a shell command in a django app and then save the output to a file.
The problem I am facing is that when running the shell command, the output tries to get saved in the url that is invoked (for example: localhost:8000/projects) which is understandable.
I want to save the output to for example:
/home/myoutput/output.txt rather than /projects or /tasks
I have to run a whole script and save it's output to the txt file later but that is easy once this is done.
Tried os.chdir() function to change directory to /desiredpath already
from subprocess import run
#the function invoked from views.py
def invoke_mpiexec():
run('echo "this is a test file" > fahadTest.txt')
FileNotFoundError at /projects
Exception Type: FileNotFoundError
First I want to say that directly calling external programs from a web request in Django is a bit of an anti-pattern. The preferred approach is to use a work queue like Celery or rq, but that comes with a bit of added complexity.
That being said, you can solve your problem with the argument shell=True:
from subprocess import run
#the function invoked from views.py
def invoke_mpiexec():
run('echo "this is a test file" > fahadTest.txt', shell=True)
Here is the documentation:
If shell is True, the specified command will be executed through the
shell. This can be useful if you are using Python primarily for the
enhanced control flow it offers over most system shells and still want
convenient access to other shell features such as shell pipes,
filename wildcards, environment variable expansion, and expansion of ~
to a user’s home directory. However, note that Python itself offers
implementations of many shell-like features (in particular, glob,
fnmatch, os.walk(), os.path.expandvars(), os.path.expanduser(), and
shutil).
Note: Using shell=True can lead to security issues:
If the shell is invoked explicitly, via shell=True, it is the
application’s responsibility to ensure that all whitespace and
metacharacters are quoted appropriately to avoid shell injection
vulnerabilities.
You should use subprocess.call with stdout argument
def invoke_mpiexec():
f = open("fahadTest.txt", "w")
subprocess.call(['echo', '"this is a test file"'], stdout=f)
or use write function
def invoke_mpiexec():
f = open('fahadTest.txt', 'w')
f.write("Now the file has more content!")
f.close()
So I figured it out.
Below is the fix:
run('mkdir -p $HOME/phdata/test/ && echo "this is a test file" > $HOME/phdata/test/fahadTest.txt', shell=True)
mkdir -p creates a directory if it doesn't exist
$HOME is used to go to the home directory and from there you can
navigate to folders.
shell=True argument is required to run it as shell command
You can also create a ssh connection and run the commands/ scripts on the remote server. For this, my approach will be to create a script on the remote server, call it through my app and provide arguments to it. Another workaround which is not that good but works is to create a script on the server using the above line and then call it.
Not sure if this is possible. I have a set of python scripts and have modified the linux PATH in ~/.bashrc so that whenever I open a terminal, the python scripts are available to run as a command.
export PATH=$PATH:/home/user/pythonlib/
my_command.py resides in the above path.
I can run my_command.py (args) from anywhere in terminal and it will run the python scripts.
I'd like to control this functionality from a different python script as this will be the quickest solution to automating my processing routines. So I need it to open a terminal and run my_command.py (args) from within the python script I'm working on.
I have tried subprocess:
import subprocess
test = subprocess.Popen(["my_command.py"], stdout=subprocess.PIPE)
output = test.communicate()[0]
While my_command.py is typically available in any terminal I launch, here I have no access to it, returns file not found.
I can start a new terminal using os then type in my_command.py, and it works
os.system("x-terminal-emulator -e /bin/bash")
So, is there a way to get the second method to accept a script you want to run from python with args?
Ubuntu 16
Thanks :)
Popen does not load the system PATH for the session you create in a python script. You have to modify the PATH in the session to include the directory to your project like so:
someterminalcommand = "my_command.py (args)"
my_env = os.environ.copy()
my_env["PATH"] = "/home/usr/mypythonlib/:" + my_env["PATH"]
combine = subprocess.Popen(shlex.split(someterminalcommand), env=my_env)
combine.wait()
This allows me to run my "my_command.py" file from a different python session just like I had a terminal window open.
If you're using Gnome, the gnome-terminal command is rather useful in this situation.
As an example of very basic usage, the following code will spawn a terminal, and run a Python REPL in it:
import subprocess
subprocess.Popen(["gnome-terminal", "-e", "python"])
Now, if you want to run a specific script, you will need to concatenate its path with python, for the last element of that list it the line that will be executed in the new terminal.
For instance:
subprocess.Popen(["gnome-terminal", "-e", "python my_script.py"])
If your script is executable, you can omit python:
subprocess.Popen(["gnome-terminal", "-e", "my_script.py"])
If you want to pass parameters to your script, simply add them to the python command:
subprocess.Popen(["gnome-terminal", "-e", "python my_script.py var1 var2"])
Note that if you want to run your script with a particular version of Python, you should specify it, by explicitly calling "python2" or "python3".
A small example:
# my_script.py
import sys
print(sys.argv)
input()
# main.py
import subprocess
subprocess.Popen(["gnome-terminal", "-e", "python3 my_script.py hello world"])
Running python3 main.py will spawn a new terminal, with ['my_script.py', 'hello', 'world'] printed, and waited for an input.
I am not able to run any commands using python subprocess module which contains * sign in the command.
I am using the call this way,
subprocess.Popen(
'cp /etc/varnida_sys/* /tmp/bucket/'.split(),
stdout=subprocess.PIPE).communicate()[0]
For this I am getting,
cp: cannot stat ‘/etc/varnida_sys/*’: No such file or directory
Why is this error coming, there is one file inside /etc/varnida_sys/genders
My investigations says that using regex like * needs some special handling. I am getting some errors in all those commands that contains *.
PS. I am not getting errors when I am running the same command through paramiko from remote host.
* is only understood by a shell (which expands it to a list of files), you need to pass shell=True to Popen(). Also, there's no need to split the command, you can use a string:
subprocess.Popen("cp /etc/varnida_sys/* /tmp/bucket/",
stdout=subprocess.PIPE, shell=True).communicate()[0]
As #triplee has suggested below, it's better to use some convenience wrapper for this task, e.g. subprocess.call():
subprocess.call("cp /etc/varnida_sys/* /tmp/bucket/", shell=True)
I have this simple script..that does not work
import subprocess
subprocess.call(["C:\Program Files\Oracle\VirtualBox\VBoxManage.exe", "VBoxManage startvm WIN7"])
I have the same thing in a bat file...which works perfectly.
cd C:\Program Files\Oracle\VirtualBox
VBoxManage startvm "WIN7"
I have the VBoxManage.exe in the PATH of Windows 8.1 (My host OS).
The python script understands the VBoxManage executable and spits out it's manual and then this ..
Syntax error: Invalid command 'VBoxManage startvm WIN7'
Could you give me a way to start a VM from inside a python script, either by invoking the .exe directly or by running the .bat file ?
Note: I have searched for the vboxshell.py file but not found it anywhere...:[
subprocess.call() expects a list of arguments, like so:
subprocess.call(['C:\Program Files\Oracle\VirtualBox\VBoxManage.exe',
'startvm',
'WIN7'])
Your code passes 'VBoxManage startvm WIN7' as a single argument to VBoxManage.exe, which expects to find only a command (e.g. 'startvm') there. The subsequent arguments ('WIN7' in this case) need to be passed separately.
In addition, there is no need to repeat the executable name when using subprocess.call(). The example from the Python docs invokes the UNIX command "ls -l" as follows:
subprocess.call(['ls', '-l'])
In other words, you don't need to repeat the 'VBoxManage' part.
The trick is to pass the command as separate arguments
import subprocess
subprocess.call(["C:\Program Files\Oracle\VirtualBox\VBoxManage.exe", "startvm", "WIN7"])
The python script I would use (source code here) would parse some arguments when called from the command line. However, I have no access to the Windows command prompt (cmd.exe) in my environment. Can I call the same script from within a Python console? I would rather not rewrite the script itself.
%run is a magic in IPython that runs a named file inside IPython as a program almost exactly like running that file from the shell. Quoting from %run? referring to %run file args:
This is similar to running at a system prompt python file args,
but with the advantage of giving you IPython's tracebacks, and of
loading all variables into your interactive namespace for further use
(unless -p is used, see below). (end quote)
The only downside is that the file to be run must be in the current working directory or somewhere along the PYTHONPATH. %run won't search $PATH.
%run takes several options which you can learn about from %run?. For instance: -p to run under the profiler.
If you can make system calls, you can use:
import os
os.system("importer.py arguments_go_here")
You want to spawn a new subprocess.
There's a module for that: subprocess
Examples:
Basic:
import sys
from subprocess import Popen
p = Popen(sys.executable, "C:\test.py")
Getting the subprocess's output:
import sys
from subprocess import Popen, PIPE
p = Popen(sys.executable, "C:\test.py", stdout=PIPE)
stdout = p.stdout
print stdout.read()
See the subprocess API Documentation for more details.