I am trying to use the subprocess module in python and trying to fetch the process id of firefox
cmd = "firefox &"
fire = subprocess.Popen(cmd,shell=True, stdout=subprocess.PIPE, preexec_fn=os.setsid)
fire_task_procs = find_task(fire.pid)
print "fire_task_procs",fire_task_procs
I think I am getting the pid of the commandline argument that I am executing.. am I doing something wrong?
I confirmed that it is not the same using the ps aux | grep firefox
If you use shell=True the pid you'll get ist that of the started shell, not that of the process you want, specially as you use & to send the process into background.
You should use the long (list) form of supplying the parameters, without &, as that makes little sense anyway if you combine it with output redirection.
Don't use the shell, instead just use
subprocess.Popen(['firefox'], stdout=subprocess.PIPE, preexec_fn=os.setsid)
However, if firefox is already running then this will not work either since in this case firefox will use some IPC to tell the existing process to open a new window and then terminates.
Related
I am running spyder on windows 10 and when I attempt to run a command similar to the following:
cmd = 'python /path/to/program.py arg1 arg2'
subprocess.run(cmd,shell=True)
The script is being run as expected but I would like to see what is being printed to screen by the executed command in the spyder ipython console. I know the program is printing things to screen as expected by other methods (running the program from a shell) so there is not an error in the script I am running.
How do I go about enabling printing for the subprocess?
The output comes in a stream called stdout. To capture it, you need to redirect it to a pipe, which then is terminated in the calling process. subprocess.run(...) has builtin support for handling this:
import subprocess
cmd = 'python /path/to/program.py arg1 arg2'.split()
proc = subprocess.run(cmd, stdout=subprocess.PIPE, universal_newlines=True)
print(proc.stdout)
As can be seen, the output is caught in the CompletedProcess object (proc) and then accessed as member data.Also, to make the output into text (a string) rather than a bytearray, I have passed the parameter universal_newlines=True.
A caveat, though, is that subprocess.run(...) runs to completion before it returns control. Therefore, this does not allow for capturing the output "live" but rather after the whole process has finsihed. If you want live capture, you must instead use subprocess.Popen(...) and then use .communicate() or some other means of communication to catch the output from the subprocess.
Another comment I like to make, is that using shell=True is not recommended. Specifically not when handling unknown or not trusted input. It leaves the interpretation of cmd to the shell which can lead to all kind of security breaches and bad behavior. Instead, split cmd into a list (e.g. as I have done) and then pass that list to subprocess.run(...) and leave out shell=True.
I have a python script that launches subprocesses using subprocess.Popen. The subprocess then launches an external command (in my case, it plays an mp3). The python script needs to be able to interrupt the subprocesses, so I used the method described here which gives the subprocess its own session ID. Unfortunately, when I close the python script now, the subprocess will continue to run.
How can I make sure a subprocess launched from a script, but given a different session ID still closes when the python script stops?
Is there any way to kill a Thread in Python?
and make sure you use it as thread
import threading
from subprocess import call
def thread_second():
call(["python", "secondscript.py"])
processThread = threading.Thread(target=thread_second) # <- note extra ','
processThread.start()
print 'the file is run in the background'
TL;DR Change the Popen params: Split up the Popen cmd (ex. "list -l" -> ["list", "-l"]) and use Shell=False
~~~
The best solution I've seen so far was just not to use shell=True as an argument for Popen, this worked because I didn't really need shell=True, I was simply using it because Popen wouldn't recognize my cmd string and I was too lazy too split it into a list of args. This caused me a lot of other problems (ex. using .terminate() becomes a lot more complicated while using shell and needs to have its session id, see here)
Simply splitting the cmd from a string to a list of args lets me use Popen.terminate() without having to give it its own session id, by not having a separate session id the process will be closed when the python script is stopped
I have a python script in blender where it has
subprocess.call(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
followed by many other code which depends on this shell script to finish. What happens is that it doesn't wait for it to finish, I don't know why? I even tried using Popen instead of call as shown:
p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
p1.wait()
and I tried using commuincate but it still didn't work:
p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True).communicate()
this shell script works great on MacOS (after changing paths) and waits when using subprocess.call(['sh', '/userA/Test/run-my-script.sh'])
but on Windows this is what happens, I run the below python script in Blender then once it gets to the subprocess line Git bash is opened and runs the shell script while blender doesn't wait for it to finish it just prints Hello in its console without waiting for the Git Bash to finish. Any help?
import bpy
import subprocess
subprocess.call(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
print('Hello')
You can use subprocess.call to do exactly that.
subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None)
Run the command described by args. Wait for command to complete, then return the returncode attribute.
Edit: I think I have a hunch on what's going on. The command works on your Mac because Macs, I believe, support Bash out of the box (at least something functionally equivalent) while on Windows it sees your attempt to run a ".sh" file and instead fires up Git Bash which I presume performs a couple forks when starting.
Because of this Python thinks that your script is done, the PID is gone.
If I were you I would do this:
Generate a unique, non-existing, absolute path in your "launching" script using the tempfile module.
When launching the script, pass the path you just made as an argument.
When the script starts, have it create a file at the path. When done, delete the file.
The launching script should watch for the creation and deletion of that file to indicate the status of the script.
Hopefully that makes sense.
You can use Popen.communicate API.
p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
sStdout, sStdErr = p1.communicate()
The command
Popen.communicate(input=None, timeout=None)
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for the process to terminate.
subprocess.run will by default wait for the process to finish.
Use subprocess.Popen and Popen.wait:
process = subprocess.Popen(['D:/Test/run-my-script.sh'],shell=True, executable="/bin/bash")
process.wait()
You could also use check_call() instead of Popen.
You can use os.system, like this:
import bpy
import os
os.system("sh "+os.path.abspath('D:/Test/run-my-script.sh'))
print('Hello')
There are apparently cases when the run command fails.
This is my workaround:
def check_has_finished(pfi, interval=1, timeout=100):
if os.path.exists(pfi):
if pfi.endswith('.nii.gz'):
mustend = time.time() + timeout
while time.time() < mustend:
try:
# Command is an ad hoc one to check if the process has finished.
subprocess.check_output('command {}'.format(pfi), shell=True)
except subprocess.CalledProcessError:
print "Caught CalledProcessError"
else:
return True
time.sleep(interval)
msg = 'command {0} not working after {1} tests. \n'.format(pfi, timeout)
raise IOError(msg)
else:
return True
else:
msg = '{} does not exist!'.format(pfi)
raise IOError(msg)
A wild try, but are you running the shell as Admin while Blender as regular user or vice versa?
Long story short (very short), Windows UAC is a sort of isolated environment between admin and regular user, so random quirks like this can happen. Unfortunately I can't remember the source of this, the closest I found is this.
My problem was the exact opposite of yours, the wait() got stuck in a infinite loop because my python REPL was fired from an admin shell and wasn't able to read the state of the regular user subprocess. Reverting to normal user shell got it fixed. It's not the first time I'm bit from this UAC snafu.
Environment: Raspberry Pi Wheezy
I have a python program that uses Popen to call another python program
from subprocess import *
oJob = Popen('sudo python mypgm.py',shell=True)
Another menu option is supposed to end the job immediately
oJob.kill()
but the job is still running??
When you add the option shell=True, python launches a shell and the shell in turn launches the process python mymgm.py. You are killing the shell process here which doesn't kill its own child that runs mymgm.py.
To ensure, that child process gets killed on oJob.kill, you need to group them all under one process group and make shell process, the group leader.
The code is,
import os
import signal
import subprocess
# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
os.killpg(pro.pid, signal.SIGTERM) # Send the signal to all the process groups
When you send SIGTERM signal to the shell process, it will kill all its child process as well.
You need to add a creation flag arg
oJob = Popen('sudo python mypgm.py',shell=True, creationflags = subprocess.CREATE_NEW_PROCESS_GROUP)
source
subprocess.CREATE_NEW_PROCESS_GROUP
A Popen creationflags parameter to specify that a new process group will be created. This flag is necessary for using os.kill() on the subprocess.
EDIT I agree with the comment on how to import stuff and why you are getting something is undefined. Also the other answer seems to be on the right track getting the pid
import subprocess as sub
oJob = sub.Popen('sudo python mypgm.py', creationflags = sub.CREATE_NEW_PROCESS_GROUP)
oJob.kill()
Warning Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input:
I am trying to compile a set of lines and execute them and append the output to text file. Instead of writing the same thing, I used a python script to compile and execute in background.
import subprocess
subprocess.call(["ifort","-openmp","mod1.f90","mod2.f90","pgm.f90","-o","op.o"])
subprocess.call(["nohup","./op.o",">","myout.txt","&"])
The program pgm.f90 is getting compliled using the ifort compiler, but the ouput is not getting appended to myout.txt. Instead it is appending output to nohup.out and the program is not running in the background even after specifying "&" in the python script.
What obvious error have I made here?
Thanks in advance
You can call a subprocess as if you were in the shell by using Popen() with the argument shell=True:
subprocess.Popen("nohup ./op.o > myout.txt &", shell=True)
This issue is that when you supply arguments as a list of elements, the subprocess library bypasses the shell and uses the exec syscall to directly run your program (in your case, "nohup"). Thus, rather than the ">" and "&" operators being interpreted by the shell to redirect your output and run in the background, they are being passed as literal arguments to the nohup command.
You can tell subprocess to execute your command via the shell, but this starts a whole extra instance of shell and can be wasteful. For a workaround, use the built-in redirection functionality in subprocess instead of using the shell primitives:
p = subprocess.Popen(['nohup', "./op.o"],
stdout=open('myout.txt', 'w'))
# process is now running in the background.
# if you want to wait for it to finish, use:
p.wait()
# or investigate p.poll() if you want to check to see if
# your process is still running.
For more information: http://docs.python.org/2/library/subprocess.html