I have a shell script which I need to invoke in the Python program.
from subprocess import *
p=Popen('some_script',stdin=PIPE)
p.communicate('S')
But after the communicate() command is executed the process goes for a deadlock. The shell script has not been written by me and I cannot modify it. I just need to kill the process after the communicate() method is executed, but the program should not exit.
Please help me in this regard.
The entire point of Popen.communicate is that it will wait until the process terminates. If this is not desired behavior, you must explicitly interact with the process' stdin/stdout/stderr.
import subprocess
p = subprocess.Popen('some_script', stdin=subprocess.PIPE)
p.stdin.write('S\n')
p.kill()
Related
I'm trying to use this answer to issue a long-running process with nohup using subprocess.Popen, then exit the connection.
From the login shell, if I run
$ nohup sleep 20 &
$ exit
I get the expected behavior, namely that I can log out while the sleep process is still running, and connect back later to check on its status. If I try to the same through Python, however, it seems as though the exit command does not get executed until the sleeping time is over.
import subprocess
HOST=<host>
sshProcess = subprocess.Popen(['ssh', HOST],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
universal_newlines = True,
bufsize = 0)
sshProcess.stdin.write("nohup sleep 20 &\n")
sshProcess.stdin.write("exit\n")
sshProcess.stdin.close()
What am I missing?
From the docs: Python Docs
Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
Alright pretty sure i got it now:
About communicate(): [...]Wait for process to terminate
So i guess your earlier Solution was better. But if you call it at the end if shouldn't be a problem or just don't call it at all if you don't need stdin or stderr as output
However according to this:StackOverflow Comment If you set preexec_fn=os.setpgrp in your Popen call it should work.
I'm experiencing an issue where a call to proc.communicate() still hangs even after calling proc.terminate().
I create tasks to be run in the background with the call
import subprocess as sub
p = sub.Popen(command, stdout=sub.PIPE, stderr=sub.PIPE, shell=False)
At the completion of the script, I call terminate(), communicate(), and wait() to gather information from the process.
p.terminate()
errorcode = p.wait()
(pout, perr) = p.communicate()
The script hangs at the call to communicate. I'd assumed that any call to communicate that follows a call to terminate would return immediately. Is there any reason why this would fail?
Edit: I'm using this method because the command is really a tight loop that won't terminate on its own. I'd like to use p.terminate() to do that, and then see what the stdout and stderr has to offer.
You don't need the first two statements. Just call communicate().
Environment: Raspberry Pi Wheezy
I have a python program that uses Popen to call another python program
from subprocess import *
oJob = Popen('sudo python mypgm.py',shell=True)
Another menu option is supposed to end the job immediately
oJob.kill()
but the job is still running??
When you add the option shell=True, python launches a shell and the shell in turn launches the process python mymgm.py. You are killing the shell process here which doesn't kill its own child that runs mymgm.py.
To ensure, that child process gets killed on oJob.kill, you need to group them all under one process group and make shell process, the group leader.
The code is,
import os
import signal
import subprocess
# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
os.killpg(pro.pid, signal.SIGTERM) # Send the signal to all the process groups
When you send SIGTERM signal to the shell process, it will kill all its child process as well.
You need to add a creation flag arg
oJob = Popen('sudo python mypgm.py',shell=True, creationflags = subprocess.CREATE_NEW_PROCESS_GROUP)
source
subprocess.CREATE_NEW_PROCESS_GROUP
A Popen creationflags parameter to specify that a new process group will be created. This flag is necessary for using os.kill() on the subprocess.
EDIT I agree with the comment on how to import stuff and why you are getting something is undefined. Also the other answer seems to be on the right track getting the pid
import subprocess as sub
oJob = sub.Popen('sudo python mypgm.py', creationflags = sub.CREATE_NEW_PROCESS_GROUP)
oJob.kill()
Warning Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input:
I need to run a subprocess from my script. The subprocess is an interactive (shell-like) application, to which I issue commands through the subprocess' stdin.
After I issue a command, the subprocess outputs the result to stdout and then waits for the next command (but does not terminate).
For example:
from subprocess import Popen, PIPE
p = Popen(args = [...], stdin = PIPE, stdout = PIPE, stderr = PIPE, shell = False)
# Issue a command:
p.stdin.write('command\n')
# *** HERE: get the result from p.stdout ***
# CONTINUE with the rest of the script once there is not more data in p.stdout
# NOTE that the subprocess is still running and waiting for the next command
# through stdin.
My problem is getting the result from p.stdout. The script needs to get the output while there is new data in p.stdout; but once there is no more data, I want to continue with the script.
The subprocess does not terminate, so I cannot use communicate() (which waits for the process to terminate).
I tried reading from p.stdout after issuing the command, like this:
res = p.stdout.read()
But the subprocess is not fast enough, and I just get empty result.
I thought about polling p.stdout in a loop until I get something, but then how do I know I got everything? And it seems wasteful anyway.
Any suggestions?
Use gevent.subprocess in gevent-1.0 to substitute the standard subprocess module. It could do the concurrency tasks using synchronous logic and won't block the script. Here is a brief tutorial about gevent.subprocess
Use circuits.io.Process in circuits-dev to wrap an asynchronous call to subprocess.
Example: https://bitbucket.org/circuits/circuits-dev/src/tip/examples/ping.py
After investigating several options I reached two solutions:
Setting the subprocess' stdout stream to be non blocking by using the fcntl module.
Using a thread to collect the subprocess' output to a proxy queue, and then reading the queue from the main thread.
I describe both solutions (and the problem and its origin) in this post.
Here's my main file:
import subprocess, time
pipe = subprocess.PIPE
popen = subprocess.Popen('pythonw -uB test_web_app.py', stdout=pipe)
time.sleep(3)
And here's test_web_app.py:
import web
class Handler:
def GET(self): pass
app = web.application(['/', 'Handler'], globals())
app.run()
When I run the main file, the program executes, but a zombie process is left hanging and I have to kill it manually. Why is this? How can I get the Popen to die when the program ends? The Popen only hangs if I pipe stdout and sleep for a bit before the program ends.
Edit -- here's the final, working version of the main file:
import subprocess, time, atexit
pipe = subprocess.PIPE
popen = subprocess.Popen('pythonw -uB test_web_app.py', stdout=pipe)
def kill_app():
popen.kill()
popen.wait()
atexit.register(kill_app)
time.sleep(3)
You have not waited for the process. Once it's done, you have to call popen.wait.
You can check if the process is terminated using the poll method of the popen object to see if it has completed.
If you don't need the stdout of the web server process, you can simply ignore the stdout option.
You can use the atexit module to implement a hook that gets called when your main file exits. This should use the kill method of the Popen object and then wait on it to make sure that it's terminated.
If your main script doesn't need to be doing anything else while the subprocess executes I'd do:
import subprocess, time
pipe = subprocess.PIPE
popen = subprocess.Popen('pythonw -uB test_web_app.py', stdout=pipe, stderr=pipe)
out, err = popen.communicate()
(I think if you specifically pipe stdout back to your program, you need to read it at some point to avoid creating zombie processes - communicate will read it in a reasonably safe way).
Or if you don't care about parsing stdout / stderr, don't bother piping them:
popen = subprocess.Popen('pythonw -uB test_web_app.py')
popen.communicate()