I am trying to execute a batch script through python and seems like subprocess.Popen is executing the command and it's getting stuck there on terminal and not printing any output though copy completed. Can you help on this.
ps_copy_command = "call copyfiles.cmd"
process=subprocess.Popen(["cmd", "/C", ps_copy_command, password], stdout=subprocess.PIPE, stderr=subprocess.PIPE);
process.wait()
print("\tCopy completed ")
output = process.stdout.read()
print (output)
If the code is stuked in output = process.stdout.read(), something similar happened to me some time ago. The problem is that process.stdout.read() wont return something until stdout has something in it. Example:
Imagine your batch script is doing some task that takes 10 seconds and then print 'Done!'
python is going to wait until 'Done!' returned by the script.
They way i solved this problems is by adding Threads. One thread is reading the stdout and the other is waiting n seconds, then i join the waiting thread and if the stdout thread has something in it, i print it
import time
import threading
def wait_thread():
seconds = 0
while seconds < 2:
time.sleep(1)
seconds += 1
return True
def stdout_thread():
global output
output = process.stdout.read()
output=None
t1 = threading.Thread(target=wait_thread)
t2 = threading.Thread(target=stdout_thread)
t1.start()
t2.start()
t1.join() # wait until waiting_thread is end
if output:
print(output)
else:
print("No output")
I am using subprocess.run to address this.
process=subprocess.run(["cmd", "/C", ps_copy_command, password], stdout=subprocess.PIPE);
print(process.returncode)
print(process.stdout)
Related
I am working on a python program which implements the cmd window.
I am using subproccess with PIPE.
If for example i write "dir" (by stdout), I use communicate() in order to get the response from the cmd and it does work.
The problem is that in a while True loop, this doesn't work more than one time, it seems like the subprocess closes itself..
Help me please
import subprocess
process = subprocess.Popen('cmd.exe', shell=False, stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=None)
x=""
while x!="x":
x = raw_input("insert a command \n")
process.stdin.write(x+"\n")
o,e=process.communicate()
print o
process.stdin.close()
The main problem is that trying to read subprocess.PIPE deadlocks when the program is still running but there is nothing to read from stdout. communicate() manually terminates the process to stop this.
A solution would be to put the piece of code that reads stdout in another thread, and then access it via Queue, which allows for reliable sharing of data between threads by timing out instead of deadlocking.
The new thread will read standard out continuously, stopping when there is no more data.
Each line will be grabbed from the queue stream until a timeout is reached(no more data in Queue), then the list of lines will be displayed to the screen.
This process will work for non-interactive programs
import subprocess
import threading
import Queue
def read_stdout(stdout, queue):
while True:
queue.put(stdout.readline()) #This hangs when there is no IO
process = subprocess.Popen('cmd.exe', shell=False, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
q = Queue.Queue()
t = threading.Thread(target=read_stdout, args=(process.stdout, q))
t.daemon = True # t stops when the main thread stops
t.start()
while True:
x = raw_input("insert a command \n")
if x == "x":
break
process.stdin.write(x + "\n")
o = []
try:
while True:
o.append(q.get(timeout=.1))
except Queue.Empty:
print ''.join(o)
I have a python system call that takes a while to finish.
os.system("call_that_takes_quite_some_time")
In the meanwhile I want to keep throwing a message that says "waiting..." every now and then till the os.system returns 0 or an error.
/
How do I do this? Is there something in python that I can "listen" to in a while loop?
os.system waits till your command execution is complete.
use subprocess.Popen you can check output or error. Popen gives handle and you can check return code using wait to find out command is successful/failure. For ex:
proc = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while proc.poll() is None:
print proc.stdout.readline() #give output from your execution/your own message
self.commandResult = proc.wait() #catch return code
You can use threading
import os
import time
import threading
def waiter():
waiter.finished = False
while not waiter.finished:
print 'Waiting...'
time.sleep(1)
os_thread = threading.Thread(target=waiter)
os_thread.daemon = True
os_thread.start()
return_value = os.system('sleep 4.9')
return_value >>= 8 # The return code is specified in the second byte
waiter.finished = True
time.sleep(3)
print 'The return value is', return_value
This will print "Waiting..." message every 1 second, and it stops after waiter.finished is set to True (in this case there will be 5 "Waiting..." messages)
But os.system is not recommended. The documentation recommends using subprocess module.
Take a look at this simple python code with Process:
from multiprocessing import Process
import time
def f(name):
time.sleep(100)
print 'hello', name
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()#Has to be terminated in 5 seconds
#p.join()
print "This Needs to be Printed Immediately"
I guess I am looking for a function like p.start(timeout).
I want to terminate the p process if it has not self-finished in like 5 seconds. How can I do that? There seems to be no such function.
If p.join() is uncommented, the following print line will have to wait 100 seconds and can not be 'Printed Immediately'.But I want it be done immediately so the p.join() has to be commented out.
Use a separate thread to start the process, wait 5 seconds, then terminate the process. Meanwhile the main thread can do the work you want to happen immediately:
from multiprocessing import Process
import time
import threading
def f(name):
time.sleep(100)
print 'hello', name
def run_process_with_timeout(timeout, target, args):
p = Process(target=target, args=args)
p.start()
time.sleep(timeout)
p.terminate()
if __name__ == '__main__':
t = threading.Thread(target=run_process_with_timeout, args=(5,f,('bob',)))
t.start()
print "This Needs to be Printed Immediately"
You might want to take a look at that SO thread.
basically their solution is to use the timeout capability of the threading module by running the process in a separate thread.
You are right, there is no such function in Python 2.x in the subprocess library.
However, with Python 3.3 you can use:
p = subprocess.Popen(...)
try:
p.wait(timeout=5)
except TimeoutError:
p.kill()
With older Python versions, you would have to write a loop that calls p.poll() and checks the returncode, e.g. once per second.
This is (like polling in general) not optimal from performance point-of-view, but it always depends on what you expect.
Try something like this:
def run_process_with_timeout(timeout, target, args):
p = Process(target=target, args=args)
running = False
second = int(time.strftime("%S"))
if second+timeout > 59:
second = (second+timeout)-60
else:
second = second+timeout
print second
while second > int(time.strftime("%S")):
if running == False:
p.start()
running = True
p.terminate()
basically just using the time module to allow a loop to run for five seconds and then moving on, this assumes timeout is given in seconds.
Though I'd point out that if this was used with the code the OP originally posted, this would work, as print was in a second function separate from the loop and would be carried out immediately after calling this function.
Why not use the timeout option of Process.join(), as in:
import sys
...
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()#Has to be terminated in 5 seconds
# print immediately and flush output
print "This Needs to be Printed Immediately"
sys.stdout.flush()
p.join(5)
if p.is_alive():
p.terminate()
I'd like to start a process, wait 2 seconds and print out whatever is in the stderr and stdout pipes so far and then exit. Here is the code I have so far and it doesn't seem to work as hoped. What am I doing wrong?
There are 3 issues:
The program as it stands prints out "done" then waits for the
suprocess to complete before printing out the first line.
As it stands, the script reads one line. How to read to the end of the current buffer?
will the subprocess exit if the calling script exits? If so, how should I modify the
function call so that the subprocess runs to completion even if the
calling script exits?
cmdStr = "./stepper.py"
proc = subprocess.Popen(cmdStr, shell=True, bufsize=-1, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
print "polling"
time.sleep(2)
print "done"
print proc.stdout.readline()
Here is what the stepper.py looks like:
out = open("stepper.log", 'w')
for idx in range(3):
time.sleep(2)
print "Idx",idx
sys.stdout.flush()
out.write("%d\n"%(idx))
print "fnished"
out.write("cloing\n")
out.close()
I run a subprocess using:
p = subprocess.Popen("subprocess",
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
This subprocess could either exit immediately with an error on stderr, or keep running. I want to detect either of these conditions - the latter by waiting for several seconds.
I tried this:
SECONDS_TO_WAIT = 10
select.select([],
[p.stdout, p.stderr],
[p.stdout, p.stderr],
SECONDS_TO_WAIT)
but it just returns:
([],[],[])
on either condition. What can I do?
Have you tried using the Popen.Poll() method. You could just do this:
p = subprocess.Popen("subprocess",
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
time.sleep(SECONDS_TO_WAIT)
retcode = p.poll()
if retcode is not None:
# process has terminated
This will cause you to always wait 10 seconds, but if the failure case is rare this would be amortized over all the success cases.
Edit:
How about:
t_nought = time.time()
seconds_passed = 0
while(p.poll() is not None and seconds_passed < 10):
seconds_passed = time.time() - t_nought
if seconds_passed >= 10:
#TIMED OUT
This has the ugliness of being a busy wait, but I think it accomplishes what you want.
Additionally looking at the select call documentation again I think you may want to change it as follows:
SECONDS_TO_WAIT = 10
select.select([p.stderr],
[],
[p.stdout, p.stderr],
SECONDS_TO_WAIT)
Since you would typically want to read from stderr, you want to know when it has something available to read (ie the failure case).
I hope this helps.
This is what i came up with. Works when you need and don't need to timeout on thep process, but with a semi-busy loop.
def runCmd(cmd, timeout=None):
'''
Will execute a command, read the output and return it back.
#param cmd: command to execute
#param timeout: process timeout in seconds
#return: a tuple of three: first stdout, then stderr, then exit code
#raise OSError: on missing command or if a timeout was reached
'''
ph_out = None # process output
ph_err = None # stderr
ph_ret = None # return code
p = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# if timeout is not set wait for process to complete
if not timeout:
ph_ret = p.wait()
else:
fin_time = time.time() + timeout
while p.poll() == None and fin_time > time.time():
time.sleep(1)
# if timeout reached, raise an exception
if fin_time < time.time():
# starting 2.6 subprocess has a kill() method which is preferable
# p.kill()
os.kill(p.pid, signal.SIGKILL)
raise OSError("Process timeout has been reached")
ph_ret = p.returncode
ph_out, ph_err = p.communicate()
return (ph_out, ph_err, ph_ret)
Here is a nice example:
from threading import Timer
from subprocess import Popen, PIPE
proc = Popen("ping 127.0.0.1", shell=True)
t = Timer(60, proc.kill)
t.start()
proc.wait()
Using select and sleeping doesn't really make much sense. select (or any kernel polling mechanism) is inherently useful for asynchronous programming, but your example is synchronous. So either rewrite your code to use the normal blocking fashion or consider using Twisted:
from twisted.internet.utils import getProcessOutputAndValue
from twisted.internet import reactor
def stop(r):
reactor.stop()
def eb(reason):
reason.printTraceback()
def cb(result):
stdout, stderr, exitcode = result
# do something
getProcessOutputAndValue('/bin/someproc', []
).addCallback(cb).addErrback(eb).addBoth(stop)
reactor.run()
Incidentally, there is a safer way of doing this with Twisted by writing your own ProcessProtocol:
http://twistedmatrix.com/projects/core/documentation/howto/process.html
Python 3.3
import subprocess as sp
try:
sp.check_call(["/subprocess"], timeout=10,
stdin=sp.DEVNULL, stdout=sp.DEVNULL, stderr=sp.DEVNULL)
except sp.TimeoutError:
# timeout (the subprocess is killed at this point)
except sp.CalledProcessError:
# subprocess failed before timeout
else:
# subprocess ended successfully before timeout
See TimeoutExpired docs.
If, as you said in the comments above, you're just tweaking the output each time and re-running the command, would something like the following work?
from threading import Timer
import subprocess
WAIT_TIME = 10.0
def check_cmd(cmd):
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
def _check():
if p.poll()!=0:
print cmd+" did not quit within the given time period."
# check whether the given process has exited WAIT_TIME
# seconds from now
Timer(WAIT_TIME, _check).start()
check_cmd('echo')
check_cmd('python')
The code above, when run, outputs:
python did not quit within the given time period.
The only downside of the above code that I can think of is the potentially overlapping processes as you keep running check_cmd.
This is a paraphrase on Evan's answer, but it takes into account the following :
Explicitly canceling the Timer object : if the Timer interval would be long and the process will exit by its "own will" , this could hang your script :(
There is an intrinsic race in the Timer approach (the timer attempt killing the process just after the process has died and this on Windows will raise an exception).
DEVNULL = open(os.devnull, "wb")
process = Popen("c:/myExe.exe", stdout=DEVNULL) # no need for stdout
def kill_process():
""" Kill process helper"""
try:
process.kill()
except OSError:
pass # Swallow the error
timer = Timer(timeout_in_sec, kill_process)
timer.start()
process.wait()
timer.cancel()