We Keep Coding

I am using subprocess RUN to exceute multiple shell scripts as per my need.Why its is getting Deadlock/stuck

This is my function which executes the scripts.
def script_execute(args,row):
"""
Executes the script using suprocess.run by using the given arguments in variable args.
"""
retry=2
att=0
timeout=600
print(args)
while att < retry:
args = [str(x) if x!=None else '' for x in args]
try:
result = run(args,stdout=DEVNULL,stderr=DEVNULL,timeout=timeout)
print(result.returncode)
if result.returncode:
current_time = str(datetime.now())
error_string ="script execution failed for arguments: " + str(result.args) +"\n"+ result.stderr
error_list = [(current_time,row['sourcepath'],row['bigdataid'],"failed",error_string,current_time,"PushDistributor")]
insert_log(error_list)
return False
else:
return True
except subprocess.TimeoutExpired:
continue
else:
break
finally:
att +=1
timeout +=300
So what I have done is introduced timeout which is set 600 secs or 10 mins initially. If the subprocess takes longer time than that it raises Timeout exception and in finally block 300 sec is added to initial timeout for 1 more retry.
I dont want to use timeout so is there any other sleek way of doing it without the use of timeout and why subprocess gets stuck or deadlock even if have set stdout and stderr as DEVNULL.
Need HELP with this.
Thanks in Advance.

Python multiprocessing - check status of each processes

I wonder if it is possible to check how long of each processes take.
for example, there are four workers and the job should take no more than 10 seconds, but one of worker take more than 10 seconds.Is there way to raise a alert after 10 seconds and before process finish the job.
My initial thought is using manager, but it seems I have wait till process finished.
Many thanks.

You can check whether process is alive after you tried to join it. Don't forget to set timeout otherwise it'll wait until job is finished.
Here is simple example for you
from multiprocessing import Process
import time
def task():
import time
time.sleep(5)
procs = []
for x in range(2):
proc = Process(target=task)
procs.append(proc)
proc.start()
time.sleep(2)
for proc in procs:
proc.join(timeout=0)
if proc.is_alive():
print "Job is not finished!"

I have found this solution time ago (somewhere here in StackOverflow) and I am very happy with it.
Basically, it uses signal to raise an exception if a process takes more than expected.
All you need to do is to add this class to your code:
import signal
class Timeout:
def __init__(self, seconds=1, error_message='TimeoutError'):
self.seconds = seconds
self.error_message = error_message
def handle_timeout(self, signum, frame):
raise TimeoutError(self.error_message)
def __enter__(self):
signal.signal(signal.SIGALRM, self.handle_timeout)
signal.alarm(self.seconds)
def __exit__(self, type, value, traceback):
signal.alarm(0)
Here is a general example of how it works:
import time
with Timeout(seconds=3, error_message='JobX took too much time'):
try:
time.sleep(10) #your job
except TimeoutError as e:
print(e)
In your case, I would add the with statement to the job that your worker need to perform. Then you catch the Exception and you do what you think is best.
Alternatively, you can periodically check if a process is alive:
timeout = 3 #seconds
start = time.time()
while time.time() - start < timeout:
if any(proces.is_alive() for proces in processes):
time.sleep(1)
else:
print('All processes done')
else:
print("Timeout!")
# do something

Use Pipe and messages
from multiprocessing import Process, Pipe
import numpy as np
caller, worker = Pipe()
val1 = ['der', 'die', 'das']
def worker_function(info):
print (info.recv())
for i in range(10):
print (val1[np.random.choice(3, 1)[0]])
info.send(['job finished'])
info.close()
def request(data):
caller.send(data)
task = Process(target=worker_function, args=(worker,))
if not task.is_alive():
print ("task is requested")
task.start()
if caller.recv() == ['job finished']:
task.join()
print ("finished")
if __name__ == '__main__':
data = {'input': 'here'}
request(data)

Kill Subprocess in Python after timeout

I searchedon internet and learned other method of implementing it
the problem now i have found out. my execution time always becomes more than that
of time out if i write stdout=subprocess.PIPE in subprocess.Popen. If i am removing it then it is taking normal execution time
import subprocess, datetime, os, time, signal
//setting time for timeout
timeout=3
start = datetime.datetime.now()
process = subprocess.Popen(["python", "/home/bourne/untitled.py"],shell=False, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
while process.poll() is None:
time.sleep(0.1)
now = datetime.datetime.now()
if (now - start).seconds > timeout:
os.kill(process.pid, signal.SIGKILL)
os.waitpid(-1, os.WNOHANG)
print "error"
print (now - start).seconds
break
print (now - start).seconds

You shouldn't spawn a new thread just for having it time out in 5 seconds and then use it's isAlive status as break condition for a busy wait. You don't need an extra thread for that, you can messure the time in the first thread.
Instead of polling the thread as often as you can, you should use a delay (time.sleep) to allow the processor to do some real work.
And you should know that if your process is generating a lot of output, it will block if you don't read it while the process is executing and let it fill up the pipe's buffer.

Thread can be handled in python VM, but process not.
so u have to use OS api to kill ur process/subprocess, such as (in linux):
os.system("kill -9 %s"%(proc.pid))
and, using thread for timing is a bad idea. how about:
start_t = time.time()
TIME_END, TIME_SLEEP = 5, 1
while time.time() - start_t < TIME_END:
if proc.poll():
break
time.sleep(TIME_SLEEP)

I have successfully solved the problem. the solution is
import subprocess, signal, os, threading, errno
from contextlib import contextmanager
class TimeoutThread(object):
def __init__(self, seconds):
self.seconds = seconds
self.cond = threading.Condition()
self.cancelled = False
self.thread = threading.Thread(target=self._wait)
def run(self):
"""Begin the timeout."""
self.thread.start()
def _wait(self):
with self.cond:
self.cond.wait(self.seconds)
if not self.cancelled:
self.timed_out()
def cancel(self):
"""Cancel the timeout, if it hasn't yet occured."""
with self.cond:
self.cancelled = True
self.cond.notify()
self.thread.join()
def timed_out(self):
"""The timeout has expired."""
raise NotImplementedError
class KillProcessThread(TimeoutThread):
def __init__(self, seconds, pid):
super(KillProcessThread, self).__init__(seconds)
self.pid = pid
def timed_out(self):
try:
os.kill(self.pid, signal.SIGKILL) // this is for linux you need to change it for windows
except OSError,e:
# If the process is already gone, ignore the error.
if e.errno not in (errno.EPERM, errno. ESRCH):
raise e
#contextmanager
def processTimeout(seconds, pid):
timeout = KillProcessThread(seconds, pid)
timeout.run()
try:
yield
finally:
timeout.cancel()
def example(cmd):
proc = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
//setting the timeout to be 1 sec
with processTimeout(1, proc.pid):
stdout,stderr=proc.communicate()
resultcode = proc.wait()
if resultcode < 0:
#print "error: %i" % resultcode
return resultcode,0
else:
return stdout,stderr
//This is used to create new subprocess and it will return output as well as error
output,err=example(["python",filepath,"5"])

Wait at most N seconds for a fork'd child PID to finish

I'm trying to use os.fork, os.waitpid, and threading to dispatch some time-consuming work to a child process, wait at most a certain amount of time for it to complete, and then continue on, leaving it running in the background. There is also a secondary timer in the child that prevents it from running for too long in the background, like so:
Fork execution
In the fork's child:
Start a thread to execute a task
If that thread runs for longer than X milliseconds:
thread.stop(), a custom method that softly stops the thread.
In the fork's parent:
If the child's PID stays running for longer than Y milliseconds:
return -1
else:
return 0
After returning a value, I want the script to terminate. The child can and should keep running if it isn't done.
The code I have tried (abbreviated) is:
class MyCustomThread(threading.Thread):
abort = False
def run(self):
counter = 0
while True:
if self.abort: break
counter += 1
if counter == 30: self.stop()
sleep(1)
return 0
def stop(self):
self.abort = True
def test():
X = 2000
Y = 500
pid = os.fork()
if pid == 0:
thread1 = MyCustomThread() #Sleeps for 30 seconds and ends.
thread1.start()
print "Started 1!"
timeout = X # say, 1000ms
while timeout > 0:
if not thread1.is_alive(): return "custom thread finished before the deadline!"
timeout -= 1
sleep(0.001)
if thread1.is_alive():
return "custom thread didn't finish before the deadline!"
thread1.stop()
exit()
else:
print pid
thread2 = Thread(target = os.waitpid, args = (pid, 0))
thread2.start()
print "Started 2!"
timeout2 = Y # say, 500ms
while timeout2 > 0:
if not thread2.is_alive(): return "child PID finished!"
timeout2 -= 1
sleep(0.001)
if thread2.is_alive():
return "child PID didn't finish yet!"
print test()
print "all done!"
The output is correct, in that I get
1171
Started 2!
Started 1!
child PID didn't finish yet!
all done!
custom thread didn't finish before the deadline!
all done!
But then the script doesn't exit! It sleeps for the remaining 28 seconds before
How do I make the main execution of this script complete after the forked parent returns a value? As I said before, the child can and should keep running in the background, it just shouldn't block execution of the next task on the parent.
I really don't even care if the child can print output to standard out--in the actual implementation, all it's doing is talking to a database, so it doesn't need to report anything interactively. The parent, however, needs to dispatch the child, wait at most Y seconds for the child to finish, and then (as far as whoever invoked the script is concerned) end the script's execution so that the next thing can be done. The other timer (X) isn't relevant, I think; it only exists to keep the child from running too long in the background.
Any ideas? I'm probably approaching this totally wrong, so "start over and do it _ way" ideas are welcome.

Try this one, it doesn't use threading, just pure fork/waitpid/alarm/sighandler:
child_exited = False
def sigh(signum, frame):
global child_exited
if signum == signal.SIGALRM:
print "custom thread didn't finish before the deadline!"
#forced exit:
exit(1)
if signum == signal.SIGCHLD:
(pid, status) = os.waitpid(-1, 0)
print "child exited with status: " + str(os.WEXITSTATUS(status))
child_exited = True
def test():
global child_exited
pid = os.fork()
if pid == 0:
signal.signal(signal.SIGALRM, sigh)
signal.alarm(30)
#do work:
print "Started 1"
time.sleep(60)
#clean exit:
exit(0)
elif (pid > 0):
signal.signal(signal.SIGCHLD, sigh)
print "Started 2"
#this sleep will return prematurely if the child exits
time.sleep(10)
if not child_exited:
print "child PID didn't finish yet!"
else:
print "fork() failed"
print test()
print "all done!"

This is not an exact answer to your question, but rather a "start over and do it _ way" idea.
You could use the multiprocessing module. The function Pool.apply_async() allows to execute a function in the background, and the returned AsyncResult object features wait() and get() methods with a timeout parameter.
Your code would essentially become (untested)
import multiprocessing
def computationally_intensive():
# whatever
p = multiprocessing.Pool(1)
deferred = p.apply_async(computationally_intensive)
try:
result = deferred.get(10) # 10 being the timeout
except multiprocessing.TimeoutError:
# Handle time out
# ...
p.terminate()

Figured it out, thanks for all the pointers!
Edit: fixed is_pid_alive function to work if called from within a child as well.
The issue was that the parent's "watcher" thread was never completing, since os.waitpid isn't a pollable/loopable function. The solution was to remove the "watcher" thread, and instead implement a polling loop that checks a pid_is_alive() function every millisecond, like so:
def pid_is_alive(pid):
try:
os.kill(pid, 0)
os.waitpid(pid, os.WNOHANG)
os.kill(pid, 0)
except OSError:
return False
return True
def test():
X = 1000 * 1000
Y = 5000
pid = os.fork()
if pid == 0:
thread1 = MyCustomThread() #Sleeps for 30 seconds and ends.
thread1.start()
print "Started 1!"
timeout = X # say, 1000ms
while timeout > 0:
if not thread1.is_alive(): return "custom thread finished before the deadline!"
timeout -= 1
sleep(0.001)
if thread1.is_alive():
return "custom thread didn't finish before the deadline!"
thread1.stop()
exit()
else:
timeout2 = Y # say, 500ms
while timeout2 > 0:
if not pid_is_alive(pid): return "child PID finished!"
timeout2 -= 1
sleep(0.001)
if pid_is_alive(pid):
print "child PID didn't finish yet!"
exit()
print test()
print "all done!"

you may use join method of Thread
thread2.join(timeout=Y)
if not thread2.is_alive():
# TODO: Post Processing

kill a function after a certain time in windows

I've read a lot of posts about using threads, subprocesses, etc.. A lot of it seems over complicated for what I'm trying to do...
All I want to do is stop executing a function after X amount of time has elapsed.
def big_loop(bob):
x = bob
start = time.time()
while True:
print time.time()-start
This function is an endless loop that never throws any errors or exceptions, period.
I"m not sure the difference between "commands, shells, subprocesses, threads, etc.." and this function, which is why I'm having trouble manipulating subprocesses.
I found this code here, and tried it but as you can see it keeps printing after 10 seconds have elapsed:
import time
import threading
import subprocess as sub
import time
class RunCmd(threading.Thread):
def __init__(self, cmd, timeout):
threading.Thread.__init__(self)
self.cmd = cmd
self.timeout = timeout
def run(self):
self.p = sub.Popen(self.cmd)
self.p.wait()
def Run(self):
self.start()
self.join(self.timeout)
if self.is_alive():
self.p.terminate()
self.join()
def big_loop(bob):
x = bob
start = time.time()
while True:
print time.time()-start
RunCmd(big_loop('jimijojo'), 10).Run() #supposed to quit after 10 seconds, but doesn't
x = raw_input('DONEEEEEEEEEEEE')
What's a simple way this function can be killed. As you can see in my attempt above, it doesn't terminate after 20 seconds and just keeps on going...
***OH also, I've read about using signal, but I"m on windows so I can't use the alarm feature.. (python 2.7)
**assume the "infinitely running function" can't be manipulated or changed to be non-infinite, if I could change the function, well I'd just change it to be non infinite wouldn't I?
Here are some similar questions, which I haven't able to port over their code to work with my simple function:
Perhaps you can?
Python: kill or terminate subprocess when timeout
signal.alarm replacement in Windows [Python]
Ok I tried an answer I received, it works.. but how can I use it if I remove the if __name__ == "__main__": statement? When I remove this statement, the loop never ends as it did before..
import multiprocessing
import Queue
import time
def infinite_loop_function(bob):
var = bob
start = time.time()
while True:
time.sleep(1)
print time.time()-start
print 'this statement will never print'
def wrapper(queue, bob):
result = infinite_loop_function(bob)
queue.put(result)
queue.close()
#if __name__ == "__main__":
queue = multiprocessing.Queue(1) # Maximum size is 1
proc = multiprocessing.Process(target=wrapper, args=(queue, 'var'))
proc.start()
# Wait for TIMEOUT seconds
try:
timeout = 10
result = queue.get(True, timeout)
except Queue.Empty:
# Deal with lack of data somehow
result = None
finally:
proc.terminate()
print 'running other code, now that that infinite loop has been defeated!'
print 'bla bla bla'
x = raw_input('done')

Use the building blocks in the multiprocessing module:
import multiprocessing
import Queue
TIMEOUT = 5
def big_loop(bob):
import time
time.sleep(4)
return bob*2
def wrapper(queue, bob):
result = big_loop(bob)
queue.put(result)
queue.close()
def run_loop_with_timeout():
bob = 21 # Whatever sensible value you need
queue = multiprocessing.Queue(1) # Maximum size is 1
proc = multiprocessing.Process(target=wrapper, args=(queue, bob))
proc.start()
# Wait for TIMEOUT seconds
try:
result = queue.get(True, TIMEOUT)
except Queue.Empty:
# Deal with lack of data somehow
result = None
finally:
proc.terminate()
# Process data here, not in try block above, otherwise your process keeps running
print result
if __name__ == "__main__":
run_loop_with_timeout()
You could also accomplish this with a Pipe/Connection pair, but I'm not familiar with their API. Change the sleep time or TIMEOUT to check the behaviour for either case.

There is no straightforward way to kill a function after a certain amount of time without running the function in a separate process. A better approach would probably be to rewrite the function so that it returns after a specified time:
import time
def big_loop(bob, timeout):
x = bob
start = time.time()
end = start + timeout
while time.time() < end:
print time.time() - start
# Do more stuff here as needed

Can't you just return from the loop?
start = time.time()
endt = start + 30
while True:
now = time.time()
if now > endt:
return
else:
print end - start

import os,signal,time
cpid = os.fork()
if cpid == 0:
while True:
# do stuff
else:
time.sleep(10)
os.kill(cpid, signal.SIGKILL)
You can also check in the loop of a thread for an event, which is more portable and flexible as it allows other reactions than brute killing. However, this approach fails if # do stuff can take time (or even wait forever on some event).