I want to replace Cron Jobs for "keeping" my program alive because it calls every XX interval whether or not the scrip is already called, creating duplicate entries.
I investigated the issue, and had a few approaches. One was to modify my program so it checks if it is already called and closes itself. The one I went after was to detach it completely from Cronjob by calling itself over and over again with execfile which works exactly how I want except the following problem:
RuntimeError: maximum recursion depth exceeded
Is there a way to keep the program in "infinite loop" without getting a Stack Overflow?
Here is my code, its a program that checks Mails, and converts them into MySQL DB entries.
imap = imaplib.IMAP4(hst)
try:
imap.login(usr, pwd)
except Exception as e:
errormsg = e
time.sleep(30)
print "IMAP error: " + str(errormsg)
execfile('/var/www/html/olotool/converter.py')
raise IOError(e)
# Authentification & Fetch Step
while True:
time.sleep(5)
'''
The script will always result in an error if there
are no mails left to check in the inbox. It then
goes into sleep mode and relaunches itself to check
if new mails have arrived.
'''
try:
imap.select("Inbox") # Tell Imap where to go
result, data = imap.uid('search', None, "ALL")
latest = data[0].split()[-1]
result, data = imap.uid('fetch', latest, '(RFC822)')
raw = data[0][1] # This contains the Mail Data
msg = email.message_from_string(raw)
except Exception as e:
disconnect(imap)
time.sleep(60)
execfile('/var/www/html/olotool/converter.py')
raise IOError(e)
I solved the problem myself with the only way I see it possible right now.
First I changed my exception in above code:
except Exception as e:
disconnect(imap)
print "Converter: No messages left"
raise os._exit(0)
# This is a special case since this Exception is
# no error thus os._exit(0) gives no false-positives
As you see I refrain from using execfile now. Instead I wrote a controller script that checks the status of my converter.py and launches it if it is not already running:
while True:
presL = os.popen('pgrep -lf python').read()
print "________________________________________"
print "Starting PIDcheck"
print "Current Processes: "
print presL # Check Processes
presRconverter = find('\d{7} python converter.py', presL)
if presRconverter:
# Store the PID
convPID = find('\d{7}', presRconverter)
print "Converter is running at PID: " + convPID
else:
print "PID Controller: Converter not running"
try:
print "PID Controller: Calling converter"
subprocess.check_call('python converter.py', shell=True)
except subprocess.CalledProcessError as e:
errormsg = e
print "Couldn't call Converter Module"
sendMail(esender,ereceiver,esubject,etext,server)
print "Error notification send"
raise IOError(e)
# If we got until here without ERROR, the call was Successfull
print "PID Controller: Call successful"
print "________________________________________"
time.sleep(60)
This method does not raise an: RuntimeError: maximum recursion depth exceeded. Also this provides you with a nohup.out file if you run the controller with command nohup python converter.py where you can see any problems for errorhandling.
I hope I could help anyone running into the same issue.
Something along the lines of this should work without having to resort to subprocess checking and such:
def check_mail_loop():
imap = imaplib.IMAP4(hst)
# Build some function to login, and, in the event of an error, sleep for n seconds and call login function again.
imap.login(usr, pwd)
while True:
try:
imap.select("Inbox")
result, data = imap.uid('search', None, "ALL")
if result and data:
latest = data[0].split()[-1]
result, data = imap.uid('fetch', latest, '(RFC822)')
raw = data[0][1] # This contains the Mail Data
msg = email.message_from_string(raw)
time.sleep(5)
except SomeRelevantException as e:
logging.log(e)
time.sleep(60)
pass
In the event of some random error that you didn't foresee, use a process control manager like supervisord or monit.
Related
What My Code is Supposed to Do: (Ran inside docker) My code uses the Mysterium network (Pretty much just a VPN but it's decentralized) to generate a list of active nodes and then cycles through each one and runs my web scraper. If the node fails the Have_Internet function then that node gets added to the blacklist.
Issue 1: I want to generate new node -> test connection (either just by ping test or short network strength test) -> Run selenium. I only want to run Selenium if the network test passes otherwise blacklist that node and try again. I cannot run the Have_Internet function in an else statement because then it will never run for some reason. I read that the else statement will only run if the try successes which is what I want, but then it never runs so I took it out.
Issue 2: I have kept checking my blacklist file and it never has anything in it so either my code block responsible for printing it fails or my network test is garbage.
Issue 3: I used some bash commands in my python code because I could not figure out a simple solution. if you made them into python lines I would be very happy.
PS. Any other general feedback is more than welcome. I am still pretty new to python :)
Cheers in advance
def Run_Selenium():
# Doing stuff
def Random_Node():
# Doing stuff
def Have_Internet():
conn = httplib.HTTPSConnection("8.8.8.8", timeout=5)
try:
conn.request("HEAD", "/")
return True
except Exception:
return False
finally:
conn.close()
if __name__ == "__main__":
while True == True:
#gets new node
random_node = Random_Node()
Temp_Random_Node = Random_Node()
try:
print("Running new myst session", flush=True)
os.system("/opt/myst/myst connection down") # stops vpn
#time.sleep(5)
#New_Myst_Command = () # makes node command
os.system(str("/opt/myst/myst connection up " + Temp_Random_Node)) # starts new connection with node
#time.sleep(5) # time to wait while dvpn starts
# catch errors when connecting to new node
except Exception as e:
print(e, flush=True)
print("Failed to connect to " + Temp_Random_Node + " moving on anyway and adding it to the blacklist", flush=True)
os.system("echo " + Temp_Random_Node + " >> /root/Dockerdata/blacklist.txt")
Connected = Have_Internet()
if Connected == True:
print("Going to try to run selenium", flush=True)
time.sleep(random.randrange(1, 20)) # 4, 305 # amount of time to wait before loading the page again
try:
Run_Selenium()
print("Just ran a successful run of selenium", flush=True)
except Exception as e:
print("Selenium failed")
print(e, flush=True)
else:
print("Failed network check")
os.system("echo " + random_node + " > /root/Dockerdata/blacklist.txt")
I have a multi-threading code in Python (firing several threads every second and closing them after), and it used to work fine. Recently, I added a new function (thread) for listening to a server for some tables (as they are streamed out from the server), through a Get Request (10 seconds timeout).
The issue is that the code works fine for about 1-2 hours and then I get the python thread error of "error: can't start new thread", with having only ~20 active threads.
I tried having a singleton pool of thread and using it, but it did not help at all.
On a side note, removing this get request from the function resolves the issue and the code runs perfectly.
Please let me know your opinions,
Thank you.
def getStreamData(self):
if (self.liveTablesTimer == None):
self.startLiveTablesTimer()
print("LiveTables timer started")
self.voidTableCount += 1 # counting for connection refresh
def separateThread():
try:
#return 0
self.streamInConnection = requests.get(self.liveTablesUrl, stream=True, verify=False, timeout=10)
#print("Live tables request sent as:", self.liveTablesUrl)
if self.streamInConnection.encoding is None:
self.streamInConnection.encoding = 'utf-8'
for line in self.streamInConnection.iter_lines(decode_unicode=True):
if line and self.userName != None:
#print("Raw stream received", line)
self.streamData.emit(line)
except:
print("getLiveTables stream link timeout")
self.streamInConnection.close()
if (self.voidTableCount>6*5): #5 min
try:
self.voidTableCount=0
pass
except:
pass
finally:
return 0
try:
print("Starting thread for receiving liveTables data")
#self.consCheck.threadExecutor.submit(separateThread)
thread = threading.Thread(target=separateThread, args=[], daemon = True)
thread.start()
except Exception as err:
print("liveTables stream error:", err)
error image
Strangely, I removed the 'verify' parameter from the request and it resolved the issue.
requests.get(self.liveTablesUrl, stream=True, timeout=10)
I wrote a program that uses threads to keep a connection alive while the main program loops until it either has an exception or is manually closed. My program runs in 1 hour intervals and the timeout for the connection is 20 minutes, thus I spawn a thread for every connection element that exist inside of my architecture. Thus, if we have two servers to connect to it connects to both these serves and stays connected and loops through each server retrieving data.
the program I wrote works correctly, however I can't seem to find a way to handle when the program it's self throws an exception. This is to say I can't find an appropriate way to dispose of the threads when the main program excepts. When the program excepts it will just hang open because of the thread not excepting as well and it won't close correctly and will have to be closed manually.
Any suggestions on how to handle cleaning up threads on program exit?
This is my thread:
def keep_vc_alive(vcenter,credentials, api):
vm_url = str(vcenter._proxy.binding.url).split('/')[2]
while True:
try:
logging.info('staying connected %s' % str(vm_url))
vcenter.keep_session_alive()
except:
logging.info('unable to call current time of vcenter %s attempting to reconnect.' % str(vm_url))
try:
vcenter = None
connected,api_version,uuid,vcenter = vcenter_open(60, api, * credentials)
except:
logging.critical('unable to call current time of vcenter %s killing application, please have administrator restart the module.' % str(vm_url))
break
time.sleep(60*10)
Then my exception clean up code is as follows, obviously I know.stop() doesn't work, but I honestly have no idea how to do what it is im trying to do.
except Abort: # Exit without clearing the semaphore
logging.exception('ApplicationError')
try:
config_values_vc = metering_config('VSphere',['vcenter-ip','username','password','api-version'])
for k in xrange(0, len(config_values_vc['username'])): # Loop through each vcenter server
vc_thread[config_values_vc['vcenter-ip'][k]].stop()
except:
pass
#disconnect vcenter
try:
for vcenter in list_of_vc_connections:
list_of_vc_connections[vcenter].disconnect()
except:
pass
try: # Close the db is it is open (db is defined)
db.close()
except:
pass
sys.exit(1)
except SystemExit:
raise
except:
logging.exception('ApplicationError')
semaphore('ComputeLoader', False)
logging.critical('Unexpected error: %s' % sys.exc_info()[0])
raise
Instead of sleeping, wait on a threading.Event():
def keep_vc_alive(vcenter,credentials, api, event): # event is a threading.Event()
vm_url = str(vcenter._proxy.binding.url).split('/')[2]
while not event.is_set(): # If the event got set, we exit the thread
try:
logging.info('staying connected %s' % str(vm_url))
vcenter.keep_session_alive()
except:
logging.info('unable to call current time of vcenter %s attempting to reconnect.' % str(vm_url))
try:
vcenter = None
connected,api_version,uuid,vcenter = vcenter_open(60, api, * credentials)
except:
logging.critical('unable to call current time of vcenter %s killing application, please have administrator restart the module.' % str(vm_url))
break
event.wait(timeout=60*10) # Wait until the timeout expires, or the event is set.
Then, in your main thread, set the event in the exception handling code:
except Abort: # Exit without clearing the semaphore
logging.exception('ApplicationError')
event.set() # keep_alive thread will wake up, see that the event is set, and exit
The generally accepted way to stop threads in python is to use the threading.Event object.
The algorithm followed usually is something like the following:
import threading
...
threads = []
#in the main program
stop_event = threading.Event()
#create thread and store thread and stop_event together
thread = threading.Thread(target=keep_vc_alive, args=(stop_event))
threads.append((thread, stop_event))
#execute thread
thread.start()
...
#in thread (i.e. keep_vc_alive)
# check is_set in stop_event
while not stop_event.is_set():
#receive data from server, etc
...
...
#in exception handler
except Abort:
#set the stop_events
for thread, stop_event in threads:
stop_event.set()
#wait for threads to stop
while 1:
#check for any alive threads
all_finished = True
for thread in threads:
if thread.is_alive():
all_finished = False
#keep cpu down
time.sleep(1)
There's lots of topics touching on part of the title, but nothing that quite satisfies the whole thing. I'm pushing a command on a remote server and need the full output after a long execution time, say 5 minutes or so. Using channel I was able to set a timeout, but when I read back stdout I got only a small portion of output. The solution seemed to be to wait for channel.exit_status_ready(). This worked on a successful call, but a failed call would never trigger the channel timeout. Having reviewed the docs, I theorize that's because the timeout only works on a read operation, and waiting for exit status doesn't qualify. Here's that attempt:
channel = ssh.get_transport().open_session()
channel.settimeout(timeout)
channel.exec_command(cmd) # return on this is not reliable
while True:
try:
if channel.exit_status_ready():
if channel.recv_ready(): # so use recv instead...
output = channel.recv(1048576)
break
if channel.recv_stderr_ready(): # then check error
error = channel.recv_stderr(1048576)
break
except socket.timeout:
print("SSH channel timeout exceeded.")
break
except Exception:
traceback.print_exc()
break
Pretty, ain't it? Wish it worked.
My first attempt at a solution was to use time.time() to get a start, then check start - time.time() > timeout. This seems straightforward, but in my present version, I output start - time.time() with a fixed timeout that should trigger a break...and see differences that double and triple the timeout with no break occurring. To save space, I'll mention my third attempt, which I've rolled up with this one. I read on here about using select.select to wait for output, and noted in the documentation that there's a timeout there as well. As you'll see from the code below, I've mixed all three methods -- channel timeout, time.time timeout, and select timeout -- yet still have to kill the process. Here's the frankencode:
channel = ssh.get_transport().open_session()
channel.settimeout(timeout)
channel.exec_command(cmd) # return on this is not reliable
print("{0}".format(cmd))
start = time.time()
while True:
try:
rlist, wlist, elist = select([channel], [], [],
float(timeout))
print("{0}, {1}, {2}".format(rlist, wlist, elist))
if rlist is not None and len(rlist) > 0:
if channel.exit_status_ready():
if channel.recv_ready(): # so use recv instead...
output = channel.recv(1048576)
break
elif elist is not None and len(elist) > 0:
if channel.recv_stderr_ready(): # then check error
error = channel.recv_stderr(1048576)
break
print("{0} - {1} = {2}".format(
time.time(), start, time.time() - start))
if time.time() - start > timeout:
break
except socket.timeout:
print("SSH channel timeout exceeded.")
break
except Exception:
traceback.print_exc()
break
Here's some typical output:
[<paramiko.Channel 3 (open) window=515488 -> <paramiko.Transport at 0x888414cL (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>], [], []
1352494558.42 - 1352494554.69 = 3.73274183273
The top line is [rlist, wlist, elist] from select, the bottom line is time.time() - start = (time.time() - start). I got this run to break by counting the iterations and breaking at the bottom of the try after looping 1000 times. timeout was set to 3 on the sample run. Which proves that we get through the try, but obviously, none of the three ways that should be timing out works.
Feel free to rip into the code if I've fundamentally misunderstood something. I'd like for this to be uber-Pythonic and am still learning.
Here's something that might help, though I'm still in the midst of testing. After struggling with timeouts of various types including a catch-all timeout for Python, and realizing that the real problem is that the server can't be trusted to terminate the process, I did this:
chan = ssh.get_transport().open_session()
cmd = "timeout {0} {1}\n".format(timeouttime, cmd)
chan.exec_command(cmd)
The server times out after timeouttime if cmd doesn't exit sooner, exactly as I'd wish, and the terminated command kills the channel. The only catch is that GNU coreutils must exist on the server. Failing that there are alternatives.
I'm having the same kind of issue. I think we can handle it with signalling. http://docs.python.org/2/library/signal.html
Here is a plain dumb example to show how it works.
import signal, time
def handler(signum, frame):
pass
# Set the signal handler and a 2-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(2)
# This is where your operation that might hang goes
time.sleep(10)
# Disable the alarm
signal.alarm(0)
So here, the alarm is set to 2 seconds. Time.sleep is called with 10 seconds. Of course, the alarm will be triggered before the sleep finishes. If you put some output after the time.sleep, you'll see that program execution resumes there.
If you want the control to continue somewhere else, wrap your hanging call in a try/except and have your handler function raise an exception.
Although I'm pretty sure it would work, I haven't tested it yet over paramiko calls.
I had a lot of problem calling the exec_command from the channel, instead I use directly the exec_command from the ssh connection and call the channel of the std output, the code that works for me is like myexec:
#!/usr/bin/env python
import paramiko
import select
def myexec(ssh, cmd, timeout):
stdin, stdout, stderr = ssh.exec_command(cmd)
channel = stdout.channel
stdin.close() #As I don't need stdin
channel.shutdown_write() #As I will not write to this channel
stdout_chunks = []
stdout_chunks.append(stdout.channel.recv(len(stdout.channel.in_buffer)))
# chunked read to prevent stalls
while not channel.closed or channel.recv_ready()
or channel.recv_stderr_ready():
# stop if channel was closed prematurely,
# and there is no data in the buffers.
got_chunk = False
readq, _, _ = select.select([stdout.channel], [], [], timeout)
for c in readq:
if c.recv_ready():
stdout_chunks.append(stdout.channel.recv(len(c.in_buffer)))
got_chunk = True
if c.recv_stderr_ready():
# make sure to read stderr to prevent stall
stderr.channel.recv_stderr(len(c.in_stderr_buffer))
got_chunk = True
if not got_chunk \
and stdout.channel.exit_status_ready() \
and not stderr.channel.recv_stderr_ready() \
and not stdout.channel.recv_ready():
# indicate that we're not going to read from this channel anymore
stdout.channel.shutdown_read() # close the channel
stdout.channel.close()
break # exit as remote side is finished and our bufferes are empty
# close all the pseudofiles
stdout.close()
stderr.close()
return (''.join(stdout_chunks), stdout.channel.recv_exit_status())
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('remotehost', username='remoteuser', password='remotepassword')
rtrval = myexec(ssh, 'remotecomand', 5*60)
ssh.close()
print rtrval
I use Debian 8 and Python 2.7.13, good luck.
Look at the bottom of this post, for final working code.
It's a working Python/CGI script which can get user-input to a CGI-script by calling another script which then sends it's commands through a local socket.
Original post:
As far as I know, there isn't any way to send user input directly to a Python/CGI script which has allready sent it's header. Like, warning the user under specific circumstances and waiting for a confirmation.
Neither have I been able to find any published solutions to this.
If I'm wrong, please correct me.
I currently have a Python script which can connect to servers, upload firmware, reboot, re-connect, change a few configuration files and such.
Sometimes, it would help alot of the user could send input to script, without having to re-launch the script and execute it from the beginning. Re-connecting over a 2G network takes too long.
I'm thinking that it must be possible to send user input to another script, which then posts it to a file, which the first/main script is watching, until it recieves the input.
It would also be nice, if the was able to stop the execution of the script, with a stop/kill input command.
As for the stop/kill command, the main script would need to have 2 threads. If it did not, it would know it should stop the script, if a process such as a large file upload is being executed, before the upload is completed.
At the same time, I think multipe users should be able to use the script at the same time. Therefore, a unique ID must be generated every time the main script launches.
Here's how I think it could be made:
Main script gets called
Global variable with a unique session ID is generated and sent to client.
Thread 1
pexpect spawns a "tail -F /var/www/cgi/tmp_cmd.log"
Thread 2
Thread status "Busy"
Connects to network element
Does its usual stuff until it reaches a point where the user needs to interact.
Prints the message to user and waits for Thread 1 with a timeout of x seconds.
Thread status "Ready"
Second script gets called by the user through AJAX with 2 headers (session ID & input)
Second script
Session ID and user input is saved to "/var/www/cgi/tmp_cmd.log"
Execution of the input script ends
Main script
Thread 1
User input recieved.
Wait for Thread 2 status to become "Ready" or ignore status if command is equals to "kill" ect.
Send user input (single line) and start Thread 1 from the beginning
Thread 2
Thread 2 status "Busy"
Input recieved and process stops/continues.
Thread 2 status "Ready"
I have made a script allready for connecting, uploading files, and running commands.
However, it cannot recieve user-input.
I could really use some good help, or someone to tell me how to approach this.
Of course, whenever the script has been completed, I will post it here or on pastebin and link to it, for other people to use. :)
Final code
With help from the post below, I have finally have the working code.
It could use Threads, but stopping/cancelling processes appeared to be way easier for me to figure out.
Client - cgi_send.py
#!/usr/bin/python
import sys, cgi, cgitb, socket
cgitb.enable()
TASKS_DIR = "/var/www/cgi-bin/tmp"
def main():
global TASKS_DIR
url = cgi.FieldStorage()
cmd = str(url.getvalue('cmd'))
sessionId = str(url.getvalue('session'))
socketLocation = TASKS_DIR + '/%s.socket' % sessionId
print 'End script Cancel task'
print '<form action=""><input type="hidden" name="session" id="session" value="'+sessionId+'" /><input type="text" name="cmd" id="cmd" value="" /><input type="submit" value="Fun!" />'
try:
sock = socket.socket(socket.AF_UNIX)
sock.setblocking(0)
sock.connect(socketLocation)
sock.send(cmd)
sock.close()
print '<br />Command sent: '+ cmd;
except IOError:
print '<br /><b>Operation failed.</b><br /> Could not write to socket: '+ socketLocation
pass
sock.close()
sys.exit();
if __name__ == '__main__':
sys.stdout.write("Content-type:text/html;charset=utf-8\r\n\r\n")
sys.stdout.write('<!DOCTYPE html>\n<html><head><title>Test</title></head><body>')
main()
print '</body></html>'
sys.exit()
Server
#!/usr/bin/python
import sys, os, socket, uuid, time, multiprocessing
# Options
TASKS_DIR = "/var/www/cgi-bin/tmp/"
def main():
sessionId = str(uuid.uuid4())
print 'Session ID: '+ sessionId
sys.stdout.write ('<br />Send test command')
sys.stdout.flush()
address = os.path.join(TASKS_DIR, '%s.socket' % sessionId)
sock = socket.socket(socket.AF_UNIX)
sock.setblocking(0)
sock.settimeout(.1)
sock.bind(address)
sock.listen(1)
taskList = [foo_task, foo_task, foo_task]
try:
for task in taskList:
print "<br />Starting new task"
runningTask = multiprocessing.Process(target=task)
runningTask.daemon = True # Needed to make KeyboardInterrupt possible when testing in shell
runningTask.start()
while runningTask.is_alive():
conn = None
try:
conn, addr = sock.accept()
data = conn.recv(100).strip()
except socket.timeout:
# nothing ready from a client
continue
except socket.error, e:
print "<br />Connection Error from client"
else:
print "<br />"+ data
sys.stdout.flush()
conn.close()
if data == "CANCEL":
# temp way to cancel our task
print "<br />Cancelling current task."
runningTask.terminate()
elif data == "QUIT":
print "<br />Quitting entire process."
runningTask.terminate()
taskList[:] = []
finally:
if conn:
conn.close()
except (KeyboardInterrupt, SystemExit):
print '\nReceived keyboard interrupt, quitting threads.'
finally:
sock.close()
os.remove(address)
def foo_task():
i = 1
while 10 >= i:
print "<br />Wating for work... "+ str(i)
sys.stdout.flush()
i = i + 1
time.sleep(1)
if __name__ == '__main__':
sys.stdout.write("Content-type:text/html;charset=utf-8\r\n\r\n")
sys.stdout.write('<!DOCTYPE html>\n<html><head><title>Test</title></head><body>')
main()
print '</body></html>'
sys.exit()
A CGI script is a pretty primitive operation. It works basically the same as any normal script you run from your command shell. An http request is made to the web server. The server starts a new process and passes the arguments in via stdin to the script. At this point, it's like a normal script.
A script can't get any more input unless it's looking for input by some means, so you are correct in assuming that once the headers are sent, the web client can no longer directly send more input, because the request is already in progress, and the response is already in progress as well.
A thread watching a file is one way to introduce a control loop to the script. Another is to open a UNIX socket to a path based on your unique ID for each instance. Then have the thread sitting on the socket for input. What you would then have to do is pass the ID back to the web client. And the client could make a call to the second script with the ID, which would then know the proper UNIX socket path to send control commands to: ie.
/tmp/script-foo/control/<id>.socket
You actually might only need 1 thread. You main thread could simply loop over checking for information on the socket, and monitoring the current operation being run in a thread or subprocess. It might be like this in pseudocode:
uid = generate_unique_id()
sock = socket.socket(AF_UNIX)
sock.bind('/tmp/script-foo/control/%s.socket' % uid)
# and set other sock options like timeout
taskList = [a,b,c]
for task in taskList:
runningTask = start task in thread/process
while runningTask is running:
if new data on socket, with timeout N ms
if command == restart:
kill runningTask
taskList = [a,b,c]
break
else:
process command
When the web client sends a command via ajax to your second script, it might look like this in pseudocode:
jobid = request.get('id')
cmd = request.get('cmd')
sock = socket.socket(socket.AF_UNIX)
sock.connect('/tmp/script-foo/control/%s.socket' % jobid)
sock.sendall(cmd)
sock.close()
Update
Based on your code update, here is a working example of what I was suggesting:
import sys
import os
import socket
import uuid
import time
# Options
TASKS_DIR = "."
def main():
sessionId = str(uuid.uuid4())
print 'Session ID: '+ sessionId
sys.stdout.write ('<br />Send test command')
sys.stdout.flush()
address = os.path.join(TASKS_DIR, '%s.socket' % sessionId)
sock = socket.socket(socket.AF_UNIX)
sock.setblocking(0)
sock.settimeout(.1)
sock.bind(address)
sock.listen(1)
fakeTasks = [foo_task, foo_task, foo_task]
try:
for task in fakeTasks:
# pretend we started a task
runningTask = task()
# runningTask = Thread(target=task)
# runningTask.start()
# while runningTask.is_alive():
while runningTask:
conn = None
try:
conn, addr = sock.accept()
data = conn.recv(100).strip()
except socket.timeout:
# nothing ready from a client
continue
except socket.error, e:
print "<br />Connection Error from client"
else:
print "<br />"+ data
sys.stdout.flush()
conn.close()
# for the thread version, you will need some
# approach to kill or interrupt it.
# This is just simulating.
if data == "CANCEL":
# temp way to cancel our task
print "<br />Cancelling current task."
runningTask = False
elif data == "QUIT":
print "<br />Quitting entire process."
runningTask = False
fakeTasks[:] = []
finally:
if conn:
conn.close()
finally:
sock.close()
os.remove(address)
def foo_task():
print 'foo task'
return True
if __name__ == '__main__':
sys.stdout.write("Content-type:text/html;charset=utf-8\r\n\r\n")
sys.stdout.write('<!DOCTYPE html>\n<html><head><title>Test</title></head><body>')
main()
print '</body></html>'
sys.exit()
Instead of using a 10 second global timeout, you set it to something small like 100ms. It loops over each task and starts it (eventually in a thread), and then tries to loop over waiting for a socket connection. If there is no connection within 100ms, it will timeout and continue to loop, while checking if the task is done. At any point, a client can connect and issue either a "CANCEL" or "QUIT" command. The socket will accept the connection, read it, and react.
You can see how you do not need multiple threads here for the solution. The only threading or subprocess you need is to run the task.