We Keep Coding

Python Threading/Request issue

I have a multi-threading code in Python (firing several threads every second and closing them after), and it used to work fine. Recently, I added a new function (thread) for listening to a server for some tables (as they are streamed out from the server), through a Get Request (10 seconds timeout).
The issue is that the code works fine for about 1-2 hours and then I get the python thread error of "error: can't start new thread", with having only ~20 active threads.
I tried having a singleton pool of thread and using it, but it did not help at all.
On a side note, removing this get request from the function resolves the issue and the code runs perfectly.
Please let me know your opinions,
Thank you.
def getStreamData(self):
if (self.liveTablesTimer == None):
self.startLiveTablesTimer()
print("LiveTables timer started")
self.voidTableCount += 1 # counting for connection refresh
def separateThread():
try:
#return 0
self.streamInConnection = requests.get(self.liveTablesUrl, stream=True, verify=False, timeout=10)
#print("Live tables request sent as:", self.liveTablesUrl)
if self.streamInConnection.encoding is None:
self.streamInConnection.encoding = 'utf-8'
for line in self.streamInConnection.iter_lines(decode_unicode=True):
if line and self.userName != None:
#print("Raw stream received", line)
self.streamData.emit(line)
except:
print("getLiveTables stream link timeout")
self.streamInConnection.close()
if (self.voidTableCount>6*5): #5 min
try:
self.voidTableCount=0
pass
except:
pass
finally:
return 0
try:
print("Starting thread for receiving liveTables data")
#self.consCheck.threadExecutor.submit(separateThread)
thread = threading.Thread(target=separateThread, args=[], daemon = True)
thread.start()
except Exception as err:
print("liveTables stream error:", err)
error image

Strangely, I removed the 'verify' parameter from the request and it resolved the issue.
requests.get(self.liveTablesUrl, stream=True, timeout=10)

Open separate CMD for the Python program from the existing Python program

I have the following code working but it halt the rest process as there is a look for checking in the anther program that I am trying to open from the python:
def RunSocket():
os.system('python "Display Lines.py"')
threading.Thread(target = RunSocket).start()
check_display_active = True
try:
while check_display_active == True:
try:
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((display_host, display_port))
client.send(str('is_active').encode())
response = client.recv(4096)
if response.decode() == "active":
break
time.sleep(0.5)
except Exception as e:
print("==========================================")
print("Display not running")
print(e)
print("==========================================")
except:
sys.exit()
I wan to run this process in separate CMD so that it can run there without halting my current process.
Please share your advice. I do not know subprocess properly I think it can be solution. But need advise.

Infinite running server-side python script?

I want to replace Cron Jobs for "keeping" my program alive because it calls every XX interval whether or not the scrip is already called, creating duplicate entries.
I investigated the issue, and had a few approaches. One was to modify my program so it checks if it is already called and closes itself. The one I went after was to detach it completely from Cronjob by calling itself over and over again with execfile which works exactly how I want except the following problem:
RuntimeError: maximum recursion depth exceeded
Is there a way to keep the program in "infinite loop" without getting a Stack Overflow?
Here is my code, its a program that checks Mails, and converts them into MySQL DB entries.
imap = imaplib.IMAP4(hst)
try:
imap.login(usr, pwd)
except Exception as e:
errormsg = e
time.sleep(30)
print "IMAP error: " + str(errormsg)
execfile('/var/www/html/olotool/converter.py')
raise IOError(e)
# Authentification & Fetch Step
while True:
time.sleep(5)
'''
The script will always result in an error if there
are no mails left to check in the inbox. It then
goes into sleep mode and relaunches itself to check
if new mails have arrived.
'''
try:
imap.select("Inbox") # Tell Imap where to go
result, data = imap.uid('search', None, "ALL")
latest = data[0].split()[-1]
result, data = imap.uid('fetch', latest, '(RFC822)')
raw = data[0][1] # This contains the Mail Data
msg = email.message_from_string(raw)
except Exception as e:
disconnect(imap)
time.sleep(60)
execfile('/var/www/html/olotool/converter.py')
raise IOError(e)

I solved the problem myself with the only way I see it possible right now.
First I changed my exception in above code:
except Exception as e:
disconnect(imap)
print "Converter: No messages left"
raise os._exit(0)
# This is a special case since this Exception is
# no error thus os._exit(0) gives no false-positives
As you see I refrain from using execfile now. Instead I wrote a controller script that checks the status of my converter.py and launches it if it is not already running:
while True:
presL = os.popen('pgrep -lf python').read()
print "________________________________________"
print "Starting PIDcheck"
print "Current Processes: "
print presL # Check Processes
presRconverter = find('\d{7} python converter.py', presL)
if presRconverter:
# Store the PID
convPID = find('\d{7}', presRconverter)
print "Converter is running at PID: " + convPID
else:
print "PID Controller: Converter not running"
try:
print "PID Controller: Calling converter"
subprocess.check_call('python converter.py', shell=True)
except subprocess.CalledProcessError as e:
errormsg = e
print "Couldn't call Converter Module"
sendMail(esender,ereceiver,esubject,etext,server)
print "Error notification send"
raise IOError(e)
# If we got until here without ERROR, the call was Successfull
print "PID Controller: Call successful"
print "________________________________________"
time.sleep(60)
This method does not raise an: RuntimeError: maximum recursion depth exceeded. Also this provides you with a nohup.out file if you run the controller with command nohup python converter.py where you can see any problems for errorhandling.
I hope I could help anyone running into the same issue.

Something along the lines of this should work without having to resort to subprocess checking and such:
def check_mail_loop():
imap = imaplib.IMAP4(hst)
# Build some function to login, and, in the event of an error, sleep for n seconds and call login function again.
imap.login(usr, pwd)
while True:
try:
imap.select("Inbox")
result, data = imap.uid('search', None, "ALL")
if result and data:
latest = data[0].split()[-1]
result, data = imap.uid('fetch', latest, '(RFC822)')
raw = data[0][1] # This contains the Mail Data
msg = email.message_from_string(raw)
time.sleep(5)
except SomeRelevantException as e:
logging.log(e)
time.sleep(60)
pass
In the event of some random error that you didn't foresee, use a process control manager like supervisord or monit.

Killing a thread with a socket in it (python)

I am pretty new at this.I am trying to build a server(chat server)
Sorry for presenting such a messing code.
There are alot of things that i am going to change about this code.
but as of now i just need help with one thing:
when i start let say more then one cleints on this ...and then just close the client i get this message:
Unhandled exception in thread started by
i have tryed to kill the thread as you can see in many places in this code. but i don't know what i am doing wrong ..
i am new at this.
any syggestions on what i should do ?
#encoding: utf-8
import socket, random, time, thread, errno, traceback
print socket.gethostname()
print "current machines IP address: "+socket.gethostbyname(socket.gethostname())
host ="10.0.0.1"# raw_input("IP: ")
# = socket.gethostbyname(socket.gethostname())
port = 12345
print host
conn_list =[None]*10
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((host, port))
sock.listen(10)
print "connected..\n\n\n\n"
def recv(conn):
while True:
try:
message = conn.recv(1024)
if "MESG" == message[1:5]:
message = message[6:].split(':')
name = str(conn)
conn_number = conn_list.index(conn)
conn_name = str(conn_number)
message = message[2]
reciever = message[0:1]
reciever = int(reciever)
for conn in conn_list:
if reciever == conn_list.index(conn):
conn.send(message)
print "Connection "+conn_name+" -----> "+str(reciever)+" :"+message+"\n"
#conn = findTheRightConnection(conn_list, conn_number)
break
else:
pass
except ValueError:
print "ValueError by %s" % (str(conn))
print conn.send("\nOpps you are not typing the correct connection number infront your message!")
except IOError:
bye(conn,conn_list)
print"Going to try to kill the thread here "
thread.quit()
thread.isAlive()
print "Still alive..."
except socket.error, v:
errorcode=v[0]
bye(conn,conn_list)
print"Going to try to kill the thread or here"
thread.quit()
thread.isAlive()
print "Still alive..."
except Exception, e:
traceback.print_exc()
finally:
thread.isAlive()
print"\nChanging the conn back to what it was... "
conn = findTheRightConnection(conn_list, conn_number)
def handle_connection(conn):
try:
recv(conn)
except socket.error, v:
errorcode=v[104]
bye(conn)
def bye(conn,conn_list):
i= 0
print "bye"
connectionName = str(conn_list.index(conn))
conn.close
conn_list = conn_list
print conn_list
for element in conn_list:
if element == conn:
conn_list[i] = None
break
i += i
print "Connection number "+connectionName+" is terminated"
print conn_list
return "Connection Terminated"
def welcome(conn,conn_list):
i = 0
for element in conn_list:
if element == None:
conn_list[i] = conn
print "Connection added in the conn_list on the slot %d" % (i)
print conn_list
return conn_list
else:
i = i+1
pass
print "The server if full! No more space left"
return conn_list
def findTheRightConnection(conn_list, number):
for conn in conn_list:
if number == conn_list.index(conn):
return conn
else:
pass
print "\nSomthing went wrong while trying to find the right connection in the method findTheRightConnection()"
return
while True:
conn, addr = sock.accept()
conn_list = welcome(conn,conn_list)
print "Got connection from : "+str(addr[0])+" by connection number: "+str(conn_list.index(conn))+"\n\n\n\n"
msg = "Welcome to the server!"
conn.send(":INFO:"+str(int(time.time()))+":"+str(len(msg))+":"+msg)
thread.start_new_thread(handle_connection, (conn,))

If you are still having trouble creating a instant messaging program in Python, you might be interested in this answer to another question.
Simple_Server.py is a minimal server implementation. A far more complex server with a variety of features can be provided on request. The complex server supports authentication, friends, private messaging, channels, filters, math evaluation, and admin controls.
MultichatClient.py is a port of a Java program written by a teacher from a networking class. The program must be run from the command line, and it must be given the server as an argument. You can use either the server's name on the network or its IP address.
Simple_Client.pyw is a more complicated client that does not require being started from the command line. When it starts, it will ask for server's name and try connecting to it while showing a progress dialog. The program will automatically try logging any errors to a file.
affinity.py is required for threadbox.py to run. (runs code on a specific thread regardless of origin)
threadbox.py is required for safetkinter.py to run. (metaclass clones classes to run using affinity)
safetkinter.py is required for Simple_Client.pyw to run. (makes tkinter safe to use with threads)

Python - User input to CGI via. Threading and reading file

Look at the bottom of this post, for final working code.
It's a working Python/CGI script which can get user-input to a CGI-script by calling another script which then sends it's commands through a local socket.
Original post:
As far as I know, there isn't any way to send user input directly to a Python/CGI script which has allready sent it's header. Like, warning the user under specific circumstances and waiting for a confirmation.
Neither have I been able to find any published solutions to this.
If I'm wrong, please correct me.
I currently have a Python script which can connect to servers, upload firmware, reboot, re-connect, change a few configuration files and such.
Sometimes, it would help alot of the user could send input to script, without having to re-launch the script and execute it from the beginning. Re-connecting over a 2G network takes too long.
I'm thinking that it must be possible to send user input to another script, which then posts it to a file, which the first/main script is watching, until it recieves the input.
It would also be nice, if the was able to stop the execution of the script, with a stop/kill input command.
As for the stop/kill command, the main script would need to have 2 threads. If it did not, it would know it should stop the script, if a process such as a large file upload is being executed, before the upload is completed.
At the same time, I think multipe users should be able to use the script at the same time. Therefore, a unique ID must be generated every time the main script launches.
Here's how I think it could be made:
Main script gets called
Global variable with a unique session ID is generated and sent to client.
Thread 1
pexpect spawns a "tail -F /var/www/cgi/tmp_cmd.log"
Thread 2
Thread status "Busy"
Connects to network element
Does its usual stuff until it reaches a point where the user needs to interact.
Prints the message to user and waits for Thread 1 with a timeout of x seconds.
Thread status "Ready"
Second script gets called by the user through AJAX with 2 headers (session ID & input)
Second script
Session ID and user input is saved to "/var/www/cgi/tmp_cmd.log"
Execution of the input script ends
Main script
Thread 1
User input recieved.
Wait for Thread 2 status to become "Ready" or ignore status if command is equals to "kill" ect.
Send user input (single line) and start Thread 1 from the beginning
Thread 2
Thread 2 status "Busy"
Input recieved and process stops/continues.
Thread 2 status "Ready"
I have made a script allready for connecting, uploading files, and running commands.
However, it cannot recieve user-input.
I could really use some good help, or someone to tell me how to approach this.
Of course, whenever the script has been completed, I will post it here or on pastebin and link to it, for other people to use. :)
Final code
With help from the post below, I have finally have the working code.
It could use Threads, but stopping/cancelling processes appeared to be way easier for me to figure out.
Client - cgi_send.py
#!/usr/bin/python
import sys, cgi, cgitb, socket
cgitb.enable()
TASKS_DIR = "/var/www/cgi-bin/tmp"
def main():
global TASKS_DIR
url = cgi.FieldStorage()
cmd = str(url.getvalue('cmd'))
sessionId = str(url.getvalue('session'))
socketLocation = TASKS_DIR + '/%s.socket' % sessionId
print 'End script Cancel task'
print '<form action=""><input type="hidden" name="session" id="session" value="'+sessionId+'" /><input type="text" name="cmd" id="cmd" value="" /><input type="submit" value="Fun!" />'
try:
sock = socket.socket(socket.AF_UNIX)
sock.setblocking(0)
sock.connect(socketLocation)
sock.send(cmd)
sock.close()
print '<br />Command sent: '+ cmd;
except IOError:
print '<br /><b>Operation failed.</b><br /> Could not write to socket: '+ socketLocation
pass
sock.close()
sys.exit();
if __name__ == '__main__':
sys.stdout.write("Content-type:text/html;charset=utf-8\r\n\r\n")
sys.stdout.write('<!DOCTYPE html>\n<html><head><title>Test</title></head><body>')
main()
print '</body></html>'
sys.exit()
Server
#!/usr/bin/python
import sys, os, socket, uuid, time, multiprocessing
# Options
TASKS_DIR = "/var/www/cgi-bin/tmp/"
def main():
sessionId = str(uuid.uuid4())
print 'Session ID: '+ sessionId
sys.stdout.write ('<br />Send test command')
sys.stdout.flush()
address = os.path.join(TASKS_DIR, '%s.socket' % sessionId)
sock = socket.socket(socket.AF_UNIX)
sock.setblocking(0)
sock.settimeout(.1)
sock.bind(address)
sock.listen(1)
taskList = [foo_task, foo_task, foo_task]
try:
for task in taskList:
print "<br />Starting new task"
runningTask = multiprocessing.Process(target=task)
runningTask.daemon = True # Needed to make KeyboardInterrupt possible when testing in shell
runningTask.start()
while runningTask.is_alive():
conn = None
try:
conn, addr = sock.accept()
data = conn.recv(100).strip()
except socket.timeout:
# nothing ready from a client
continue
except socket.error, e:
print "<br />Connection Error from client"
else:
print "<br />"+ data
sys.stdout.flush()
conn.close()
if data == "CANCEL":
# temp way to cancel our task
print "<br />Cancelling current task."
runningTask.terminate()
elif data == "QUIT":
print "<br />Quitting entire process."
runningTask.terminate()
taskList[:] = []
finally:
if conn:
conn.close()
except (KeyboardInterrupt, SystemExit):
print '\nReceived keyboard interrupt, quitting threads.'
finally:
sock.close()
os.remove(address)
def foo_task():
i = 1
while 10 >= i:
print "<br />Wating for work... "+ str(i)
sys.stdout.flush()
i = i + 1
time.sleep(1)
if __name__ == '__main__':
sys.stdout.write("Content-type:text/html;charset=utf-8\r\n\r\n")
sys.stdout.write('<!DOCTYPE html>\n<html><head><title>Test</title></head><body>')
main()
print '</body></html>'
sys.exit()

A CGI script is a pretty primitive operation. It works basically the same as any normal script you run from your command shell. An http request is made to the web server. The server starts a new process and passes the arguments in via stdin to the script. At this point, it's like a normal script.
A script can't get any more input unless it's looking for input by some means, so you are correct in assuming that once the headers are sent, the web client can no longer directly send more input, because the request is already in progress, and the response is already in progress as well.
A thread watching a file is one way to introduce a control loop to the script. Another is to open a UNIX socket to a path based on your unique ID for each instance. Then have the thread sitting on the socket for input. What you would then have to do is pass the ID back to the web client. And the client could make a call to the second script with the ID, which would then know the proper UNIX socket path to send control commands to: ie.
/tmp/script-foo/control/<id>.socket
You actually might only need 1 thread. You main thread could simply loop over checking for information on the socket, and monitoring the current operation being run in a thread or subprocess. It might be like this in pseudocode:
uid = generate_unique_id()
sock = socket.socket(AF_UNIX)
sock.bind('/tmp/script-foo/control/%s.socket' % uid)
# and set other sock options like timeout
taskList = [a,b,c]
for task in taskList:
runningTask = start task in thread/process
while runningTask is running:
if new data on socket, with timeout N ms
if command == restart:
kill runningTask
taskList = [a,b,c]
break
else:
process command
When the web client sends a command via ajax to your second script, it might look like this in pseudocode:
jobid = request.get('id')
cmd = request.get('cmd')
sock = socket.socket(socket.AF_UNIX)
sock.connect('/tmp/script-foo/control/%s.socket' % jobid)
sock.sendall(cmd)
sock.close()
Update
Based on your code update, here is a working example of what I was suggesting:
import sys
import os
import socket
import uuid
import time
# Options
TASKS_DIR = "."
def main():
sessionId = str(uuid.uuid4())
print 'Session ID: '+ sessionId
sys.stdout.write ('<br />Send test command')
sys.stdout.flush()
address = os.path.join(TASKS_DIR, '%s.socket' % sessionId)
sock = socket.socket(socket.AF_UNIX)
sock.setblocking(0)
sock.settimeout(.1)
sock.bind(address)
sock.listen(1)
fakeTasks = [foo_task, foo_task, foo_task]
try:
for task in fakeTasks:
# pretend we started a task
runningTask = task()
# runningTask = Thread(target=task)
# runningTask.start()
# while runningTask.is_alive():
while runningTask:
conn = None
try:
conn, addr = sock.accept()
data = conn.recv(100).strip()
except socket.timeout:
# nothing ready from a client
continue
except socket.error, e:
print "<br />Connection Error from client"
else:
print "<br />"+ data
sys.stdout.flush()
conn.close()
# for the thread version, you will need some
# approach to kill or interrupt it.
# This is just simulating.
if data == "CANCEL":
# temp way to cancel our task
print "<br />Cancelling current task."
runningTask = False
elif data == "QUIT":
print "<br />Quitting entire process."
runningTask = False
fakeTasks[:] = []
finally:
if conn:
conn.close()
finally:
sock.close()
os.remove(address)
def foo_task():
print 'foo task'
return True
if __name__ == '__main__':
sys.stdout.write("Content-type:text/html;charset=utf-8\r\n\r\n")
sys.stdout.write('<!DOCTYPE html>\n<html><head><title>Test</title></head><body>')
main()
print '</body></html>'
sys.exit()
Instead of using a 10 second global timeout, you set it to something small like 100ms. It loops over each task and starts it (eventually in a thread), and then tries to loop over waiting for a socket connection. If there is no connection within 100ms, it will timeout and continue to loop, while checking if the task is done. At any point, a client can connect and issue either a "CANCEL" or "QUIT" command. The socket will accept the connection, read it, and react.
You can see how you do not need multiple threads here for the solution. The only threading or subprocess you need is to run the task.