I want to use Redis's pubsub feature to implement comet, but pubsub doesn't have timeout, so if I use ps.listen(), it will block, even if client closes browser.
Greenlet has a timeout feature when spawn process, but I don't know how to combine them.
Flask's pseudo code:
#app.route('/')
def comet():
rc = redis.Redis()
ps = rc.pubsub()
ps.subscribe('foo')
for item in ps.listen():
if item['type'] == 'message':
return item['data']
# ps.listen() will block, so how to make it timeout after 30 s?
Because you're not threading (and I'm assuming this is intentional and in some cases wise) you must use a type of interrupt. Signals are a type of interrupt on Unix systems to allow you to return to a callback during a call that could block.
This example of a file open which will never return is in line with what you want to do. It's taken from http://docs.python.org/library/signal.html#module-signal
But a warning. Because Python uses a Global Interpreter Lock to perform OS signal handling it is subject to some stability problems. These problems should be rare normally though.
import signal, os
def handler(signum, frame):
print 'Signal handler called with signal', signum
raise IOError("Couldn't open device!")
# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)
# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)
signal.alarm(0) # Disable the alarm
p = redis.pubsub(
ignore_subscribe_messages=True
)
p.subscribe(
DB_PREFIX + CHANEL_KEY
)
message = None
timeout = 20
stop_time = time.time() + timeout
# little hack for setting get_message polling timeout
# because redis-py have bug and first call this method processed
# without timeout
while time.time() < stop_time:
message = p.get_message(timeout=stop_time - time.time())
if message:
break
if message:
data = json.loads(message["data"])
else:
raise HTTPRequestTimeout
Related
I have a script that uses a server-sent event library to connect with a server that pushes events to me regularly. The issue is that the stream will freeze after a long time and I will have to restart the script manually and this is not maintainable. The structure of the current code looks like this
def listen(self):
print("listening to events .....")
try:
url = settings.EVENT_URL + "/v1/events"
auth_key = settings.KEY
headers = {
"Authorization": "Basic " + auth_key,
"Accept": "text/event-stream",
}
response = self.with_urllib3(url, headers)
client = sseclient.SSEClient(response)
for event in client.events():
# the script freezes here.
logger.info(event.data)
process(event.data)
I have tried doing something like
def start(self):
def wait():
time.sleep(10 * 60)
background = threading.Thread(name = 'background', target = self.listen)
background.daemon = True
background.start()
wait()
try:
self.start()
except:
self.start()
finally:
self.start()
But I don't know if this will work mainly because a daemon thread will keep running in the background which means I will have copies of the task running after a while.
What I need is a better way to call a function and after some elapsed time return from the function and recall it again immediately. Thanks for any help.
You could consider a construction using the signal module like shown below. As a note though, the SIGALRM signal is not compatible with Windows.
import signal
TIMEOUT = 5
def _handle_alarm(_, __):
raise TimeoutError("Some useful message")
def listen():
print("Starting to listen...")
import time
time.sleep(10)
while True:
try:
timer = signal.signal(signal.SIGALRM, _handle_alarm)
timer.alarm(TIMEOUT)
listen()
except TimeoutError:
pass
In my Python application, I have a function that consumes message from Amazon SQS FIFO queue.
def consume_msgs():
sqs = boto3.client('sqs',
region_name='us-east-1',
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
print('STARTING WORKER listening on {}'.format(QUEUE_URL))
while 1:
response = sqs.receive_message(
QueueUrl=QUEUE_URL,
MaxNumberOfMessages=1,
WaitTimeSeconds=10,
)
messages = response.get('Messages', [])
for message in messages:
try:
print('{} > {}'.format(threading.currentThread().getName(), message.get('Body')))
body = json.loads(message.get('Body'))
sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=message.get('ReceiptHandle'))
except Exception as e:
print('Exception in worker > ', e)
sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=message.get('ReceiptHandle'))
time.sleep(10)
In order to scale up, I am using multi threading to process messages.
if __name__ == '__main__:
for i in range(3):
t = threading.Thread(target=consume_msgs, name='worker-%s' % i)
t.setDaemon(True)
t.start()
while True:
print('Waiting')
time.sleep(5)
The application runs as service. If I need to deploy new release, it has to be restarted. Is there a way have the threads exist gracefully when main process is being terminated? In stead of killing the threads abruptly, they finish with current message first and stop receiving the next messages.
Since your threads keep looping, you cannot just join them, but you need to signal them it's time to break out of the loop too in order to be able to do that. This docs hint might be useful:
Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
With that, I've put the following example together, which can hopefully help a bit:
from threading import Thread, Event
from time import sleep
def fce(ident, wrap_up_event):
cnt = 0
while True:
print(f"{ident}: {cnt}", wrap_up_event.is_set())
sleep(3)
cnt += 1
if wrap_up_event.is_set():
break
print(f"{ident}: Wrapped up")
if __name__ == '__main__':
wanna_exit = Event()
for i in range(3):
t = Thread(target=fce, args=(i, wanna_exit))
t.start()
sleep(5)
wanna_exit.set()
A single event instance is passed to fce which would just keep running endlessly, but when done with each iteration, before going back to the top check, if the event has been set to True. And before exiting from the script, we set this event to True from the controlling thread. Since the threads are no longer marked as daemon threads, we do not have to explicitly join them.
Depending on how exactly you want to shutdown your script, you will need to handle the incoming signal (SIGTERM perhaps) or KeyboardInterrupt exception for SIGINT. And perform your clean-up before exiting, the mechanics of which remain the same. Apart from not letting python just stop execution right away, you need to let your threads know they should not re-enter the loop and wait for them to be joined.
The SIGINT is a bit simpler, because it's exposed as a python exception and you could do for instance this for the "main" bit:
if __name__ == '__main__':
wanna_exit = Event()
for i in range(3):
t = Thread(target=fce, args=(i, wanna_exit))
t.start()
try:
while True:
sleep(5)
print('Waiting')
except KeyboardInterrupt:
pass
wanna_exit.set()
You can of course send SIGINT to a process with kill and not only from the controlling terminal.
I have a thread waiting on input, but in the event that no input is provided, I need to exit the program. How can i exit the program? in this example the exit should be triggered by keyboard ctrl+c however I would also like to do this without interaction ie via a timeout or other event.
import threading
import signal
import sys
import time
shutdown = False
def shutdownHook(sigNum, currentStackFrame):
global shutdown
print('shutdown')
shutdown = True
def readInput():
print('readInput')
print(sys.stdin.readline())
print('done reading input')
if __name__ == '__main__':
signal.signal(signal.SIGINT, shutdownHook)
signal.signal(signal.SIGTERM, shutdownHook)
inputThread = threading.Thread(name='input', target=readInput)
inputThread.start()
print('started input')
while not shutdown:
time.sleep(1)
print('waiting ' + str(shutdown))
print('current thread' + str(threading.current_thread()))
print('end of program ' + str(shutdown))
sys.exit(0)
You may use signal.alarm() to send a SIGALRM to your program after a certain amount of time (define here in second):
if __name__ == '__main__':
# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, shutdownHook)
signal.alarm(5)
Here is the complete working example from the documentation:
Here is a minimal example program. It uses the alarm() function to
limit the time spent waiting to open a file; this is useful if the
file is for a serial device that may not be turned on, which would
normally cause the os.open() to hang indefinitely. The solution is to
set a 5-second alarm before opening the file; if the operation takes
too long, the alarm signal will be sent, and the handler raises an
exception.
import signal, os
def handler(signum, frame):
print('Signal handler called with signal', signum)
raise OSError("Couldn't open device!")
# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)
# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)
signal.alarm(0) # Disable the alarm
As for why your program is not quitting is because quoted the doc
Python signal handlers are always executed in the main Python thread,
even if the signal was received in another thread. This means that
signals can’t be used as a means of inter-thread communication. You
can use the synchronization primitives from the threading module
instead. Besides, only the main thread is allowed to set a new signal handler.
That means your thread cannot receive no signals the way you design the program. In fact if you try to set a signal in your thread you will receive a ValueError:
ValueError: signal only works in main thread
That's why your program keeps turning after receiving a SIGTERM. Because the thread did not received the signal.
See here: Kill python thread using os for alternative solution.
Make the thread as Deamon thread, this way it will also shutdown when main thread is exited.
inputThread = threading.Thread(name='input', target=readInput)
inputThread.setDaemon(True) # add this line
inputThread.start()
Also you can add a time lapse for no activity within specified period.
time_limit_for_shutdown_in_secs = 10
secs = 0
while not shutdown:
if secs > time_limit_for_shutdown_in_secs: break
time.sleep(1)
print('waiting ' + str(shutdown))
secs += 1
print('current thread' + str(threading.current_thread()))
print('end of program ' + str(shutdown))
sys.exit(0)
I need to process frames from a webcam and send a few selected frames to a remote websocket server. The server answers immediately with a confirmation message (much like an echo server).
Frame processing is slow and cpu intensive so I want to do it using a separate thread pool (producer) to use all the available cores. So the client (consumer) just sits idle until the pool has something to send.
My current implementation, see below, works fine only if I add a small sleep inside the producer test loop. If I remove this delay I stop receiving any answer from the server (both the echo server and from my real server). Even the first answer is lost, so I do not think this is a flood protection mechanism.
What am I doing wrong?
import tornado
from tornado.websocket import websocket_connect
from tornado import gen, queues
import time
class TornadoClient(object):
url = None
onMessageReceived = None
onMessageSent = None
ioloop = tornado.ioloop.IOLoop.current()
q = queues.Queue()
def __init__(self, url, onMessageReceived, onMessageSent):
self.url = url
self.onMessageReceived = onMessageReceived
self.onMessageSent = onMessageSent
def enqueueMessage(self, msgData, binary=False):
print("TornadoClient.enqueueMessage")
self.ioloop.add_callback(self.addToQueue, (msgData, binary))
print("TornadoClient.enqueueMessage done")
#gen.coroutine
def addToQueue(self, msgTuple):
yield self.q.put(msgTuple)
#gen.coroutine
def main_loop(self):
connection = None
try:
while True:
while connection is None:
try:
print("Connecting...")
connection = yield websocket_connect(self.url)
print("Connected " + str(connection))
except Exception, e:
print("Exception on connection " + str(e))
connection = None
print("Retry in a few seconds...")
yield gen.Task(self.ioloop.add_timeout, time.time() + 3)
try:
print("Waiting for data to send...")
msgData, binaryVal = yield self.q.get()
print("Writing...")
sendFuture = connection.write_message(msgData, binary=binaryVal)
print("Write scheduled...")
finally:
self.q.task_done()
yield sendFuture
self.onMessageSent("Sent ok")
print("Write done. Reading...")
msg = yield connection.read_message()
print("Got msg.")
self.onMessageReceived(msg)
if msg is None:
print("Connection lost")
connection = None
print("main loop completed")
except Exception, e:
print("ExceptionExceptionException")
print(e)
connection = None
print("Exit main_loop function")
def start(self):
self.ioloop.run_sync(self.main_loop)
print("Main loop completed")
######### TEST METHODS #########
def sendMessages(client):
time.sleep(2) #TEST only: wait for client startup
while True:
client.enqueueMessage("msgData", binary=False)
time.sleep(1) # <--- comment this line to break it
def testPrintMessage(msg):
print("Received: " + str(msg))
def testPrintSentMessage(msg):
print("Sent: " + msg)
if __name__=='__main__':
from threading import Thread
client = TornadoClient("ws://echo.websocket.org", testPrintMessage, testPrintSentMessage)
thread = Thread(target = sendMessages, args = (client, ))
thread.start()
client.start()
My real problem
In my real program I use a "window like" mechanism to protect the consumer (an autobahn.twisted.websocket server): the producer can send up to a maximum number of un-acknowledge messages (the webcam frames), then stops waiting for half of the window to free up.
The consumer sends a "PROCESSED" message back acknowleding one or more messages (just a counter, not by id).
What I see on the consumer log is that the messages are processed and the answer is sent back but these acks vanish somewhere in the network.
I have little experience with asynchio so I wanted to be sure that I'm not missing any yield, annotation or something else.
This is the consumer side log:
2017-05-13 18:59:54+0200 [-] TX Frame to tcp4:192.168.0.5:48964 : fin = True, rsv = 0, opcode = 1, mask = -, length = 21, repeat_length = None, chopsize = None, sync = False, payload = {"type": "PROCESSED"}
2017-05-13 18:59:54+0200 [-] TX Octets to tcp4:192.168.0.5:48964 : sync = False, octets = 81157b2274797065223a202250524f434553534544227d
This is neat code. I believe the reason you need a sleep in your sendMessages thread is because, otherwise, it keeps calling enqueueMessage as fast as possible, millions of times per second. Since enqueueMessage does not wait for the enqueued message to be processed, it keeps calling IOLoop.add_callback as fast as it can, without giving the loop enough opportunity to execute the callbacks.
The loop might make some progress running on the main thread, since you're not actually blocking it. But the sendMessages thread adds callbacks much faster than the loop can handle them. By the time the loop has popped one message from the queue and has begun to process it, millions of new callbacks are added already, which the loop must execute before it can advance to the next stage of message-processing.
Therefore, for your test code, I think it's correct to sleep between calls to enqueueMessage on the thread.
I wrote a program that uses threads to keep a connection alive while the main program loops until it either has an exception or is manually closed. My program runs in 1 hour intervals and the timeout for the connection is 20 minutes, thus I spawn a thread for every connection element that exist inside of my architecture. Thus, if we have two servers to connect to it connects to both these serves and stays connected and loops through each server retrieving data.
the program I wrote works correctly, however I can't seem to find a way to handle when the program it's self throws an exception. This is to say I can't find an appropriate way to dispose of the threads when the main program excepts. When the program excepts it will just hang open because of the thread not excepting as well and it won't close correctly and will have to be closed manually.
Any suggestions on how to handle cleaning up threads on program exit?
This is my thread:
def keep_vc_alive(vcenter,credentials, api):
vm_url = str(vcenter._proxy.binding.url).split('/')[2]
while True:
try:
logging.info('staying connected %s' % str(vm_url))
vcenter.keep_session_alive()
except:
logging.info('unable to call current time of vcenter %s attempting to reconnect.' % str(vm_url))
try:
vcenter = None
connected,api_version,uuid,vcenter = vcenter_open(60, api, * credentials)
except:
logging.critical('unable to call current time of vcenter %s killing application, please have administrator restart the module.' % str(vm_url))
break
time.sleep(60*10)
Then my exception clean up code is as follows, obviously I know.stop() doesn't work, but I honestly have no idea how to do what it is im trying to do.
except Abort: # Exit without clearing the semaphore
logging.exception('ApplicationError')
try:
config_values_vc = metering_config('VSphere',['vcenter-ip','username','password','api-version'])
for k in xrange(0, len(config_values_vc['username'])): # Loop through each vcenter server
vc_thread[config_values_vc['vcenter-ip'][k]].stop()
except:
pass
#disconnect vcenter
try:
for vcenter in list_of_vc_connections:
list_of_vc_connections[vcenter].disconnect()
except:
pass
try: # Close the db is it is open (db is defined)
db.close()
except:
pass
sys.exit(1)
except SystemExit:
raise
except:
logging.exception('ApplicationError')
semaphore('ComputeLoader', False)
logging.critical('Unexpected error: %s' % sys.exc_info()[0])
raise
Instead of sleeping, wait on a threading.Event():
def keep_vc_alive(vcenter,credentials, api, event): # event is a threading.Event()
vm_url = str(vcenter._proxy.binding.url).split('/')[2]
while not event.is_set(): # If the event got set, we exit the thread
try:
logging.info('staying connected %s' % str(vm_url))
vcenter.keep_session_alive()
except:
logging.info('unable to call current time of vcenter %s attempting to reconnect.' % str(vm_url))
try:
vcenter = None
connected,api_version,uuid,vcenter = vcenter_open(60, api, * credentials)
except:
logging.critical('unable to call current time of vcenter %s killing application, please have administrator restart the module.' % str(vm_url))
break
event.wait(timeout=60*10) # Wait until the timeout expires, or the event is set.
Then, in your main thread, set the event in the exception handling code:
except Abort: # Exit without clearing the semaphore
logging.exception('ApplicationError')
event.set() # keep_alive thread will wake up, see that the event is set, and exit
The generally accepted way to stop threads in python is to use the threading.Event object.
The algorithm followed usually is something like the following:
import threading
...
threads = []
#in the main program
stop_event = threading.Event()
#create thread and store thread and stop_event together
thread = threading.Thread(target=keep_vc_alive, args=(stop_event))
threads.append((thread, stop_event))
#execute thread
thread.start()
...
#in thread (i.e. keep_vc_alive)
# check is_set in stop_event
while not stop_event.is_set():
#receive data from server, etc
...
...
#in exception handler
except Abort:
#set the stop_events
for thread, stop_event in threads:
stop_event.set()
#wait for threads to stop
while 1:
#check for any alive threads
all_finished = True
for thread in threads:
if thread.is_alive():
all_finished = False
#keep cpu down
time.sleep(1)