I've played around with threading before in Python, but decided to give the asyncio module a try, especially since you can cancel a running task, which seemed like a nice detail. However, for some reason, I can't wrap my head around it.
Here's what I wanted to implement (sorry if I'm using incorrect terminology):
a downloader thread that downloads the same file every x seconds, checks its hash against the previous download and saves it if it's different.
a webserver thread that runs in the background, allowing control (pause, list, stop) of the downloader thread.
I used aiohttp for the webserver.
This is what I have so far:
class aiotest():
def __init__(self):
self._dl = None # downloader future
self._webapp = None # web server future
self.init_server()
def init_server(self):
print('Setting up web interface')
app = web.Application()
app.router.add_route('GET', '/stop', self.stop)
print('added urls')
self._webapp = app
#asyncio.coroutine
def _downloader(self):
while True:
try:
print('Downloading and verifying file...')
# Dummy sleep - to be replaced by actual code
yield from asyncio.sleep(random.randint(3,10))
# Wait a predefined nr of seconds between downloads
yield from asyncio.sleep(30)
except asyncio.CancelledError:
break
#asyncio.coroutine
def _supervisor(self):
print('Starting downloader')
self._dl = asyncio.async(self._downloader())
def start(self):
loop = asyncio.get_event_loop()
loop.run_until_complete(self._supervisor())
loop.close()
#asyncio.coroutine
def stop(self):
print('Received STOP')
self._dl.cancel()
return web.Response(body=b"Stopping... ")
This class is called by:
t = aiotest()
t.start()
This doesn't work of course, and I feel that this is a horrible piece of code.
What's unclear to me:
I stop the downloader in the stop() method, but how would I go about stopping the webserver (e.g. in a shutdown() method)?
Does the downloader need a new event loop, or can I use the loop returned by asyncio.get_event_loop()?
Do I really need something like the supervisor for what I'm trying to implement? This seems so clunky. And how do I get supervisor to keep running instead of ending after a single execution as it does now?
One last, more general question: is asyncio supposed to replace the threading module (in the future)? Or does each have its own application?
I appreciate all the pointers, remarks and clarifications!
Why current code is not working:
You're running event loop until self._supervisor() is complete. self._supervisor() creates task (it happens immediately) and finishes immediately.
You're trying to run event loop until _supervisor complete, but how and when are you going start server? I think event loop should be running until server stopped. _supervisor or other stuff can be added as task (to same event loop). aiohttp already has function to start server and event loop - web.run_app, but we can do it manually.
Your questions:
Your server will run until you stop it. You can start/stop different
coroutines while your server working.
You need only one event loop for different coroutines.
I think you don't need supervisor.
More general question: asyncio helps you to run different
functions parallel in single thread in single process. That's why
asyncio is so cool and fast. Some of your sync code with threads you
can rewrite using asyncio and it's coroutines. Moreover: asyncio can
interact with threads and processes.
It can be useful in case you still need threads and processes: here's example.
Useful notes:
It's better to use term coroutine instead of thread while we talk about asyncio coroutines that are not threads
If you use Python 3.5, you can use async/await syntax
instead of coroutine/yield from
I rewrote your code to show you idea. How to check it: run program, see console, open http://localhost:8080/stop, see console, open http://localhost:8080/start, see console, type CTRL+C.
import asyncio
import random
from contextlib import suppress
from aiohttp import web
class aiotest():
def __init__(self):
self._webapp = None
self._d_task = None
self.init_server()
# SERVER:
def init_server(self):
app = web.Application()
app.router.add_route('GET', '/start', self.start)
app.router.add_route('GET', '/stop', self.stop)
app.router.add_route('GET', '/kill_server', self.kill_server)
self._webapp = app
def run_server(self):
# Create server:
loop = asyncio.get_event_loop()
handler = self._webapp.make_handler()
f = loop.create_server(handler, '0.0.0.0', 8080)
srv = loop.run_until_complete(f)
try:
# Start downloader at server start:
asyncio.async(self.start(None)) # I'm using controllers here and below to be short,
# but it's better to split controller and start func
# Start server:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
# Stop downloader when server stopped:
loop.run_until_complete(self.stop(None))
# Cleanup resources:
srv.close()
loop.run_until_complete(srv.wait_closed())
loop.run_until_complete(self._webapp.shutdown())
loop.run_until_complete(handler.finish_connections(60.0))
loop.run_until_complete(self._webapp.cleanup())
loop.close()
#asyncio.coroutine
def kill_server(self, request):
print('Server killing...')
loop = asyncio.get_event_loop()
loop.stop()
return web.Response(body=b"Server killed")
# DOWNLOADER
#asyncio.coroutine
def start(self, request):
if self._d_task is None:
print('Downloader starting...')
self._d_task = asyncio.async(self._downloader())
return web.Response(body=b"Downloader started")
else:
return web.Response(body=b"Downloader already started")
#asyncio.coroutine
def stop(self, request):
if (self._d_task is not None) and (not self._d_task.cancelled()):
print('Downloader stopping...')
self._d_task.cancel()
# cancel() just say task it should be cancelled
# to able task handle CancelledError await for it
with suppress(asyncio.CancelledError):
yield from self._d_task
self._d_task = None
return web.Response(body=b"Downloader stopped")
else:
return web.Response(body=b"Downloader already stopped or stopping")
#asyncio.coroutine
def _downloader(self):
while True:
print('Downloading and verifying file...')
# Dummy sleep - to be replaced by actual code
yield from asyncio.sleep(random.randint(1, 2))
# Wait a predefined nr of seconds between downloads
yield from asyncio.sleep(1)
if __name__ == '__main__':
t = aiotest()
t.run_server()
Related
I am writing a pyModbus server with asyncio, based on this example.
Alongside the server I've got a serial device which I'm communicating with and a server updating task.
One task should check the status of the serial device every 500ms.
The server updating task should check if there are any changes in the status of the serial device and update the info on the server. Moreover, if there is a request waiting on the server it should call another task which will send necessary info to the serial device.
My three questions are:
How should I stop the server politely? For now the app is running only in console so it is stopped by ctrl+c - how can I stop the server without causing an avalanche of errors?
How can I implement tasks to be executed cyclically (let's say I want to frefresh the server data every 500ms)? I've found the aiocron module but as far as I can tell its functionalities are a bit limtied as it is intended just for calling functions in intervals.
How can I politely cancel all the tasks before stopping the server (the infinitely, cyclically running ones) when closing the app?
Thanks!
EDIT:
Speaking of running cyclical tasks and cancelling them - is this a proper way to do that? This doesn't rise any errors but does it clean eveything correctly? (I created this sketch compiling a dozen of questions on stackoverflow, I am not sure if this makes sense)
import asyncio
async def periodic():
try:
while True:
print('periodic')
await asyncio.sleep(1)
except asyncio.CancelledError as ex:
print('task1', type(ex))
raise
async def periodic2():
try:
while True:
print('periodic2')
await asyncio.sleep(0.5)
except asyncio.CancelledError as ex:
print('task2', type(ex))
raise
async def main():
tasks = []
task = asyncio.create_task(periodic())
tasks.append(task)
task2 = asyncio.create_task(periodic2())
tasks.append(task2)
for task in tasks:
await task
if __name__ == "__main__":
try:
asyncio.run(main())
except KeyboardInterrupt:
pass
Imagine an async scenario with a function calling a blocking read() in a loop which for that reason gets executed in a ThreadPoolExecutor:
def read_from_io(loop, event_queue):
while True:
event = some_io_resource.read()
asyncio.run_coroutine_threadsafe(event_queue.put(event), loop=loop)
event_queue = asyncio.Queue()
asyncio.set_event_loop(loop := asyncio.new_event_loop())
with concurrent.futures.ThreadPoolExecutor() as pool:
reader_task = loop.run_in_executor(pool, read_from_io, loop, event_queue)
try:
loop.run_forever()
except KeyboardInterrupt:
#
# how to signal reader_task to stop?
Is there a clean way to stop this task? From what I read in tutorials and documentation neither reader_task.cancel() nor pool.shutdown() has an effect in this (threaded) scenario.
The next common answer you find involves a threading.Event() being fired in the main thread - but that doesn't work with the blocking function in read_from_io, right?
The only generic way to terminate a blocking function I came up with was to "send" an exception to that thread, like described here or here. But that looks ugly:
def raise_exception(thread, exception_type):
res = ctypes.pythonapi.PyThreadState_SetAsyncExc(
thread.get_id(),
ctypes.py_object(exception_type))
if res > 1:
ctypes.pythonapi.PyThreadState_SetAsyncExc(thread.get_id(), 0)
print('Exception raise failure')
With Python3.11+, asyncio, concurrent and stuff - is there a more elegant way to stop a blocking task created for exactly that reason?
I have an app that runs in asyncio loop. I wish to start a thread which will execute a piece of blocking code:
def run(self)->bool:
self._running = True
while self._running:
#MOVE FORWORD / BACKWORDS
if kb.is_pressed('x'):
self.move_car("forward")
elif kb.is_pressed('w'):
self.move_car("backward")
return True
until I decide to stop it and manually set the self._running = False by calling:
def stop(self):
self._running = False
These are both methods of a class controlling the whole operation of a raspberry pi robot-car I made.
I want this to run on a separate thread so that my main application can still listen to my input and stop the thread while the other thread is running in this while loop you can see above.
How can I achieve that?Note For sending the start and stop signal I use http requests but this does not affect the core of my question.
You can run the code of your blocking function run inside the default executor of the loop. This is documented here Executing code in thread or process pools.
async def main():
# asumming you have a class `Interface`
# that contains `run` and an async method `listen_for_stop`.
loop = asyncio.get_running_loop()
inter = Interface()
run_task = loop.run_in_executor(None, inter.run)
results = await asyncio.gather(run_task, inter.listen_for_stop())
With asyncio.gather you await for the execution of the two tasks concurrently.
Also you should check Running Tasks Concurrently.
I have an XPUB/XSUB device and a number of mock publishers running in one process. In a separate process, I want to connect a subscriber and print received message to the terminal. Below I will show two variants of a simple function to do just that. I have these functions wrapped as command-line utilities.
My problem is that the asyncio variant never receives messages.
On the other hand, the non-async variant works just fine. I have tested all cases for ipc and tcp transports. The publishing process never changes in my tests, except when I restart it to change transport. The messages are short strings and published roughly once per second, so we're not looking at performance problem.
The subscriber program sits indefinitely at the line msg = await sock.receive_multipart(). In the XPUB/XSUB device I have instrumentation that shows the forwarding of the sock.setsockopt(zmq.SUBSCRIBE, channel.encode()) message, same as when the non-async variant connects.
The asyncio variant (not working, as described)
def subs(url, channel):
import asyncio
import zmq
import zmq.asyncio
ctx = zmq.asyncio.Context.instance()
sock = ctx.socket(zmq.SUB)
sock.connect(url)
sock.setsockopt(zmq.SUBSCRIBE, channel.encode())
async def task():
while True:
msg = await sock.recv_multipart()
print(' | '.join(m.decode() for m in msg))
try:
asyncio.run(task())
finally:
sock.setsockopt(zmq.LINGER, 0)
sock.close()
The regular blocking variant (works fine)
def subs(url, channel):
import zmq
ctx = zmq.Context.instance()
sock = ctx.socket(zmq.SUB)
sock.connect(url)
sock.setsockopt(zmq.SUBSCRIBE, channel.encode())
def task():
while True:
msg = sock.recv_multipart()
print(' | '.join(m.decode() for m in msg))
try:
task()
finally:
sock.setsockopt(zmq.LINGER, 0)
sock.close()
For this particular tool there is no need to use asyncio. However, I am experiencing this problem elsewhere in my code too, where an asynchronous recv never receives. So I'm hoping that by clearing it up in this simple case I'll understand what's going wrong in general.
My versions are
import zmq
zmq.zmq_version() # '4.3.2'
zmq.__version__ # '19.0.2'
I'm on MacOS 10.13.6.
I'm fully out of ideas. Internet, please help!
A working async variant is
def subs(url, channel):
import asyncio
import zmq
import zmq.asyncio
ctx = zmq.asyncio.Context.instance()
async def task():
sock = ctx.socket(zmq.SUB)
sock.connect(url)
sock.setsockopt(zmq.SUBSCRIBE, channel.encode())
try:
while True:
msg = await sock.recv_multipart()
print(' | '.join(m.decode() for m in msg))
finally:
sock.setsockopt(zmq.LINGER, 0)
sock.close()
asyncio.run(task())
I conclude that, when using asyncio zmq, sockets must be created with a call running on the event loop from which the sockets will be awaited. Even though the original form did not do anything fancy with event loops, it appears that the socket has an event loop different from that used by asyncio.run. I'm not sure why, and I didn't open an issue with pyzmq because their docs show usage as in this answer, without comment.
Edit in response to a comment:
asyncio.run always creates a new event loop, so the loop presumably created for the sockets instantiated outside of the co-routine passed to asyncio.run (as in the asyncio variant in the original question) is obviously different.
I am trying to run the following test using pika 0.10.0 from github:
import logging
import sys
import pika
import threading
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
URL = 'amqp://guest:guest#127.0.0.1:5672/%2F?socket_timeout=0.25'
class BaseConsumer(threading.Thread):
def run(self):
self._connection = None
self._channel = None
self.connect()
self.open_channel()
self.consume()
def connect(self):
parameters = pika.URLParameters(URL)
self._connection = pika.BlockingConnection(parameters)
def open_channel(self):
self._channel = self._connection.channel()
self._channel.exchange_declare(exchange='exc1', exchange_type='topic', passive=False,
durable=False, auto_delete=False, internal=False, arguments=None)
self._channel.queue_declare(queue='test', passive=False, durable=False,
exclusive=False, auto_delete=False, arguments=None)
self._channel.queue_bind(
'test', 'exc1', routing_key='rk', arguments=None)
def consume(self):
self._channel.basic_consume(self.on_message, 'test')
try:
self._channel.start_consuming()
except KeyboardInterrupt:
logging.info("Stop consuming now!")
self._channel.stop_consuming()
self._connection.close()
def on_message(self, channel, method_frame, header_frame, body):
print method_frame.delivery_tag
print body
channel.basic_ack(delivery_tag=method_frame.delivery_tag)
c1 = BaseConsumer()
c1.setDaemon(False)
c1.start()
The script is connecting to my MQ and is apparently able to consume messages from the MQ. The problem is that I have no way of stoping the thread. Pressing CTRL-C on my keyboard only causes "^C" to appear in the console without interrupting the consume.
Question is, how do I make pika stop consuming when it is running inside of a thread ? I would like to note that I am following the guidelines of creating the connection in the consumer thread.
If after starting the thread with c1.start() I also do an infinite while loop and log something from there then pressing CTRL-C will end the while loop but the consumer thread will still ignore any additional CTRL-C.
Side question: is it possible to stop consuming inside the thread with some outside signalling like a threading.Condition or something ? I don't see how i can interfere with start_consuming.
Question: ... from there then pressing CTRL-C will end the while loop
Add a def stop() to your BaseConsumer,
catch the KeyboardInterrupt and call stop().
try:
BaseConsumer.run()
except KeyboardInterrupt:
BaseConsumer.stop()
Read pika: asynchronous_consumer_example