I'm trying to understand coroutines in python, but I have some troubles grasping how I could end one.
I try to understand the following code :
async def send_recieve():
async with websockets.connect(*parameters*) as _ws:
async def send():
while True:
#function send...
async def recieve():
while True:
#function recieve...
if #condition met:
break
send_result, receive_result = await asyncio.gather(send(), receive())
asyncio.run(send_receive())
When the condition is met, the recieve function is ended, but the send function keep working and I can't end the whole send_recieve async.
I tried to sum up the code to be more clear, I can share the whole version if it's easier to understand.
I get that I miss a condition in the send function fulfilled when the recieve function is ended but I can't understand how I can write it.
If I try to add loop.stop() if the condition is met, it raises the error "RuntimeError: Event loop stopped before Future completed."
The whole code is the following :
async def send_receive():
print(f'Connecting websocket to url ${URL}')
async with websockets.connect(
URL,
extra_headers=(("Authorization", auth_key),),
ping_interval=5,
ping_timeout=20
) as _ws:
await asyncio.sleep(0.3)
print("Receiving SessionBegins ...")
session_begins = await _ws.recv()
print(session_begins)
print("Sending messages ...")
async def send():
while True:
try:
data = stream.read(FRAMES_PER_BUFFER)
data = base64.b64encode(data).decode("utf-8")
json_data = json.dumps({"audio_data":str(data)})
await _ws.send(json_data)
except websockets.exceptions.ConnectionClosedError as e:
print(e)
assert e.code == 4008
break
except Exception as e:
assert False, "Not a websocket 4008 error"
await asyncio.sleep(0.01)
return True
async def receive():
while True:
try:
result_str = await _ws.recv()
majtext = json.loads(result_str)['text']
print(majtext)
except websockets.exceptions.ConnectionClosedError as e:
print(e)
assert e.code == 4008
return
except Exception as e:
assert False, "Not a websocket 4008 error"
if json.loads(result_str)['message_type'] == 'FinalTranscript':
break
send_result, receive_result = await asyncio.gather(send(), receive())
loop = asyncio.get_event_loop()
loop.run_until_complete(send_receive())
how could I tell the send function to end when the receive function is ended
asyncio.gather() waits for both functions to finish. You can instead wait for either function to finish by replacing:
send_result, receive_result = await asyncio.gather(send(), receive())
with:
await asyncio.wait(
[asyncio.create_task(send()), asyncio.create_task(receive())],
return_when=asyncio.FIRST_COMPLETED
)
(Note that "results" of send and receive you retrieved from gather() don't make sense since neither function returns a useful value.)
Related
I am learning Python here with asyncio. I have a Javascript background and am expecting it to work similar to js.
Unfortunately, it says that my coroutine does not have the columns from the chain data frame in the code that proceeds this excerpt. My impression is that it is not awaiting correctly.
async def get_chain(symbol):
try:
chain = await options.get_options_chain(symbol, req["expiration"])
return chain
except Exception as err:
print(err, flush=True)
async def get_price(symbol):
try:
price = await si.get_live_price(symbol)
return price
except Exception as err:
print(err, flush=True)
#app.route("/dev/api/option/rankings", methods=["POST"])
#cross_origin(origin="*")
async def get_option_rankings():
# get saved db symbols
req = request.get_json()
arr = []
for symbol in req["symbols"]:
# get stock price
chain = get_chain(symbol)
price = get_price(symbol)
print(chain, flush=True)
print(price, flush=True)
How do I make it so that I am stuck awaiting something if the value of it isn't true?
for example :
await stuff() # Gets executed
await continueif(x, True) # This only returns when x is True
await morestuff() # Will get to this part when the above statement is executed, which only happens if the condition is true
...
without using while loop
So, ideally
x = True
continueif(x, True)
Should immediately let us pass through
but,
x = False
continueif(x, True)
Should never let us pass through until or unless the value of x becomes True sometime during the execution of the program.
I've created something around your code sample to create something minimally run-able. I've used asyncio.Queue, however if your requirement is for a single event asyncio.Event may be a better choice.
It's worth keeping in mind that asyncio is not multi-processing, rather there is one event loop that shares a single thread while some IO (typically) is blocking.
import asyncio
from time import perf_counter
async def make_true(myx: asyncio.Queue):
await asyncio.sleep(1.0)
await myx.put(False)
await asyncio.sleep(1.0)
await myx.put('hi')
await asyncio.sleep(1.0)
await myx.put(True)
async def stuff():
print(f'{perf_counter()} doing stuff')
async def continueif(myx: asyncio.Queue, condition: bool):
while True:
msg = await myx.get()
if msg == condition:
print(f'{perf_counter()} {msg} is {condition}')
break
else:
print(f'{perf_counter()} {msg} is not {condition}')
async def morestuff():
print(f'{perf_counter()} doing more stuff')
async def main():
x = asyncio.Queue()
asyncio.create_task(make_true(x))
await stuff()
await continueif(x, True)
await morestuff()
if __name__ == '__main__':
asyncio.run(main())
Output is:
249083.3551413 doing stuff
249084.3697383 False is not True
249085.3773555 hi is not True
249086.3854538 True is True
249086.385835 doing more stuff
By the way, the while True is only required to parse multiple messages from the asyncio.Queue. If True is the only message the while True: can be omitted.
I have written code for async pool below. in __aexit__ i'm cancelling the _worker tasks after the tasks get finished. But when i run the code, the worker tasks are not getting cancelled and the code is running forever. This what the task looks like: <Task pending coro=<AsyncPool._worker() running at \async_pool.py:17> wait_for=<Future cancelled>>. The asyncio.wait_for is getting cancelled but not the worker tasks.
class AsyncPool:
def __init__(self,coroutine,no_of_workers,timeout):
self._loop = asyncio.get_event_loop()
self._queue = asyncio.Queue()
self._no_of_workers = no_of_workers
self._coroutine = coroutine
self._timeout = timeout
self._workers = None
async def _worker(self):
while True:
try:
ret = False
queue_item = await self._queue.get()
ret = True
result = await asyncio.wait_for(self._coroutine(queue_item), timeout = self._timeout,loop= self._loop)
except Exception as e:
print(e)
finally:
if ret:
self._queue.task_done()
async def push_to_queue(self,item):
self._queue.put_nowait(item)
async def __aenter__(self):
assert self._workers == None
self._workers = [asyncio.create_task(self._worker()) for _ in range(self._no_of_workers)]
return self
async def __aexit__(self,type,value,traceback):
await self._queue.join()
for worker in self._workers:
worker.cancel()
await asyncio.gather(*self._workers, loop=self._loop, return_exceptions =True)
To use the Asyncpool:
async def something(item):
print("got", item)
await asyncio.sleep(item)
async def main():
async with AsyncPool(something, 5, 2) as pool:
for i in range(10):
await pool.push_to_queue(i)
asyncio.run(main())
The Output in my terminal:
The problem is that your except Exception exception clause also catches cancellation, and ignores it. To add to the confusion, print(e) just prints an empty line in case of a CancelledError, which is where the empty lines in the output come from. (Changing it to print(type(e)) shows what's going on.)
To correct the issue, change except Exception to something more specific, like except asyncio.TimeoutError. This change is not needed in Python 3.8 where asyncio.CancelledError no longer derives from Exception, but from BaseException, so except Exception doesn't catch it.
When you have an asyncio task created and then cancelled, you still have the task alive that need to be "reclaimed". So you want to await worker for it. However, once you await such a cancelled task, as it will never give you back the expected return value, the asyncio.CancelledError will be raised and you need to catch it somewhere.
Because of this behavior, I don't think you should gather them but to await for each of the cancelled tasks, as they are supposed to return right away:
async def __aexit__(self,type,value,traceback):
await self._queue.join()
for worker in self._workers:
worker.cancel()
for worker in self._workers:
try:
await worker
except asyncio.CancelledError:
print("worker cancelled:", worker)
This appears to work. The event is a counting timer and when it expires it cancels the tasks.
import asyncio
from datetime import datetime as dt
from datetime import timedelta as td
import random
import time
class Program:
def __init__(self):
self.duration_in_seconds = 20
self.program_start = dt.now()
self.event_has_expired = False
self.canceled_success = False
async def on_start(self):
print("On Start Event Start! Applying Overrides!!!")
await asyncio.sleep(random.randint(3, 9))
async def on_end(self):
print("On End Releasing All Overrides!")
await asyncio.sleep(random.randint(3, 9))
async def get_sensor_readings(self):
print("getting sensor readings!!!")
await asyncio.sleep(random.randint(3, 9))
async def evauluate_data(self):
print("checking data!!!")
await asyncio.sleep(random.randint(3, 9))
async def check_time(self):
if (dt.now() - self.program_start > td(seconds = self.duration_in_seconds)):
self.event_has_expired = True
print("Event is DONE!!!")
else:
print("Event is not done! ",dt.now() - self.program_start)
async def main(self):
# script starts, do only once self.on_start()
await self.on_start()
print("On Start Done!")
while not self.canceled_success:
readings = asyncio.ensure_future(self.get_sensor_readings())
analysis = asyncio.ensure_future(self.evauluate_data())
checker = asyncio.ensure_future(self.check_time())
if not self.event_has_expired:
await readings
await analysis
await checker
else:
# close other tasks before final shutdown
readings.cancel()
analysis.cancel()
checker.cancel()
self.canceled_success = True
print("cancelled hit!")
# script ends, do only once self.on_end() when even is done
await self.on_end()
print('Done Deal!')
async def main():
program = Program()
await program.main()
Consider the following code:
async def remote_data_retriever():
remote = Remote(sock_path)
while True:
Cached.update_global(remote.get_global())
await asyncio.sleep(RTR_RETR_INTERVAL)
async def on_message(websocket, path):
async for message in websocket:
data = Cached.get_global()
await websocket.send(json.dumps(data.__dict__))
if __name__ == '__main__':
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
ssl_context.load_cert_chain(RTR_CERT_PATH)
app_server = websockets.serve(on_message, RTR_LISTEN_HOST, RTR_LISTEN_PORT, ssl=ssl_context)
try:
asyncio.get_event_loop().run_until_complete(app_server)
print('1')
asyncio.get_event_loop().run_until_complete(remote_data_retriever())
print('2')
asyncio.get_event_loop().run_forever()
except Exception as e:
print(e)
This code will print '1' and then never print '2'. How to correctly schedule a coroutine so it does NOT block on the following call
asyncio.get_event_loop().run_until_complete(remote_data_retriever())
?
run_until_complete(task) starts task at once and waits for its real end, not for await. But your second task uses while True which never ends.
You should rather add task to loop using create_task(task) and start loop later with run_forever().
asyncio.get_event_loop().create_task(app_server)
print('1')
asyncio.get_event_loop().create_task(remote_data_retriever())
print('2')
asyncio.get_event_loop().run_forever()
And then both tasks will run in one loop and await will stop one task to start another task.
Example code which everyone can run
import asyncio
async def task_1():
number = 0
while True:
number += 1
print('task1', number)
await asyncio.sleep(1)
async def task_2():
number = 0
while True:
number += 1
print('task2', number)
await asyncio.sleep(1)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
try:
loop.create_task(task_1())
print('1')
loop.create_task(task_2())
print('2')
loop.run_forever()
except Exception as e:
print(e)
Your exact suggestion doesn't work as create_task() throws exception (correctly) claiming app_server is not a coroutine. However, based on your proposed code, I've been able to get it to work like this:
loop = asyncio.get_event_loop()
try:
asyncio.ensure_future(app_server, loop=loop)
print('1')
loop.create_task(remote_data_retriever())
print('2')
loop.run_forever()
except Exception as e:
print(e)
I'm having a bit of trouble deciding how to break out of this coroutine if the except statement catches. Normally, I would just use:
def f(x):
try:
foo_var = next(a_volitile_generator(x))
except Exception:
print('it broked')
return
yield foo_var
However! I'm going to spawn a gazillion of these generator funcs, make them coroutines, and finally Futureize them & load them into an event-loop. Here's the issue:
loop = asyncio.get_event_loop()
queue = asyncio.Queue(maxsize=100)
async def f(x, result_queue, *, loop=loop):
while _some_condition() is True:
try:
foo_var = await a_volitile_async_generator(x)
except Exception:
print('it broked')
# HELP BELOW
raise StopAsyncIteration # <--ONLY OPTION????????
await result_queue.put(foo_var)
returning anything into the event-loop is a no-no in asyncio because you'll break everything...even if that value is None.
Aside from that, let's also say I'd like to avoid raising StopAsyncIteration...I'm trying to figure out if that's the only way to do this or if I have other options:
As mentioned by #dirn in the comments, you can replace your raise StopAsyncIteration line with a simple break:
async def f(x, result_queue, *, loop=loop):
while _some_condition() is True:
try:
foo_var = await a_volitile_async_generator(x)
except Exception:
print('it broked')
break
await result_queue.put(foo_var)
JFYI, I guess, you're doing something wrong.
yield from a_volitile_async_generator(x)
If a_volitile_async_generator is a real async generator, it won't work: you can't use yield from with it.