We Keep Coding

Why the program stuck at the await statement if using while loop and using lock.locked() checking

I am writing a code to simulate getting data from a website which takes 3-5seconds with multiple selenium drivers but I cannot even get one browser working. I want to implement while loop inside an async function but the program seems stuck at the await asyncio.sleep(random.uniform(3, 5)) statement so the lock is never released so other async basically cannot enter the async with lock_browser:
import asyncio
import random
import time
async def work(lock_browser,x,y):
while True:
if lock_browser.locked() is False:
async with lock_browser:
await asyncio.sleep(random.uniform(3, 5)) #simulate only one browser connect to the website at a time
print(x,y)
break
async def main(x):
tasks=[]
lock_browser = asyncio.Lock()
for y in range(5):
tasks.append(work(lock_browser, x,y))
await asyncio.gather(*tasks)
if __name__ == "__main__":
start = time.perf_counter()
for x in range(5):
asyncio.run(main(x))
end = time.perf_counter()
duration=end-start
print(f'Time elapsed:{duration:.2f}s')
The program will run if I remove the while & break statement. What are the possible causes that the program stuck inside the work() function??

Cancelling asyncio task run in executor

I'm scraping some websites, paralelizing requests library using asyncio:
def run():
asyncio.run(scrape());
def check_link(link):
#.... code code code ...
response = requests.get(link)
#.... code code code ...
write_some_stats_into_db()
async def scrape():
#.... code code code ...
task = asyncio.get_event_loop().run_in_executor(check_link(link));
#.... code code code ...
if done:
for task in all_tasks:
task.cancel();
I only need to find one 'correct' link, after that, I can stop the program. However, because the check_link is run in executor, it's threads are automatically daemonized, thus even after calling taks.cancel(), I have to wait for all of the other still running check_link to complete.
Do you have any ideas how to 'force-kill' the other running checks in the thread executor?

You can do it the following way, actually from my point of view, if you do not have to use asyncio for the task, use only threads without any async loop, since it makes your code more complicated.
import asyncio
from random import randint
import time
from functools import partial
# imagine that this is links array
LINKS = list(range(1000))
# how many thread-worker you want to have simultaneously
WORKERS_NUM = 10
# stops the app
STOP_EVENT = asyncio.Event()
STOP_EVENT.clear()
def check_link(link: str) -> int:
"""checks link in another thread and returns result"""
time.sleep(3)
r = randint(1, 11)
print(f"{link}____{r}\n")
return r
async def check_link_wrapper(q: asyncio.Queue):
"""Async wrapper around sync function"""
loop = asyncio.get_event_loop()
while not STOP_EVENT.is_set():
link = await q.get()
if not link:
break
value = await loop.run_in_executor(None, func=partial(check_link, link))
if value == 10:
STOP_EVENT.set()
print("Hurray! We got TEN !")
async def feeder(q: asyncio.Queue):
"""Send tasks and "poison pill" to all workers"""
# send tasks to workers
for link in LINKS:
await q.put(link)
# ask workers to stop
for _ in range(WORKERS_NUM):
await q.put(None)
async def amain():
"""Main async function of the app"""
# maxsize is one since we want the app
# to stop as fast as possible if stop condition is met
q = asyncio.Queue(maxsize=1)
# we create separate task, since we do not want to await feeder
# we are interested only in workers
asyncio.create_task(feeder(q))
await asyncio.gather(
*[check_link_wrapper(q) for _ in range(WORKERS_NUM)],
)
if __name__ == '__main__':
asyncio.run(amain())

Get async input from console in Python [duplicate]

I'm trying to create a WebSocket command line client that waits for messages from a WebSocket server but waits for user input at the same time.
Regularly polling multiple online sources every second works fine on the server, (the one running at localhost:6789 in this example), but instead of using Python's normal sleep() method, it uses asyncio.sleep(), which makes sense because sleeping and asynchronously sleeping aren't the same thing, at least not under the hood.
Similarly, waiting for user input and asynchronously waiting for user input aren't the same thing, but I can't figure out how to asynchronously wait for user input in the same way that I can asynchronously wait for an arbitrary amount of seconds, so that the client can deal with incoming messages from the WebSocket server while simultaneously waiting for user input.
The comment below in the else-clause of monitor_cmd() hopefully explains what I'm getting at:
import asyncio
import json
import websockets
async def monitor_ws():
uri = 'ws://localhost:6789'
async with websockets.connect(uri) as websocket:
async for message in websocket:
print(json.dumps(json.loads(message), indent=2, sort_keys=True))
async def monitor_cmd():
while True:
sleep_instead = False
if sleep_instead:
await asyncio.sleep(1)
print('Sleeping works fine.')
else:
# Seems like I need the equivalent of:
# line = await asyncio.input('Is this your line? ')
line = input('Is this your line? ')
print(line)
try:
asyncio.get_event_loop().run_until_complete(asyncio.wait([
monitor_ws(),
monitor_cmd()
]))
except KeyboardInterrupt:
quit()
This code just waits for input indefinitely and does nothing else in the meantime, and I understand why. What I don't understand, is how to fix it. :)
Of course, if I'm thinking about this problem in the wrong way, I'd be very happy to learn how to remedy that as well.

You can use the aioconsole third-party package to interact with stdin in an asyncio-friendly manner:
line = await aioconsole.ainput('Is this your line? ')

Borrowing heavily from aioconsole, if you would rather avoid using an external library you could define your own async input function:
async def ainput(string: str) -> str:
await asyncio.get_event_loop().run_in_executor(
None, lambda s=string: sys.stdout.write(s+' '))
return await asyncio.get_event_loop().run_in_executor(
None, sys.stdin.readline)

Borrowing heavily from aioconsole, there are 2 ways to handle.
start a new daemon thread:
import sys
import asyncio
import threading
from concurrent.futures import Future
async def run_as_daemon(func, *args):
future = Future()
future.set_running_or_notify_cancel()
def daemon():
try:
result = func(*args)
except Exception as e:
future.set_exception(e)
else:
future.set_result(result)
threading.Thread(target=daemon, daemon=True).start()
return await asyncio.wrap_future(future)
async def main():
data = await run_as_daemon(sys.stdin.readline)
print(data)
if __name__ == "__main__":
asyncio.run(main())
use stream reader:
import sys
import asyncio
async def get_steam_reader(pipe) -> asyncio.StreamReader:
loop = asyncio.get_event_loop()
reader = asyncio.StreamReader(loop=loop)
protocol = asyncio.StreamReaderProtocol(reader)
await loop.connect_read_pipe(lambda: protocol, pipe)
return reader
async def main():
reader = await get_steam_reader(sys.stdin)
data = await reader.readline()
print(data)
if __name__ == "__main__":
asyncio.run(main())

How to await a select.select call in python asyncio

I have a python 3.6 program where I am using the asyncio package event loops. One of my data sources comes from an api which was not build around asyncio. My connection object contains a member called _connection which is just a python socket. Right now I can use this in a select statement to tell when data is ready.
async def run(self):
while True:
if select.select([self._q._connection], [], [])[0]:
msg = self._q.receive()
print(msg)
What I would really like is...
async def run(self):
while True:
if await select.select([self._q._connection], [], [])[0]:
msg = self._q.receive()
print(msg)
I know there is a sock_recv function in the asyncio event loop however I need the api to do the actual reading and decoding. I tried this but it would just fall through the await which I guess makes sense since I said 0 bytes.
async def run(self):
while True:
print('A')
await asyncio.get_event_loop().sock_recv(self._q._connection, 0)
print('B')
msg = self._q.receive()
print(msg)
The only solution I can think of for now is to add a small timeout to the select and then call asyncio.sleep while there is no data but this seems like an inefficent approach. I wish there was something like asyncio.select. Do anyone want to recommend another approach?
EDIT: Right now I have come up with this. I don't like it because it adds an extra quarter second latency (probably doesn't matter much for my application but it still bugs me.)
async def run(self):
while True:
if select.select([self._q._connection], [], [], 0)[0]:
print(self._q.receive())
else:
await asyncio.sleep(0.25)

You could use loop.add_reader to wait for the read availability of your socket:
async def watch(fd):
future = asyncio.Future()
loop.add_reader(fd, future.set_result, None)
future.add_done_callback(lambda f: loop.remove_reader(fd))
await future
async def run(self):
while True:
await watch(self._q._connection)
msg = self._q.receive()
print(msg)
However, it'll be very tricky to avoid all the blocking IO calls of the library you mentioned without rewriting it completely. Instead, I'd recommend to use the loop.run_in_executor method to schedule the blocking IO calls in a thread pool:
async def run(self):
loop = asyncio.get_event_loop()
while True:
msg = await loop.run_in_executor(None, self._q.receive)
print(msg)

Conditional in a coroutine based on whether it was called again?

I am trying to translate this key "debouncing" logic from Javascript to Python.
function handle_key(key) {
if (this.state == null) {
this.state = ''
}
this.state += key
clearTimeout(this.timeout)
this.timeout = setTimeout(() => {
console.log(this.state)
}, 500)
}
handle_key('a')
handle_key('b')
The idea is that subsequent key presses extend the timeout. The Javascript version prints:
ab
I don't want to translate the JS timeout functions, I'd rather have idiomatic Python using asyncio. My attempt in Python (3.5) is below, but it doesn't work as global_state is not actually updated when I expect.
import asyncio
global_state = ''
#asyncio.coroutine
def handle_key(key):
global global_state
global_state += key
local_state = global_state
yield from asyncio.sleep(0.5)
#if another call hasn't modified global_state we print it
if local_state == global_state:
print(global_state)
#asyncio.coroutine
def main():
yield from handle_key('a')
yield from handle_key('b')
ioloop = asyncio.get_event_loop()
ioloop.run_until_complete(main())
It prints:
a
ab
I have looked into asyncio Event, Queue and Condition but it isn't clear to me how to use them for this. How would you implement the desired behavior using Python's asyncio?
EDIT
Some more details on how I'd like to use handle_keys. I have an async function that checks for key presses.
#asyncio.coroutine
def check_keys():
keys = driver.get_keys()
for key in keys:
yield from handle_key(key)
Which in turn is scheduled along with other program tasks
#asyncio.coroutine
def main():
while True:
yield from check_keys()
yield from do_other_stuff()
ioloop = asyncio.get_event_loop()
ioloop.run_until_complete(main())
Qeek's use of asyncio.create_task and asyncio.gather makes sense. But how would I use it within a loop like this? Or is there another way to schedule the async tasks that would allow handle_keys calls to "overlap"?
Actual code on GitHub if you are interested.

Why your code doesn't work now?
Both handle_key javascript functions don't block execution. Each just clear timeout callback and set new one. It happens immediately.
Coroutines work another way: using yield from or newer syntax await on coroutine means that we want to resume execution flow only after this coroutine if fully done:
async def a():
await asyncio.sleep(1)
async def main():
await a()
await b() # this line would be reached only after a() done - after 1 second delay
asyncio.sleep(0.5) in your code - is not setting callback by timeout, but code that should be done before handle_key finsihed.
Let's try to make code work
You can create task to start execution some coroutine "in background". You can also cancel task (just like you do with clearTimeout(this.timeout)) if you don't want it to be finished.
Python version that emulates your javascript snippet:
import asyncio
from contextlib import suppress
global_state = ''
timeout = None
async def handle_key(key):
global global_state, timeout
global_state += key
# cancel previous callback (clearTimeout(this.timeout))
if timeout:
timeout.cancel()
with suppress(asyncio.CancelledError):
await timeout
# set new callback (this.timeout = setTimeout ...)
async def callback():
await asyncio.sleep(0.5)
print(global_state)
timeout = asyncio.ensure_future(callback())
async def main():
await handle_key('a')
await handle_key('b')
# both handle_key functions done, but task isn't finished yet
# you need to await for task before exit main() coroutine and close loop
if timeout:
await timeout
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main())
finally:
loop.close()
Idiomatic?
While code above works, it is not how asyncio should be used. Your javascript code based on callbacks, while asyncio usually is about to avoid using of callbacks.
It's hard to demonstrate difference on your example since it's callback based by nature (key handling - is some sort of global callback) and doesn't have more async logic. But this understanding would be important later when you'll add more async operations.
Right now I advice you to read about async/await in modern javascript (it's similar to Python's async/await) and look at examples comparing it to callbacks/promises. This article looks good.
It'll help you understand how you can use coroutine-based approach in Python.
Upd:
Since buttons.check needs to periodically call driver.get_buttons() you'll have to use loop. But it can be done as task along with your event loop.
If you had some sort of button_handler(callback) (this is usually how different libs allow to handle user input) you could use it to set some asyncio.Future directly and avoid loop.
Consider possibility write some little gui app with asyncio from the beginning. I think it may help you to better understand how you can adapt your existing project.
Here's some pseudo-code that shows background task to handle
buttons and using asyncio to handle some simple UI events/states logic:
.
import asyncio
from contextlib import suppress
# GUI logic:
async def main():
while True:
print('We at main window, popup closed')
key = await key_pressed
if key == 'Enter':
print('Enter - open some popup')
await popup()
# this place wouldn't be reached until popup is not closed
print('Popup was closed')
elif key == 'Esc':
print('Esc - exit program')
return
async def popup():
while True:
key = await key_pressed
if key == 'Esc':
print('Esc inside popup, let us close it')
return
else:
print('Non escape key inside popup, play sound')
# Event loop logic:
async def button_check():
# Where 'key_pressed' is some global asyncio.Future
# that can be used by your coroutines to know some key is pressed
while True:
global key_pressed
for key in get_buttons():
key_pressed.set_result(key)
key_pressed = asyncio.Future()
await asyncio.sleep(0.01)
def run_my_loop(coro):
loop = asyncio.get_event_loop()
# Run background task to process input
buttons_task = asyncio.ensure_future(button_check())
try:
loop.run_until_complete(main())
finally:
# Shutdown task
buttons_task.cancel()
with suppress(asyncio.CancelledError):
loop.run_until_complete(buttons_task)
loop.close()
if __name__ == '__main__':
run_my_loop(main())

What's wrong
Basically the yield from xy() is very similar to the normal function call. The difference between function call and yield from is that the function call immediately start processing called function. The yield from statement insert called coroutine into queue inside event loop and give control to event loop and it decide which coroutine in it's queue will be processed.
Here is the explanation of what you code does:
It adds the main into event loop's queue.
The event loop start processing coroutine in the queue.
The queue contains only the main coroutine so it starts that.
The code hits the yield from handle_key('a').
It adds the handle_key('a') in the event loop's queue.
The event loop now contains the main and handle_key('a') but the main cannot be started because it is waiting for the result of the handle_key('a').
So the event loop starts the handle_key('a').
It will do some stuff until it hits the yield from asyncio.sleep(0.5).
Now the event loop contains main(), handle_key('a') and sleep(0.5).
The main() is waiting for result from handle_key('a').
The handle_key('a') is waiting for result from sleep(0.5).
The sleep has no dependency so it can be started.
The asyncio.sleep(0.5) returns None after 0.5 second.
The event loop takes the None and return it into the handle_key('a') coroutine.
The return value is ignored because it isn't assign into anything
The handle_key('a') prints the key (because nothing change the state)
The handle_key coroutine at the end return None (because there isn't return statement).
The None is returned to the main.
Again the return value is ignored.
The code hits the yield from handle_key('b') and start processing new key.
It run same steps from step 5 (but with the key b).
How to fix it
The main coroutinr replace with this:
#asyncio.coroutine
def main(loop=asyncio.get_event_loop()):
a_task = loop.create_task(handle_key('a'))
b_task = loop.create_task(handle_key('b'))
yield from asyncio.gather(a_task, b_task)
The loop.create_task adds into the event loop's queue the handle_key('a') and handle_key('b') and then the yield from asyncio.gather(a_task, b_task) give control to the event loop. The event loop from this point contains handle_key('a'), handle_key('b'), gather(...) and main().
The main() wiating for result from gather()
The gather() waiting until all tasks given as parameters are finished
The handle_key('a') and handle_key('b') has no dependencies so they can be started.
The event loop now contains 2 coroutine which can start but which one will it pick? Well... who knows it is implementation depended. So for better simulation of pressed keys this one replace should be a little better:
#asyncio.coroutine
def main(loop=asyncio.get_event_loop()):
a_task = loop.create_task(handle_key('a'))
yield from asyncio.sleep(0.1)
b_task = loop.create_task(handle_key('b'))
yield from asyncio.gather(a_task, b_task)
Python 3.5 bonus
From the documentation:
Coroutines used with asyncio may be implemented using the async def statement.
The async def type of coroutine was added in Python 3.5, and is recommended if there is no need to support older Python versions.
It means that you can replace:
#asyncio.coroutine
def main():
with newer statement
async def main():
If you start using the new syntax then you have to also replace yield from with await.