I'd like to keep a client app running that uses async.run to start its main function:
async def main():
nc = await nats.connect(nats_url)
js = nc.jetstream()
await js.add_stream(name=stream, subjects=[subject])
await js.subscribe(subject, qgroup, cb=do_work)
if __name__ == "__main__":
asyncio.run(main())
Running as above, of course, the program completes immediately. What's the preferred way to keep a client running using asyncio?
After looking at the Python asyncio docs, the closest example i found is this:
Note the asyncio.sleep inside a loop, similar to the traditional pattern of an infinite loop around a blocking function.
async def main():
nc = await nats.connect(nats_url)
js = nc.jetstream()
await js.add_stream(name=stream, subjects=[subject])
await js.subscribe(subject, qgroup, cb=do_work)
while True:
print("loop")
await asyncio.sleep(10)
if __name__ == "__main__":
asyncio.run(main())
I don't know js but for a regular nats client instead of
while True:
print("loop")
await asyncio.sleep(10)
you should add
nc.wait(count=1000)
for getting 1000 messages, or
while True:
print('got a new message:')
nc.wait(count=1)
for running forever
Related
I have two bots, one is using pydle for IRC, like:
async def start_ircbot ():
try:
client = MyOwnBot(NICK,
realname=REALNAME,
sasl_username=SASL_USERNAME,
sasl_password=SASL_PASSWORD,
sasl_identity=SASL_IDENTITY,)
loop = asyncio.get_event_loop()
asyncio.ensure_future(client.connect(HOST, PORT, tls=True, tls_verify=False), loop=loop)
loop.run_forever()
loop.close()
except Exception as e:
print (e)
and another is using telethon for Telegram:
#client.on(events.NewMessage)
async def my_event_handler(event):
...
async def start_client ():
print ("Telegram monitor started...")
await client.start()
await client.run_until_disconnected()
Both of them work without problem separately.
Now, I want to integrate both of them, I tried to launch both of them in my main function like this,
import Notifier
...
async def main():
await asyncio.gather (Notifier.start_client (), start_ircbot ())
asyncio.run(main())
It starts without issue but my_event_handler seems never to get new messages. If I swap the order of functions:
await asyncio.gather (start_ircbot (), Notifier.start_client ())
The script will be stuck at launching, I suspect it has to be something within events loops and tried some different methods but without luck, could anyone shed light on this for me?
Newer Python versions are removing the loop parameter from most methods, so you should try to avoid using it. As long as you don't use asyncio.run (which creates a new loop) or you don't create a new loop yourself, both libraries should be using the default loop from the main thread (I can't speak for pydle, but Telethon does this).
As long as the asyncio event loop is running, Telethon should have no trouble receiving updates. You can use client.loop to make sure it's using the same loop:
tlclient = TelegramClient(...)
irclient = MyOwnBot(...)
#tlclient.on(events.NewMessage)
async def my_event_handler(event):
...
async def main():
await tlclient.start()
await irclient.connect(HOST, PORT, tls=True, tls_verify=False), loop=tlclient.tlclient)
await tlclient.run_until_disconnected() # similar to loop.run_forever() but stops when the client disconnects
client.loop.run_until_complete(main())
I am trying to do something similar like C# ManualResetEvent but in Python.
I have attempted to do it in python but doesn't seem to work.
import asyncio
cond = asyncio.Condition()
async def main():
some_method()
cond.notify()
async def some_method():
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
main()
I want the some_method to start then wait until signaled to start again.
This code is not complete, first of all you need to use asyncio.run() to bootstrap the event loop - this is why your code is not running at all.
Secondly, some_method() never actually starts. You need to asynchronously start some_method() using asyncio.create_task(). When you call an "async def function" (the more correct term is coroutinefunction) it returns a coroutine object, this object needs to be driven by the event loop either by you awaiting it or using the before-mentioned function.
Your code should look more like this:
import asyncio
async def main():
cond = asyncio.Condition()
t = asyncio.create_task(some_method(cond))
# The event loop hasn't had any time to start the task
# until you await again. Sleeping for 0 seconds will let
# the event loop start the task before continuing.
await asyncio.sleep(0)
cond.notify()
# You should never really "fire and forget" tasks,
# the same way you never do with threading. Wait for
# it to complete before returning:
await t
async def some_method(cond):
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
asyncio.run(main())
I use async create_task to run my task in background, but my_task() method not be executed.
async def my_task():
print("starting my task...")
time.sleep(2)
print("finished my task.")
if __name__ == '__main__':
print("1111")
loop = asyncio.get_event_loop()
loop.create_task(my_task())
print("2222")
the result is
1111
2222
There is a simple fix for this, but you still need to use await which you cannot avoid because create tasks returns a coroutine
import asyncio
async def my_task():
print("starting my task...")
await asyncio.sleep(2)
print("finished my task.")
if __name__ == '__main__':
print("1111")
loop = asyncio.get_event_loop()
loop.run_until_complete(my_task())
# or use can use asyncio.run(my_task())
print("2222")
EDIT: Made changes for pyfile, instead of notebook, thanks user4815162342 for pointing out
I'm programming a server in Python3, which takes screenshot and sends it over websockets. I have coroutine for handling connection and I would like to create another coroutine for taking screenshot at some interval. Screenshot coroutine will probably run in different thread and I will need to propagate the result to some shared variable with read-write lock, to be able to send it. My questions: (result should be multiplatform, if possible)
How is it possible to schedule tasks like this? I created server which runs forever, and I can create periodical coroutine, but somehow I can't put them together in one loop.
What is a good way to propagate the result from one thread (or coroutine, if server is single threaded) to another?
I found this piece of code similar to this and I can't get it to work (second coroutine doesn't execute). Can someone correct this with and without multithreading?
async def print_var():
global number
await asyncio.sleep(2)
print(number)
async def inc_var():
global number
await asyncio.sleep(5)
number += 1
number = 0
asyncio.get_event_loop().run_until_complete(print_var())
asyncio.async(inc_var)
asyncio.get_event_loop().run_forever()
Post-answer edit
In the end after more hours of googling, I actually got it to work on a single thread, so there's no danger of race condition. (But I'm still not sure what ensure_future does, and why it isn't called on event loop.)
users = set()
def register(websocket):
users.add(websocket)
def unregister(websocket):
users.remove(websocket)
async def get_screenshot():
global screenshot
while True:
screenshot = screenshot()
await asyncio.sleep(0.2)
async def server(websocket, path):
global screenshot
register(websocket)
try:
async for message in websocket:
respond(screenshot)
finally:
unregister(websocket)
def main():
asyncio.get_event_loop().run_until_complete(
websockets.serve(server, 'localhost', 6789))
asyncio.ensure_future(get_screenshot())
asyncio.get_event_loop().run_forever()
main()
In Python 3.7:
import asyncio
import websockets
CAPTURE_INTERVAL = 1
running = True
queues = set()
async def handle(ws, path):
queue = asyncio.Queue()
queues.add(queue)
while running:
data = await queue.get()
if not data:
break
await ws.send(data)
def capture_screen():
# Do some work here, preferably in C extension without holding the GIL
return b'screenshot data'
async def main():
global running
loop = asyncio.get_running_loop()
server = await websockets.serve(handle, 'localhost', 8765)
try:
while running:
data = await loop.run_in_executor(None, capture_screen)
for queue in queues:
queue.put_nowait(data)
await asyncio.sleep(CAPTURE_INTERVAL)
finally:
running = False
for queue in queues:
queue.put_nowait(None)
server.close()
await server.wait_closed()
if __name__ == '__main__':
asyncio.run(main())
Please note, this is only for demonstrating the producer-consumer fan-out pattern. The queues are not essential - you can simply send data to all server.sockets in main() directly, while in handle() you should worry about incoming websocket messages. For example, client may control image compression rate like this:
import asyncio
import websockets
CAPTURE_INTERVAL = 1
DEFAULT = b'default'
qualities = {}
async def handle(ws, path):
try:
async for req in ws:
qualities[ws] = req
finally:
qualities.pop(ws, None)
def capture_screen():
# Do some work here, preferably in C extension without holding the GIL
return {
DEFAULT: b'default screenshot data',
b'60': b'data at 60% quality',
b'80': b'data at 80% quality',
}
async def main():
loop = asyncio.get_running_loop()
server = await websockets.serve(handle, 'localhost', 8765)
try:
while True:
data = await loop.run_in_executor(None, capture_screen)
for ws in server.sockets:
quality = qualities.get(ws, DEFAULT)
if quality not in data:
quality = DEFAULT
asyncio.create_task(ws.send(data[quality]))
await asyncio.sleep(CAPTURE_INTERVAL)
finally:
server.close()
await server.wait_closed()
if __name__ == '__main__':
asyncio.run(main())
I am using the Sanic as the server and try to handle multiple request concurrently.
I have used the await for the encode function(I use for loop to simulate do something) but when I try the time curl http://0.0.0.0:8000/ in two separate consoles, it doesn't run concurrently.
I have searched google but only find event_loop but it is to schedule registered conroutines.
How do I await the for loop so the requests won't be blocked?
Thank you.
from sanic import Sanic
from sanic import response
from signal import signal, SIGINT
import asyncio
import uvloop
app = Sanic(__name__)
#app.route("/")
async def test(request):
# await asyncio.sleep(5)
await encode()
return response.json({"answer": "42"})
async def encode():
print('encode')
for i in range(0, 300000000):
pass
asyncio.set_event_loop(uvloop.new_event_loop())
server = app.create_server(host="0.0.0.0", port=8000)
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(server)
signal(SIGINT, lambda s, f: loop.stop())
try:
loop.run_forever()
except:
loop.stop()
Running for i in range() is blocking. If you change that to put your await asyncio.sleep(5) into the encode method, you will see that it operates as expected.
#app.route("/")
async def test(request):
await encode()
return response.json({"answer": "42"})
async def encode():
print('encode')
await asyncio.sleep(5)
When you call await encode() and encode is a blocking method, then it still is going to block because you are not "awaiting" anything else. Your thread is still locked up.
You could also add another worker:
app.create_server(worker=2)
Try looking through this answer
Since the async handler is actually running in an eventloop, it is running asynchronously as callback rather than concurrently.
loop.run_forever() would call loop._run_once over and over again to run all the registered event, each await would stop the coroutine and yield control back to the eventloop and eventloop arrange to run the next event.
So basically if you don't want blocking in a long running for-loop, you need to manually hand over the control back to the eventloop inside the for-loop, see the issue about relinquishing control:
async def encode():
print('encode')
for i in range(0, 300000000):
await asyncio.sleep(0)
Here is a quote from Guido:
asyncio.sleep(0) means just that -- let any other tasks run and then
come back here.