Firstly, an apology. I'm pretty new to Python. I come from a Java/C# coding background. I'm loving Pythons simplicity in many ways, but also finding some standards hard to pin down.
For instance, I've successfully managed to get a Discord Bot running. The async methods are working well. But I would like to schedule a job to run every (say) 30 minutes. However, when I type asyncio.run(job()), Python tells me that "run" is not an attribute of asyncio. I'm really not sure why it would say that. Heck, is asyncio even the "right" way to do this?
Is it possible the discord import has masked it in some way? I think I might need to get a book or something!
Again, thanks. I did try a search on this, but nothing came up!
on_ready is called when the discord bot gets initiated so one way is to attach your job to it:
import discord
import asyncio
client = discord.Client()
#client.event
async def on_ready():
while True:
await asyncio.sleep(30*60) # every 30 minutes
job()
client.run(os.environ.get('DISCORD_BOT_SECRET')) # provide your API token here!!
asyncio.sleep is a non-blocking sleep -- if one were to use time.sleep here, then the bot would wait for time.sleep to complete and would be unresponsive to any other messages coming in. But what await asyncio.sleep does is yield control back to the event loop which can take care of other bot functions. Only after 30 minutes control will return to on_ready.
Note that while your job runs it will block your bot which is a problem for jobs tasking longer than a few seconds. If your job is I/O-based (e.g. fetching websites) you can use async I/O operations (such as aiohttp) to keep it responsive. If your job is CPU-based you might have to use multiple processes, such as subprocess.Popen if your job can be invoked with a terminal command.
The modern way to schedule a job every 30 minutes is using discord.ext.tasks:
import asyncio
import contextlib
import discord
from discord.ext import tasks
client = discord.Client()
#tasks.loop(minutes=30) # every 30 minutes
async def job():
with contextlib.suppress(Exception):
... # your code here
job.start()
client.run(os.environ.get('DISCORD_BOT_SECRET')) # provide your API token here!!
The function job() will be called every 30 minutes by the Discord client's event loop. A couple notes here:
If job() throws an exception, the ENTIRE LOOP will stop. That is right, not just this invocation, but the entire loop will exit. That is why I added a with-statement to catch all exceptions, this is equivalent to try: ... except: pass.
While job() is running, the other parts of your bot are blocked. This is because async is not implemented using multiple threads but using coroutines in a single thread. A way around this could be to make your code async (e.g. using aiohttp for web requests) or to start another process using subprocess.Popen.
Related
Context: I am making a discord bot, and the mute command comes with a duration specifier for when the user should be unmuted again (this is being done via assigning a role to the user that removes their ability to send messages).
Idea 1: I could make a loop that checks every, say, 30 seconds and looks through to see which mutes have expired and cancel them.
Idea 2: Each time a mute is processed, I could await asyncio.sleep(however long) and then cancel it.
I would like to ask - which is more idiomatic, and also more importantly, which is more efficient? The first one has the advantage that it only has one coroutine running whereas the last one spawns a new one for each individual case (I don't really know how many that might be, but let's say around 10 concurrent cases maximum). However, the last one is easier for me to implement and feels like a more direct solution, on top of making the unmute happen exactly on time instead of on a timed loop.
There is one idea to make a loop that awaits until the next task and then when it processes that, queue up the next one, but then you can't insert tasks at the front of the queue.
TL;DR - if I need to schedule multiple events, do I run a loop to constantly check the scheduler, or do I open a coroutine for each and just await asyncio.sleep(however long) each of them?
Your second idea seems the most accurate and straightforward as you can simply use asyncio.create_task to run an unmute coroutine after a certain amount of time. By using asyncio.create_task, you do not need to return or yield back the unmute coroutine to be gathered later or submit it to be checked by a loop at an established interval, for awaiting the returned response from asyncio.create_task will run the task in the background:
import asyncio
async def unmute(delay, user_id):
await asyncio.sleep(delay)
#run unmute process
async def mute(user_id):
#your mute function
mute_for = await do_mute(user_id) #some mute function
u = asyncio.create_task(unmute(mute_for, user_id))
await u #submitting the task to be run in the background
I'm trying to run an infinite loop to check a web server every 20 seconds, and if it finds something, I want it to send a message to a discord channel via my discord bot. However, I'm neither quite sure how asyncio works because I haven't used async/await much nor do I know how to actually implement this.
I've tried a few things:
async def poll():
...
await send()
threading.Thread(target = poll).start()
This fails because I don't await the poll function. If I don't include the async in async def poll that obviously fails because then the await send() isn't valid.
async def poll():
...
await send()
asyncio.run_coroutine_threadsafe(poll(), asyncio.get_event_loop()) # this freezes the rest of my program and prevents my discord bot from working correctly
asyncio.run_coroutine_threadsafe(poll(), asyncio.new_event_loop()) # this warns me saying "coroutine poll was never awaited" and doesn't seem to execute the loop
I doubt I'm supposed to use threads with asyncio. But how would I make an infinite loop run in parallel to the rest of my code?
If you want this for a discord bot and using discord.py, then you can use discord.ext.tasks.loop or a background task.
Loop Example: Here
Background Task Example: Here
None of this will affect your bot, until you are not using a blocking module like requests or time.sleep()
Hi I am new to asyncio and concept of event loops (Non-blocking IO)
async def subWorker():
...
async def firstWorker():
await subWorker()
async def secondWorker():
await asyncio.sleep(1)
loop = asyncio.get_event_loop()
asyncio.ensure_future(firstWorker())
asyncio.ensure_future(secondWorker())
loop.run_forever()
here, when code starts, firstWorker() is executed and paused until it encounters await subWorker(). While firstWorker() is waiting, secondWorker() gets started.
Question is, when firstWorker() encounters await subWorker() and gets paused, the computer then will execute subWorker() and secondWorker() at the same time. Since the program has only 1 thread now, and I guess the single thread does secondWorker() work. Then who executes subWorker()? If single thread can only do 1 thing at a time, who else does the other jobs?
The assumption that subWorker and secondWorker execute at the same time is false.
The fact that secondWorker simply sleeps means that the available time will be spent in subWorker.
asyncio by definition is single-threaded; see the documentation:
This module provides infrastructure for writing single-threaded concurrent code
The event loop executes a task at a time, switching for example when one task is blocked while waiting for I/O, or, as here, voluntarily sleeping.
This is a little old now, but I've found the visualization from the gevent docs (about 1 screen down, beneath "Synchronous & Asynchronous Execution") to be helpful while teaching asynchronous flow control to coworkers: http://sdiehl.github.io/gevent-tutorial/
The most important point here is that only one coroutine is running at any one time, even though many may be in process.
I've a trouble understand the reason for the below code, hope someone can shed some lights on this. I'm new in async programming.
This is from websockets documentation
#!/usr/bin/env python
import asyncio
import websockets
async def echo(websocket, path):
async for message in websocket:
await websocket.send(message)
asyncio.get_event_loop().run_until_complete(
websockets.serve(echo, 'localhost', 8765))
asyncio.get_event_loop().run_forever()
I have some questions about asyncio design and how I can take advantage of this.
First of all, looking at the last two lines. If I understand correctly, shouldn't run_until_complete shutdown after finished its work? How can the second loop kept it alive with no jobs submitted into the loop?
Secondly, I was trying to build a back_end that can process some data from the front_end using websockets and return the calculation in realtime. There will be two kinds of tasks, one is bit longer that need computation power a bit but will happen one time for a session, and bunch of streaming data processing that need to be send back immediately (90 frames per seconds).
For the bigger tasks, should I just fire up another websocket server that process the longer work, and use the main websocket to consume on it? Or use another process to do the work in a chained async function? And for the smaller tasks, what would go wrong if I do the same as above?
TLDR:
Async programming is hammering my brain.
Maintain a nonblocking session
Which process hard work and light work simultaneously
The light work should have zero latency
The hard work should be as fast as possible but not effecting the light work.
Thanks!!
In my Python console app, I have a simple for loop that looks like this:
for package in page.packages:
package.load()
# do stuff with package
Every time it's called, package.load() makes a series of HTTP requests. Since page.packages typically contains thousands of packages, the load() call is becoming a substantial bottleneck for my app.
To speed things up, I've thought about using the multiprocessing module to do parallelization, but that would still waste a lot of resources because the threads are network-bound, not CPU-bound: instead of having 1 thread waiting around doing nothing, you'd have 4 of them. Is it possible to use asynchrony instead to somehow just use one/a few threads, but make sure they're never waiting around for the network?
asyncio is an excellent fit for this, but you will need to convert your HTTP loading code Package.load to async using something like aiohttp. For example:
async def load(self):
with self._session.get(self.uri, params) as resp:
data = resp.read() # etc
The sequential loop you've had previously would be expressed as:
async def load_all_serial(page):
for package in page.packages:
await package.load()
But now you also have the option to start the downloads in parallel:
async def load_all_parallel(page):
# just create tasks, do not await them yet
tasks = [package.load() for package in page.packages]
# now let them run, and wait until all have been completed
await asyncio.gather(*tasks)
Calling either of these async functions from synchronous code is as simple as:
loop = asyncio.get_event_loop()
loop.run_until_complete(load_all_parallel(page))