We Keep Coding

How to really make this operation async in python?

I have very recently started checking out asyncio for python. The use case is that, I hit an endpoint /refresh_data and the data is refreshed (from an S3 bucket). But this should be a non blocking operation, other API endpoints should still be able to be serviced.
So in my controller, I have:
def refresh_data():
myservice.refresh_data()
return jsonify(dict(ok=True))
and in my service I have:
async def refresh_data():
try:
s_result = await self.s3_client.fetch()
except (FileNotFound, IOError) as e:
logger.info("problem")
gather = {i.pop("x"):i async for i in summary_result}
# ... some other stuff
and in my client:
async def fetch():
result = pd.read_parquet("bucket", "pyarrow", cols, filts).to_dict(orient="col1")
return result
And when I run it, I see this error:
TypeError: 'async for' requires an object with __aiter__ method, got coroutine
I don't know how to move past this. Adding async definitely makes it return a coroutine type - but, either I have implemented this messily or I haven't fully understood asyncio package in Python. I have been working off of simple examples but I'm not sure what's wrong with what I've done here.

Concurrent execution of two python methods

I'm creating a script that is posting a message to both discord and twitter, depending on some input. I have to methods (in separate .py files), post_to_twitter and post_to_discord. What I want to achieve is that both of these try to execute even if the other fails (e.g. if there is some exception with login).
Here is the relevant code snippet for posting to discord:
def post_to_discord(message, channel_name):
client = discord.Client()
#client.event
async def on_ready():
channel = # getting the right channel
await channel.send(message)
await client.close()
client.run(discord_config.token)
and here is the snippet for posting to twitter part (stripped from the try-except blocks):
def post_to_twitter(message):
auth = tweepy.OAuthHandler(twitter_config.api_key, twitter_config.api_key_secret)
auth.set_access_token(twitter_config.access_token, twitter_config.access_token_secret)
api = tweepy.API(auth)
api.update_status(message)
Now, both of these work perfectly fine on their own and when being called synchronously from the same method:
def main(message):
post_discord.post_to_discord(message)
post_tweet.post_to_twitter(message)
However, I just cannot get them to work concurrently (i.e. to try to post to twitter even if discord fails or vice-versa). I've already tried a couple of different approaches with multi-threading and with asyncio.
Among others, I've tried the solution from this question. But got an error No module named 'IPython'. When I omitted the IPython line, changed the methods to async, I got this error: RuntimeError: Cannot enter into task <ClientEventTask state=pending event=on_ready coro=<function post_to_discord.<locals>.on_ready at 0x7f0ee33e9550>> while another task <Task pending name='Task-1' coro=<main() running at post_main.py:31>> is being executed..
To be honest, I'm not even sure if asyncio would be the right approach for my use case, so any insight is much appreciated.
Thank you.

In this case running the two things in completely separate threads (and completely separate event loops) is probably the easiest option at your level of expertise. For example, try this:
import post_to_discord, post_to_twitter
import concurrent.futures
def main(message):
with concurrent.futures.ThreadPoolExecutor() as pool:
fut1 = pool.submit(post_discord.post_to_discord, message)
fut2 = pool.submit(post_tweet.post_to_twitter, message)
# here closing the threadpool will wait for both futures to complete
# make exceptions visible
for fut in (fut1, fut2):
try:
fut.result()
except Exception as e:
print("error: ", e)

Python 3.9: How to correctly await a lock in non-async function

TL;DR: Q: How to lock protect a shared resource from a sync function in py39?
Sorry if this was already answered - I just couldn't find it easily.
I'm struggling how to correctly await a lock when accessing a shared resourced.
My API is accessing a backend API which spends forever logging in, so I'm caching the auth token, and renewing whenever it expires:
async def _login():
global connection_lock
while True:
async with connection_lock:
# return a new connection or the one which has been cached in the meantime
# or raise whichever error received while attempting to login
class BackendSystem:
#property
def client(self):
if self._client is None or cache().get('headers') is None:
self._client = loop.run_until_complete(_login())
return self._client
Now the issue is that in certain cases several requests (likely in parallel) to flask will be issued to flask in parallel, causing this error:
File "/usr/lib/python3.6/asyncio/base_events.py", line 471, in run_until_complete
self.run_forever()
File "/usr/lib/python3.6/asyncio/base_events.py", line 425, in run_forever
raise RuntimeError('This event loop is already running')
And I did find suggestions to install nest-asyncio however I'm not sure that is truly what I need TBH. If I understand correctly then the loops isn't nested (ie. run from within each other, but I could very well be wrong), but rather I'm trying to use the same loop which is already running - and possibly this part is illegal?
It just strikes me as incredible that it should be this hard to do the very basic lock protection of a shared resource from a function which isn't async, and which I don't have any intent to convert into being async.

I created a Reproducible Example (it helps a lot !) :
import asyncio
connection_lock = asyncio.Lock()
loop = asyncio.get_event_loop()
class Client:
def __init__(self, username: str):
self.username = username
def __str__(self):
return f"Client({self.username!r})"
async def _login(username) -> Client:
global connection_lock
while True:
print(f"{username} will lock")
async with connection_lock:
print(f"{username} got the lock")
await asyncio.sleep(5) # sleep a bit
print(f"{username} has finished")
return Client(username)
class BackendSystem:
def __init__(self, username: str):
self._client = None
self.username = username
#property
def client(self):
if self._client is None:
self._client = loop.run_until_complete(_login(self.username))
return self._client
def main1():
def do_something(username: str):
print(BackendSystem(username).client)
for username in ["Steffen", "Lenormju"]:
do_something(username)
def main2():
async def do_something(username: str):
print(BackendSystem(username).client)
future = asyncio.gather(
do_something("Steffen"), do_something("Lenormju")
)
results = loop.run_until_complete(future)
return results
if __name__ == '__main__':
print("main1 ================")
main1()
print("main2 ================")
main2()
The output is :
main1 ================
Steffen will lock
Steffen got the lock
Steffen has finished
Client('Steffen')
Lenormju will lock
Lenormju got the lock
Lenormju has finished
Client('Lenormju')
main2 ================
Lenormju will lock
Lenormju got the lock
Steffen will lock
Traceback (most recent call last):
File "C:/PycharmProjects/stack_overflow/68159604.py", line 62, in <module>
main2()
File "C:/PycharmProjects/stack_overflow/68159604.py", line 54, in main2
results = loop.run_until_complete(future)
File "C:\Program Files\Python3.6.8\lib\asyncio\base_events.py", line 484, in run_until_complete
return future.result()
File "C:/PycharmProjects/stack_overflow/68159604.py", line 49, in do_something
print(BackendSystem(username).client)
File "C:/PycharmProjects/stack_overflow/68159604.py", line 35, in client
self._client = loop.run_until_complete(_login(self.username))
File "C:\Program Files\Python3.6.8\lib\asyncio\base_events.py", line 471, in run_until_complete
self.run_forever()
File "C:\Program Files\Python3.6.8\lib\asyncio\base_events.py", line 425, in run_forever
raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
The problem occurs when there are concurrent requests, the main1 shows that sequential works fine. It confirms that you encounter it when the code is run by Flask (already in an event loop).
A more minimal example is :
import asyncio
loop = asyncio.get_event_loop()
if __name__ == '__main__':
async def lock_5_seconds():
await asyncio.sleep(5)
def do_synchronously_something_async():
loop.run_until_complete(lock_5_seconds())
async def do_something_async():
do_synchronously_something_async()
loop.run_until_complete(
asyncio.gather(
do_something_async(),
do_something_async()
)
)
Looking at the asyncio.loop.run_until_complete documentation :
Run until the future [...] has completed.
As explained in this answer :
Event loop's method such as run_forever or run_until_complete — are just a ways to start event loop in general.
You must not call them several times if you never stopped it. Flask starts it once for you, you should not start it again yourself.
But I guess (re)starting the event loop was not what you meant.
Your actual problem is that you want to synchronously call an async function. Indeed, async functions get scheduled to run on the event loop. You can't just call them directly.
But this question's answers tell you to just call loop.run_until_complete, which does not work in your case because you already have an event loop running.
Here is a discussion about a similar case on Reddit : Calling async functions from synchronous functions. And one on StackOverflow from someone using FastAPI : Python: Call asynchronous code from synchronous method when there is already an event loop running.
The conclusion is that there is NO way to do exactly what you want.
I think you should change your design : the call to your client property is currently synchronous, although it has to call asynchronous (slowwww) code (the _login function).
You did not say what the rest of your code does, but if it's a wrapper around a remote API I recommend something like that :
async def login():
...
class BackendSystem():
async def ensure_is_logged():
...
async def call_this_api_method():
await self.ensure_is_logged()
...
and embrace asynchronous code.
Or just don't make your login function async. Mixing both is a recipe for headaches.

Can you have an async handler in Lambda Python 3.6?

I've made Lambda functions before but not in Python. I know in Javascript Lambda supports the handler function being asynchronous, but I get an error if I try it in Python.
Here is the code I am trying to test:
async def handler(event, context):
print(str(event))
return {
'message' : 'OK'
}
And this is the error I get:
An error occurred during JSON serialization of response: <coroutine object handler at 0x7f63a2d20308> is not JSON serializable
Traceback (most recent call last):
File "/var/lang/lib/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/var/lang/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/var/lang/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/var/runtime/awslambda/bootstrap.py", line 149, in decimal_serializer
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <coroutine object handler at 0x7f63a2d20308> is not JSON serializable
/var/runtime/awslambda/bootstrap.py:312: RuntimeWarning: coroutine 'handler' was never awaited
errortype, result, fatal = report_fault(invokeid, e)
EDIT 2021:
Since this question seems to be gaining traction, I assume people are coming here trying to figure out how to get async to work with AWS Lambda as I was. The bad news is that even now more than a year later, there still isn't any support by AWS to have an asynchronous handler in a Python-based Lambda function. (I have no idea why, as NodeJS-based Lambda functions can handle it perfectly fine.)
The good news is that since Python 3.7, there is a simple workaround in the form of asyncio.run:
import asyncio
def lambda_handler(event, context):
# Use asyncio.run to synchronously "await" an async function
result = asyncio.run(async_handler(event, context))
return {
'statusCode': 200,
'body': result
}
async def async_handler(event, context):
# Put your asynchronous code here
await asyncio.sleep(1)
return 'Success'
Note: The selected answer says that using asyncio.run is not the proper way of starting an asynchronous task in Lambda. In general, they are correct because if some other resource in your Lambda code creates an event loop (a database/HTTP client, etc.), it's wasteful to create another loop and it's better to operate on the existing loop using asyncio.get_event_loop.
However, if an event loop does not yet exist when your code begins running, asyncio.run becomes the only (simple) course of action.

Not at all. Async Python handlers are not supported by AWS Lambda.
If you need to use async/await functionality in your AWS Lambda, you have to define an async function in your code (either in Lambda files or a Lambda Layer) and call asyncio.get_event_loop().run_until_complete(your_async_handler()) inside your sync handler.
Please note that asyncio.run (introduced in Python 3.7) is not a proper way to call an async handler in AWS Lambda execution environment since Lambda tries to reuse the execution context for subsequent invocations. The problem here is that asyncio.run creates a new EventLoop and closes the previous one. If you have opened any resources or created coroutines attached to the closed EventLoop from previous Lambda invocation you will get «Event loop closed» error. asyncio.get_event_loop().run_until_complete allows you to reuse the same loop. See related StackOverflow question.
AWS Lambda documentation misleads its readers a little by introducing synchronous and asynchronous invocations. Do not mix it up with sync/async Python functions. Synchronous refers to invoking AWS Lambda with further waiting for the result (blocking operation). The function is called immediately and you get the response as soon as possible. Whereas using an asynchronous invocation you ask Lambda to schedule the function execution and do not wait for the response at all. When the time comes, Lambda still will call the handler function synchronously.

Don't use run() method and call run_until_complete()
import json
import asyncio
async def my_async_method():
await some_async_functionality()
def lambda_handler(event, context):
loop = asyncio.get_event_loop()
result = loop.run_until_complete(my_async_method())
return {
'statusCode': 200,
'body': json.dumps('Hello Lambda')
}

How to use Tornado.gen.coroutine in TCP Server?

i write a Tcp Server with Tornado.
here is the code:
#! /usr/bin/env python
#coding=utf-8
from tornado.tcpserver import TCPServer
from tornado.ioloop import IOLoop
from tornado.gen import *
class TcpConnection(object):
def __init__(self,stream,address):
self._stream=stream
self._address=address
self._stream.set_close_callback(self.on_close)
self.send_messages()
def send_messages(self):
self.send_message(b'hello \n')
print("next")
self.read_message()
self.send_message(b'world \n')
self.read_message()
def read_message(self):
self._stream.read_until(b'\n',self.handle_message)
def handle_message(self,data):
print(data)
def send_message(self,data):
self._stream.write(data)
def on_close(self):
print("the monitored %d has left",self._address)
class MonitorServer(TCPServer):
def handle_stream(self,stream,address):
print("new connection",address,stream)
conn = TcpConnection(stream,address)
if __name__=='__main__':
print('server start .....')
server=MonitorServer()
server.listen(20000)
IOLoop.instance().start()
And i face some eorror assert self._read_callback is None, "Already reading",i guess the eorror is because multiple commands to read from socket at the same time.and then i change the function send_messages with tornado.gen.coroutine.here is code:
#gen.coroutine
def send_messages(self):
yield self.send_message(b'hello \n')
response1 = yield self.read_message()
print(response1)
yield self.send_message(b'world \n')
print((yield self.read_message()))
but there are some other errors. the code seem to stop after yield self.send_message(b'hello \n'),and the following code seem not to execute.
how should i do about it ? If you're aware of any Tornado tcpserver (not HTTP!) code with tornado.gen.coroutine,please tell me.I would appreciate any links!

send_messages() calls send_message() and read_message() with yield, but these methods are not coroutines, so this will raise an exception.
The reason you're not seeing the exception is that you called send_messages() without yielding it, so the exception has nowhere to go (the garbage collector should eventually notice and print the exception, but that can take a long time). Whenever you call a coroutine, you should either use yield to wait for it to finish, or IOLoop.current().spawn_callback() to run the coroutine in the "background" (this tells Tornado that you do not intend to yield the coroutine, so it will print the exception as soon as it occurs). Also, whenever you override a method you should read the documentation to see whether coroutines are allowed (when you override TCPServer.handle_stream() you can make it a coroutine, but __init__() may not be a coroutine).
Once the exception is getting logged, the next step is to fix it. You can either make send_message() and read_message() coroutines (getting rid of the handle_message() callback in the process), or you can use tornado.gen.Task() to call coroutine-style code from a coroutine. I generally recommend using coroutines everywhere.