I have the following function.
async def f(url):
async with httpx.AsyncClient() as client:
async with client.stream('GET', url) as resp:
async for c in resp.aiter_bytes():
yield c
And I'm writing a test code to test if the function yields.
#pytest.mark.asyncio
async def test():
c = None
async for c in f(URL):
c = chunk
break # there will be no error if removing this line. However, I don't need to loop all the items
assert c
However, the code runs but with the following warnings/errors. What's the error?
lib\site-packages\httpx_client.py:2012: UserWarning: Unclosed <httpx.AsyncClient object at 0x000001AC1E8CF3A0>. See https://www.python-httpx.org/async/#opening-and-closing-clients for details.
Is the "Unclosed" warning caused by break?
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001AC1D884DC0>
Traceback (most recent call last):
File "C:\anaconda3\envs\s\lib\asyncio\proactor_events.py", line 116, in __del__
self.close()
File "C:\anaconda3\envs\s\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\anaconda3\envs\s\lib\asyncio\base_events.py", line 719, in call_soon
self._check_closed()
File "C:\anaconda3\envs\s\lib\asyncio\base_events.py", line 508, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
The error disappears if removing the break statement.
In the answer of Async generator is not an iterator?, the following code will get rid of the error. However, it needs to call a -private- "special" function __anext()__().
#pytest.mark.asyncio
async def test():
it = f(URL):
c = it.__anext__()
assert c
Related
I am trying to make some API calls to firebase using the google.cloud.firestore.AsyncClient module. Fire2 is just a wrapper for this object.
The traceback suggests its a problem with asyncio, but if I make a dummy await it actually runs just fine. What is the problem and why does it occur?
Sample code:
import Fire2
class Test:
def __init__(self):
doc = asyncio.run(self.wait())
async def wait(self):
doc = await Fire2("test1").get(g1) # gives the error
# doc = await asyncio.sleep(1) # runs without error
return doc
def test(self):
x = Test2(p1)
class Test2:
def __init__(self, p):
doc = asyncio.run(self.run(p))
print(doc.to_dict())
async def run(self, p):
doc = await Fire2('test2').get(p)
return doc
p1 = 'foo'
g1 = 'bar'
h = Test()
h.test()
Traceback:
Traceback (most recent call last):
File "<project path>\scratch.py", line 137, in <module>
h.test()
File "<project path>\scratch.py", line 123, in test
x = Test2(p1)
File "<project path>\scratch.py", line 127, in __init__
doc = asyncio.run(self.run(p))
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\runners.py", line 44, in run
return loop.run_until_complete(main)
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete
return future.result()
File "<project path>\scratch.py", line 131, in run
doc = await Fire2('test2').get(p)
File "<project path>\<Fire2 file>", line 422, in get
res = await self._collection.document(doc_id).get()
File "<project path>\venv\lib\site-packages\google\cloud\firestore_v1\async_document.py", line 364, in get
response_iter = await self._client._firestore_api.batch_get_documents(
File "<project path>\venv\lib\site-packages\google\api_core\grpc_helpers_async.py", line 171, in error_remapped_callable
call = callable_(*args, **kwargs)
File "<project path>\venv\lib\site-packages\grpc\aio\_channel.py", line 165, in __call__
call = UnaryStreamCall(request, deadline, metadata, credentials,
File "<project path>\venv\lib\site-packages\grpc\aio\_call.py", line 553, in __init__
self._send_unary_request_task = loop.create_task(
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 431, in create_task
self._check_closed()
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'UnaryStreamCall._send_unary_request' was never awaited
Process finished with exit code 1
You are making two separate calls to asyncio.run. We can see from the stack trace that the error is coming from inside the grpc library. I suspect that the same grpc connection is being used both times, and that during its initialization it integrates itself with the async event loop in order to maintain the connection in the background between calls.
The problem is that every time you call asyncio.run, you will create a new event loop, and every time it returns that event loop will be closed. So by the time you call this function again, and the grpc layer tries to do its work, it realizes that the original event loop has been closed and it returns this error.
To fix this, You can have a single call to asyncio.run, typically calling a "main" function, or similar. This would then make calls to async functions which perform the actual work.
If the problem persists then there might be some issues in the “Fire2”class.
For more reference you can check this link which outlines high-level asyncio APIs to work with coroutines and Tasks with the help of various examples.
Use pytest with using fixture with scope="session" in conftest.py
#pytest.fixture(scope="session")
def event_loop():
loop = get_event_loop()
yield loop
loop.close()
Here's a simplified scenario for the problem I'm having.
import asyncio
async def c():
print("yes")
def b():
asyncio.run(c())
async def a():
b()
asyncio.run(a())
I'm expecting the program to print "yes". However, I get this instead:
Traceback (most recent call last):
File [redacted], line 12, in <module>
asyncio.run(a())
File "/usr/lib64/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib64/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
File [redacted], line 10, in a
b()
File [redacted], line 7, in b
asyncio.run(c())
File "/usr/lib64/python3.7/asyncio/runners.py", line 34, in run
"asyncio.run() cannot be called from a running event loop")
RuntimeError: asyncio.run() cannot be called from a running event loop
sys:1: RuntimeWarning: coroutine 'c' was never awaited
What would you think would be the solution for this problem?
(Also, can this be done using purely asyncio?)
Don't call asyncio.run() twice in the same program. Instead, once the event loop is running, create a task and await it. A sync function can create a new task but can't await it; it can return it as an awaitable. The async calling function can then perform the await.
import asyncio
async def c():
print("yes")
def b():
return asyncio.create_task(c())
async def a():
task = b()
await task
asyncio.run(a())
>>>yes
I have a project that depends on Scrapy 2.3.0 which uses Twisted 20.3.0 as its network engine. I am trying to convert the callback based approach used by Scrapy to coroutines and run it with Python's asyncio. To make a HTTP request, one needs to create a new Request object which takes a callback that handles the sucess cases and a errback that handles the failures. So I wrote the following method to do that conversion:
import asyncio
from scrapy import http
# ...
async def request(
self, url: str, method: str, *, headers: dict = None, cookies: dict = None,
body: str = None, meta: dict = None
) -> http.Response:
fut = asyncio.get_running_loop().create_future()
def fire_success(response):
if not fut.done():
fut.set_result(response)
return None
def fire_failure(failure):
if not fut.done():
fut.set_exception(failure.value)
return None
scrapy_request = http.Request(
url,
method=method,
headers=headers,
cookies=cookies,
body=body,
meta=meta,
callback=fire_success,
errback=fire_failure,
dont_filter=True
)
self._crawler.engine.crawl(scrapy_request, self._crawler.spider)
return await fut
But when an exception is raised, I get just the traceback before return await fut, does this means that failure.value set in
def fire_failure(failure):
if not fut.done():
fut.set_exception(failure.value)
return None
has no traceback?
The following example reproduces that problem:
import asyncio
async def a():
await asyncio.sleep(2)
raise Exception('Hello from a()')
async def b():
await a()
async def c(fut: asyncio.Future):
try:
await b()
except Exception as e:
fut.set_exception(e.with_traceback(None))
async def d(fut):
return await fut
async def main():
fut = asyncio.get_running_loop().create_future()
await asyncio.gather(d(fut), c(fut))
asyncio.run(main())
where I removed the traceback by calling e.with_traceback(None), this is the incomplete traceback (it's missing the call to await b()):
Traceback (most recent call last):
File "tmp.py", line 23, in <module>
asyncio.run(main())
File ".../python/3.8.6/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File ".../python/3.8.6/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "tmp.py", line 21, in main
await asyncio.gather(d(fut), c(fut))
File "tmp.py", line 17, in d
return await fut
Exception: Hello from a()
I have an async function that I'm trying to get the return variable from but I'm not able to get it to work for some reason, I've tried a few different things from googling but theyre all returning a similar error.
I have this function:
#bot.command()
async def start(ctx):
"""starts the server"""
try:
status = server.status()
await ctx.channel.send("The server is already online!")
except:
os.chdir(".\\Minecraft")
file = subprocess.Popen("Start.bat")
await ctx.channel.send("starting the server")
starting = True
while starting == True:
time.sleep(int(delay))
with open("outfile.txt") as outfile:
for line in outfile:
if "Done" in line:
await ctx.channel.send("server has loaded")
starting = False
return file
else:
continue
and i return the variable file
But then when I try to get the variable in another function
#bot.command()
async def kill(ctx):
"""shuts down the server"""
x = start(ctx)
print(x)
x.terminate()
I get the error:
<coroutine object Command.__call__ at 0x040F7B28>
Ignoring exception in command kill:
Traceback (most recent call last):
File "C:\Users\TheRi\AppData\Local\Programs\Python\Python38-32\lib\site-packages\discord\ext\commands\core.py", line 83, in wrapped
ret = await coro(*args, **kwargs)
File "c:/Users/TheRi/OneDrive/Desktop/Python/MinecraftDiscordBot/minecraftBot.py", line 113, in kill
x.terminate()
AttributeError: 'coroutine' object has no attribute 'terminate'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\TheRi\AppData\Local\Programs\Python\Python38-32\lib\site-packages\discord\ext\commands\bot.py", line 892, in invoke
await ctx.command.invoke(ctx)
File "C:\Users\TheRi\AppData\Local\Programs\Python\Python38-32\lib\site-packages\discord\ext\commands\core.py", line 797, in invoke
await injected(*ctx.args, **ctx.kwargs)
File "C:\Users\TheRi\AppData\Local\Programs\Python\Python38-32\lib\site-packages\discord\ext\commands\core.py", line 92, in wrapped
raise CommandInvokeError(exc) from exc
discord.ext.commands.errors.CommandInvokeError: Command raised an exception: AttributeError: 'coroutine' object has no attribute 'terminate'
C:\Users\TheRi\AppData\Local\Programs\Python\Python38-32\lib\asyncio\events.py:81: RuntimeWarning: coroutine 'Command.__call__' was never awaited
self._context.run(self._callback, *self._args)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
The first line seems to be where I've attempted to print x, to see if I could see what was going on, and the rest is the error. It doesn't seem to be returning any value but just the co routine itself?
I've tried changing the way I refrence the function, x = start(ctx),x = start(),x = start etc.
Is there something I'm doing wrong? How can I return the variable.
You need to await coroutines; i.e. x = await start(ctx)
I'm trying to experiment with the new await syntax for coroutines in Python 3.5.
I had a simple example like this:
#! /usr/bin/env python
import asyncio
#asyncio.coroutine
def adder(*args, delay):
while True:
yield from asyncio.sleep(delay)
print(sum(args))
def main():
asyncio.Task(adder(1, 2, 3, delay=5))
asyncio.Task(adder(10, 20, delay=3))
loop = asyncio.get_event_loop()
loop.run_forever()
if __name__ == "__main__":
main()
I changed the yield from line to use the await keyword:
await asyncio.sleep(delay)
And I get SyntaxError:
File "./test.py", line 8
await asyncio.sleep(delay)
^
SyntaxError: invalid syntax
So I try await (asyncio.sleep(delay)) just to see what happens:
Task exception was never retrieved
future: <Task finished coro=<adder() done, defined at c:\python35\Lib\asyncio\coroutines.py:198> exception=NameError("name 'await' is not defined",)>
Traceback (most recent call last):
File "c:\python35\Lib\asyncio\tasks.py", line 239, in _step
result = coro.send(value)
File "c:\python35\Lib\asyncio\coroutines.py", line 200, in coro
res = func(*args, **kw)
File "./test.py", line 8, in adder
await (asyncio.sleep(delay))
NameError: name 'await' is not defined
Task exception was never retrieved
future: <Task finished coro=<adder() done, defined at c:\python35\Lib\asyncio\coroutines.py:198> exception=NameError("name 'await' is not defined",)>
Traceback (most recent call last):
File "c:\python35\Lib\asyncio\tasks.py", line 239, in _step
result = coro.send(value)
File "c:\python35\Lib\asyncio\coroutines.py", line 200, in coro
res = func(*args, **kw)
File "./test.py", line 8, in adder
await (asyncio.sleep(delay))
NameError: name 'await' is not defined
Am I using the keyword wrong? Why is await not defined? I got my await syntax from this post.
Just to cover all my bases:
$ /usr/bin/env python --version
Python 3.5.0
Edit:
I suppose adding the parens to the await line is trying to call a function await() - so that's why that doesn't work and gives me a NameError. But why doesn't the keyword get recognized in either example?
You must declare your function as async to use await. Replacing yield from is not sufficient.
#! /usr/bin/env python
import asyncio
#asyncio.coroutine
async def adder(*args, delay):
while True:
await asyncio.sleep(delay)
print(sum(args))
def main():
asyncio.Task(adder(1, 2, 3, delay=5))
asyncio.Task(adder(10, 20, delay=3))
loop = asyncio.get_event_loop()
loop.run_forever()
if __name__ == "__main__":
main()
You need to declare your function as async to make this work. The await keyword will only be enabled if you include an a function declared with async def in your file – see PEP 492 for details.
Per PEP 492, the await keyword is only recognized in functions defined with the new async def syntax, which removes the need for a __future__ import.