We Keep Coding

Mocking an Async Property in Python

The below example class has a property bar that is awaitable, as in async_main() because it (theoretically) does some IO work before returning the answer to everything.
class Foo:
#property
async def bar(self):
return 42
async def async_main():
f = Foo()
print(await f.bar)
I'm having trouble testing it, as the usual suspects of Mock, MagicMock, and AsyncMock don't work with properties as expected. My current workaround is:
f.bar = some_awaitable()
since this makes f.bar a 'field' that can be awaited, but unfortunately I need to access it multiple times while it's under test, which yields RuntimeError: cannot reuse already awaited coroutine on the second access of course.
Is there an established way to mock an async property like this?

The easiest way that I can think of is to patch bar again with an async property for the purposes of your test.
I am assuming you have some other method on Foo that you want to test, and that method calls its bar.
code.py
from asyncio import run
class Foo:
#property
async def bar(self) -> int:
return 42
async def func(self) -> int:
return await self.bar
async def main():
f = Foo()
print(await f.func())
if __name__ == '__main__':
run(main())
test.py
from unittest import IsolatedAsyncioTestCase
from unittest.mock import patch
from . import code
class FooTestCase(IsolatedAsyncioTestCase):
async def test_func(self) -> None:
expected_output = 69420
#property
async def mock_bar(_foo_self: code.Foo) -> int:
return expected_output
with patch.object(code.Foo, "bar", new=mock_bar):
f = code.Foo()
# Just to see that our mocking worked:
self.assertEqual(expected_output, await f.bar)
# Should call `bar` property again:
output = await f.func()
self.assertEqual(expected_output, output)
References: patch docs.

How to mock an async function returning a tuple?

I need to test the following function
class C:
async def f():
a, b = await self.g() # need to mock g
However, the following test got the error of TypeError: object tuple can't be used in 'await' expression
#pytest.mark.asyncio
async def test_f():
sut = C()
sut.g = MagicMock(return_value=(1,2)) # TypeError: object tuple can't be used in 'await' expression
await sut.f()
sut.g.assert_called_once()

Use AsyncMock instead of the MagicMock:
from unittest.mock import AsyncMock
#pytest.mark.asyncio
async def test_f():
sut = C()
sut.g = AsyncMock(return_value=(1,2))
await sut.f()
sut.g.assert_called_once()
AsyncMock is part of the standard library since Python 3.8; if you need a backport for older versions, install and use the asyncmock package.

Turn a sync function into an awaitable function

I've got a situation where I can't be sure if a function will be sync, or async. In other words, the function has a type signature of Union[Callable[..., Awaitable[Any]], Callable[..., Any]].
I can't seem to find a good (non-deprecated) way of turning the above type signature into a consistent type of Callable[..., Awaitable[Any]].
The only way I was able to find how to do this, is by doing this:
import asyncio
import inspect
from typing import Any, Awaitable, Callable, List, Union
def force_awaitable(function: Union[Callable[..., Awaitable[Any]], Callable[..., Any]]) -> Callable[..., Awaitable[Any]]:
if inspect.isawaitable(function):
# Already awaitable
return function
else:
# Make it awaitable
return asyncio.coroutine(function)
However, asyncio.coroutine (which is normally used as a decorator) is deprecated since Python 3.8. https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine
The alternative provided does not work for me here, since I don't use asyncio.coroutine as a decorator. Unlike a decorator, the async keyword can't be used as a function.
How do I turn a sync function (not awaitable) into an async function (awaitable)?
Other considered options
Above I've shown that you can detect if something is awaitable. This allows us to change how to call like so:
def my_function():
pass
if inspect.isawaitable(function):
await my_function()
else:
my_function()
However, this feels clunky, can create messy code, and creates unnecessary checks inside big loops. I want to be able to define how to call the function before I'm entering my loop.
In NodeJS I would simply await the sync function:
// Note: Not Python!
function sync() {}
await sync();
When I try to do the same thing in Python, I'm greeted with an error:
def sync():
pass
await sync() # AttributeError: 'method' object has no attribute '__await__'

You can return an async function with the original function object in its scope:
def to_coroutine(f:Callable[..., Any]):
async def wrapper(*args, **kwargs):
return f(*args, **kwargs)
return wrapper
def force_awaitable(function: Union[Callable[..., Awaitable[Any]], Callable[..., Any]]) -> Callable[..., Awaitable[Any]]:
if inspect.iscoroutinefunction(function):
return function
else:
return to_coroutine(function)
Now, if function is not awaitable, force_awaitable will return a coroutine function that contains function.
def test_sync(*args, **kwargs):
pass
async def main():
await force_awaitable(test_sync)('foo', 'bar')
asyncio.run(main())

How to timeout an async test in pytest with fixture?

I am testing an async function that might get deadlocked. I tried to add a fixture to limit the function to only run for 5 seconds before raising a failure, but it hasn't worked so far.
Setup:
pipenv --python==3.6
pipenv install pytest==4.4.1
pipenv install pytest-asyncio==0.10.0
Code:
import asyncio
import pytest
#pytest.fixture
def my_fixture():
# attempt to start a timer that will stop the test somehow
asyncio.ensure_future(time_limit())
yield 'eggs'
async def time_limit():
await asyncio.sleep(5)
print('time limit reached') # this isn't printed
raise AssertionError
#pytest.mark.asyncio
async def test(my_fixture):
assert my_fixture == 'eggs'
await asyncio.sleep(10)
print('this should not print') # this is printed
assert 0
--
Edit: Mikhail's solution works fine. I can't find a way to incorporate it into a fixture, though.

Convenient way to limit function (or block of code) with timeout is to use async-timeout module. You can use it inside your test function or, for example, create a decorator. Unlike with fixture it'll allow to specify concrete time for each test:
import asyncio
import pytest
from async_timeout import timeout
def with_timeout(t):
def wrapper(corofunc):
async def run(*args, **kwargs):
with timeout(t):
return await corofunc(*args, **kwargs)
return run
return wrapper
#pytest.mark.asyncio
#with_timeout(2)
async def test_sleep_1():
await asyncio.sleep(1)
assert 1 == 1
#pytest.mark.asyncio
#with_timeout(2)
async def test_sleep_3():
await asyncio.sleep(3)
assert 1 == 1
It's not hard to create decorator for concrete time (with_timeout_5 = partial(with_timeout, 5)).
I don't know how to create texture (if you really need fixture), but code above can provide starting point. Also not sure if there's a common way to achieve goal better.

There is a way to use fixtures for timeout, one just needs to add the following hook into conftest.py.
Any fixture prefixed with timeout must return a number of seconds(int, float) the test can run.
The closest fixture w.r.t scope is chosen. autouse fixtures have lesser priority than explicitly chosen ones. Later one is preferred. Unfortunately order in the function argument list does NOT matter.
If there is no such fixture, the test is not restricted and will run indefinitely as usual.
The test must be marked with pytest.mark.asyncio too, but that is needed anyway.
# Add to conftest.py
import asyncio
import pytest
_TIMEOUT_FIXTURE_PREFIX = "timeout"
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_setup(item: pytest.Item):
"""Wrap all tests marked with pytest.mark.asyncio with their specified timeout.
Must run as early as possible.
Parameters
----------
item : pytest.Item
Test to wrap
"""
yield
orig_obj = item.obj
timeouts = [n for n in item.funcargs if n.startswith(_TIMEOUT_FIXTURE_PREFIX)]
# Picks the closest timeout fixture if there are multiple
tname = None if len(timeouts) == 0 else timeouts[-1]
# Only pick marked functions
if item.get_closest_marker("asyncio") is not None and tname is not None:
async def new_obj(*args, **kwargs):
"""Timed wrapper around the test function."""
try:
return await asyncio.wait_for(
orig_obj(*args, **kwargs), timeout=item.funcargs[tname]
)
except Exception as e:
pytest.fail(f"Test {item.name} did not finish in time.")
item.obj = new_obj
Example:
#pytest.fixture
def timeout_2s():
return 2
#pytest.fixture(scope="module", autouse=True)
def timeout_5s():
# You can do whatever you need here, just return/yield a number
return 5
async def test_timeout_1():
# Uses timeout_5s fixture by default
await aio.sleep(0) # Passes
return 1
async def test_timeout_2(timeout_2s):
# Uses timeout_2s because it is closest
await aio.sleep(5) # Timeouts
WARNING
Might not work with some other plugins, I have only tested it with pytest-asyncio, it definitely won't work if item is redefined by some hook.

I just loved Quimby's approach of marking tests with timeouts. Here's my attempt to improve it, using pytest marks:
# tests/conftest.py
import asyncio
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_pyfunc_call(pyfuncitem: pytest.Function):
"""
Wrap all tests marked with pytest.mark.async_timeout with their specified timeout.
"""
orig_obj = pyfuncitem.obj
if marker := pyfuncitem.get_closest_marker("async_timeout"):
async def new_obj(*args, **kwargs):
"""Timed wrapper around the test function."""
try:
return await asyncio.wait_for(orig_obj(*args, **kwargs), timeout=marker.args[0])
except (asyncio.CancelledError, asyncio.TimeoutError):
pytest.fail(f"Test {pyfuncitem.name} did not finish in time.")
pyfuncitem.obj = new_obj
yield
def pytest_configure(config: pytest.Config):
config.addinivalue_line("markers", "async_timeout(timeout): cancels the test execution after the specified amount of seconds")
Usage:
#pytest.mark.asyncio
#pytest.mark.async_timeout(10)
async def potentially_hanging_function():
await asyncio.sleep(20)
It should not be hard to include this to the asyncio mark on pytest-asyncio, so we can get a syntax like:
#pytest.mark.asyncio(timeout=10)
async def potentially_hanging_function():
await asyncio.sleep(20)
EDIT: looks like there's already a PR for that.

Python - object MagicMock can't be used in 'await' expression

When I was trying to mock an async function in unittest with MagicMock, I got this exception:
TypeError: object MagicMock can't be used in 'await' expression
With sample code like:
# source code
class Service:
async def compute(self, x):
return x
class App:
def __init__(self):
self.service = Service()
async def handle(self, x):
return await self.service.compute(x)
# test code
import asyncio
import unittest
from unittest.mock import patch
class TestApp(unittest.TestCase):
#patch('__main__.Service')
def test_handle(self, mock):
loop = asyncio.get_event_loop()
app = App()
res = loop.run_until_complete(app.handle('foo'))
app.service.compute.assert_called_with("foo")
if __name__ == '__main__':
unittest.main()
How should I fix it with built-in python3 libraries?

In python 3.8+ you can make use of the AsyncMock
async def test_that_mock_can_be_awaited():
mock = AsyncMock()
mock.x.return_value = 123
result = await mock.x()
assert result == 123
The class AsyncMock object will behave so the object is recognized as an async function, and the result of a call is an awaitable.
>>> mock = AsyncMock()
>>> asyncio.iscoroutinefunction(mock)
True
>>> inspect.isawaitable(mock())
True

I ended up with this hack.
# monkey patch MagicMock
async def async_magic():
pass
MagicMock.__await__ = lambda x: async_magic().__await__()
It only works for MagicMock, not other pre-defined return_value

You can get mocks to return objects that can be awaited by using a Future. The following is a pytest test case, but something similar should be possible with unittest.
async def test_that_mock_can_be_awaited():
mock = MagicMock(return_value=Future())
mock.return_value.set_result(123)
result = await mock()
assert result == 123
In your case, since you're patching Service (which gets passed in as mock), mock.return_value = Future() should do the trick.

shaun shia provided really good universal solution, but i found what in python 3.8 you can use just #patch('__main__.Service', new=AsyncMock)

I found this comment very useful when trying to await a mock object in Python < 3.8. You simply create a child class AsyncMock that inherits from MagicMock and overwrites a __call__ method to be a coroutine:
class AsyncMock(MagicMock):
async def __call__(self, *args, **kwargs):
return super(AsyncMock, self).__call__(*args, **kwargs)
Then, inside your test, do:
#pytest.mark.asyncio
async def test_my_method():
# Test "my_method" coroutine by injecting an async mock
my_mock = AsyncMock()
assert await my_method(my_mock)
you might also want to install pytest-asyncio

To override async classes one needs to tell patch that the return_value needs to be AsyncMock. So use
#patch('__main__.Service', return_value=AsyncMock(Service))
def test_handle(self, mock):
loop = asyncio.get_event_loop()
app = App()
res = loop.run_until_complete(app.handle('foo'))
app.service.compute.assert_called_with("foo")
With that Service will be a MagicMock, but Service() will return an AsyncMock instance of Service.

If you'd like a pytest-mock solution compatible with < py3.8, I did something like this.
class AsyncMock(MagicMock):
async def __call__(self, *args, **kwargs):
return super().__call__(*args, **kwargs)
def test_my_method(mocker):
my_mock = mocker.patch("path.to.mocked.thing", AsyncMock())
my_mock.return_value = [1, 2, 3]
assert my_method()
Definitely borrowed from Tomasz's solution here!