I am trying to forward messages to internal topics in faust. As suggested by faust in this example:
https://faust.readthedocs.io/en/latest/playbooks/vskafka.html
I have the following code
in_topic = app.topic("in_topic", internal=False, partitions=1)
batching = app.topic("batching", internal=True, partitions=1)
....
#app.agent(in_topic)
async def process(stream):
async for event in stream:
event.forward(batching)
yield
But i always get the following error when runnning my pytest:
AttributeError: 'str' object has no attribute 'forward'
Was this feature removed or do i need to specify the topic differently to get an event, or is this even an issue with pytest ?
You are using the syntax backwards, thats why!
in_topic = app.topic("in_topic", internal=False, partitions=1)
batching = app.topic("batching", internal=True, partitions=1)
....
#app.agent(in_topic)
async def process(stream):
async for event in stream:
batching.send(value=event)
yield
Should actually work.
Edit:
This is also the only they that I could make correct use of pytest with faust.
The sink method is only usable if you delete all but the last sink of the mocked agent, which does not seem intuitive.
If you first import your sink and then add this decorator to your test, everything should work fine:
from my_path import my_sink, my_agent
from unittest.mock import patch, AsyncMock
#patch(__name__ + '.my_sink.send', new_callable=AsyncMock)
def test_my_agent(test_app)
async with my_agent.test_context() as agent:
payload = 'This_works_fine'
agent.put(payload)
my_sink.send.assert_called_with(value=payload)
try the next one:
#app.agent(in_topic, sink=[batching])
async def process(stream):
async for event in stream:
yield event
The answer from #Florian Hall works fine. I also found out that the forward method is only implemented for Events, in my case i received a str. The reason could be the difference between Faust Topics and Channels. Another thing is that using pytest and the Faust test_context() behaves weird, for example it forces you to yield even if you don't have a sink defined.
Related
I have very recently started checking out asyncio for python. The use case is that, I hit an endpoint /refresh_data and the data is refreshed (from an S3 bucket). But this should be a non blocking operation, other API endpoints should still be able to be serviced.
So in my controller, I have:
def refresh_data():
myservice.refresh_data()
return jsonify(dict(ok=True))
and in my service I have:
async def refresh_data():
try:
s_result = await self.s3_client.fetch()
except (FileNotFound, IOError) as e:
logger.info("problem")
gather = {i.pop("x"):i async for i in summary_result}
# ... some other stuff
and in my client:
async def fetch():
result = pd.read_parquet("bucket", "pyarrow", cols, filts).to_dict(orient="col1")
return result
And when I run it, I see this error:
TypeError: 'async for' requires an object with __aiter__ method, got coroutine
I don't know how to move past this. Adding async definitely makes it return a coroutine type - but, either I have implemented this messily or I haven't fully understood asyncio package in Python. I have been working off of simple examples but I'm not sure what's wrong with what I've done here.
I'm creating a script that is posting a message to both discord and twitter, depending on some input. I have to methods (in separate .py files), post_to_twitter and post_to_discord. What I want to achieve is that both of these try to execute even if the other fails (e.g. if there is some exception with login).
Here is the relevant code snippet for posting to discord:
def post_to_discord(message, channel_name):
client = discord.Client()
#client.event
async def on_ready():
channel = # getting the right channel
await channel.send(message)
await client.close()
client.run(discord_config.token)
and here is the snippet for posting to twitter part (stripped from the try-except blocks):
def post_to_twitter(message):
auth = tweepy.OAuthHandler(twitter_config.api_key, twitter_config.api_key_secret)
auth.set_access_token(twitter_config.access_token, twitter_config.access_token_secret)
api = tweepy.API(auth)
api.update_status(message)
Now, both of these work perfectly fine on their own and when being called synchronously from the same method:
def main(message):
post_discord.post_to_discord(message)
post_tweet.post_to_twitter(message)
However, I just cannot get them to work concurrently (i.e. to try to post to twitter even if discord fails or vice-versa). I've already tried a couple of different approaches with multi-threading and with asyncio.
Among others, I've tried the solution from this question. But got an error No module named 'IPython'. When I omitted the IPython line, changed the methods to async, I got this error: RuntimeError: Cannot enter into task <ClientEventTask state=pending event=on_ready coro=<function post_to_discord.<locals>.on_ready at 0x7f0ee33e9550>> while another task <Task pending name='Task-1' coro=<main() running at post_main.py:31>> is being executed..
To be honest, I'm not even sure if asyncio would be the right approach for my use case, so any insight is much appreciated.
Thank you.
In this case running the two things in completely separate threads (and completely separate event loops) is probably the easiest option at your level of expertise. For example, try this:
import post_to_discord, post_to_twitter
import concurrent.futures
def main(message):
with concurrent.futures.ThreadPoolExecutor() as pool:
fut1 = pool.submit(post_discord.post_to_discord, message)
fut2 = pool.submit(post_tweet.post_to_twitter, message)
# here closing the threadpool will wait for both futures to complete
# make exceptions visible
for fut in (fut1, fut2):
try:
fut.result()
except Exception as e:
print("error: ", e)
I'm trying to mock a single request to an external URL but in the documentation exists just examples to internal request (starting with '/'), it's impossible to add routers who not start with '/' on the current version of aiohttp.
I'm using pytest and pytest-aiohttp, here are an example of the request code:
import aiohttp
import asyncio
async def fetch(client):
async with client.get('http://python.org') as resp:
return resp.status, (await resp.text())
async def main():
async with aiohttp.ClientSession() as client:
html = await fetch(client)
print(html)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
The kind of assertion that I want to do is very simple, like check the status code, the headers, and the content.
You can patch (using asynctest.patch) your ClientSession. But in this case you need to implement simple ResponseContextManager with .status, async .text() (async .json()), etc. methods and attrs.
In your comment (below your question), you say you're actually looking to mock aiohttp responses. For that, I've been using the 3rd-party library aioresponses: https://github.com/pnuckowski/aioresponses
I use it for integration testing, where it seems better than directly mocking or patching aiohttp methods.
I made it into a little pytest fixture like so:
#pytest.fixture
def aiohttp_mock():
with aioresponses() as aioresponse_mock:
yield aioresponse_mock
which I can then call like an aiohttp client/session: aiohttp_mock.get(...)
Edit from the future: We actually went back to mocking aiohttp methods because aioresponses currently lacks the ability to verify the args that were used in the call. We decided that verifying the args was a requirement for us.
I've written a simple HTTP client using aiohttp and I'm trying to test it by patching aiohttp.ClientSession and aiohttp.ClientResponse. However, it appears as though the unittest.mock.patch decorator is not respecting my asynchronous code. At a guess, I would say it's some kind of namespacing mismatch.
Here's a minimal example:
from aiohttp import ClientSession
async def is_ok(url:str) -> bool:
async with ClientSession() as session:
async with session.request("GET", url) as response:
return (response.status == 200)
I'm using an asynchronous decorator for testing, as described in this answer. So here's my attempted test:
import unittest
from unittest.mock import MagicMock, patch
from aiohttp import ClientResponse
from my.original.module import is_ok
class TestClient(unittest.TestCase):
#async_test
#patch("my.original.module.ClientSession", spec=True)
async def test_client(self, mock_client):
mock_response = MagicMock(spec=ClientResponse)
mock_response.status = 200
async def _mock_request(*args, **kwargs):
return mock_response
mock_client.request = mock_response
status = await is_ok("foo")
self.assertTrue(status)
My is_ok coroutine works fine when it's used in, say, __main__, but when I run the test, it gives me an error that indicates that the session.request function has not been mocked per my patch call. (Specifically it says "Could not parse hostname from URL 'foo'", which it should if it weren't mocked.)
I am unable to escape this behaviour. I have tried:
Importing is_ok after the mocking is done.
Various combinations of assigning mocks to mock_client and mock_client.__aenter__, setting mock_client.request to MagicMock(return_value=mock_response), or using mock_client().request, etc.
Writing a mock ClientSession with specific __aenter__ and __aexit__ methods and using it in the new argument to patch.
None of these appear to make a difference. If I put assertions into is_ok to test that ClientSession is an instance of MagicMock, then these assertions fail when I run the test (as, again, they would when the code is not patched). That leads me to my namespacing mismatch theory: That is, the event loop is running in a different namespace to which patch is targeting.
Either that, or I'm doing something stupid!
Mocking ClientSession is discouraged.
Recommended way is creation fake server and sending real requests to it.
Take a look on aiohttp example.
I'm new to tornado.
What I want is to write some functions to fetch webpages asynchronously. Since no requesthandlers, apps, or servers involved here, I think I can use tornado.httpclient.AsyncHTTPClient alone.
But all the sample codes seem to be in a tornado server or requesthandler. When I tried to use it alone, it never works.
For example:
def handle(self,response):
print response
print response.body
#tornado.web.asynchronous
def fetch(self,url):
client=tornado.httpclient.AsyncHTTPClient()
client.fetch(url,self.handle)
fetch('http://www.baidu.com')
It says "'str' object has no attribute 'application'", but I'm trying to use it alone?
or :
#tornado.gen.coroutine
def fetch_with_coroutine(url):
client=tornado.httpclient.AsyncHTTPClient()
response=yield http_client.fetch(url)
print response
print response.body
raise gen.Return(response.body)
fetch_with_coroutine('http://www.baidu.com')
doesn't work either.
Earlier, I tried pass a callback to AsyncHTTPHandler.fetch, then start the IOLoop, It works and the webpage source code is printed. But I can't figure out what to do with the ioloop.
#tornado.web.asynchronous can only be applied to certain methods in RequestHandler subclasses; it is not appropriate for this usage.
Your second example is the correct structure, but you need to actually run the IOLoop. The best way to do this in a batch-style program is IOLoop.current().run_sync(fetch_with_coroutine). This starts the IOLoop, runs your callback, then stops the IOLoop. You should run a single function within run_sync(), and then use yield within that function to call any other coroutines.
For a more complete example, see https://github.com/tornadoweb/tornado/blob/master/demos/webspider/webspider.py
Here's an example I've used in the past...
from tornado.httpclient import AsyncHTTPClient
from tornado.ioloop import IOLoop
AsyncHTTPClient.configure(None, defaults=dict(user_agent="MyUserAgent"))
http_client = AsyncHTTPClient()
def handle_response(response):
if response.error:
print("Error: %s" % response.error)
else:
print(response.body)
async def get_content():
await http_client.fetch("https://www.integralist.co.uk/", handle_response)
async def main():
await get_content()
print("I won't wait for get_content to finish. I'll show immediately.")
if __name__ == "__main__":
io_loop = IOLoop.current()
io_loop.run_sync(main)
I've also detailed how to use Pipenv with tox.ini and Flake8 with this tornado example so others should be able to get up and running much more quickly https://gist.github.com/fd603239cacbb3d3d317950905b76096