I am writing an API client for a REST API that allows for requests the be batched together in and send as one request
I am struggling to figure out how to structure the client. I've come up with two designs so far (pseudo Python code):
Async Method
When calls to the api are made (make_api_request), the method waits for the bulk request to be made. Once the bulk request is made by calling the resolve coroutine, the request coroutine releases control back to he make_api_request method which processes the response for that specific request and returns it
import asyncio
class FakeAPIClient:
def __init__(self):
"""
"""
pass
async def make_api_request(self):
"""
returns response after request is made
"""
resp = await self.request()
#response processing
return resp
async def request(self):
"""
I block until the bulk request is made
Once it is make, I return the response from the single request
"""
return "RESPONSE AFTER DONE BLOCKING "
async def resolve(self):
"""
Make the bulk request and release the make_api_request calls and return the assocaited responses
"""
return None
async def example():
api = FakeAPIClient()
future1 = api.make_api_request()
future2 = api.make_api_request()
future3 = api.make_api_request()
#3 requests are sent in bulk
response1, response2, response3, _ = asyncio.gather(
future1,
future2,
future3,
api.resolve()
)
asyncio.run(example())
Lookup method
When calls to the api are made (make_api_request), a lookup ID is returned and the request is put into storage. When the resolve method is called, the bulk request is called, the requests in storage are sent as a bulk request and an object is returned that can be used to find the lookup id's corresponding response.
class FakeAPIClient:
def __init__(self):
pass
def make_api_request(self):
"""
adds request to a queue of requests that will be resolved when resolve method is called
also provide a callback for post request processing
returns a unique id of the request
"""
return "UNIQUE ID"
def resolve(self):
"""
makes the bulk request
takes the responses and associates them with the request id
calls the callback associated with the response for post request processing
returns an object with a method that returns the responses when provided the request id
"""
return "RESPONSE LOOKUP"
api = FakeAPIClient()
lookup_id1 = api.make_api_request()
lookup_id2 = api.make_api_request()
lookup_id3 = api.make_api_request()
lookup_object = api.resolve()
response1 = lookup_object.get(lookup_id1)
response2 = lookup_object.get(lookup_id2)
response3 = lookup_object.get(lookup_id3)
I don't really love either of these solutions, but I can't think of any alternatives. I'd assume there are known patterns for solving this problem, what are they?
Related
I'm trying to work with a third party API and I am having problems with sending the request when using the requests or even urllib.request.
Somehow when I use http.client I am successful sending and receiving the response I need.
To make life easier for me, I created an API class below:
class API:
def get_response_data(self, response: http.client.HTTPResponse) -> dict:
"""Get the response data."""
response_body = response.read()
response_data = json.loads(response_body.decode("utf-8"))
return response_data
The way I use it is like this:
api = API()
rest_api_host = "api.app.com"
connection = http.client.HTTPSConnection(rest_api_host)
token = "my_third_party_token"
data = {
"token":token
}
payload = json.loads(data)
headers = {
# some headers
}
connection.request("POST", "/some/endpoint/", payload, headers)
response = connection.getresponse()
response_data = api.get_response_data(response) # I get a dictionary response
This workflow works for me. Now I just want to write a test for the get_response_data method.
How do I instantiate a http.client.HTTPResponse with the desired output to be tested?
For example:
from . import API
from unittest import TestCase
class APITestCase(TestCase):
"""API test case."""
def setUp(self) -> None:
super().setUp()
api = API()
def test_get_response_data_returns_expected_response_data(self) -> None:
"""get_response_data() method returns expected response data in http.client.HTTPResponse"""
expected_response_data = {"token": "a_secret_token"}
# I want to do something like this
response = http.client.HTTPResponse(expected_response_data)
self.assertEqual(api.get_response_data(response), expected_response_data)
How can I do this?
From the http.client docs it says:
class http.client.HTTPResponse(sock, debuglevel=0, method=None, url=None)
Class whose instances are returned upon successful connection. Not instantiated directly by user.
I tried looking at socket for the sock argument in the instantiation but honestly, I don't understand it.
I tried reading the docs in
https://docs.python.org/3/library/http.client.html#http.client.HTTPResponse
https://docs.python.org/3/library/socket.html
Searched the internet on "how to test http.client.HTTPResponse" but I haven't found the answer I was looking for.
I need help responding to an HTTP POST request. I am trying to use slack slash and you need to respond in three seconds to their http post request. My code is like this
#app.route('/recover', methods = ['POST'])
def recover(data):
if verify_signature(slack_signing_signature):
send_message()
data = {"response_type":"in_channel"}
return jsonify(data)
My problem is that how can i return a response to slack first then do the actual stuff that I need?
You can use Thread;
from threading import Thread
def slack_it(sign):
if verify_signature(sign):
send_message()
#app.route('/recover', methods = ['POST'])
def recover(data):
Thread(target=slack_it, args=(slack_signing_signature,), daemon=False).start()
return jsonify({"response_type":"in_channel"})
I am trying to implement a type of custom authentication by using aiohttp something like the example in this link but I also need request body. Here is an example for requests:
class CustomAuth(AuthBase):
def __init__(self, secretkey):
self.secretkey = secretkey
def get_hash(self, request):
if request.body:
data = request.body.decode('utf-8')
else:
data = "{}"
signature = hmac.new(
str.encode(self.secretkey),
msg=str.encode(data),
digestmod=hashlib.sha256
).hexdigest().upper()
return signature
def __call__(self, request):
request.headers["CUSTOM-AUTH"] = self.get_hash(request)
return request
I've looked into tracing and BasicAuth but they are useless in my situation. On on_request_start request body is not ready, on on_request_chunk_sent headers have already been sent. A solution like BasicAuth don't have access the request data at all.
Do you have any idea?
Thanks in advance.
I want to create a middleware function in Flask that logs details from the request and the response. The middleware should run after the Response is created, but before it is sent back. I want to log:
The request's HTTP method (GET, POST, or PUT)
The request endpoint
The response HTTP status code, including 500 responses. So, if an exception is raised in the view function, I want to record the resulting 500 Response before the Flask internals send it off.
Some options I've found (that don't quite work for me):
The before_request and after_request decorators. If I could access the request data in after_request, my problems still won't be solved, because according to the documentation
If a function raises an exception, any remaining after_request functions will not be called.
Deferred Request Callbacks - there is an after_this_request decorator described on this page, which decorates an arbitrary function (defined inside the current view function) and registers it to run after the current request. Since the arbitrary function can have info from both the request and response in it, it partially solves my problem. The catch is that I would have to add such a decorated function to every view function; a situation I would very much like to avoid.
#app.route('/')
def index():
#after_this_request
def add_header(response):
response.headers['X-Foo'] = 'Parachute'
return response
return 'Hello World!'
Any suggestions?
My first answer is very hacky. There's actually a much better way to achieve the same result by making use of the g object in Flask. It is useful for storing information globally during a single request. From the documentation:
The g name stands for “global”, but that is referring to the data being global within a context. The data on g is lost after the context ends, and it is not an appropriate place to store data between requests. Use the session or a database to store data across requests.
This is how you would use it:
#app.before_request
def gather_request_data():
g.method = request.method
g.url = request.url
#app.after_request
def log_details(response: Response):
g.status = response.status
logger.info(f'method: {g.method}\n url: {g.url}\n status: {g.status}')
return response
Gather whatever request information you want in the function decorated with #app.before_request and store it in the g object.
Access whatever you want from the response in the function decorated with #app.after_request. You can still refer to the information you stored in the g object from step 1. Note that you'll have to return the response at the end of this function.
you can use flask-http-middleware for it link
from flask import Flask
from flask_http_middleware import MiddlewareManager, BaseHTTPMiddleware
app = Flask(__name__)
class MetricsMiddleware(BaseHTTPMiddleware):
def __init__(self):
super().__init__()
def dispatch(self, request, call_next):
url = request.url
response = call_next(request)
response.headers.add("x-url", url)
return response
app.wsgi_app = MiddlewareManager(app)
app.wsgi_app.add_middleware(MetricsMiddleware)
#app.get("/health")
def health():
return {"message":"I'm healthy"}
if __name__ == "__main__":
app.run()
Every time you make request, it will pass the middleware
Okay, so the answer was staring me in the face the whole time, on the page on Deferred Request Callbacks.
The trick is to register a function to run after the current request using after_this_request from inside the before_request callback. This is the code snippet of what worked for me:
#app.before_request
def log_details():
method = request.method
url = request.url
#after_this_request
def log_details_callback(response: Response):
logger.info(f'method: {method}\n url: {url}\n status: {response.status}')
These are the steps:
Get the required details from the response in the before_request callback and store them in some variables.
Then access what you want of the response in the function you decorate with after_this_request, along with the variables you stored the request details in earlier.
Apologies for asking with what may be considered redundant, but I'm finding it extremely difficult to figure out what are the current recommended best practices for using asyncio and aiohttp.
I'm working with an API that ultimately returns a link to a generated CSV file. There are two steps in using the API.
Submit request the triggers a long running process and returns a status URL.
Poll the status URL until the status_code is 201 and then get the URL of the CSV file from the headers.
Here's a stripped down example of how I can successfully do this synchronously with requests.
import time
import requests
def submit_request(id):
"""Submit request to create CSV for specified id"""
body = {'id': id}
response = requests.get(
url='https://www.example.com/endpoint',
json=body
)
response.raise_for_status()
return response
def get_status(request_response):
"""Check whether the CSV has been created."""
status_response = requests.get(
url=request_response.headers['Location']
)
status_response.raise_for_status()
return status_response
def get_data_url(id, poll_interval=10):
"""Submit request to create CSV for specified ID, wait for it to finish,
and return the URL of the CSV.
Wait between status requests based on poll_interval.
"""
response = submit_request(id)
while True:
status_response = get_status(response)
if status_response.status_code == 201:
break
time.sleep(poll_interval)
data_url = status_response.headers['Location']
return data_url
What I'd like to do is be able to submit a group of requests at once, and then wait on all of them to be finished. But I'm not clear on how to structure this with asyncio and aiohttp.
One option would be to first submit all of the requests and then use await.gather (or something) to get all of the status URLs. Then start another event loop where I continuously poll the status_urls until they have all completed and I end up with a list of data URLs.
Alternatively, I suppose I could create a single function that submits the request, gets the status URL, and then polls that until it completes. In that case I would just have a single event loop where I submit each of the IDs that I want processed.
If some pseudo code for those options would be useful I can try to provide it. I've looked at a lot of different examples where you submit requests for a bunch of URLs asynchronously -- this for example -- but I'm finding that I get a bit lost when trying to translate them to this slightly more complicated scenario where I submit the request and then get back a new URL to poll.
FYI based on the comments above my current solution is something like this.
import asyncio
import aiohttp
async def get_data_url(session, id):
url = 'https://www.example.com/endpoint'
body = {'id': id}
async with session.post(url=url, json=body) as response:
response.raise_for_status()
status_url = response.headers['Location']
while True:
async with session.get(url=status_url) as status_response:
status_response.raise_for_status()
if status_response.status == 201:
return status_response.headers['Location']
await asyncio.sleep(10)
async def main(access_token, id):
headers = {'token': access_token}
async with aiohttp.ClientSession(headers=headers) as session:
data_url = await get_data_url(session, id)
return data_url
This works though I'm still not sure on best practices for submitting a set of IDs. I think asyncio.gather would work but it looks like it's deprecated. Ideally I would have a queue of say 100 IDs and only have 5 requests running at any given time. I've found some examples like this but they depend on asyncio.Queue which is also deprecated.