I wrote a certain API wrapper using Python's requests library.
When it gets a response using requests.get, it attempts to parse as json and takes the raw content if it doesn't work:
resp = requests.get(url, ...)
try:
resp_content = resp.json()
except ValueError:
resp_content = resp.content
return resp_content
This is correct for my purposes. The problem is how long it takes when the downloaded response is an image file, for example, if it is large, then it takes an extremely long time between entering the try, and failing the json parse and entering the except.
(I don't know if it takes super long for the .json() to error at all, or if once it errors it then takes a while to get into the except.)
Is there a way to see if resp is json-parsable without attempting to parse it with .json()? Something like resp.is_json, so I can instantly know which branch to take (resp.json() or resp.content), instead of waiting 30 seconds (large files can take minutes).
Depending on the consistency of the response, you could check if the returned headers include content-type application/json:
resp.headers.get('content-type') == 'application/json'
You can check if the content-type application/json is in the response headers:
'application/json' in response.headers.get('Content-Type', '')
(Addressing Daniel Kats comment in a previous response)
You could check if the returned headers include Content-Type application/json:
response.headers.get('Content-Type').startswith('application/json')
By using startswith, you're accounting for all allowed formats from https://www.w3.org/Protocols/rfc1341/4_Content-Type.html.
That doesn't guaranty that it will be a valid JSON, but at least that will catch responses which aren't being declared as JSON.
If using Session and not directly requests.(METHOD)
from requests import Session
from simplejson.errors import JSONDecodeError
class MySession(Session):
def __init__(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
def request(self, *args, **kwargs):
res = super().request(*args, **kwargs)
json = res.json
def wrapper():
try:
return json()
except JSONDecodeError:
return None
res.json = wrapper
return res
session = MySession()
res = session.get("https://api64.ipify.org")
if res.json():
print("ok")
Have you tried -
#resp_content = resp.json()
if resp_content.ok:
resp_content.json()
else:
resp_content = resp.content
I would check the first few 100 bytes and count the number of json characters like {":. Or you could check for image signatures (JFIF, PNG, GIF89A)..
Related
I have a function like this:
def get_some_data(api_url, **kwargs)
# some logic on generating headers
# some more logic
response = requests.get(api_url, headers, params)
return response
I need to create a fake/mock "api_url", which, when made request to, would generate a valid response.
I understand how to mock the response:
def mock_response(data):
response = requests.Response()
response.status_code = 200
response._content = json.dumps(data)
return response
But i need to make the test call like this:
def test_get_some_data(api_url: some_magic_url_path_that_will_return_mock_response):
Any ideas on how to create an url path returning a response within the scope of the test (only standard Django, Python, pytest, unittest) would be very much appreciated
The documentation is very well written and more than clear on how to mock whatever you want. But, let say you have a service that makes the 3rd party API call:
def foo(url, params):
# some logic on generating headers
# some more logic
response = requests.get(url, headers, params)
return response
In your test you want to mock the return value of this service.
#patch("path_to_service.foo")
def test_api_call_response(self, mock_response):
mock_response.return_value = # Whatever the return value you want it to be
# Here you call the service as usual
response = foo(..., ...)
# Assert your response
While implementing a rest API in python flask, I have used several options to return a file (any type) , read it and save it to local repository of request but encountered with multiple errors as below:
Case 1:
def download_file():
return send_file('any_file.pdf')
r = requests.get(url = 'http://localhost:5000/download').read()
has responded with a error Response object has no attribute read/text/content
Case 2:
def download_file():
file = open('any_file.pdf','r').read()
return file
r = requests.get(url = 'http://localhost:5000/download')
has responded with a error Return doesn't accept this
So How can I do this as flask is not allowing to return a file without response object and response object is not readable and doesn't support to save that file directly.
The Flask server code in Case 1 is correct. A more complete example:
#app.route('/download')
def download_file():
# Some logic here
send_file('any_file.pdf')
However the Response object returned by requests.get doesn't have a read method. The correct way is to use:
Response.content: Content of the response, in bytes.
So, the client code should be:
r = requests.get('http://localhost:5000/download')
bytes = r.content
# Now do something with bytes, for example save it:
with open('downloaded_file.ext', 'wb') as f:
f.write(bytes)
I am developping a JSON API with Python Flask.
What I want is to always return JSON, with a error message indicating any error that occured.
That API also only accept JSON data in the POST body, but Flask by default return a HTML error 400 if it can't read the data as JSON.
Preferably, I d also like to not force the user to send the Content-Type header, and if raw or text content-type, try to parse the body as JSON nonetheless.
In short, I need a way to validate that the POST body's is JSON, and handle the error myself.
I've read about adding decorator to request to do that, but no comprehensive example.
You have three options:
Register a custom error handler for 400 errors on the API views. Have this error return JSON instead of HTML.
Set the Request.on_json_loading_failed method to something that raises a BadRequest exception subclass with a JSON payload. See Custom Errors in the Werkzeug exceptions documentation to see how you can create one.
Put a try: except around the request.get_json() call, catch the BadRequest exception and raise a new exception with a JSON payload.
Personally, I'd probably go with the second option:
from werkzeug.exceptions import BadRequest
from flask import json, Request, _request_ctx_stack
class JSONBadRequest(BadRequest):
def get_body(self, environ=None):
"""Get the JSON body."""
return json.dumps({
'code': self.code,
'name': self.name,
'description': self.description,
})
def get_headers(self, environ=None):
"""Get a list of headers."""
return [('Content-Type', 'application/json')]
def on_json_loading_failed(self):
ctx = _request_ctx_stack.top
if ctx is not None and ctx.app.config.get('DEBUG', False):
raise JSONBadRequest('Failed to decode JSON object: {0}'.format(e))
raise JSONBadRequest()
Request.on_json_loading_failed = on_json_loading_failed
Now, every time request.get_json() fails, it'll call your custom on_json_loading_failed method and raise an exception with a JSON payload rather than a HTML payload.
Combining the options force=True and silent=True make the result of request.get_json be None if the data is not parsable, then a simple if allow you to check the parsing.
from flask import Flask
from flask import request
#app.route('/foo', methods=['POST'])
def function(function = None):
print "Data: ", request.get_json(force = True, silent = True);
if request.get_json() is not None:
return "Is JSON";
else:
return "Nope";
if __name__ == "__main__":
app.run()
Credits to lapinkoira and Martijn Pieters.
You can try to decode JSON object using python json library.
The main idea is to take plain request body and try to convert to JSON.E.g:
import json
...
# somewhere in view
def view():
try:
json.loads(request.get_data())
except ValueError:
# not a JSON! return error
return {'error': '...'}
# do plain stuff
The code below sets up _request if the HTTP method is GET, then it has an ifstatement for handling PUT POST and PATCH.
I'm trying to have one single request setup statement for all method types.
Is this possible? It appears to me that there is no way to add data to a prepared request, and if this is true then perhaps I'm stuck with needing two different ways of setting up a request, one way for GET and one way for PUT, PATCH and POST.
def fetch_from_api(self):
s = Session()
headers = { "Authorization" : REST_API_AUTHORIZATION_HEADER}
_request = Request(self.method, self.url_for_api, headers=headers)
if self.method in ['POST', 'PATCH', 'PUT']:
headers['content-type'] = 'application/x-www-form-urlencoded'
_request = Request(self.method, self.url_for_api, headers=headers, data=self.postdata)
prepped = _request.prepare()
self.api_response = s.send(prepped)
The question is a little old and hopefully #DukeDougal already has a solution. Maybe this will help others, though.
The first thing I notice in the example is that a Request object is created near the beginning of the method. Then, if the method is "POST", "PATCH", or "PUT", the Request constructor is called again to get another object. In that case, the first object is gone. It was created unnecessarily.
When a data= argument isn't given to the Request constructor, it's the same as specifying data=None. Take advantage of that and call the constructor only once, then a data value won't need to be added to an existing Request (or PreparedRequest) object:
def fetch_from_api(self):
s = Session()
headers = {'Authorization': REST_API_AUTHORIZATION_HEADER}
data = None # Assume no data until method is checked
if self.method in ['POST', 'PATCH', 'PUT']:
headers['content-type'] = 'application/x-www-form-urlencoded'
data = self.postdata # Add the data
# Now headers and data are ready, get a Request object
_request = Request(self.method, self.url_for_api, headers=headers, data=data)
prepped = _request.prepare()
self.api_response = s.send(prepped)
If you look at the requests.Request model, it looks like you can set the data attribute if needed:
some_request = Request(method, url, headers=headers)
if # ...we decide we need to add data:
some_request.data = data
Looking at the model, it appears that this would work, because when you prepare the request later on, it looks at the instance's data attribute.
EDIT:
But reading your question a bit more closely, it looks like you want to add data to a prepared_request. I guess you could create your own prepared_request and pass the data in specifically when you call the prepare method, but I don't see how that helps? It seems like you want to just branch and maybe add data or maybe not?
Anyway, the above seems it could potentially simplify your code slightly to the following:
def fetch_from_api(self):
s = Session()
headers = { "Authorization" : REST_API_AUTHORIZATION_HEADER}
_request = Request(self.method, self.url_for_api, headers=headers)
if self.method in ['POST', 'PATCH', 'PUT']:
headers['content-type'] = 'application/x-www-form-urlencoded'
_request.data = self.postdata
prepped = _request.prepare()
self.api_response = s.send(prepped)
(But that doesn't look much simpler to me. What are we trying to achieve? Also, it seems weird to have a method called fetch_from_api that could also be POSTing or PUTing data. As a dev, I would not be expecting that to be the case from the name.)
In the past, I've done stuff like this as a result of having to sign requests: I have to create them in one place and then hand them off to a class that knows how to create signatures, which then hands them back. In other words, you can certainly edit requests before preparing and sending them on their way.
Anyway, I haven't tried any of this, but it's similar to some things I've done in the past with requests, so it looks legit, but I would be concerned about what you are attempting to achieve and whether or not things are being crammed together which maybe should not be.
I am using HTTPforHumans, requests module.
import requests
def pv_request(url, methods, data=None, headers=None, type=None):
try:
if 'POST' in methods:
return requests.post(url=url, headers=headers, data=data).json()
elif 'GET' in methods:
return requests.get(url=url, headers=headers, data=data).json()
elif 'PUT' in methods:
if type == 'music':
return requests.put(url=url, headers=headers, data=data).json()
elif type == 'image':
return requests.put(url=url, headers=headers, data=open(data, 'rb')).json()
except requests.exceptions.ConnectionError:
return None
Might not be in the lines of what you are looking for, but here is my all-in-on purpose request handler.
I'm trying to build a simple proxy using Flask and requests. The code is as follows:
#app.route('/es/<string:index>/<string:type>/<string:id>',
methods=['GET', 'POST', 'PUT']):
def es(index, type, id):
elasticsearch = find_out_where_elasticsearch_lives()
# also handle some authentication
url = '%s%s%s%s' % (elasticsearch, index, type, id)
esreq = requests.Request(method=request.method, url=url,
headers=request.headers, data=request.data)
resp = requests.Session().send(esreq.prepare())
return resp.text
This works, except that it loses the status code from Elasticsearch. I tried returning resp (a requests.models.Response) directly, but this fails with
TypeError: 'Response' object is not callable
Is there another, simple, way to return a requests.models.Response from Flask?
Ok, found it:
If a tuple is returned the items in the tuple can provide extra information. Such tuples have to be in the form (response, status, headers). The status value will override the status code and headers can be a list or dictionary of additional header values.
(Flask docs.)
So
return (resp.text, resp.status_code, resp.headers.items())
seems to do the trick.
Using text or content property of the Response object will not work if the server returns encoded data (such as content-encoding: gzip) and you return the headers unchanged. This happens because text and content have been decoded, so there will be a mismatch between the header-reported encoding and the actual encoding.
According to the documentation:
In the rare case that you’d like to get the raw socket response from the server, you can access r.raw. If you want to do this, make sure you set stream=True in your initial request.
and
Response.raw is a raw stream of bytes – it does not transform the response content.
So, the following works for gzipped data too:
esreq = requests.Request(method=request.method, url=url,
headers=request.headers, data=request.data)
resp = requests.Session().send(esreq.prepare(), stream=True)
return resp.raw.read(), resp.status_code, resp.headers.items()
If you use a shortcut method such as get, it's just:
resp = requests.get(url, stream=True)
return resp.raw.read(), resp.status_code, resp.headers.items()
Flask can return an object of type flask.wrappers.Response.
You can create one of these from your requests.models.Response object r like this:
from flask import Response
return Response(
response=r.reason,
status=r.status_code,
headers=dict(r.headers)
)
I ran into the same scenario, except that in my case my requests.models.Response contained an attachment. This is how I got it to work:
return send_file(BytesIO(result.content), mimetype=result.headers['Content-Type'], as_attachment=True)
My use case is to call another API in my own Flask API. I'm just propagating unsuccessful requests.get calls through my Flask response. Here's my successful approach:
headers = {
'Authorization': 'Bearer Muh Token'
}
try:
response = requests.get(
'{domain}/users/{id}'\
.format(domain=USERS_API_URL, id=hit['id']),
headers=headers)
response.raise_for_status()
except HTTPError as err:
logging.error(err)
flask.abort(flask.Response(response=response.content, status=response.status_code, headers=response.headers.items()))