I am building an API using Flask and Flask-Restful. The API might be accessed by different sort of tools (web apps, automated tools, etc.) and one of the requirement is to provide different representations (let's say json and csv for the sake of the example)
As explained in the restful doc, it's easy to change the serialization based on the content type, so for my CSV serialization I've added this:
#api.representation('text/csv')
def output_csv(data, code, headers=None):
#some CSV serialized data
data = 'some,csv,fields'
resp = app.make_response(data)
return resp
And it's working when using curl and passing the correct -H "Accept: text/csv" parameter.
The issue is that since some browsers might be routed to a url directly to download a csv file, I would like to be able to force my serialization via a url parameter for example http://my.domain.net/api/resource?format=csv where the format=csvwould have the same effect as -H "Accept: text/csv".
I've gone through both Flask and Flask-Restful documentation and I don't see how to correctly handle this.
Simply create a sub-class of Api and override the mediatypes method:
from werkzeug.exceptions import NotAcceptable
class CustomApi(Api):
FORMAT_MIMETYPE_MAP = {
"csv": "text/csv",
"json": "application/json"
# Add other mimetypes as desired here
}
def mediatypes(self):
"""Allow all resources to have their representation
overriden by the `format` URL argument"""
preferred_response_type = []
format = request.args.get("format")
if format:
mimetype = FORMAT_MIMETYPE_MAP.get(format)
preferred_response_type.append(mimetype)
if not mimetype:
raise NotAcceptable()
return preferred_response_type + super(CustomApi, self).mediatypes()
Basically you want to retrieve parameters from GET method. Please refer to:
How do I get the url parameter in a Flask view
Related
I would like to write a FastAPI endpoint, with a Swagger page (or something similar) that will accept a non-encoded URL as input. It should preferably use GET, not POST method.
Here's an example of a GET endpoint that does require double-URL encoding.
#app.get("/get_from_dub_encoded/{double_encoded_url}")
async def get_from_dub_encoded(double_encoded_url: str):
"""
Try https%253A%252F%252Fworld.openfoodfacts.org%252Fapi%252Fv0%252Fproduct%252F7622300489434.json
"""
original_url = urllib.parse.unquote(urllib.parse.unquote(double_encoded_url))
response = requests.get(original_url)
return response.json()
Which generates a Swagger interface as below.
The following PUT request does solve my problem, but the simplicity of a GET request with a form is better for my co-workers.
class InputModel(BaseModel):
unencoded_url: AnyUrl = Field(description="An unencoded URL for an external resource", format="url")
#app.post("/unencoded-url")
def unencoded_url(inputs: InputModel):
response = requests.get(inputs.unencoded_url)
return response.json()
How can I deploy a convenient interface like that without requiring users to write the payload for a PUT request or to perform double URL encoding?
This post has some helpful related discussion, but doesn't explicitly address the FORM solution: How to pass URL as a path parameter to a FastAPI route?
You can use Form instead of query parameters as payload.
from fastapi import FastAPI, Form
import requests
app = FastAPI()
#app.post("/")
def get_url(url: str = Form()):
"""
Try https://world.openfoodfacts.org/api/v0/product/7622300489434.json
"""
response = requests.get(url)
return response.json()
Swagger interface would look like:
You'll need to install python-multipart.
Tip: Don't use async endpoint if you are using requests or any non-async library.
I need to setup a /metrics endpoint so that Prometheus can consume statistics about an endpoint. How do I go about doing this?
I have this in my Flask app:
#app.route('/metrics')
def metrics():
def generateMetrics():
metrics = ""
... some string builder logic
return metrics
response = make_response(generateMetrics(), 200)
response.mimetype = "text/plain"
return response
Is this the best way? What is the difference between returning a String (just returning metrics here) and returning plain text? Why do I need the mimetype?
Is this the best way?
There are several ways to set the MIME type, better explained and discussed in this StackOverflow question. Your way works fine and gets the job done.
What is the difference between returning a String and returning plain text?
If you return a string Flask will automatically handle some of the Response logic for you. This means using some default values. If you set up two different endpoints you'll see that the difference turns out to be that your code returns the following header:
Content-Type:"text/plain; charset=utf-8"
While just returning a string (and default MIME type) would return the following header:
Content-Type:"text/html; charset=utf-8"
Why do I need the mimetype?
You might say that it is technically more correct, given that your response is simply plain text, and not HTML. However, a more forcing reason for needing it would be that a third party system you are using (Prometheus) relies on or cares about the contents of the "Content-Type" header. If they do, then you must set it for them to accept it.
Example code
For the Content-Type header demonstration I used the following example Python code:
from flask import Flask, make_response
app = Flask(__name__)
def generateMetrics():
return "hello world"
#app.route('/metrics')
def metrics():
response = make_response(generateMetrics(), 200)
response.mimetype = "text/plain"
return response
#app.route('/metrics2')
def metrics2():
return generateMetrics()
I then viewed the returned body and headers using Postman.
I try to use falcon package in my project. Problem is I didn't find a way to get body data from the HTTP post request.
I used code from example, but req.stream.read() doesn't return JSON as expected.
The code is:
raw_json = req.stream.read()
result.json(raw_json, encoding='utf-8')
resp.body = json.dumps(result_json, encoding='utf-8')
How to get the POST data?
Thanks for any help
in falcon 2, if you work with json type, use req.media
for example:
import falcon
from json import dumps
class Resource(object):
def on_post(self, req, resp, **kwargs):
result = req.media
# do your job
resp.body = dumps(result)
api = falcon.API()
api.add_route('/test', Resource())
Little digging into the problem led to the following linked issue on github. It states that falcon framework at least in its version 0.3 and working with Python 2 didn't parse data 'POSTed' as string if they are aptly escaped. We could use more information on what data you are trying to send over POST request and in what format is that being sent, as in if its being send as simple text, or with Header Information Content-Type:application/json, or if its coming through an HTML form.
While the exact issue is not clear from the question I could still suggest trying to use bounded_stream instead of stream as in:
raw_json = req.bounded_stream.read()
result.json(raw_json, encoding='utf-8')
resp.body = json.dumps(result_json, encoding='utf-8')
for the official documentation suggests use of bounded_stream where uncertain conditions such as Content-Length undefined or 0, or if header information is missing altogether.
bounded_stream is described as the following in the official falcon documentation.
File-like wrapper around stream to normalize certain differences between the native input objects employed by different WSGI servers. In particular, bounded_stream is aware of the expected Content-Length of the body, and will never block on out-of-bounds reads, assuming the client does not stall while transmitting the data to the server.
Falcon receives the HTTP requests data as buffer object as passed by WSGI wrapper which receives the data from client, and its possible it doesn't run proper parsing on top of the data to convert to a more usable data structure for performance reasons.
Big thanks to Ryan (and Prateek Jain) for the answer.
The solution is simply to put app.req_options.auto_parse_form_urlencoded=True. For example:
import falcon
class ThingsResource(object):
def on_post(self, req, resp):
value = req.get_param("value", required=True)
#do something with value
app = falcon.API()
app.req_options.auto_parse_form_urlencoded=True
things = ThingsResource()
app.add_route('/things', things)
The field you're looking for is somewhat confusingly named, but it's req.media:
Returns a deserialized form of the request stream. When called, it will attempt to deserialize the request stream using the Content-Type header as well as the media-type handlers configured via falcon.RequestOptions.
If the request is JSON, req.media already contains a python dict.
I have added changes in request.py in falcon framework to parse application/x-www-form-urlencoded and multipart/from-data.
I have raised pull request - https://github.com/falconry/falcon/pull/1236 but it is not yet merged in master.
Check this - https://github.com/branelmoro/falcon
I have added new code to parse POST, PUT and DELETE application/x-www-form-urlencoded and multipart/form-data.
Text fields will be available in req.form_data dictionary and upload file buffer stream will be available in req.files dictionary.
I hope this will help to access POST and GET parameters separately and we will be able to upload files as well.
Good thing about the change is that it will not load entire uploaded file in memory.
Below is sample code to show how to use POST, PUT and DELETE application/x-www-form-urlencoded and multipart/form-data:
import falcon
class Resource(object):
def on_post(self, req, resp):
# req.form_data will return dictionary of text field names and their values
print(req.form_data)
# req.form_data will return dictionary of file field names and
# their buffer class FileStream objects as values
print(req.files)
# support we are uploading a image.jpg in `pancard` file field then
# req.files["pancard"] will be FileStream buffer object
# We can use set_max_upload_size method to set maximum allowed
# file size let say 1Mb = 1*1024*1024 bytes for this file
req.files["pancard"].set_max_upload_size(1*1024*1024)
# We can use uploadto method to upload file on required path (Note: absolute filepath is required)
# This method returns boolean - `True` on successful upload
# and if upload is unsuccessful then it returns `False` and sets error on failure.
path = "/tmp/" + req.files["pancard"].name
response = req.files["pancard"].uploadto("/tmp/" + path)
print(response)
# Once file is uploaded sucessfully, we can check it's size
print(req.files["pancard"].size)
# If file is not uploaded sucessfully, we can check it's error
print(req.files["pancard"].error)
resp.body = "Done file upload"
resp.status = falcon.HTTP_200
# falcon.API instances are callable WSGI apps
app = falcon.API()
things = Resource()
# things will handle post requests to the '/post_path' URL path
app.add_route('/post_path', things)
Do let me know if you have any doubts.
So far... for me bounded_stream.read() and stream.read() both get the posted data as type str. I have only found one way around the issue so far:
def on_post(self, req, resp):
posted_data = json.loads(req.stream.read())
print(str(type(posted_data)))
print(posted_data)
Loading the string into a json dict once the posted data is received is my only solution that I can come up with
Here's something I used while designing an API.
import falcon
import json
class VerifierResource():
def on_post(self, req, resp):
credentials = json.loads(req.stream.read())
if credentials['username'] == USER \
and credentials['passwd'] == PASSWORD:
resp.body = json.dumps({"status": "verified"})
else:
resp.body = json.dumps({"status": "invalid"})
api = falcon.API()
api.add_route('/verify', VerifierResource())
This returns a serialized JSON with corresponding response body.
there is a sample way to get media from body. I use to get the body in the post method:
def on_post(req,resp)
arguments = {}
# get body media on post method
body = req.get_media()
if 'something' in body:
arguments['something'] = body['something']
send body content type Media-Type and print resp or use in code, but if want to send JSON body your code should cover give JSON parameters.
Do let me know if you have any doubts.
I am completely new to Python. I am using GitLab which offers system hook feature wherein i can specify a URL and it will send event details in the form of JSON POST data. When i create a RequestBin URL and provide that URL in GitLab's system hook, then in case of any event such as project creation, it sends the event details and i can see the same in RequestBin as shown in the snapshot below.
Now, i want to fetch this JSON data in some variable so i can process it as per my need but i'm not sure how to do read that data.
I've seen some posts which explain how to read JSON data but as you can see below in the screenshot, the FORM/POST PARAMETERS is showing as None. It's the raw body that contains all the details (in JSON format):
I have tried reading the data using Java and it works with the code shown below:
String recv;
String recvbuff="";
BufferedReader buffread = new BufferedReader(new InputStreamReader(request.getInputStream()));
while ((recv = buffread.readLine()) != null)
recvbuff += recv;
buffread.close();
System.out.println(recvbuff);
out.println(recvbuff);
Looking for something similar in Python.
I would suggest using CherryPy. It's a neat Python library that allows you to build a simple webserver application, it's fits pretty nicely in your use case: it can easily accept JSON requests (http://docs.cherrypy.org/en/latest/basics.html#dealing-with-json).
If you write a file called myserver.py with the following code:
#!/usr/bin/python3
import cherrypy
class Root(object):
#cherrypy.expose
#cherrypy.tools.json_in()
def index(self):
data = cherrypy.request.json
# You can manipulate here your json data as you wish
print(data['name'])
if __name__ == '__main__':
cherrypy.quickstart(Root(), '/')
You can simply launch the server with the command line:
python3 myserver.py
And test it with the following curl command:
curl -H "Content-Type: application/json" -POST http://127.0.0.1:8080 -d '{"name": "test", "path": "/"}'
You will then see test printed in your server log.
Your Flask application doesn't return any data so you're not going to see anything returned. You need to return something like:
return "test data"
Your screenshot is only showing the request not the response. You sent no form encoded parameters, which is why it's showing "None".
The correct Content-type for JSON is: application/json
I'm working on an API wrapper. The spec I'm trying to build to has the following request in it:
curl -H "Content-type:application/json" -X POST -d data='{"name":"Partner13", "email":"example#example.com"}' http://localhost:5000/
This request produces the following response from a little test server I setup to see exatly what headers/params etc are sent as. This little script produces:
uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data={"name":"Partner13", "email":"example#example.com"}
So that above is the result I want my python script to create when it hits the little test script.
I'm using the python requests module, which is the most beautiful HTTP lib I have ever used. So here is my python code:
uri = "http://localhost:5000/"
headers = {'content-type': 'application/json' }
params = {}
data = {"name":"Partner13", "email":"example#exmaple.com"}
params["data"] = json.dumps(data)
r = requests.post(uri, data=params, headers=headers)
So simple enough stuff. Set the headers, and create a dictionary for the POST parameters. That dictionary has one entry called "data" which is the JSON string of the data I want to send to the server. Then I call the post. However, the result my little test script gives back is:
uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data=%7B%22name%22%3A+%22Partner13%22%2C+%22email%22%3A+%22example%40example.com%22%7D
So essentially the json data I wanted to send under the data parameter has been urlendcoded.
Does anyone know how to fix this? I have looked through the requests documentation and cannot seem to find a way to not auto urlencode the send data.
Thanks very much,
Kevin
When creating the object for the data keyword, simply assign a variable the result of json.dumps(data).
Also, because HTTP POST can accept both url parameters as well as data in the body of the request, and because the requests.post function has a keyword argument named "params", it might be better to use a different variable name for readability. The requests docs use the variable name "payload", so thats what I use.
data = {"name":"Partner13", "email":"example#exmaple.com"}
payload = json.dumps(data)
r = requests.post(uri, data=payload, headers=headers)
Requests automatically URL encodes dictionaries passed as data here. John_GG's solution works because rather than posting a dictionary containing the JSON encoded string in the 'data' field it simply passes the JSON encoded string directly: strings are not automatically encoded. I can't say I understand the reason for this behaviour in Requests but regardless, it is what it is. There is no way to toggle this behaviour off that I can find.
Best of luck with it, Kevin.