We Keep Coding

Python Flask/JSON: string indices must be integers

I am trying to write a webserver with Flask and e.g. have following function for post:
#app.route('/v1/tasks', methods=['POST'])
def post():
data=request.get_json()
title=data["title"]
tasks.append(json.dumps({"id": len(tasks), "title": title, "is_completed": "false"}))
index=len(tasks)-1
return json.dumps({"id":index}), 200
title=data["title"] throws following error:
TypeError: string indices must be integers
The input format for POST should be:
{title: "Test Task 2"}
I am confused because I saw a different post function, where access the contents of the JSON worked like this:
#app.route('/post', methods=['POST'])
def post():
data=request.get_json()
dictionary[data["key"]]=data["value"]
data["message"]="success"
return json.dumps(data)
What do I need to change so that I can access the title from the input JSON?
Thank you for your help!

There is nothing wrong. Verify if you are using the correct MIME media type for posting JSON, which is application/json. Test your Flask server with this code and see if it works correctly.
import requests
response = requests.post('http://localhost:5000/v1/tasks', json={'title': "Test Task 2"})
print(response.status_code)

Python Falcon - get POST data

I try to use falcon package in my project. Problem is I didn't find a way to get body data from the HTTP post request.
I used code from example, but req.stream.read() doesn't return JSON as expected.
The code is:
raw_json = req.stream.read()
result.json(raw_json, encoding='utf-8')
resp.body = json.dumps(result_json, encoding='utf-8')
How to get the POST data?
Thanks for any help

in falcon 2, if you work with json type, use req.media
for example:
import falcon
from json import dumps
class Resource(object):
def on_post(self, req, resp, **kwargs):
result = req.media
# do your job
resp.body = dumps(result)
api = falcon.API()
api.add_route('/test', Resource())

Little digging into the problem led to the following linked issue on github. It states that falcon framework at least in its version 0.3 and working with Python 2 didn't parse data 'POSTed' as string if they are aptly escaped. We could use more information on what data you are trying to send over POST request and in what format is that being sent, as in if its being send as simple text, or with Header Information Content-Type:application/json, or if its coming through an HTML form.
While the exact issue is not clear from the question I could still suggest trying to use bounded_stream instead of stream as in:
raw_json = req.bounded_stream.read()
result.json(raw_json, encoding='utf-8')
resp.body = json.dumps(result_json, encoding='utf-8')
for the official documentation suggests use of bounded_stream where uncertain conditions such as Content-Length undefined or 0, or if header information is missing altogether.
bounded_stream is described as the following in the official falcon documentation.
File-like wrapper around stream to normalize certain differences between the native input objects employed by different WSGI servers. In particular, bounded_stream is aware of the expected Content-Length of the body, and will never block on out-of-bounds reads, assuming the client does not stall while transmitting the data to the server.
Falcon receives the HTTP requests data as buffer object as passed by WSGI wrapper which receives the data from client, and its possible it doesn't run proper parsing on top of the data to convert to a more usable data structure for performance reasons.

Big thanks to Ryan (and Prateek Jain) for the answer.
The solution is simply to put app.req_options.auto_parse_form_urlencoded=True. For example:
import falcon
class ThingsResource(object):
def on_post(self, req, resp):
value = req.get_param("value", required=True)
#do something with value
app = falcon.API()
app.req_options.auto_parse_form_urlencoded=True
things = ThingsResource()
app.add_route('/things', things)

The field you're looking for is somewhat confusingly named, but it's req.media:
Returns a deserialized form of the request stream. When called, it will attempt to deserialize the request stream using the Content-Type header as well as the media-type handlers configured via falcon.RequestOptions.
If the request is JSON, req.media already contains a python dict.

I have added changes in request.py in falcon framework to parse application/x-www-form-urlencoded and multipart/from-data.
I have raised pull request - https://github.com/falconry/falcon/pull/1236 but it is not yet merged in master.
Check this - https://github.com/branelmoro/falcon
I have added new code to parse POST, PUT and DELETE application/x-www-form-urlencoded and multipart/form-data.
Text fields will be available in req.form_data dictionary and upload file buffer stream will be available in req.files dictionary.
I hope this will help to access POST and GET parameters separately and we will be able to upload files as well.
Good thing about the change is that it will not load entire uploaded file in memory.
Below is sample code to show how to use POST, PUT and DELETE application/x-www-form-urlencoded and multipart/form-data:
import falcon
class Resource(object):
def on_post(self, req, resp):
# req.form_data will return dictionary of text field names and their values
print(req.form_data)
# req.form_data will return dictionary of file field names and
# their buffer class FileStream objects as values
print(req.files)
# support we are uploading a image.jpg in `pancard` file field then
# req.files["pancard"] will be FileStream buffer object
# We can use set_max_upload_size method to set maximum allowed
# file size let say 1Mb = 1*1024*1024 bytes for this file
req.files["pancard"].set_max_upload_size(1*1024*1024)
# We can use uploadto method to upload file on required path (Note: absolute filepath is required)
# This method returns boolean - `True` on successful upload
# and if upload is unsuccessful then it returns `False` and sets error on failure.
path = "/tmp/" + req.files["pancard"].name
response = req.files["pancard"].uploadto("/tmp/" + path)
print(response)
# Once file is uploaded sucessfully, we can check it's size
print(req.files["pancard"].size)
# If file is not uploaded sucessfully, we can check it's error
print(req.files["pancard"].error)
resp.body = "Done file upload"
resp.status = falcon.HTTP_200
# falcon.API instances are callable WSGI apps
app = falcon.API()
things = Resource()
# things will handle post requests to the '/post_path' URL path
app.add_route('/post_path', things)
Do let me know if you have any doubts.

So far... for me bounded_stream.read() and stream.read() both get the posted data as type str. I have only found one way around the issue so far:
def on_post(self, req, resp):
posted_data = json.loads(req.stream.read())
print(str(type(posted_data)))
print(posted_data)
Loading the string into a json dict once the posted data is received is my only solution that I can come up with

Here's something I used while designing an API.
import falcon
import json
class VerifierResource():
def on_post(self, req, resp):
credentials = json.loads(req.stream.read())
if credentials['username'] == USER \
and credentials['passwd'] == PASSWORD:
resp.body = json.dumps({"status": "verified"})
else:
resp.body = json.dumps({"status": "invalid"})
api = falcon.API()
api.add_route('/verify', VerifierResource())
This returns a serialized JSON with corresponding response body.

there is a sample way to get media from body. I use to get the body in the post method:
def on_post(req,resp)
arguments = {}
# get body media on post method
body = req.get_media()
if 'something' in body:
arguments['something'] = body['something']
send body content type Media-Type and print resp or use in code, but if want to send JSON body your code should cover give JSON parameters.
Do let me know if you have any doubts.

Ajax post to Python server: No JSON object could be decoded

I am getting this error from the server:
ValueError: No JSON object could be decoded
This is the ajax request from the client-side
function searchDB(profile_url) {
console.log(profile_url);
$.ajax({
type: 'POST',
url:'http://127.0.0.1:5000/update_greenhouse',
data: JSON.stringify(profile_url),
contentType: "application/json"
})
}
Server code:
#app.route('/update_greenhouse', methods=['GET', 'POST', 'OPTIONS'])
def update_gh():
y = request.data
print(y)
json.loads(y)
I am setting the contentType to be JSON. Why is the server giving me that error?

I have no idea why, but it was "OPTIONS" in the methods=[] caused the issue. Removed it and everything worked.

It seems that you're not sending proper JSON to the server. Just setting the contentType isn't enough. Your data (profile_url) needs to be encoded properly, as JSON object.
Try something like this in your request code:
data: JSON.stringify({'url': profile_url})
Your server will then get a basic dictionary back to use. If you really want to just pass a string, you can forgo the json.loads() on the server and operate direction from request.data, but for building anything more than a hello world app, I'd strongly suggest using a more structured format like JSON.
You also need to return a response (or error) for every Flask request. If you're building a JSON-based API, I'd suggest using Flask's jsonify. Something along the lines of
return flask.jsonify({'status': 'updated'})
on success, possibly with details about whatever the updated state is. If there are errors, you should return a response with the proper error code, like so:
return flask.make_response(jsonify({'error': 'Error details'}), 418)
Replacing 418 with whatever the appropriate HTTP error code may be.

Why does JSON returned from the django rest framework have forward slashes in the response?

My response code
from rest_framework.response import Response
import json
responseData = { 'success' : True }
return Response(json.dumps(responseData))
How it appears on doing curl or accessing the response through the browser.
"{\"success\": true}"
Why the forward slashes? How do I remove them?

You are rendering the data to JSON twice. Remove your json.dumps() call.
From the Django REST documentation:
Unlike regular HttpResponse objects, you do not instantiate Response objects with rendered content. Instead you pass in unrendered data, which may consist of any Python primitives.
The Django REST framework then takes care of producing JSON for you. Since you gave it a string, that string was JSON encoded again:
>>> import json
>>> responseData = { 'success' : True }
>>> print json.dumps(responseData)
{"success": true}
>>> print json.dumps(json.dumps(responseData))
"{\"success\": true}"
The framework uses Content Negotiation to determine what serialisation format to use; that way your API clients can also request that the data is encoded as YAML or XML, for example.
Also see the Responses documentation:
REST framework supports HTTP content negotiation by providing a Response class which allows you to return content that can be rendered into multiple content types, depending on the client request.

Unit testing Django JSON View

I'm trying to write some unit tests for some Django json_view views and I'm having trouble passing the json_string to the view. I posted a related question yesterday about passing a json string to a Django view from the JS, the issue was that in my JS I was just passing the json string where I needed to be passing the string as the attribute of an object, because I was failing to do this the string was being taken as the key for the resulting query dict. I'm having a similar problem again except that this time it is form a Django unit test to the Django View. Here is a simplified version of my code which produces the same result.
class MyTestCase(TestCase):
def setUp(self):
self.u = User.objects.create_user('test','test','test')
self.u.is_active = True
self.u.save()
self.client.login(username='test',password='test')
def test_create_object_from_form(self):
"""Test the creation of the Instance from the form data."""
import json
json_string json.dumps({'resource':{'type':'book','author':'John Doe'}})
print(json_string)
response = self.client.post(reverse('ajax_view'),
{'form':json_string},'json')
self.assetNotContains(response,'error')
and the view looks like this
#json_view
def ajax_view(request):
"""Process the incoming form data."""
if request.method == 'POST':
print(request.POST)
form_data = json.loads(request.POST['form'])
resource_data = form_data['resource']
form = MyUserForm(resource_data)
if form.is_valid():
...
Here is what the two print statements produce when the test is run. The json_string is
{"resource": {"type": "book", "author": "John Doe"}}
and the query dict looks like
<QueryDict: {u'{\'form\': \'{"resource": {"type": "book", "author": "John Doe"}}\'}': [u'']}>
I'm total newbie with JS and ajax, so don't worry about hurting my pride, the answer is probably so close it could jump up and bite me.

Final edit
I originally stated that header HTTP_X_REQUESTED_WITH='XMLHttpRequest' was necessary in the post call but this is currently false while in tests. This header is necessary for the csrf middleware but csrf is disabled in tests. However, I still believe it is a good practice to put in test even if middleware disables csrf since most javascript library already pass this header by default when doing ajax. Also, if another piece of code that is not disabled ever use the is_ajax method, you won't need to debug your unittest for hours to figure out that the header was missing.
The problem is with the content-type because when django gets a value in there that is different than text/html, it doesn't use the default post data handling which is to format your data like in a query: type=book&author=JohnDoe for example.
Then the fixed code is:
response = self.client.post(reverse('ajax_view'),
{'form':json_string},
HTTP_X_REQUESTED_WITH='XMLHttpRequest')
Here's how I'm using it myself:
post_data = {
"jsonrpc" : "2.0", "method": method, "params" : params, "id" : id }
return client.post('/api/json/',
json.dumps(post_data), "text/json",
HTTP_X_REQUESTED_WITH='XMLHttpRequest')
to do some json-rpc. Notice that since I pass a different content-type than the default value, my data is passed as is in the post request.

Thank you to #Eric_Fortin for turning me on to the header, it does not however resolve my issue with the malformed query dictionary using 'client.post'. Once I made the change from POST to GET with the XMLHttpRequest header my query dictionary straitened itself out. Here is the current solution:
response = self.client.get(reverse('ajax_view'),
{'form':json_string},'json',
HTTP_X_REQUESTED_WITH='XMLHttpRequest')
this is only a partial answer since this request is going to change data on the server and should be POST not a GET.
Edit:
Here is the final code in my test that works for passing a JSON string via POST to my view:
response = self.client.post(reverse('ajax_view'),
{'form':json.dumps(json_dict)})
Now printing from the view shows that the query dictionary is well formed.
<QueryDict: {u'form': [u'{"resource": {"status": "reviewed", "name": "Resource Test", "description": "Unit Test"}}']}>
I found the answer while tinkering with one of my co-workers, removing the content_type 'json' fixed the malformed query dictionary. The view that is being tested does not make use of or call the 'HttpRequest.is_ajax()', sending the header XMLHttpRequest' has no impact on my issue, though including the header would constitute good-form since this post is an ajax request.