We Keep Coding

Python - Accept POST (raw body) data

I am completely new to Python. I am using GitLab which offers system hook feature wherein i can specify a URL and it will send event details in the form of JSON POST data. When i create a RequestBin URL and provide that URL in GitLab's system hook, then in case of any event such as project creation, it sends the event details and i can see the same in RequestBin as shown in the snapshot below.
Now, i want to fetch this JSON data in some variable so i can process it as per my need but i'm not sure how to do read that data.
I've seen some posts which explain how to read JSON data but as you can see below in the screenshot, the FORM/POST PARAMETERS is showing as None. It's the raw body that contains all the details (in JSON format):
I have tried reading the data using Java and it works with the code shown below:
String recv;
String recvbuff="";
BufferedReader buffread = new BufferedReader(new InputStreamReader(request.getInputStream()));
while ((recv = buffread.readLine()) != null)
recvbuff += recv;
buffread.close();
System.out.println(recvbuff);
out.println(recvbuff);
Looking for something similar in Python.

I would suggest using CherryPy. It's a neat Python library that allows you to build a simple webserver application, it's fits pretty nicely in your use case: it can easily accept JSON requests (http://docs.cherrypy.org/en/latest/basics.html#dealing-with-json).
If you write a file called myserver.py with the following code:
#!/usr/bin/python3
import cherrypy
class Root(object):
#cherrypy.expose
#cherrypy.tools.json_in()
def index(self):
data = cherrypy.request.json
# You can manipulate here your json data as you wish
print(data['name'])
if __name__ == '__main__':
cherrypy.quickstart(Root(), '/')
You can simply launch the server with the command line:
python3 myserver.py
And test it with the following curl command:
curl -H "Content-Type: application/json" -POST http://127.0.0.1:8080 -d '{"name": "test", "path": "/"}'
You will then see test printed in your server log.

Your Flask application doesn't return any data so you're not going to see anything returned. You need to return something like:
return "test data"
Your screenshot is only showing the request not the response. You sent no form encoded parameters, which is why it's showing "None".
The correct Content-type for JSON is: application/json

How can I send multipart/form-data that contains json object and image file via curl? And how can I treat it with Django Rest Framework?

For example I tried that command via terminal:
curl -F "profileImage=#/home/user/image.jpg" -F "{'firstName':'hello'};type=application/json" http://127.0.0.1:8000/api/v1/signup/
Then I received the request object like that:
print request.FILES
# <MultiValueDict: {u'profileImage': [<InMemoryUploadedFile: image.jpg (image/jpeg)>]}>
print request.DATA
# <QueryDict: {u"{'firstName':'hello'};content-type": [u'application/json']}>
The image is ok, but QueryDict is not represented correctly - all JSON file is a key and content-type
is a value.
In Django use these parsers:
parser_classes = (MultiPartParser, FormParser, JSONParser,)
I necessary need to send text data via JSON structure .

If you want to POST with multipart/form-data content-type, you can't also specify application/json. The simple fix for this is to send the form data in url-encoded format. Try this command:
curl -F "profileImage=#/home/user/image.jpg" -F "firstName=hello" http://127.0.0.1:8000/api/v1/signup/

I came up with a solution like this:
class MultiPartJSONParser(parsers.MultiPartParser):
def parse(self, stream, media_type=None, parser_context=None):
dataAndFiles = super(MultiPartJSONParser, self).parse(stream, media_type, parser_context)
try:
jsonData = json.loads(dataAndFiles.data.get('data'))
except ValueError as exc:
raise parsers.ParseError('JSON parse error - %s' % six.text_type(exc))
# make data mutable, insert json data, and remove raw data
dataAndFiles.data = data = dataAndFiles.data.copy()
data.update(jsonData)
del data['data']
return dataAndFiles
It assumes that the JSON part is sent as a form field called 'data'.
If you want to get really slick, you could probably parse the data field according to the media type value of the Accept header. Alternatively, you could POST the data as a file and parse according to its content type.
Note that to use this parser class, you'll need to set the following default settings:
REST_FRAMEWORK = {
'FORM_METHOD_OVERRIDE': None,
'FORM_CONTENT_OVERRIDE': None,
}
That is due to this bug: https://github.com/tomchristie/django-rest-framework/issues/1346

the parameter to specify the content-type is just "type="
curl -F "profileImage=#/home/user/image.jpg" -F "{'firstName':'hello'};type=application/json" http://127.0.0.1:8000/api/v1/signup/
However, I don't think that will allow the JSONParser to take the information..., but you can try :)

Here :-
curl -vvv -X POST -H "Content-Type:multipart/form-data" -H "Accept:application/json" -H -F "username=sample" -F "password=something" -F "image=#Mercury.gif" http://127.0.0.1:8000/api/objects
No need to type accepts application/json. Django will automatically treat these as dictionary objects.
Print request.DATA and request.FILES will give you
<QueryDict: {u'username': [u'sample'] , u'password': [u'something']}>
<MultiValueDict: {u'image': [<InMemoryUploadedFile: Mercury.gif (image/gif)>]}>

Thanks xjtian - link that you pasted in comment + some researches make solution.
So:
If you want sending json data + file - send two requests. First will create object on backend and give id to app. Second will update object with file using gived id. More info there.
If you don't care about data types - send request via "multipart/form-data".
Thanks to all for answers, you realy helps, but I think this one is more appropriate for REST architecture.

POST method to update XML file

I'm trying to use the following code to update a particular tag (<a>) text on file.xml
#app.route('/model', methods = ['POST'])
def api_post():
if request.headers['Content-Type'] == 'text/xml':
f = open("/home/file.xml", "w")
f.write(request.data)
f.close()
but the test with curl is not working...
curl -H "Content-type: text/xml" -X POST http://192.168.1.12:8080/model -d "<a>hello</a>"
Could someone help since I can't find any examples of flask and XML for the POST method?

Possibilities:
You're routing /data POSTs to api_post(), but you're sending
your curl test to /model.
HTTP header names are case-insensitive, but Python dictionary keys
are not. You're posting 'Content-type' but checking 'Content-Type'.
Suggestion: Expand upon "not working..." by providing any error messages or log entries which do not make sense to you. If there are no traces of trouble, add diagnostics until there are.

Python httplib POST request and proper formatting

I'm currently working on a automated way to interface with a database website that has RESTful webservices installed. I am having issues with figure out the proper formatting of how to properly send the requests listed in the following site using python.
https://neesws.neeshub.org:9443/nees.html
Particular example is this:
POST https://neesws.neeshub.org:9443/REST/Project/731/Experiment/1706/Organization
<Organization id="167"/>
The biggest problem is that I do not know where to put the XML formatted part of the above. I want to send the above as a python HTTPS request and so far I've been trying something of the following structure.
>>>import httplib
>>>conn = httplib.HTTPSConnection("neesws.neeshub.org:9443")
>>>conn.request("POST", "/REST/Project/731/Experiment/1706/Organization")
>>>conn.send('<Organization id="167"/>')
But this appears to be completely wrong. I've never actually done python when it comes to webservices interfaces so my primary question is how exactly am I supposed to use httplib to send the POST Request, particularly the XML formatted part of it? Any help is appreciated.

You need to set some request headers before sending data. For example, content-type to 'text/xml'. Checkout the few examples,
Post-XML-Python-1
Which has this code as example:
import sys, httplib
HOST = www.example.com
API_URL = /your/api/url
def do_request(xml_location):
"""HTTP XML Post requeste"""
request = open(xml_location,"r").read()
webservice = httplib.HTTP(HOST)
webservice.putrequest("POST", API_URL)
webservice.putheader("Host", HOST)
webservice.putheader("User-Agent","Python post")
webservice.putheader("Content-type", "text/xml; charset=\"UTF-8\"")
webservice.putheader("Content-length", "%d" % len(request))
webservice.endheaders()
webservice.send(request)
statuscode, statusmessage, header = webservice.getreply()
result = webservice.getfile().read()
print statuscode, statusmessage, header
print result
do_request("myfile.xml")
Post-XML-Python-2
You may get some idea.

Should a webserver ignore the request body when request method is not POST or PUT?

I was working on a simple API server using tornado and all requests require the parameter access_token. I was playing with curl, and was surprised to find that DELETE and GET requests will not extract this value from the request body--they only allow this param to be passed via the query string.
ie, when I do
curl -i -X DELETE -d access_token=1234 http://localhost:8888/
In the delete method of my web handler, this returns None:
self.get_argument('access_token', None)
However, when I do
curl -i -X DELETE http://localhost:8888/?access_token=1234
This yields "1234" as expected:
self.get_argument('access_token', None)
I examined the tornado source, and found that the body is only parsed for POST and PUT requests: https://github.com/facebook/tornado/blob/4b346bdde80c1e677ca0e235e04654f8d64b365c/tornado/httpserver.py#L258
Is it correct to ignore the request body for GET, HEAD, and DELETE requests, or is this a choice made by the authors of tornado?

This is correct per the HTTP/1.1 protocol specification.
DELETE and GET requests do not accept entity data enclosed in the request.
According to the definition, get requests retrieve their entity data from the request URI.
HEAD requests are defined as identical to GET requests except that the server should not return a message body in the response.
Therefore the authors of tornado were correct to ignore the "post" data for GET, HEAD, and DELETE.
See HTTP/1.1 Method Definitions

It is a good idea to not to accept requests with the payload if they are not POST or PUT. Just because of security reasons. Some servers, e.g. lighttpd, return server error in this case.