I have a file, and I need to send its contents to a function. What I am currently doing is as follows:
$ curl --data-urlencode "$( cat my_file.xml)" MyIP
# in function
def receiver(request):
print request.POST
This returns something like:
<QueryDict: {u'\ufeff<?xml version ': [u' "1.0" encoding = "UTF-8"?>\n<Orders>\n ...</Orders>']}>
In other words, the curl command is correctly sending the data to the django function; however, it is not easily retrievable since the POST data is split up between the request.POST key (`u'\ufeff
In addition the xml file is not properly url-encoded, so I don't think I can use curl -d #file.
It seems to be taking '\ufeff<?xml version ' as the QueryDict key. How would I pass the file contents here and retrieve it as a single string so I can parse the xml?
There is a very easy (hack) solution for this, use string concatenation on the request.POST dict to re-combine the information:
contents = request.POST.keys()[0] + "=" + request.POST.values()[0]
Related
I am trying to get value of id i.e. "id": 59 which is in the curl output in form of json.
Below is the curl output in json:
[{"id":59,"description":"This is a demo project","name":"Demo_Project","name_with_namespace":"sam / Demo_Project","path":"demo_project","path_with_namespace":"sam/demo_project","created_at":"2020-03-02T08:43:13.664Z","default_branch":"master","tag_list":[],"ssh_url_to_repo":"ssh://git#od-test.od.com:2222/sam/demo_project.git","http_url_to_repo":"https://od-test.od.com/gitlab/sam/demo_project.git","web_url":"https://od-test.od.com/gitlab/sam/demo_project","readme_url":"https://od-test.od.com/gitlab/sam/demo_project/blob/master/README.md","avatar_url":null,"star_count":0,"forks_count":0,"last_activity_at":"2020-04-09T09:28:09.860Z","namespace":{"id":2259,"name":"sam","path":"sam","kind":"user","full_path":"sam","parent_id":null,"avatar_url":"https://secure.gravatar.com/avatar/755db8ssqaq50dcc9d189c53523b?s=80\u0026d=identicon","web_url":"https://od-test.od.com/gitlab/sam"}}]
I am using python to parse the json and get the value of id.
I have tried the following command to do the same but got an error.
curl --header "PRIVATE-TOKEN: 9999ayayayar66" "https://od-test.od.com/gitlab/api/v4/search?scope=projects&search=demo_project" | python -c 'import sys, json; print(json.load(sys.stdin)["id"])'
Error:
Can anyone help me the correct python command to get the value of id.
Thanks in advance.
The JSON contains an array of objects but you are treating it like it is a single object:
import sys, json; print(json.load(sys.stdin)["id"])
You're basically saying "here is a collection of objects, give me the ID of the object". It doesn't make sense.
If you assume you only ever want the ID of the first object in the array, you can use this:
import sys, json; print(json.load(sys.stdin)[0]["id"])
The root of you JSON object is actually an array, so you should get the first item out of it first:
curl --header "PRIVATE-TOKEN: 9999ayayayar66" "https://od-test.od.com/gitlab/api/v4/search?scope=projects&search=demo_project" | python -c 'import sys, json; print(json.load(sys.stdin)[0]["id"])'
You have to write print(json.load(sys.stdin)[0]["id"]) because json response is a list of dictionaries, not a dictionary.
Since you also tagged Python Requests, your CURL command could be translated to:
import requests
headers = {
'PRIVATE-TOKEN': '9999ayayayar66',
}
params = (
('scope', 'projects'),
('search', 'demo_project'),
)
response = requests.get('https://od-test.od.com/gitlab/api/v4/search', headers=headers, params=params)
Then you can get the id value from the JSON Response Content with response.json()[0]["id"]. This uses the built-in JSON decoder to convert the JSON response to a list of dictionaries.
I have run into some trouble with the issue, that request.data sometimes is a dict (especially when testing) and sometimes a QueryDict instance (when using curl).
This is especially a problem because apparently there is a big difference when calling a view using curl like so:
curl -X POST --data "some_float=1.23456789012123123" "http://localhost:8000/myview"
Or using the django_webtest client like so:
class APIViewTest(WebTest):
def test_testsomething(self):
self.app.post(url=url, params=json.dumps({some_float=1.26356756467}))
And then casting that QueryDict to a dict like so
new_dict = dict(**request.data)
my_float = float(new_dict['some_float'])
Everything works fine in the tests, as there request.data is a dict, but in production the view crashes because new_dict['some_float'] is actually a list with one element, and not as expected a float.
I have considered fixing the issue like so:
if type(request.data) is dict:
new_dict = dict(**request.data)
else:
new_dict = dict(**request.data.dict())
which feels very wrong as the tests would only test line 2, and (some? all?) production code would run line 4.
So while I am wondering why QueryDict behaves in this way, I would rather know why and when response.data is a QueryDict in the first place. And how I can use django tests to simulate this behavior. Having different conditions for production and testing systems is always troublesome and sometimes unavoidable, but in this case I feel like it could be fixed. Or is this a specific issue related to django_webtest?
Your test isn't a reflection of your actual curl call.
In your test, you post JSON, which is then available as a dict from request.data. But your curl call posts standard form data, which is available as a QueryDict. This behaviour is managed by the parsers attribute of your view or the DEFAULT_PARSER_CLASSES settings - and further note that this is functionality specifically provided by django-rest-framework, not Django itself.
Really you should test the same thing as you are doing; either send JSON from curl or get your test to post form-data.
When your request content_type is "application/x-www-form-urlencoded", request.Data become QueryDict.
see FormParser class.
https://github.com/encode/django-rest-framework/blob/master/rest_framework/parsers.py
And
QueryDict has get lists method. but it can't fetch dict value.
convert name str to array.
<input name="items[name]" value="Example">
<input name="items[count]" value="5">
https://pypi.org/project/html-json-forms/
And define custom form paser.
class CustomFormParser(FormParser):
"""
Parser for form data.
"""
media_type = 'application/x-www-form-urlencoded'
def parse(self, stream, media_type=None, parser_context=None):
"""
Parses the incoming bytestream as a URL encoded form,
and returns the resulting QueryDict.
"""
parser_context = parser_context or {}
encoding = parser_context.get('encoding', settings.DEFAULT_CHARSET)
data = QueryDict(stream.read(), encoding=encoding)
return parse_json_form(data.dict()) # return dict
And overwite DEFAULT_PARSER_CLASSES.
https://www.django-rest-framework.org/api-guide/settings/#default_parser_classes
I am trying to convert a raw curl command to use Python Request Module with no luck. This is a simple request to query JBoss Mgmt interface but it does not parse my JSON correctly.
16:34:26,868 DEBUG [org.jboss.as.domain.http.api] (HttpManagementService-threads - 15) Unable to construct ModelNode 'Invalid character: o'
Python version
Python 2.7.6
Working raw cURL command:
/usr/bin/curl --digest -v -L -D - 'http://brenn:!12rori#localhost:9990/management' --header Content-Type:application/json '-d {"operation":"read-attribute","name":"server-state","json.pretty":1}'
In python code i read in my REST/cURL payload like so
import requests
----
def readconfigfile():
with open('jboss_modification.cfg') as f:
lines = f.readlines()
return lines
The config file looks like so
{"operation":"read-attribute","name":"server-state","json.pretty":1}
I convert the str format from readconfigfile() to a dictionary as follows
def converttodictionary(incominglines):
commands = []
for lin in incominglines:
#dumps = json.dumps(lin)
obj = json.loads(lin)
commands.append(obj)
return commands
The python code to execute this request is as follows
def applyconfig(lines):
url="http://localhost:9990/management"
auth=HTTPBasicAuth('brenn', '!12rori')
s = requests.Session()
re=s.get(url, auth=HTTPDigestAuth('brenn', '!12rori')) ##200 RESP
s.headers.update({'Content-Type': 'application/json'})
for line in lines:
payload=line
r=s.post(url,payload)
print(r.text)
Any help much appreciated?
Note: This question has been updated a few times as I resolved other issues....
The issues was...
Initial JSON requestfailed because when i read it from file python interpreted as a str.
Converted to dictionary using json.loads and server accepted request but could not parse the JSON with illegal character error
Converted this json back to a str using json.dumps -- which to my mind looks like what i was trying to do in the first place -- and this now works
read JSON file as per def readconfigfile(): above
convert to json/dictionary as per def converttodictionary above: json.loads(lin)
Convert this json "back" to a string using json.dumps and POST as follows
payload = json.dumps(command)
r = session.post(url, payload,auth=HTTPDigestAuth('brenn', '!12rori')
)
I don't know a single thing of perl but from a big perl script, I managed to get the relevant parts and make a HTTP request. So, this perl code works perfectly.
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
use HTTP::Request::Common;
my $ua = new LWP::UserAgent;
my $request = "X-CTCH-PVer: 0000001\r\n";
my $method_url = "http://localhost:8088/ctasd/GetStatus";
my $response = $ua->request (POST $method_url,Content => $request);
my $data = $response->status_line . "\n";
print $data;
print $response->content;
The above code outputs:
200 OK
X-CTCH-PVer: 0000001
From my understanding, it's doing a POST to a URL with the specified data. With that base, my python code looks like:
#!/usr/bin/python
import urllib
url = "http://localhost:8088/ctasd/GetStatus"
data = urllib.urlencode([("X-CTCH-PVer", "0000001")])
print urllib.urlopen(url, data).read()
But, this returns response as:
X-CTCH-Error: Missing protocol header "X-CTCH-PVer"
Please help me in making a Python equivalent of perl code.
So, the actual thing was, the $request in Perl was literally been sent as POST data without any change. Now I get why is the name content in Perl.
#!/usr/bin/python
import urllib
url = "http://localhost:8088/ctasd/GetStatus"
print urllib.urlopen(url, "X-CTCH-PVer: 0000001").read()
Worked. I actually found out about this after capturing the traffic in both cases and analysing it in wireshark.
The error is because you are not sending the header, you are making/sending a urlencoded string, hence the function urllib.urlencode
Try setting the request with actual headers:
#!/usr/bin/python
import urllib2
request = urllib2.Request("http://localhost:8088/ctasd/GetStatus", headers={"X-CTCH-PVer" : "0000001"})
contents = urllib2.urlopen(request).read()
Stupid question, but takes a long time to figure out. I'm trying to post an API request to parse.com using python. The json array file is meant to be read by item and transferred. The problem is that whenever I do json.load(datafile) the text format is changed to the one that is not acceptable by the server. In particular instead of having {"field":"value"} I have now {u'field:'value'} which gives a server error. It must be some simple encoding problem, just having hard time figuring out.
Many thanks!
#! /usr/bin/env python
import os
import json
APPLICATION_ID="XXXXXXXXXXXXXXXXXXXX"
REST_API_KEY="XXXXXXXXXXXXXXXXXXXXXX"
File_to_Post="../Data/area.json"
command="cat %s" % (File_to_Post)
# Parse the JSON
fileh = open(File_to_Post)
objs = json.load(fileh)
for o in objs:
command1 = 'curl -X POST \
-H "X-Parse-Application-Id: %s" \
-H "X-Parse-REST-API-Key: %s" \
-H "Content-Type: application/json" \
-d "%s" \
https://api.parse.com/1/classes/Areaobject' % (APPLICATION_ID, REST_API_KEY, o)
os.system(command1)
fileh.close()
When you load it, the JSON becomes a Python object. If you want to turn it back into JSON then you need to use json.dump() or json.dumps().