I was to automate action using python requests, when encountered strange behavior.
I want to generate reports from a certain domain, to do so I have to send a POST request providing parameters in form of an XML. In devtools it looks like this:
xmlWe: <REPORTPARS><PROC_ID>11</PROC_ID>
..... and so on
when I send report data by python requests, it works perfectly whe provided a with dictionary:
data = dict(xmlWe = '<REPORTPARS><PROC_ID>11</PROC_ID>(...) '
r = s.post(URL_generate, data=data))
IMO, its kind of strange, dictionary is a type in python so why would this server handle or expect this type?
I was trying to send this XML as text or JSON, adding the corresponding header 'Content-type' but without success. The server will return 777 Java nullptr exception, which is the same as in case of sending any other nonsense datatype. So it looks like he was expecting some sort of dictionary.
My problem is, my company uses pl/sql for automation, so I will finally have to use this language. There, in http_utils data can be sent merely by write_text, so I'm restricted to sending string (can be dumped JSON).
What do You think, Is there an option for this server will accept report_pars as a string by any chance?
You need to send a xml directly:
import requests
headers = {'Content-Type': 'application/xml'}
r = requests.post(URL_generate, data=xmlWe, headers=headers)
This is more of a python question than Splunk but would be helpful if anyone had done this... specifically here, there's a discussion of sending multiple metrics in a single POST to the server. The example they provide is using curl and wrapping the entire payload in single quotes ('), e.g.
curl -k http://<IP address or host name or load balancer name>:8088/services/collector \
-H "Authorization: Splunk 98a1e071-bc35-410b-8642-78ce7d829083"
\
-d '{"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
{"time": 1505511013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'
My question is how to do the same thing in python – i.e. you can't wrap multiple JSON objects in single quotes like in the curl command - that just makes the entire payload a string. Is there some other wrapper that can be used for this purpose?
So, this works:
payload = {"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
But this does not:
payload = {"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
{"time": 1505511013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}
FYI, then the POST looks like:
resp = requests.post(splunkurl,json=payload,headers=headers)
Well, "multiple json objects" is not a valid json, until it's a list of objects.
Generally, python doesn't care (just like any other network tool), json is just data format, and you need a different one. So you need to construct text payload yourself, i.e. json.dumps(payload1) + json.dumps(payload2), and send it via your network client as "raw" data.
I highly doubt that mainstream http libraries provide such usecase out of the box.
Not sure on reason for downvotes, i.e. requests library (which is kinda standard de-facto for high-level networking) have smart processing for payloads:
requests.post(url, data={'v1': 1, 'v2': 2}) # will encode it as form data
requests.post(url, json={'v1': 1, 'v2': 2}) # will encode as json
requests.post(url, data="{'v1': 1}{'v2': 2}") # will send as-is
Json has nothing to do with http itself, it's just a way to serialize data. Most clients will eventually use urllib, which doesn't care at all, the only question is if library gives easy way to send data raw
Inside integration tests, I try to make a POST request with a nested JSON parameter (this is just one of the approaches I tried):
test_function(admin_client):
admin_client.post(some_url,
json.dumps(some_nested_json),
content_type='application/json',
headers={'Content-type': 'application/json; charset=utf-8'})
I've also tried all sorts of different combinations for the parameters etc., everything I could find on the web, but I can't get the proper JSON output in the received request.JSON.
I get one of these three cases:
request.POST contains the first level of JSON plus the arrays of second level JSON keys
request.POST contains the first level of JSON plus an empty second level JSON
request.POST is empty, but request.body contains the whole nested JSON, in expected format
What am I missing here? I'm using Python 2.7.
This is expected behavior in django. Json requests are not in in request.POST but in request.body. You need to manually do json.loads(request.body) in your view.
Here is a small quote from this answer:
import requests
import json
data = {"data" : "24.3"}
data_json = json.dumps(data)
headers = {'Content-type': 'application/json'}
response = requests.post(url, data=data_json, headers=headers)
Does anyone know for sure whether it matters whether you have
data = {"data" : "24.3"}
or
data = {"data" : 24.3} ?
You are already giving a string to requests.post(), because you convert your dictionary to a JSON document with json.dumps(). It doesn't matter to requests what this string contains.
It only matters to whatever server you are sending this data; it is that server that will decode the JSON document and use your number or string.
Note that requests can do the JSON conversion for you. There is no need to use json.dumps() here, just pass your dictionary to the json keyword argumnet:
import requests
data = {"data" : "24.3"}
response = requests.post(url, json=data)
This also takes care of setting the Content-Type header to application/json.
There are two unrelated questions in your post.
The first is:
Does anyone know for sure whether it matters whether you have
data = {"data" : "24.3"}
or
data = {"data" : 24.3} ?
Yes, it does matter!
They are completely different things.
Treating them the same would make JSON format usage obsolete.
If server expects key "data" to be JSON data type number and you send it as a JSON data type string instead, a HTTP status code 400 should be responded.
If server does not report any error it means that this particular key is not being used in server and/or it is not being validated in server.
If server does treat them the same it is idiotic rather than redundant. That is why JSON format is being used in the first place.
The second is:
Does Python Requests POST need numerical data to be a string rather
than a float?
This question title is explained in Martijn Pieters's answer.
Reading the documentation for Python's requests library, the API for the basic request function has several parameters that seem redundant:
data -- (optional) Dictionary or list of tuples [(key, value)] (will be form-encoded), bytes, or file-like object to send in the body of the Request.
json -- (optional) json data to send in the body of the Request.
files -- (optional) Dictionary of 'name': file-like-objects (or {'name': file-tuple}) for multipart encoding upload. file-tuple can be a 2-tuple ('filename', fileobj), 3-tuple ('filename', fileobj, 'content_type') or a 4-tuple ('filename', fileobj, 'content_type', custom_headers), where 'content-type' is a string defining the content type of the given file and custom_headers a dict-like object containing additional headers to add for the file.
The same is the case for the rest of the interface, e.g., post, get, etc. I find this a bit confusing.
Suppose I'm trying to send files to an endpoint somewhere along with instructions on how to use them. Can I/would there ever be any reason not to just jam everything into the data parameter, rather than use files, json, etc.?
In other words, is there any difference between:
requests.post('http://some.api.endpoint',
data={'command': 'foo',
'options': {'the_thing':'gimmie', 'the_answer': 42},
'the_file': open('myfile.pdf', 'rb')})
versus
requests.post('http://some.api.endpoint',
data={'command': 'foo'},
json=json.dumps({'options': {'the_thing':'gimmie', 'the_answer': 42}}),
files={'the_file': open('myfile.pdf', 'rb')})
The most natural reading of the quoted bit of the documentation is that those two calls would be equivalent...?
Maybe I'm misreading the documentation for the data parameter, and it can't take a dictionary of files? It's a little unclear to me. Does the documentation explanation of data mean "dictionary or list [of triples, bytes, or file-like object[s]], or does it mean "[dictionary or list of tuples], bytes, or file-like object? Either is a little bit grammatically uncomfortable.
Using the files argument states "for multipart encoding upload". A multi part http response is not default when using the data argument, therefore the two are not equivalent. See What is http multipart request?
Specifically, using the files argument generates a multipart/form-data POST as opposed to application/x-www-form-urlencoded.