I was to automate action using python requests, when encountered strange behavior.
I want to generate reports from a certain domain, to do so I have to send a POST request providing parameters in form of an XML. In devtools it looks like this:
xmlWe: <REPORTPARS><PROC_ID>11</PROC_ID>
..... and so on
when I send report data by python requests, it works perfectly whe provided a with dictionary:
data = dict(xmlWe = '<REPORTPARS><PROC_ID>11</PROC_ID>(...) '
r = s.post(URL_generate, data=data))
IMO, its kind of strange, dictionary is a type in python so why would this server handle or expect this type?
I was trying to send this XML as text or JSON, adding the corresponding header 'Content-type' but without success. The server will return 777 Java nullptr exception, which is the same as in case of sending any other nonsense datatype. So it looks like he was expecting some sort of dictionary.
My problem is, my company uses pl/sql for automation, so I will finally have to use this language. There, in http_utils data can be sent merely by write_text, so I'm restricted to sending string (can be dumped JSON).
What do You think, Is there an option for this server will accept report_pars as a string by any chance?
You need to send a xml directly:
import requests
headers = {'Content-Type': 'application/xml'}
r = requests.post(URL_generate, data=xmlWe, headers=headers)
I am trying to get an API query into python. The command line
curl -X POST https://xxxx.com/api/auth/refreshToken -d access_token
it will give output in text string.access_tokenis a hexadecimal variable that remains constant throughout. I would like to make this call from python so that I can loop through different ids and analyze the output. Any ideas?
Many thanks.
You can use requests library https://2.python-requests.org/en/master/user/quickstart/#make-a-request
Maybe something like this :
response = requests.post('https://httpbin.org/post', data = {'key':'value'})
Then you can utilise the response and process it
This is more of a python question than Splunk but would be helpful if anyone had done this... specifically here, there's a discussion of sending multiple metrics in a single POST to the server. The example they provide is using curl and wrapping the entire payload in single quotes ('), e.g.
curl -k http://<IP address or host name or load balancer name>:8088/services/collector \
-H "Authorization: Splunk 98a1e071-bc35-410b-8642-78ce7d829083"
\
-d '{"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
{"time": 1505511013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'
My question is how to do the same thing in python – i.e. you can't wrap multiple JSON objects in single quotes like in the curl command - that just makes the entire payload a string. Is there some other wrapper that can be used for this purpose?
So, this works:
payload = {"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
But this does not:
payload = {"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
{"time": 1505511013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}
FYI, then the POST looks like:
resp = requests.post(splunkurl,json=payload,headers=headers)
Well, "multiple json objects" is not a valid json, until it's a list of objects.
Generally, python doesn't care (just like any other network tool), json is just data format, and you need a different one. So you need to construct text payload yourself, i.e. json.dumps(payload1) + json.dumps(payload2), and send it via your network client as "raw" data.
I highly doubt that mainstream http libraries provide such usecase out of the box.
Not sure on reason for downvotes, i.e. requests library (which is kinda standard de-facto for high-level networking) have smart processing for payloads:
requests.post(url, data={'v1': 1, 'v2': 2}) # will encode it as form data
requests.post(url, json={'v1': 1, 'v2': 2}) # will encode as json
requests.post(url, data="{'v1': 1}{'v2': 2}") # will send as-is
Json has nothing to do with http itself, it's just a way to serialize data. Most clients will eventually use urllib, which doesn't care at all, the only question is if library gives easy way to send data raw
When sending data through python-requests a GET request, I have a need to specifically add something at the beginning of the query string. I have tried passing the data in through dicts and json strings with no luck.
The request as it appears when produced by requests:
/apply/.../explain?%7B%22......
The request as it appears when produced by their interactive API documentation (Swagger):
/apply/.../explain?record=%7B%22....
Where the key-value pairs of my data follow the excerpt above.
Ultimately, I think the missing piece is the record= that gets produced by their documentation. It is the only piece that is different from what is produced by Requests.
At the moment I've got it set up something like this:
import requests
s = requests.Session()
s.auth = requests.auth.HTTPBasicAuth(username,password)
s.verify = certificate_path
# with data below being a dictionary of the values I need to pass.
r = s.get(url,data=data)
I am trying to include an image of the documentation below, but don't yet have enough reputation to do so:
apply/model/explain documentation
'GET' requests don't have data, that's for 'POST' and friends.
You can send the query string arguments using params kwarg instead:
>>> params = {'record': '{"'}
>>> response = requests.get('http://www.example.com/explain', params=params)
>>> response.request.url
'http://www.example.com/explain?record=%7B%22'
From the comments i felt the need to explain this.
http://example.com/sth?key=value&anotherkey=anothervalue
Let's assume you have a url like the above in order to call with python requests you only have to write
response = requests.get('http://example.com/sth', params={
'key':'value',
'anotherkey':'anothervalue'
})
Have in mind that if your value or your keys have any special character in them they will be escaped thats the reason for the %7B%2 part of url in your question.
I use python requests.post function to send json queries to my django app.
r = requests.post(EXTERNAL_SERVER_ADDRESS, data={'123':'456', '456':'789'})
But on the external server request.POST object looks like this:
<QueryDict: {'123': ['456'], '456': ['789']}>
Why does it happen? How can I just send a dict?
requests is not doing anything here. Presumably your receiving server is Django; that's just how it represents data from a request. request.POST['123'] would still give '456'.
You are sending a dict, Django transform this JSON in this QueryDict objetc automatically when it receives the message. If you want to parse it to an dict, do:
myDict = dict(queryDict.iterlists())