I was to automate action using python requests, when encountered strange behavior.
I want to generate reports from a certain domain, to do so I have to send a POST request providing parameters in form of an XML. In devtools it looks like this:
xmlWe: <REPORTPARS><PROC_ID>11</PROC_ID>
..... and so on
when I send report data by python requests, it works perfectly whe provided a with dictionary:
data = dict(xmlWe = '<REPORTPARS><PROC_ID>11</PROC_ID>(...) '
r = s.post(URL_generate, data=data))
IMO, its kind of strange, dictionary is a type in python so why would this server handle or expect this type?
I was trying to send this XML as text or JSON, adding the corresponding header 'Content-type' but without success. The server will return 777 Java nullptr exception, which is the same as in case of sending any other nonsense datatype. So it looks like he was expecting some sort of dictionary.
My problem is, my company uses pl/sql for automation, so I will finally have to use this language. There, in http_utils data can be sent merely by write_text, so I'm restricted to sending string (can be dumped JSON).
What do You think, Is there an option for this server will accept report_pars as a string by any chance?
You need to send a xml directly:
import requests
headers = {'Content-Type': 'application/xml'}
r = requests.post(URL_generate, data=xmlWe, headers=headers)
This is more of a python question than Splunk but would be helpful if anyone had done this... specifically here, there's a discussion of sending multiple metrics in a single POST to the server. The example they provide is using curl and wrapping the entire payload in single quotes ('), e.g.
curl -k http://<IP address or host name or load balancer name>:8088/services/collector \
-H "Authorization: Splunk 98a1e071-bc35-410b-8642-78ce7d829083"
\
-d '{"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
{"time": 1505511013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'
My question is how to do the same thing in python – i.e. you can't wrap multiple JSON objects in single quotes like in the curl command - that just makes the entire payload a string. Is there some other wrapper that can be used for this purpose?
So, this works:
payload = {"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
But this does not:
payload = {"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
{"time": 1505511013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}
FYI, then the POST looks like:
resp = requests.post(splunkurl,json=payload,headers=headers)
Well, "multiple json objects" is not a valid json, until it's a list of objects.
Generally, python doesn't care (just like any other network tool), json is just data format, and you need a different one. So you need to construct text payload yourself, i.e. json.dumps(payload1) + json.dumps(payload2), and send it via your network client as "raw" data.
I highly doubt that mainstream http libraries provide such usecase out of the box.
Not sure on reason for downvotes, i.e. requests library (which is kinda standard de-facto for high-level networking) have smart processing for payloads:
requests.post(url, data={'v1': 1, 'v2': 2}) # will encode it as form data
requests.post(url, json={'v1': 1, 'v2': 2}) # will encode as json
requests.post(url, data="{'v1': 1}{'v2': 2}") # will send as-is
Json has nothing to do with http itself, it's just a way to serialize data. Most clients will eventually use urllib, which doesn't care at all, the only question is if library gives easy way to send data raw
I'm trying to imitate the following cURL command to POST my url-encoded list to a form:
curl "https://website.com/update/" --data "simChangesList=%5B%7B%22simId%22%3A760590802%2C%22changeType%22%3A2%2C%22targetValue%22%3A%220003077%22%2C%22effectiveDate%22%3Anull%7D%5D" --compressed
The data I need to url-encode is the simChangesList. Essentially, this data is decoded as simChangesList: [{"simId":760590802,"changeType":2,"targetValue":000307,"effectiveDate":null}]
With the following script, I come across a Required List parameter 'simChangesList' is not present error.
Here's my script:
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
enc = urllib.quote('simChangesList: [{"simId":760590802,"changeType":2,"targetValue":000307,"effectiveDate":null}]')
auth = session.post(uri, data=enc, headers=headers)
print auth.text
However, executing the above script returns the above missing list parameter error.
How can I pass in the missing list parameter when my encoded list looks like this:
simChangesList%3A%20%5B%7B%22simId%22%3A760590802%2C%22changeType%22%3A2%2C%22targetValue%22%3A000307%2C%22effectiveDate%22%3Anull%7D%5D
Notice it's missing the '=' operator, why can I not execute my script and make it closely resemble my cURL command to post my url-encoded information?
I also encountered a similar situation landlord, but I just prompted one of the parameters inside json does not exist, is looking for a solution.
On the landlord to pass parameters, it is recommended to add a json.dumps (data)
try it.
I've just begun my journey into the realms of the CherryPy module in Python. It's pretty awesome how easy it makes it to setup a server with a RESTful api, however I've ran into an issue. My code is only acknowledging the first parameter in a query string.
I want to make a GET request like so:
curl -X GET 127.0.0.1:8080/api/sum/?a=2&b=3
My Python code is as follows:
import cherrypy
class getSum:
exposed = True
def GET(self, **params):
a = float(params['a'])
b = float(params['b'])
return a+b
if __name__ == '__main__':
cherrypy.tree.mount(getSum(), '/api/sum/', {'/': {'request.dispatch': cherrypy.dispatch.MethodDispatcher()}})
cherrypy.engine.start()
cherrypy.engine.block()
After fiddling with the code for a while I've come to the following diagnosis:
When the query string is being read into params, instead of starting a new entry in the dictionary when it reaches the '&' it just stops instead.
Has anyone got any advice on how to read multiple parameters in as a single query string?
With Thanks, Sean.
The Python script posted in my question is actually a working implementation of what I wanted to do. The problem was instead with the curl request that I was making.
The '&' character in my curl request was truncating my bash command, resulting in this: curl -X GET 127.0.0.1:8080/api/sum/?a=2
The fix for this is to wrap my url in quotes like so: curl -X GET "127.0.0.1:8080/api/sum/?a=2&b=3"
When executing this command 5.0 is returned to the terminal, as expected.
This following post contains a useful discussion on the use of ampersands in curl requests: How to include an '&' character in a bash curl statement
how do I get the whole raw http request in the python framework bottle?
I need something like this:
GET\n
myurl.com\n
/\n
attribute=value
&att2=value2
I need this to sign my http api requests
As far as I can tell from the docs you can't get the data in raw format.
What you can do is reconstruct it using bottle.request.data and bottle.request.headers. That may be enough for your purposes.
If you just want to print the request you can do the following:
headers_string = ['{}: {}'.format(h, request.headers.get(h)) for h in request.headers.keys()]
print('URL={}, method={}\nheaders:\n{}'.format(request.url, request.method, '\n'.join(headers_string)))