I have an AWS Lambda that is sitting behind a API Gateway and taking POST requests from GitHub on Pull Requests. I'm trying to process the Payload so that I can make decisions based on the Pull Request status.
My issue is that I'm not able to get the dictionary key value. This is a slimmed down version of the dictionary that I'm getting in the Event variable of the Lambda (the payload from GitHub):
dict = {'version': 2.0, 'routeKey': 'POST /', 'body': '{"action": "synchronize","number": 1,"pull_request": {"url":"https://myurl.com"}'}
print(dict['body']['action'])
I need to get the value of the pull_request url. I get the following when I try to process the dictionary:
print(dict['version']) # 2.0
print(dict['body']['pull_request']['url']) #TypeError: string indices must be integers
Why is there a single quote in the dictionary that is preventing me from accessing keys and values?
How can I get the value of the pull request url from this key?
It is because dict['body'] is a string not a dictionary, you have to convert that to a dictionary before you can access.
Also it seems to be a malformed JSON string, missing a closing '}'
{"action": "synchronize","number": 1,"pull_request": {"url":"https://myurl.com"}
If you fix that you can then use json.loads
from json import loads
event = {'version': 2.0, 'routeKey': 'POST /', 'body': '{"action": "synchronize","number": 1,"pull_request": {"url":"https://myurl.com"}}'}
print(loads(event['body'])['pull_request']['url'])
This is more of a python question than Splunk but would be helpful if anyone had done this... specifically here, there's a discussion of sending multiple metrics in a single POST to the server. The example they provide is using curl and wrapping the entire payload in single quotes ('), e.g.
curl -k http://<IP address or host name or load balancer name>:8088/services/collector \
-H "Authorization: Splunk 98a1e071-bc35-410b-8642-78ce7d829083"
\
-d '{"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
{"time": 1505511013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'
My question is how to do the same thing in python – i.e. you can't wrap multiple JSON objects in single quotes like in the curl command - that just makes the entire payload a string. Is there some other wrapper that can be used for this purpose?
So, this works:
payload = {"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
But this does not:
payload = {"time": 1505501013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west- 1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":999311222774,"metric_name":"total"}}
{"time": 1505511013.000,"source":"disk","host":"host_99","fields":
{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}
FYI, then the POST looks like:
resp = requests.post(splunkurl,json=payload,headers=headers)
Well, "multiple json objects" is not a valid json, until it's a list of objects.
Generally, python doesn't care (just like any other network tool), json is just data format, and you need a different one. So you need to construct text payload yourself, i.e. json.dumps(payload1) + json.dumps(payload2), and send it via your network client as "raw" data.
I highly doubt that mainstream http libraries provide such usecase out of the box.
Not sure on reason for downvotes, i.e. requests library (which is kinda standard de-facto for high-level networking) have smart processing for payloads:
requests.post(url, data={'v1': 1, 'v2': 2}) # will encode it as form data
requests.post(url, json={'v1': 1, 'v2': 2}) # will encode as json
requests.post(url, data="{'v1': 1}{'v2': 2}") # will send as-is
Json has nothing to do with http itself, it's just a way to serialize data. Most clients will eventually use urllib, which doesn't care at all, the only question is if library gives easy way to send data raw
Inside integration tests, I try to make a POST request with a nested JSON parameter (this is just one of the approaches I tried):
test_function(admin_client):
admin_client.post(some_url,
json.dumps(some_nested_json),
content_type='application/json',
headers={'Content-type': 'application/json; charset=utf-8'})
I've also tried all sorts of different combinations for the parameters etc., everything I could find on the web, but I can't get the proper JSON output in the received request.JSON.
I get one of these three cases:
request.POST contains the first level of JSON plus the arrays of second level JSON keys
request.POST contains the first level of JSON plus an empty second level JSON
request.POST is empty, but request.body contains the whole nested JSON, in expected format
What am I missing here? I'm using Python 2.7.
This is expected behavior in django. Json requests are not in in request.POST but in request.body. You need to manually do json.loads(request.body) in your view.
When sending data through python-requests a GET request, I have a need to specifically add something at the beginning of the query string. I have tried passing the data in through dicts and json strings with no luck.
The request as it appears when produced by requests:
/apply/.../explain?%7B%22......
The request as it appears when produced by their interactive API documentation (Swagger):
/apply/.../explain?record=%7B%22....
Where the key-value pairs of my data follow the excerpt above.
Ultimately, I think the missing piece is the record= that gets produced by their documentation. It is the only piece that is different from what is produced by Requests.
At the moment I've got it set up something like this:
import requests
s = requests.Session()
s.auth = requests.auth.HTTPBasicAuth(username,password)
s.verify = certificate_path
# with data below being a dictionary of the values I need to pass.
r = s.get(url,data=data)
I am trying to include an image of the documentation below, but don't yet have enough reputation to do so:
apply/model/explain documentation
'GET' requests don't have data, that's for 'POST' and friends.
You can send the query string arguments using params kwarg instead:
>>> params = {'record': '{"'}
>>> response = requests.get('http://www.example.com/explain', params=params)
>>> response.request.url
'http://www.example.com/explain?record=%7B%22'
From the comments i felt the need to explain this.
http://example.com/sth?key=value&anotherkey=anothervalue
Let's assume you have a url like the above in order to call with python requests you only have to write
response = requests.get('http://example.com/sth', params={
'key':'value',
'anotherkey':'anothervalue'
})
Have in mind that if your value or your keys have any special character in them they will be escaped thats the reason for the %7B%2 part of url in your question.
I use python requests.post function to send json queries to my django app.
r = requests.post(EXTERNAL_SERVER_ADDRESS, data={'123':'456', '456':'789'})
But on the external server request.POST object looks like this:
<QueryDict: {'123': ['456'], '456': ['789']}>
Why does it happen? How can I just send a dict?
requests is not doing anything here. Presumably your receiving server is Django; that's just how it represents data from a request. request.POST['123'] would still give '456'.
You are sending a dict, Django transform this JSON in this QueryDict objetc automatically when it receives the message. If you want to parse it to an dict, do:
myDict = dict(queryDict.iterlists())