How can I format this JSON String into a JSON object? - python

I have the following JSON document being sent from an SNS HTTPs subscription to an API Gateway endpoint (backed by a Python2.7 Lambda function). In this case, I have a Cloudwatch Alarm that's configured to send to SNS, which then sends to this Gateway endpoint. SNS packages that alarm (which is JSON) into the "body" field of the message it sends to Gateway.
I need to extract the JSON document out of the "body" field, but by this point it's been properly mangled with escape characters and new lines and json.loads() is not liking it at all.
How can I read the value of "body" back into a JSON document using Python2.7 within a Lambda function? I've tried cleaning it by removing '\n' and '\', but I'm just striking out!
Here is the JSON as received by Lambda:
{
"body": "{\n \"Type\" : \"Notification\",\n \"MessageId\" : \"944c9xxx3-c98d636ff2c7\",\n \"TopicArn\" : \"arn:aws:sns:us-west-2:xxx6xx:sxxxr-sns-topic\",\n \"Subject\" : \"ALARM: \\\"hhh\\\" in US West (Oregon)\",\n \"Message\" : \"{\\\"AlarmName\\\":\\\"hhh\\\",\\\"AlarmDescription\\\":null,\\\"AWSAccountId\\\":\\\"8xxx\\\",\\\"NewStateValue\\\":\\\"ALARM\\\",\\\"NewStateReason\\\":\\\"Threshold Crossed: 1 out of the last 1 datapoints [0.333370380661336 (13/06/18 18:06:00)] was greater than or equal to the threshold (0.1) (minimum 1 datapoint for OK -> ALARM transition).\\\",\\\"StateChangeTime\\\":\\\"2018-06-13T18:16:56.457+0000\\\",\\\"Region\\\":\\\"US West (Oregon)\\\",\\\"OldStateValue\\\":\\\"INSUFFICIENT_DATA\\\",\\\"Trigger\\\":{\\\"MetricName\\\":\\\"CPUUtilization\\\",\\\"Namespace\\\":\\\"AWS/EC2\\\",\\\"StatisticType\\\":\\\"Statistic\\\",\\\"Statistic\\\":\\\"AVERAGE\\\",\\\"Unit\\\":null,\\\"Dimensions\\\":[{\\\"name\\\":\\\"InstanceId\\\",\\\"value\\\":\\\"i-07bxxx26\\\"}],\\\"Period\\\":300,\\\"EvaluationPeriods\\\":1,\\\"ComparisonOperator\\\":\\\"GreaterThanOrEqualToThreshold\\\",\\\"Threshold\\\":0.1,\\\"TreatMissingData\\\":\\\"\\\",\\\"EvaluateLowSampleCountPercentile\\\":\\\"\\\"}}\",\n \"Timestamp\" : \"2018-06-13T18:16:56.486Z\",\n \"SignatureVersion\" : \"1\",\n \"Signature\" : \"fFunXkjjxxxvF7Kmxxx\",\n \"SigningCertURL\" : \"https://sns.us-west-2.amazonaws.com/SimpleNotificationService-xxx.pem\",\n \"UnsubscribeURL\" : \"https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=axxxd\"\n}",
"resource": "/message",
"requestContext": {
"requestTime": "13/Jun/2018:18:16:56 +0000",
"protocol": "HTTP/1.1",
"resourceId": "m4sxxxq",
"apiId": "2v2cthhh",
"resourcePath": "/message",
"httpMethod": "POST",
"requestId": "f41e8-8cbd-57ad9e625d12",
"extendedRequestId": "xxx",
"path": "/stage/message",
"stage": "stage",
"requestTimeEpoch": 1528913816627,
"identity": {
"userArn": null,
"cognitoAuthenticationType": null,
"accessKey": null,
"caller": null,
"userAgent": "Amazon Simple Notification Service Agent",
"user": null,
"cognitoIdentityPoolId": null,
"cognitoIdentityId": null,
"cognitoAuthenticationProvider": null,
"sourceIp": "xxx",
"accountId": null
},
"accountId": "xxx"
},
"queryStringParameters": {
"id": "CBxxx69"
},
"httpMethod": "POST",
"pathParameters": null,
"headers": {
"Content-Type": "text/plain; charset=UTF-8",
"Via": "1.1 xxx.cloudfront.net (CloudFront)",
"Accept-Encoding": "gzip,deflate",
"CloudFront-Is-SmartTV-Viewer": "false",
"x-amz-sns-subscription-arn": "arn:aws:sns:us-west-2:xxx:sxxx-nxxx-sns-topic:xxx",
"CloudFront-Forwarded-Proto": "https",
"X-Forwarded-For": "54.240.xxx, 54.182.xxx",
"CloudFront-Viewer-Country": "US",
"User-Agent": "Amazon Simple Notification Service Agent",
"X-Amzn-Trace-Id": "Root=1-5b21xxx53acea6642317ed4",
"x-amz-sns-topic-arn": "arn:aws:sns:us-west-2:xxxx:sxxxier-sns-topic",
"Host": "2vxxx.execute-api.us-west-2.amazonaws.com",
"X-Forwarded-Proto": "https",
"X-Amz-Cf-Id": "xxx",
"CloudFront-Is-Tablet-Viewer": "false",
"X-Forwarded-Port": "443",
"x-amz-sns-message-type": "Notification",
"CloudFront-Is-Mobile-Viewer": "false",
"x-amz-sns-message-id": "xxx",
"CloudFront-Is-Desktop-Viewer": "true"
},
"stageVariables": null,
"path": "/message",
"isBase64Encoded": false
}

if i use your pasted sample as a raw string, it works well:
>>> j = r'''...your sample pasted here...'''
>>> data = json.loads(j)
>>> bodydata = json.loads(data['body'])
>>> bodydata['Type']
u'Notification'
seems, that what you pasted above is the repr form, printed out with Python

Related

How to request GET method in specific url using Postman?

I want to get some results using Postman.
expected result :
{
"status": "200",
"message": "success",
"results": {
"count": "6816",
"data": [
{
"timestamp": "2022-07-17T10:47:03Z",
"sender": "0x2F664960a7FBdaDA8e133DbAF1Bfc27DCCBADBfb",
"receiver": "0xA68A135Ccd37E720000fC30CFcc453b15F8040df",
"value": "0",
"status": "1",
"message": "",
"transaction_hash": "0x1a03528eecf165678d7fbcb9ceeee45a98b02f10a5ce30236ef29ee5b3747c84",
"block_number": "2472",
"contract_address": null,
"trace_address": [
"0",
"1",
"1"
],
"trace_type": "staticcall",
"sub_traces": null,
"transaction_index": "0",
"gas_used": "2715",
"gas_limit": "29039032",
"external_receiver": "0",
"input": "0x0d2020dd476f7665726e616e6365436f6e74726163740000000000000000000000000000"
},
...
]
}
}
When I request GET Method for example,
API Docs : https://docs.wemix.com/v/en/dapp-developer/api-reference/account-apis#get-internal-transactions-by-address-and-get-internal-transactions-by-transaction-hash
GET https://explorerapi.test.wemix.com/v1/accounts/0xA68A135Ccd37E720000fC30CFcc453b15F8040df/internal-transactions
api-key: 1ba5e446edf1997f67b51bf9e60b3fbba6fa1bf84301115292805d7e24f43539RESPONSE DATA
I requests GET method using 'api-key' using Postman
but, my response is,
{
"status": "401",
"message": "Unauthorized"
}
How to fix it to get proper response?
Or, is there a good way to call API with Python?
In addition to Khalil's answer - to do the same in python you could do the following:
import requests
headers = {'api-key': '1ba5e446edf1997f67b51bf9e60b3fbba6fa1bf84301115292805d7e24f43539'}
response = requests.get('https://explorerapi.test.wemix.com/v1/accounts/0xA68A135Ccd37E720000fC30CFcc453b15F8040df/internal-transactions', headers=headers)
print(response)
<Response [200]> which will get you the data that you desire.
The API Key that was provided had RESPONSE DATA at the end which was invalid as the API key.
On the Headers tab, you need to add a key api-key with the api key 1ba5e446edf1997f67b51bf9e60b3fbba6fa1bf84301115292805d7e24f43539 in the value.

JSON Payload, AWS Lambda (Python), API Gateway

I am receiving a payload via API Gateway to my Lambda function.
{
"resource": "/onfleet-webhook-production",
"path": "/onfleet-webhook-production",
"httpMethod": "POST",
"headers": {
"accept": "application/json",
"content-type": "application/json",
"Host": "fje4zhwf83.execute1-api.us-east-1.amazonaws.com",
"X-Amzn-Trace-Id": "Root=1-60884247-298cc6854b722b00668b0553",
"X-Forwarded-For": "54.84.188.171",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https",
"X-Onfleet-Signature": "539370210b482cbb4f7d728ecb33a690c89ff70dc574475000788d0fedfdd25471a45e909f42a951d8ccbb53c22ab0a548f3c7a7fa965b697ab13544c92ff08b"
},
"multiValueHeaders": {
"accept": ["application/json"],
"content-type": ["application/json"],
"Host": ["fje4zhwf83.execute-api.us-east-1.amazonaws.com"],
"X-Amzn-Trace-Id": ["Root=1-60884247-298cc6854b722b00668b0553"],
"X-Forwarded-For": ["54.84.188.171"],
"X-Forwarded-Port": ["443"],
"X-Forwarded-Proto": ["https"],
"X-Onfleet-Signature": ["539370210b482cbb4f7d728ecb33a690c89ff70dc574475000788d0fedfdd25471a45e909f42a951d8ccbb53c22ab0a548f3c7a7fa965b697ab13544c92ff08b"]
},
"queryStringParameters": null,
"multiValueQueryStringParameters": null,
"pathParameters": null,
"stageVariables": null,
"requestContext": {
"resourceId": "0xos2v",
"resourcePath": "/onfleet-webhook-production",
"httpMethod": "POST",
"extendedRequestId": "ec9LLEw0oAMFkwA=",
"requestTime": "27/Apr/2021:16:56:39 +0000",
"path": "/default/onfleet-webhook-production",
"accountId": "406135884221",
"protocol": "HTTP/1.1",
"stage": "default",
"domainPrefix": "fje4zhwf83",
"requestTimeEpoch": 1619542599421,
"requestId": "9485a44a-44f7-436c-9e98-0cb7d9034513",
"identity": {
"cognitoIdentityPoolId": null,
"accountId": null,
"cognitoIdentityId": null,
"caller": null,
"sourceIp": "54.84.188.171",
"principalOrgId": null,
"accessKey": null,
"cognitoAuthenticationType": null,
"cognitoAuthenticationProvider": null,
"userArn": null,
"userAgent": null,
"user": null
},
"domainName": "fje4zhwf83.execut1e-api.us-east-1.amazonaws.com",
"apiId": "fje4zhwf83"
},
"body": "{\"taskId\":\"f7XbmRxXvfntB086PseFXpZ9\",\"actionContext\":{\"type\":\"WORKER\",\"id\":\"C~s63JVCJTtD12J4BCY2Y4cN\"},\"triggerId\":0,\"triggerName\":\"taskStarted\",\"workerId\":\"C~s63JVCJTtD12J4BCY2Y4cN\",\"adminId\":null,\"data\":{\"task\":{\"id\":\"f7XbmRxXvfntB086PseFXpZ9\",\"timeCreated\":1619535291000,\"timeLastModified\":1619535394486,\"organization\":\"hFz5MmFqcMoMwjYmvvPbBo5U\",\"shortId\":\"4334535c\",\"trackingURL\":\"https://onf.lt/4334535c0c\",\"worker\":\"C~s63JVCJTtD12J4BCY2Y4cN\",\"merchant\":\"hFz5MmFqcMoMwjYmvvPbBo5U\",\"executor\":\"hFz5MmFqcMoMwjYmvvPbBo5U\",\"creator\":\"YG5INuyrKz4dggsW7Vo0q*Nv\",\"dependencies\":[],\"state\":2,\"completeAfter\":null,\"completeBefore\":null,\"pickupTask\":false,\"notes\":\"\",\"completionDetails\":{\"failureNotes\":\"\",\"failureReason\":\"NONE\",\"events\":[],\"actions\":[],\"time\":null,\"firstLocation\":[],\"lastLocation\":[],\"unavailableAttachments\":[]},\"feedback\":[],\"metadata\":[],\"overrides\":{},\"quantity\":0,\"serviceTime\":0,\"identity\":{\"failedScanCount\":0,\"checksum\":null},\"appearance\":{\"triangleColor\":null},\"container\":{\"type\":\"WORKER\",\"worker\":\"C~s63JVCJTtD12J4BCY2Y4cN\"},\"trackingViewed\":false,\"estimatedCompletionTime\":null,\"estimatedArrivalTime\":null,\"destination\":{\"id\":\"yIDUKM65Ck8bwNEP8z4W42Ea\",\"timeCreated\":1619535291000,\"timeLastModified\":1619535291451,\"location\":[-71.25429253636486,42.244868727926736],\"address\":{\"apartment\":\"\",\"state\":\"Massachusetts\",\"postalCode\":\"02030\",\"number\":\"35\",\"street\":\"Strawberry Hill Street\",\"city\":\"Dover\",\"country\":\"United States\"},\"notes\":\"\",\"metadata\":[],\"googlePlaceId\":null,\"warnings\":[]},\"recipients\":[{\"id\":\"5DoYN3*udPr8eUuEELGPq6yk\",\"timeCreated\":1571336170000,\"timeLastModified\":1619535291465,\"name\":\"s Test\",\"phone\":\"+16378343616\",\"notes\":\"Testing Email\",\"organization\":\"hFz5MmFqcMoMwjYmvvPbBo5U\",\"skipSMSNotifications\":false,\"metadata\":[]}]}},\"time\":1619535394615}",
"isBase64Encoded": false
}
I can access the values within this JSON;
ex . data = event.get('path')
however, i can not acess to any of the variables inside the 'body'
`event.get('body')` doesn't seem to work at all
how can I access the values within the body like 'taskId', 'timeCreated" etc..
I have tried different things like json.loads(event). but they all end up with the same outcome.
Your body is a string. You have to use ast parse it to valid python map:
import ast
body = ast.literal_eval(event['body'])
then you should be able to use body as regular map.
Alternative
import json
body = json.loads("{}".format(event['body']))

Getting a request error 422 upon get request

I have seen some posts about this but most of them used requests.post and I want a get request.
I am making a program that simply goes to a url and gets a list of orders.
This is the example response:
{
"page": 1,
"pageCount": 22,
"orderCount": 214,
"orders": [
{
"id": "c1497823-370c-4c7a-82cd-dacddb36fc30",
"productId": "1a641ba5-38df-4acb-86f7-f5c031e538a0",
"email": "demoemail#autobuy.io",
"ipAddress": "127.0.0.1",
"total": 18,
"currency": "USD",
"gateway": "Stripe",
"isComplete": true,
"createdAtUtc": "2019-10-15T01:44:10.5446599+00:00"
},
{
"id": "228f4ca4-5001-4c19-8350-f960f13d35a7",
"productId": "a0041cc0-2bc6-40a2-9084-5880bae5ecec",
"email": "demoemail#autobuy.io",
"ipAddress": "127.0.0.1",
"total": 50,
"currency": "USD",
"gateway": "Stripe",
"isComplete": true,
"createdAtUtc": "2019-10-15T01:43:17.8322919+00:00"
},
{
"id": "71aed9b2-4bd2-4a49-9e6a-82119e6e05bf",
"productId": "2e0ac75b-bfea-42f1-ad60-ead17825162a",
"email": "demoemail#autobuy.io",
"ipAddress": "127.0.0.1",
"total": 6,
"currency": "USD",
"gateway": "Stripe",
"isComplete": true,
"createdAtUtc": "2019-10-14T23:54:44.6217478+00:00"
}
]
}
This is what I get:
<Response [422]>
My code:
import requests
url = "https://autobuy.io/api/Orders?page=1"
headers = {
"Cache-Control": "no-cache",
"Pragma": "no-cache",
"APIKey": "<APIKEY>"
}
req = requests.get(url, headers=headers)
print(req)
Documentation for the api I'mtrying to use https://api.autobuy.io/?version=latest
edit: on their site it says
HEADERS
APIKey
2cdbdc48-b297-41ad-a234-329db0d2dbea
AutoBuy Api Key found in your shop settings
but then when I remove my Cache-Control and Pragma headers I get an error that has to do with my headers being cached (cause the site is behind a CDN?)
Ended up being a bug in the site, my dashboard was showing an invalid key.

Need help formatting/parsing this list

I'm not a developer so sorry if this is a dumb question or if my terminology is incorrect. I'm writing a script to make calls to our CMDB's API but i'm not sure how to handle the data that is being sent back from it. It appears to be a list type but I can't reference anything by key names. Is there a way to convert it to something that i can easily manipulate and pull data out of?
Here is my code:
import requests
import json
r=requests.post('API.URL', data={'grant_type': 'password', 'client_id':'#######', 'username': 'user', 'password': 'password'})
json_data = json.loads(r.content)
token = json_data['access_token']
data ={
"filters": [
{
"fieldId": "937905400191ae67dd03ab4b79968fcbaa264b1a75",
"operator": "eq",
"value": "hostname"
}
],
"fields":[
'9426b6ddf3cb971488517145e39efc5aa7f16fec46',
'9343f8800b3917f26533954918a6388ae8c863507f',
'9379053db492ece14816704ef5a9e3e567e217511b',
'9343f93fc4c8422bcf24e74a9a86035bb7d0248b00',
'941ba290776d6f51ce35664246927b958330a753b2'
],
"association": "Configuration Item",
"busObId": "93dada9f640056ce1dc67b4d4bb801f69104894dc8",
"includeAllFields": 'false',
"pageNumber": 0,
"pageSize": 300,
"scope": "Global",
"scopeOwner": "(None)",
"searchName": "APItest"
}
payload = json.dumps(data)
headers = {'Content-Type': 'application/json', 'Accept': 'application/json', 'Authorization':'bearer '+token}
search=requests.post('http://API.URL', headers=headers, data=payload)
search_json = json.loads(search.content)
bo = search_json['businessObjects']
print(bo)
Here's the response:
[
{
"busObRecId": "9423ad7d617390fdc956ee4302a69d0ccf1a37a4c1",
"hasError": false,
"links": [
{
"url": "http://URL",
"name": "Delete Record"
}
],
"fields": [
{
"displayName": "Business Sponsor",
"name": "Business Sponsor",
"value": "",
"html": null,
"dirty": false,
"fieldId": "9426b6ddf3cb971488517145e39efc5aa7f16fec46"
},
{
"displayName": "Owned By",
"name": "Owned By",
"value": "John Doe",
"html": null,
"dirty": false,
"fieldId": "9343f8800b3917f26533954918a6388ae8c863507f"
},
{
"displayName": "Asset Status",
"name": "Asset Status",
"value": "Active",
"html": null,
"dirty": false,
"fieldId": "9379053db492ece14816704ef5a9e3e567e217511b"
},
{
"displayName": "Description",
"name": "Description",
"value": "Automation Server",
"html": null,
"dirty": false,
"fieldId": "9343f93fc4c8422bcf24e74a9a86035bb7d0248b00"
},
{
"displayName": "Data Center Location",
"name": "Data Center Location",
"value": "",
"html": null,
"dirty": false,
"fieldId": "941ba290776d6f51ce35664246927b958330a753b2"
}
],
"errorMessage": null,
"busObPublicId": "9423ad7d617390fdc956ee4302a69d0ccf1a37a4c1",
"busObId": "93dada9f640056ce1dc67b4d4bb801f69104894dc8",
"errorCode": null
}
]
type() shows the object bo as a list and len() says it only has one element so I'm not sure how to pull data out of it without hacking away at it stripping out characters.
The reason why you cannot reference anything by key names is the fact that your output is a list. A list which is only containing one single dictionary element. If you print out
bo[0]
you get the whole data, without the [ and ] symbols. As for the dictionary now we can access different elements by keys, e.g.:
print(bo["busObId"])
will return the following value:
93dada9f640056ce1dc67b4d4bb801f69104894dc
Let's say you would want to print out the fieldId of the first element of "fields". You can do it the following way:
print(bo[0]["fields"][0]["fieldId"])
Hope this helped.

Am I doing it wrong or django rest api framework / coreapi has a bug?

I'm experiencing some troubles with coreapi and Django REST API. Perhaps someone can help me understand what is wrong.
On update and partial_update the schema returns the same fields twice. Here's an example:
"update": {
"_type": "link",
"url": "/lindex/puddle/{name}/",
"action": "put",
"encoding": "application/json",
"fields": [
{
"name": "name", <-- Here
"required": true,
"location": "path",
"schema": {
"_type": "string",
"description": "A unique value identifying this puddle.",
"title": "name"
}
},
{
"name": "name", <-- Here
"required": true,
"location": "form",
"schema": {
"_type": "string",
"description": "",
"title": "Name"
}
},
When coreapi does this call
create = client.action(schema, ['api', 'v1', 'puddleatom', 'update'],
{ "name": "1NEW",
#"names": "RAW_TR_BOS_CDR_201701-NEW",
"ingestion_flow": "ING-RAW_TR_BOS_CDR",
"status": "OK",
"location": "hdfs://hdfs-prod/user/hive/warehouse/repl_jmailer_tr.db/bos_cdr_201701",
"comment": "This atom contains data about BOS_CDR in 201701",
"records": 29452695,
"size": 2022540894,
"files": 10,
"puddle": "RAW_TR_BOS_CDR"
})
the computed URL is wrong:
http://172.17.0.2:8000/api/v1/puddleatom/{name}/
This happens because the function _get_params in /usr/local/lib/python2.7/dist-packages/coreapi/transports/http.py
allocates the parameters in the form but not in the path
key[status], value[OK]
location:form
key[comment], value[This atom contains data about BOS_CDR in 201701]
location:form
-------------------
name is processed just once into the _get_params function and the destination is the form
key[name], value[1NEW]
location:form
--------------------
key[records], value[29452695]
location:form
key[puddle], value[RAW_TR_BOS_CDR]
location:form
key[ingestion_flow], value[ING-RAW_TR_BOS_CDR]
location:form
key[files], value[10]
location:form
key[size], value[2022540894]
location:form
key[location], value[hdfs://hdfs-prod/user/hive/warehouse/repl_jmailer_tr.db/bos_cdr_201701]
location:form
What could be wrong here? The _get_params in /usr/local/lib/python2.7/dist-packages/coreapi/transports/http.py or the definition of the schema where the same fields is reported twice?

Categories