We Keep Coding

How to parse nested JSON in python

I'm struggling to access some values in this nested json in python.
How can I access this ['Records'][0]['s3']['bucket']['name'] ? I did search a lot to find a simple python snippet, but no luck. Thanks in advance!
{
"Records": [
{
"eventName": "xxxxxxx",
"userIdentity": {
"principalId": "AWS:XXXXXXXXXXXXXX"
},
"requestParameters": {
"sourceIPAddress": "XX.XX.XX.XX"
},
"responseElements": {
"x-amz-request-id": "8CXXXXXXXXXXHRQX",
"x-amz-id-2": "doZ3+gxxxxxxx"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "X-Event",
"bucket": {
"name": "bucket-name",
"ownerIdentity": {
"principalId": "xxxxxxx"
},
"arn": "arn:aws:s3:::bucket-name"
},
"object": {
"key": "object.png",
"sequencer": "0060XXXXXXX75X"
}
}
}
]
}

Since this is a string, use the json.loads method from the inbuilt JSON library.
import json
json_string = # your json string
parsed_string = json.loads(json_string)
print(parsed_string) # it will be a python dict
print(parsed_string['Records'][0]['s3']['bucket']['name']) # prints the string

Have you tried running your example? If you're loading the json from elsewhere, you'd need to convert it to this native dictionary object using the json library (as mentioned by others, json.loads(data))
kv = {
"Records": [
{
"eventName": "xxxxxxx",
"userIdentity": {
"principalId": "AWS:XXXXXXXXXXXXXX"
},
"requestParameters": {
"sourceIPAddress": "XX.XX.XX.XX"
},
"responseElements": {
"x-amz-request-id": "8CXXXXXXXXXXHRQX",
"x-amz-id-2": "doZ3+gxxxxxxx"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "X-Event",
"bucket": {
"name": "bucket-name",
"ownerIdentity": {
"principalId": "xxxxxxx"
},
"arn": "arn:aws:s3:::bucket-name"
},
"object": {
"key": "object.png",
"sequencer": "0060XXXXXXX75X"
}
}
}
]
}
print("RESULT:",kv['Records'][0]['s3']['bucket']['name'])
RESULT: bucket-name

Filter MongoDB query to find documents only if a field in a list of objects is not empty

I have a MongoDB document structure like following:
Structure
{
"stores": [
{
"items": [
{
"feedback": [],
"item_category": "101",
"item_id": "10"
},
{
"feedback": [],
"item_category": "101",
"item_id": "11"
}
]
},
{
"items": [
{
"feedback": [],
"item_category": "101",
"item_id": "10"
},
{
"feedback": ["A feedback"],
"item_category": "101",
"item_id": "11"
},
{
"feedback": [],
"item_category": "101",
"item_id": "12"
},
{
"feedback": [],
"item_category": "102",
"item_id": "13"
},
{
"feedback": [],
"item_category": "102",
"item_id": "14"
}
],
"store_id": 500
}
]
}
This is a single document in a collection. Some field are deleted to produce minimal representation of the data.
What I want is to get items only if the feedback field in the items array is not empty. The expected result is:
Expected result
{
"stores": [
{
"items": [
{
"feedback": ["A feedback"],
"item_category": "101",
"item_id": "11"
}
],
"store_id": 500
}
]
}
This is what I tried based on examples in this, which I think pretty same situation, but it didn't work. What's wrong with my query, isn't it the same situation in zipcode search example in the link? It returns everything like in the first JSON code, Structure:
What I tried
query = {
'date': {'$gte': since, '$lte': until},
'stores.items': {"$elemMatch": {"feedback": {"$ne": []}}}
}
Thanks.

Please try this :
db.yourCollectionName.aggregate([
{ $match: { 'date': { '$gte': since, '$lte': until }, 'stores.items': { "$elemMatch": { "feedback": { "$ne": [] } } } } },
{ $unwind: '$stores' },
{ $match: { 'stores.items': { "$elemMatch": { "feedback": { "$ne": [] } } } } },
{ $unwind: '$stores.items' },
{ $match: { 'stores.items.feedback': { "$ne": [] } } },
{ $group: { _id: { _id: '$_id', store_id: '$stores.store_id' }, items: { $push: '$stores.items' } } },
{ $project: { _id: '$_id._id', store_id: '$_id.store_id', items: 1 } },
{ $group: { _id: '$_id', stores: { $push: '$$ROOT' } } },
{ $project: { 'stores._id': 0 } }
])
We've all these stages as you need to operate on an array of arrays, this query is written assuming you're dealing with a large set of data, Since you're filtering on dates just in case if your documents size is way less after first $match then you can avoid following $match stage which is in between two $unwind's.
Ref 's :
$match,
$unwind,
$project,
$group

This aggregate query gets the needed result (using the provided sample document and run from the mongo shell):
db.stores.aggregate( [
{ $unwind: "$stores" },
{ $unwind: "$stores.items" },
{ $addFields: { feedbackExists: { $gt: [ { $size: "$stores.items.feedback" }, 0 ] } } },
{ $match: { feedbackExists: true } },
{ $project: { _id: 0, feedbackExists: 0 } }
] )

Google assistant sdk [CUSTOM TYPES ]: capture monetary values ($10, $20. $50) from a custom action

I want to use a custom type to capture $10, $5, $20, etc values in my custom action because there's no a Schema.org-defined type for this kind of data. I add this to the json file, but it doesn't work.
This is running on raspbian.
Using google assistant SDK 1.0.1 version.
The action is made for Spanish language.
{
"locale": "es",
"manifest": {
"displayName": "Imprimir",
"invocationName": "Imprimir",
"category": "PRODUCTIVITY"
},
"actions": [
{
"name": "com.example.actions.Imprimir",
"availability": {
"deviceClasses": [
{
"assistantSdkDevice": {}
}
]
},
"intent": {
"name": "com.example.intents.Imprimir",
"parameters": [
{
"name": "cantidad",
"type": "SchemaOrg_Number"
},
{
"name": "valor",
"type": "$Valor"
}
],
"trigger": {
"queryPatterns": [
"imprimir $SchemaOrg_Number:cantidad tickets de $Valor:valor"
]
}
},
"fulfillment": {
"staticFulfillment": {
"templatedResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "De acuerdo"
}
},
{
"deviceExecution": {
"command": "com.example.commands.Imprimir",
"params": {
"cantidad": "$cantidad",
"valor": "$valor"
}
}
}
]
}
}
}
}
],
"types": [
{
"name": "$Valor",
"entities": [
{
"key": "1$",
"synonyms": [
"1 dolar"
]
},
{
"key": "2$",
"synonyms": [
"2 dolares"
]
},
{
"key": "5$",
"synonyms": [
"5 dolares"
]
},
{
"key": "10$",
"synonyms": [
"10 dolares"
]
},
{
"key": "20$",
"synonyms": [
"20 dolares"
]
}
]
}
]
}
It doesn't show me any apparent error. This is what happend when I talk the order:
ON_END_OF_UTTERANCE
ON_END_OF_UTTERANCE
ON_RECOGNIZING_SPEECH_FINISHED:
{"text": "imprimir dos ticket de $10"}
ON_RESPONDING_STARTED:
{"is_error_response": false}
ON_RESPONDING_FINISHED
ON_CONVERSATION_TURN_FINISHED:
{"with_follow_on_turn": false}
The assistant tell me that she doesn't understant.
I'm working in a spanish gaction.
I know that my custom type is not working because the event is not captured. I'm sure the json file doesn't have errors because if I ignore the "Valor" type, replace it for $SchemaOrg_Number and omit word "dolar" the gaction works well.
I already tried with $SchemaOrg_priceCurrency, but it capture the type of currency like dollar, sol, yen, euro, etc...

Your synonyms list should include each possible combination. If your query has "$10" as the string, that needs to match as a synonym or the query would not match at all.

Not able to extract my friend images from Facebook API using python

I have got this response from the Facebook Graph API:
{
"taggable_friends": {
"data": [
{
"name": "Friend1 Name",
"picture": {
"data": {
"url": "https://fb-s-c-a.akamaihd.net/h-ak-fbx/v/t1.0-1/p200x200/completeUrl1"
}
},
"id": "response1d"
},
{
"name": "Friend2 name",
"picture": {
"data": {
"url": "https://fb-s-a-a.akamaihd.net/h-ak-fbx/v/t1.0-1/p200x200/completeURL2"
}
},
"id": "responseid2"
}
],
"paging": {
"cursors": {
"before": "xyz",
"after": "abc"
},
"next": "NextpageURl"
}
},
"id": "xxxxxxxxx"
}
I am willing to extract the URL part of the graph API response with field taggable_friends.
I have tried something like this:
for friends in data_json_liked_pages['taggable_friends']['data']:
friend_url = friends['picture']['data']['url']
print friend_url
I am getting the following error:
Exception Type: TypeError
Exception Value: list indices must be integers, not str
What can I do to improve this?

API.ai Actions on Google API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: ": Cannot find field."

I am using python to create webhook for Assistat app. I am able to ask user for location permission, but as soon as user gives consent, I receive following error
UnparseableJsonResponse
API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: ": Cannot find field.".
I have checked my webhook server and no request comes to it. This looks like some issue at API.ai side. Below is the Debug response from Actions console when using Python client
{
"assistantToAgentDebug": {
"curlCommand": "curl -v '<URL>'{\"user\":{\"userId\":\"<USED_ID>\",\"locale\":\"en-US\"},\"conversation\":{\"conversationId\":\"1504592665563\",\"type\":\"ACTIVE\",\"conversationToken\":\"[\\\"defaultwelcomeintent-followup\\\"]\"},\"inputs\":[{\"intent\":\"actions.intent.PERMISSION\",\"rawInputs\":[{\"inputType\":\"VOICE\",\"query\":\"yes\"}],\"arguments\":[{\"name\":\"PERMISSION\",\"textValue\":\"true\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"}]},\"device\":{\"location\":{\"coordinates\":{\"latitude\":37.4219806,\"longitude\":-122.0841979}}},\"isInSandbox\":true}'",
"assistantToAgentJson": {
"user": {
"userId": "<USED_ID>",
"locale": "en-US"
},
"conversation": {
"conversationId": "1504592665563",
"type": "ACTIVE",
"conversationToken": "[\"defaultwelcomeintent-followup\"]"
},
"inputs": [
{
"intent": "actions.intent.PERMISSION",
"rawInputs": [
{
"inputType": "VOICE",
"query": "yes"
}
],
"arguments": [
{
"name": "PERMISSION",
"textValue": "true"
}
]
}
],
"surface": {
"capabilities": [
{
"name": "actions.capability.AUDIO_OUTPUT"
},
{
"name": "actions.capability.SCREEN_OUTPUT"
}
]
},
"device": {
"location": {
"coordinates": {
"latitude": 37.4219806,
"longitude": -122.0841979
}
}
},
"isInSandbox": true
}
},
"agentToAssistantDebug": {
"agentToAssistantJson": {
"message": "Unexpected apiai response format: Empty speech response",
"apiResponse": {
"id": "<ID>",
"timestamp": "2017-09-05T06:24:41.711Z",
"lang": "en",
"result": {},
"status": {
"code": 200,
"errorType": "success"
},
"sessionId": "1504592665563"
}
}
},
"sharedDebugInfo": [
{
"name": "GOOGLE_SYSTEM_ACTION",
"debugInfo": "Your query was handled by Actions on Google."
},
{
"name": "GOOGLE_SYSTEM_ACTION",
"debugInfo": "Your query was handled by Actions on Google."
},
{
"name": "ResponseValidation",
"subDebugEntry": [
{
"name": "UnparseableJsonResponse",
"debugInfo": "API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: \": Cannot find field.\"."
}
]
}
]
}
Using Python library Flask-Assistant
How can I resolve this issue?
UPDATE
Node JS client works... what is the issue with Python client?
Action Console Debug response
{
"assistantToAgentDebug": {
"curlCommand": "curl -v '<URL>'{\"user\":{\"userId\":\"<USER_ID>\",\"locale\":\"en-US\"},\"conversation\":{\"conversationId\":\"<ID>\",\"type\":\"ACTIVE\",\"conversationToken\":\"[\\\"_actions_on_google_\\\",\\\"defaultwelcomeintent-followup\\\"]\"},\"inputs\":[{\"intent\":\"actions.intent.PERMISSION\",\"rawInputs\":[{\"inputType\":\"VOICE\",\"query\":\"yes\"}],\"arguments\":[{\"name\":\"PERMISSION\",\"textValue\":\"true\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]},\"device\":{\"location\":{\"coordinates\":{\"latitude\":37.4219806,\"longitude\":-122.0841979},\"formattedAddress\":\"Googleplex, Mountain View, CA 94043, United States\",\"zipCode\":\"94043\",\"city\":\"Mountain View\"}},\"isInSandbox\":true}'",
"assistantToAgentJson": {
"user": {
"userId": "<USER_ID>",
"locale": "en-US"
},
"conversation": {
"conversationId": "<ID>",
"type": "ACTIVE",
"conversationToken": "[\"_actions_on_google_\",\"defaultwelcomeintent-followup\"]"
},
"inputs": [
{
"intent": "actions.intent.PERMISSION",
"rawInputs": [
{
"inputType": "VOICE",
"query": "yes"
}
],
"arguments": [
{
"name": "PERMISSION",
"textValue": "true"
}
]
}
],
"surface": {
"capabilities": [
{
"name": "actions.capability.AUDIO_OUTPUT"
}
]
},
"device": {
"location": {
"coordinates": {
"latitude": 37.4219806,
"longitude": -122.0841979
},
"formattedAddress": "Googleplex, Mountain View, CA 94043, United States",
"zipCode": "94043",
"city": "Mountain View"
}
},
"isInSandbox": true
}
},
"agentToAssistantDebug": {
"agentToAssistantJson": {
"conversationToken": "[\"_actions_on_google_\",\"defaultwelcomeintent-followup\"]",
"expectUserResponse": true,
"expectedInputs": [
{
"inputPrompt": {
"richInitialPrompt": {
"items": [
{
"simpleResponse": {
"textToSpeech": "Given permission"
}
}
]
}
},
"possibleIntents": [
{
"intent": "assistant.intent.action.TEXT"
}
]
}
],
"responseMetadata": {
"status": {
"code": 14
},
"queryMatchInfo": {
"queryMatched": true,
"intent": "Default Welcome Intent - fallback"
}
}
}
}
}
Request from Actions server to my Node JS webhook server
{ originalRequest:
{ source: 'google',
version: '2',
data:
{ isInSandbox: true,
surface: [Object],
inputs: [Array],
user: [Object],
device: [Object],
conversation: [Object] } },
id: '<ID>',
timestamp: '2017-09-06T05:43:21.342Z',
lang: 'en',
result:
{ source: 'agent',
resolvedQuery: 'actions_intent_PERMISSION',
speech: '',
action: 'DefaultWelcomeIntent.DefaultWelcomeIntent-fallback',
actionIncomplete: false,
parameters: {},
contexts: [ [Object], [Object], [Object], [Object], [Object] ],
metadata:
{ intentId: '<ID>',
webhookUsed: 'true',
webhookForSlotFillingUsed: 'false',
nluResponseTime: 2,
intentName: 'Default Welcome Intent - fallback' },
fulfillment: { speech: 'Given permission', messages: [Array] },
score: 1 },
status: { code: 200, errorType: 'success' },
sessionId: '<SID>'
}
API.ai Intent settings

The most likely reason you're not getting any hits on your webhook is that you don't have an intent registered to get the reply.
You can do this by creating an Intent with the Event set to actions_intent_PERMISSION.
See also the following answers on SO:
Unable to accept the permission prompt on Actions on Google
Permission response not handled correctly