I'm developing an AWS lambda function that is triggered from an event bridge and then putting another event using python
but struggling to retrieve a value from a variable in the Json string
below is the code
import json, boto3
client = boto3.client('events')
def lambda_handler(event, context):
testV2_dict={
"K1" : event['e1'] ,
"K2" : event['e2']
}
#converting python to json as (put_event - Details) section is expecting json
testV2=json.dumps(testV2_dict)
response = client.put_events(
Entries=
[
{
"DetailType": "test",
"Source": "test",
"Detail": "{ \"testK\": \"testV\",\"testK2\": \""+ testV2 +"\" }"
}
]
)
tried to add Details on different ways,
"Detail": "{ \"testK\": \"testV\",\"testK2\": \""+ testV2 +"\" }" and still getting error as Malformated Details
and if i deleted the ++, I'm getting word testV2 itself not the value from the above
How can I retrieve the value of testV2 in the Details inside the event?
You don't have enough escaping in there. If testV2 is supposed to be a JSON string emebedded in a JSON string embedded in as JSON string, then you need more string escapes. I would let json.dumps handle that:
import json
event = {'e1': 99, 'e2': 101}
testV2_dict={
"K1" : event['e1'] ,
"K2" : event['e2']
}
testV2=json.dumps(testV2_dict)
detail = {
"testK": "testV",
"testK2": testV2
}
Entries= [
{
"DetailType": "test",
"Source": "test",
"Detail": json.dumps(detail),
}
]
print(Entries)
Output:
[{'DetailType': 'test', 'Source': 'test', 'Detail': '{"testK": "testV", "testK2": "{\\"K1\\": 99, \\"K2\\": 101}"}'}]
I don't think you need to do any thing the output of boto3 client has always been in JSON string.
Related
I am using python 3.6.3a. I would like to generate payload for each of the json records. I am using each variable to access the record. How to assign variable value (each in this case) in payload? I tried {each} and other methods but didn't work.
code snippet below.
json_records = [{"description":"<p>This is scenario1<\/p>","owner":"deb",
"priority":"high"},
{"description":"<p>This is scenario2<\/p>","owner":"deb",
"priority":"medium"}]
json_object = json.loads(json_records)
for each in json_object:
payload = """
{
"subject": "test",
"fieldValues": [
{each}
]
}
"""
There are two ways to approach this problem.
One way could be creating a dict() object and inserting keys as you wish, then json.dumps(object) to convert into string payload as in:
import json
json_records = [{"description":"This is scenario1</p>","owner":"deb","priority":"high"}
,{"description":"This is scenario2</p>","owner":"deb","priority":"medium"}]
for obj in json_records:
payload = dict()
payload['subject'] = 'test'
for key,value in obj.items():
payload['fieldName'] = {
key:value
}
print(json.dumps(payload))
#{"subject": "test", "fieldName": {"priority": "high"}}
#{"subject": "test", "fieldName": {"priority": "medium"}}
Second way is to create a textual payload from string as in, however if you need a valid JSON at the end, this would require a post-step of validation (something like try json.loads(payload) - So I'd just use the first method. I would use this method only if I have a specific requirements to generate the payload in a certain way.
import json
json_records = [{"description":"This is scenario1</p>","owner":"deb","priority":"high"}
,{"description":"This is scenario2</p>","owner":"deb","priority":"medium"}]
# json_object = json.loads(json_records) # json.loads works only on byte-like strings. your object is already in python in this case.
for obj in json_records:
payload = """
{
"subject": "test",
"fieldValues": [
%s
]
}
""" % (obj["priority"])
print(payload)
#{
# "subject": "test",
# "fieldValues": [
# high
# ]
# }
#
#
# {
# "subject": "test",
# "fieldValues": [
# medium
# ]
# }
You could make payload a Template string and use it to put the data in each JSON record into the format you want. Bracket {} characters have not special meaning in Templates, which is what makes using them easy.
Doing that will create a valid string representation of a dictionary containing everything. You can turn this into an actual Python dictionary data-structure using the ast.literal_eval() function, and then convert that into JSON string format — which I think is the final format you're after.
rom ast import literal_eval
import json
from string import Template
from textwrap import dedent
json_records = '''[{"description":"<p>This is scenario1<\/p>","owner":"deb",
"priority":"high"},
{"description":"<p>This is scenario2<\/p>","owner":"deb",
"priority":"medium"}]'''
json_object = json.loads(json_records)
payload = Template(dedent("""
{
"subject": "test",
"fieldValues": [
$each
]
}""")
)
for each in json_object:
obj = literal_eval(payload.substitute(dict(each=each)))
print(json.dumps(obj, indent=2))
Output:
{
"subject": "test",
"fieldValues": [
{
"description": "<p>This is scenario1</p>",
"owner": "deb",
"priority": "high"
}
]
}
{
"subject": "test",
"fieldValues": [
{
"description": "<p>This is scenario2</p>",
"owner": "deb",
"priority": "medium"
}
]
}
I am using the following Python function in AWs Lambda:
import json
import boto3
from boto3.dynamodb.conditions import Key, Attr
#always start with the lambda_handler
def lambda_handler(event, context):
# make the connection to dynamodb
dynamodb = boto3.resource('dynamodb')
# select the table
table = dynamodb.Table("test")
response = table.query(
KeyConditionExpression=Key('coursename').eq('intro')
)
data = response['Items']
return {'body' : data}
It outputs the following JSON - notice the "body" key? This is creating some issues when I try to use the response in my app because I have to reference the "body" as part of the response.
JSON response from Lambda
{
"body": [{
"coursename": "introto1",
"course-lesson-name": "Welcome to One! "
}, {
"coursename": "introto2",
"course-lesson-name": "What is One?"
}, {
"coursename": "introto2",
"course-lesson-name": "What Can We do with One?"
}]
}
This is the JSON format I need my Python function to output. Can this be done in AWS Lambda?
JSON format I need:
[{
"coursename": "introto1",
"course-lesson-name": "Welcome to One! "
}, {
"coursename": "introto2",
"course-lesson-name": "What is One?"
}, {
"coursename": "introto2",
"course-lesson-name": "What Can We do with One?"
}]
The response is in that format because you are explicitly wrapping it in a JSON object with a body property. Change return {'body' : data} to return data.
A simple one, but I've just not yet been able to wrap my head around parsing nested lists and json structures in Python...
Here is the raw message I am trying to parse.
{
"Records": [
{
"messageId": "1b9c0952-3fe3-4ab4-a8ae-26bd5d3445f8",
"receiptHandle": "AQEBy40IsvNDy33dOhn4KB8+7apBecWpSuw5OgL9sw/Nf+tM2esLgqmWjGsd4n0oqB",
"body": "{\n \"Type\" : \"Notification\",\n \"MessageId\" : \"dce5c301-029f-55e1-8cee-959b1ad4e500\",\n \"TopicArn\" : \"arn:aws:sns:ap-southeast-2:062497424678:vid\",\n \"Message\" : \"ChiliChallenge.mp4\",\n \"Timestamp\" : \"2020-01-16T07:51:39.807Z\",\n \"SignatureVersion\" : \"1\",\n \"Signature\" : \"oloRF7SzS8ipWQFZieXDQ==\",\n \"SigningCertURL\" : \"https://sns.ap-southeast-2.amazonaws.com/SimpleNotificationService-a.pem\",\n \"UnsubscribeURL\" : \"https://sns.ap-southeast-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:ap-southeast-2:062478:vid\"\n}",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1579161099897",
"SenderId": "AIDAIY4XD42",
"ApproximateFirstReceiveTimestamp": "1579161099945"
},
"messageAttributes": {},
"md5OfBody": "1f246d643af4ea232d6d4c91f",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:ap-southeast-2:062497424678:vid",
"awsRegion": "ap-southeast-2"
}
]
}
I am trying to extract the Message in the body section, ending up with a string as "ChiliChallenge.mp4\"
Thanks!
Essentially I just keep getting either TypeError: string indices must be integers or parsing the body but not getting any further into the list without an error.
Here's my attempt:
import json
with open ("event_testing.txt", "r") as myfile:
event=myfile.read().replace('\n', '')
str(event)
event = json.loads(event)
key = event['Records'][0]['body']
print(key)
you can use json.loads to load string
with open ("event_testing.txt", "r") as fp:
event = json.loads(fp.read())
key = json.loads(event['Records'][0]['body'])['Message']
print(key)
'ChiliChallenge.mp4'
Say your message is phrase,
I rebuild your code like:
phrase_2 = phrase["Records"]
print(phrase_2[0]["body"])
Then it works clearly. Because beginning of the Records, it looks like an array so you need to organized it.
I have a JSON file as follows :
{
"desired":{
"property1":{
"port":"/dev/usbserial",
"rx":{
"watchdoginterval":3600
},
"state":{
"path":"/Users/user1"
},
"enabled":"true",
"active":{
"enabled":"true"
}
},
"property2":{
"signal_interrupt":"USR2",
"signal_description_path":"/tmp/logger.log"
},
"property3":{
"periodmins":40
},
}
}
I am having issues trying to convert this into a string for use with AWS IoT. The function I am using is deviceShadowHandler.shadowUpdate(JSONPayload, customShadowCallback_Update, 5)
Where JSONPayload should be the JSON string.
I have tried :
with open('JSONfile.json' , 'r') as f:
dict = json.load(f)
JSONPayload = str(dict)
but I receive an "Invalid JSON file error".
An attempt to manually create a literal string from the jSON file gets messy with complaints about "EOL while scanning string literal" etc.
What is the best solution to solve this? I am new to JSON and stuff and Python.
Trailing commas are not allowed in JSON.
{
"desired":{
"property1":{
"port":"/dev/usbserial",
"rx":{
"watchdoginterval":3600
},
"state":{
"path":"/Users/user1"
},
"enabled":"true",
"active":{
"enabled":"true"
}
},
"property2":{
"signal_interrupt":"USR2",
"signal_description_path":"/tmp/logger.log"
},
"property3":{
"periodmins":40
} # <- no comma there
}
}
Why does this code give a KeyError?
output_format = """
{
"File": "{filename}",
"Success": {success},
"ErrorMessage": "{error_msg}",
"LogIdentifier": "{log_identifier}"
}
"""
print output_format.format(filename='My_file_name',
success=True,
error_msg='',
log_identifier='123')
Error message:
KeyError: ' "File"'
You need to double the outer braces; otherwise Python thinks { "File".. is a reference too:
output_format = '{{ "File": "{filename}", "Success": {success}, "ErrorMessage": "{error_msg}", "LogIdentifier": "{log_identifier}" }}'
Result:
>>> print output_format.format(filename='My_file_name',
... success=True,
... error_msg='',
... log_identifier='123')
{ "File": "My_file_name", "Success": True, "ErrorMessage": "", "LogIdentifier": "123" }
If, indicentally, you are producing JSON output, you'd be better off using the json module:
>>> import json
>>> print json.dumps({'File': 'My_file_name',
... 'Success': True,
... 'ErrorMessage': '',
... 'LogIdentifier': '123'})
{"LogIdentifier": "123", "ErrorMessage": "", "Success": true, "File": "My_file_name"}
Note the lowercase true in the output, as required by the JSON standard.
As mentioned by Tudor in a comment to another answer, the Template class was the solution that worked best for me. I'm dealing with nested dictionaries or list of dictionaries and handling those were not as straightforward.
Using Template though the solution is quite simple.
I start with a dictionary that is converted into a string. I then replace all instances of { with ${ which is the Template identifier to substitute a placeholder.
The key point of getting this to work is using the Template method safe_substitute. It will replace all valid placeholders like ${user_id} but ignore any invalid ones that are part of the dictionary structure, like ${'name': 'John', ....
After the substitution is done I remove any leftovers $ and convert the string back to a dictionary.
In the code bellow, resolve_placeholders returns a dictionary where each key matches a placeholder in the payload string and the value is substituted by the Template class.
from string import Template
.
.
.
payload = json.dumps(payload)
payload = payload.replace('{', '${')
replace_values = self.resolve_placeholders(payload)
if replace_values:
string_template = Template(payload)
payload = string_template.safe_substitute(replace_values)
payload = payload.replace('${', '{')
payload = json.loads(payload)
To extend on Martijn Pieters answer and comment:
According to MArtijn' comment, escaping the {..} pairs that are not placeholders is they way to go with nested dictionaries. I haven't succeded in doing that, so I suggest the following method.
For nested dictionaries I tried doubling up on any { and } of the nested dictionaries.
a='{{"names":{{"a":"{name}"}}}}'
a.format(name=123) output:
output: '{"names":{"a":"123"}}'
But this makes using format to change values inside a json string, a over-complex method, so I use a twist on the format command.
I replace ${param_name} in a json string. For example:
My predefined JSON looks like this:
my_json_dict = {
'parameter': [
{
'name': 'product',
'value': '${product}'
},
{
'name': 'suites',
'value': '${suites}'
},
{
'name': 'markers',
'value': '${markers}'
}
]
}
I provide this dictionary as values to replace instead of the parameters
parameters = {
'product': 'spam',
'suites': 'ham',
'markers': 'eggs'
}
And use this code to do the replacment
json_str = json.dumps(my_json_dict)
for parameter_name, parameter_value in parameters.iteritems():
parameter_name = '${'+parameter_name+'}'
json_str = json_str.replace(parameter_name, parameter_value)
json_dict = json.loads(json_str)