We Keep Coding

How to push the data from dynamodb through stream

Below is the json file
[
{
"year": 2013,
"title": "Rush",
"actors": [
"Daniel Bruhl",
"Chris Hemsworth",
"Olivia Wilde"
]
},
{
"year": 2013,
"title": "Prisoners",
"actors": [
"Hugh Jackman",
"Jake Gyllenhaal",
"Viola Davis"
]
}
]
Below is the code to push to dynamodb. I have created testjsonbucket bucket name, moviedataten.json is the filename and saved above json.Create a dynamodb with Primary partition key as year (Number) and
Primary sort key as title (String).
import json
from decimal import Decimal
import json
import boto3
s3 = boto3.resource('s3')
obj = s3.Object('testjsonbucket', 'moviedataten.json')
body = obj.json
#def lambda_handler(event,context):
# print (body)
def load_movies(movies, dynamodb=None):
if not dynamodb:
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Movies')
for movie in movies:
year = int(movie['year'])
title = movie['title']
print("Adding movie:", year, title)
table.put_item(Item=movie)
def lambda_handler(event, context):
movie_list = json.loads(body, parse_float=Decimal)
load_movies(movie_list)
I want to push in to ElasticSearch from dynamodb.
I have created a Elastic Domain https://xx.x.x.com/testelas
I have gone through the link https://aws.amazon.com/blogs/compute/indexing-amazon-dynamodb-content-with-amazon-elasticsearch-service-using-aws-lambda/
I clicked Managestream also
My Requirement:
Any change in Dynamodb has to reflect in the Elasticsearch?

This lambda just writing the document to DynamoDb, and I will not recommend adding the code in this lambda to push the same object to Elastic search, as lambda function should perform a single task and pushing the same document to ELK should be managed as a DynamoDB stream.
What if ELK is down or not available how you will manage this in lambda?
What if you want to disable this in future? you will need to modify lambda instead of controlling this from AWS API or AWS console, all you need to just disable the stream when required no changes on above lambda side code
What if you want to move only modify or TTL item to elastic search?
So create Dyanodb Stream that pushes the document to another Lambda that is responsible to push the document to ELK, with this option you can also push old and new both items.
You can look into this article too that describe another approach data-streaming-from-dynamodb-to-elasticsearch
For above approach look into this GitHub project dynamodb-stream-elasticsearch.
const { pushStream } = require('dynamodb-stream-elasticsearch');
const { ES_ENDPOINT, INDEX, TYPE } = process.env;
function myHandler(event, context, callback) {
console.log('Received event:', JSON.stringify(event, null, 2));
pushStream({ event, endpoint: ES_ENDPOINT, index: INDEX, type: TYPE })
.then(() => {
callback(null, `Successfully processed ${event.Records.length} records.`);
})
.catch((e) => {
callback(`Error ${e}`, null);
});
}
exports.handler = myHandler;

DynamoDB has a built in feature (DynamoDB streams) that will handle the stream part of this question.
When you configure this you have the choice of the following configurations:
KEYS_ONLY — Only the key attributes of the modified item.
NEW_IMAGE — The entire item, as it appears after it was modified.
OLD_IMAGE — The entire item, as it appeared before it was modified.
NEW_AND_OLD_IMAGES — Both the new and the old images of the item.
This will produce an event that looks like the following
{
"Records":[
{
"eventID":"1",
"eventName":"INSERT",
"eventVersion":"1.0",
"eventSource":"aws:dynamodb",
"awsRegion":"us-east-1",
"dynamodb":{
"Keys":{
"Id":{
"N":"101"
}
},
"NewImage":{
"Message":{
"S":"New item!"
},
"Id":{
"N":"101"
}
},
"SequenceNumber":"111",
"SizeBytes":26,
"StreamViewType":"NEW_AND_OLD_IMAGES"
},
"eventSourceARN":"stream-ARN"
},
{
"eventID":"2",
"eventName":"MODIFY",
"eventVersion":"1.0",
"eventSource":"aws:dynamodb",
"awsRegion":"us-east-1",
"dynamodb":{
"Keys":{
"Id":{
"N":"101"
}
},
"NewImage":{
"Message":{
"S":"This item has changed"
},
"Id":{
"N":"101"
}
},
"OldImage":{
"Message":{
"S":"New item!"
},
"Id":{
"N":"101"
}
},
"SequenceNumber":"222",
"SizeBytes":59,
"StreamViewType":"NEW_AND_OLD_IMAGES"
},
"eventSourceARN":"stream-ARN"
},
{
"eventID":"3",
"eventName":"REMOVE",
"eventVersion":"1.0",
"eventSource":"aws:dynamodb",
"awsRegion":"us-east-1",
"dynamodb":{
"Keys":{
"Id":{
"N":"101"
}
},
"OldImage":{
"Message":{
"S":"This item has changed"
},
"Id":{
"N":"101"
}
},
"SequenceNumber":"333",
"SizeBytes":38,
"StreamViewType":"NEW_AND_OLD_IMAGES"
},
"eventSourceARN":"stream-ARN"
}
]
}
As you're already familiar with Lambda it makes sense to use a Lambda function to consume the records and then iterate through them to process them in the Elasticsearch format before adding them to your index.
When doing this make sure that you iterate through each record as there may be multiple depending on your configuration.
For more information on the steps required for the Lambda side of the function check out the Tutorial: Using AWS Lambda with Amazon DynamoDB streams page.

Using the Firestore REST API to Update a Document Field

I've been searching for a pretty long time but I can't figure out how to update a field in a document using the Firestore REST API. I've looked on other questions but they haven't helped me since I'm getting a different error:
{'error': {'code': 400, 'message': 'Request contains an invalid argument.', 'status': 'INVALID_ARGUMENT', 'details': [{'#type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'oil', 'description': "Error expanding 'fields' parameter. Cannot find matching fields for path 'oil'."}]}]}}
I'm getting this error even though I know that the "oil" field exists in the document. I'm writing this in Python.
My request body (field is the field in a document and value is the value to set that field to, both strings received from user input):
{
"fields": {
field: {
"integerValue": value
}
}
}
My request (authorizationToken is from a different request, dir is also a string from user input which controls the directory):
requests.patch("https://firestore.googleapis.com/v1beta1/projects/aethia-resource-management/databases/(default)/documents/" + dir + "?updateMask.fieldPaths=" + field, data = body, headers = {"Authorization": "Bearer " + authorizationToken}).json()

Based on the the official docs (1,2, and 3), GitHub and a nice article, for the example you have provided you should use the following:
requests.patch("https://firestore.googleapis.com/v1beta1/projects{projectId}/databases/{databaseId}/documents/{document_path}?updateMask.fieldPaths=field")
Your request body should be:
{
"fields": {
"field": {
"integerValue": Value
}
}
}
Also keep in mind that if you want to update multiple fields and values you should specify each one separately.
Example:
https://firestore.googleapis.com/v1beta1/projects/{projectId}/databases/{databaseId}/documents/{document_path}?updateMask.fieldPaths=[Field1]&updateMask.fieldPaths=[Field2]
and the request body would have been:
{
"fields": {
"field": {
"integerValue": Value
},
"Field2": {
"stringValue": "Value2"
}
}
}
EDIT:
Here is a way I have tested which allows you to update some fields of a document without affecting the rest.
This sample code creates a document under collection users with 4 fields, then tries to update 3 out of 4 fields (which leaves the one not mentioned unaffected)
from google.cloud import firestore
db = firestore.Client()
#Creating a sample new Document “aturing” under collection “users”
doc_ref = db.collection(u'users').document(u'aturing')
doc_ref.set({
u'first': u'Alan',
u'middle': u'Mathison',
u'last': u'Turing',
u'born': 1912
})
#updating 3 out of 4 fields (so the last should remain unaffected)
doc_ref = db.collection(u'users').document(u'aturing')
doc_ref.update({
u'first': u'Alan',
u'middle': u'Mathison',
u'born': 2000
})
#printing the content of all docs under users
users_ref = db.collection(u'users')
docs = users_ref.stream()
for doc in docs:
print(u'{} => {}'.format(doc.id, doc.to_dict()))
EDIT: 10/12/2019
PATCH with REST API
I have reproduced your issue and it seems like you are not converting your request body to a json format properly.
You need to use json.dumps() to convert your request body to a valid json format.
A working example is the following:
import requests
import json
endpoint = "https://firestore.googleapis.com/v1/projects/[PROJECT_ID]/databases/(default)/documents/[COLLECTION]/[DOCUMENT_ID]?currentDocument.exists=true&updateMask.fieldPaths=[FIELD_1]"
body = {
"fields" : {
"[FIELD_1]" : {
"stringValue" : "random new value"
}
}
}
data = json.dumps(body)
headers = {"Authorization": "Bearer [AUTH_TOKEN]"}
print(requests.patch(endpoint, data=data, headers=headers).json())

I found the official documentation to not to be of much use since there was no example mentioned. This is the API end-point for your firestore database
PATCH https://firestore.googleapis.com/v1beta1/projects/{YOUR_PROJECT_ID}/databases/(default)/documents/{COLLECTION_NAME}/{DOCUMENT_NAME}
the following code is the body of your API request
{
"fields": {
"first_name": {
"stringValue":"Kurt"
},
"name": {
"stringValue":"Cobain"
},
"band": {
"stringValue":"Nirvana"
}
}
}
The response you should get upon successful update of the database should look like
{
"name": "projects/{YOUR_PROJECT_ID}/databases/(default)/documents/{COLLECTION_ID/{DOC_ID}",
{
"fields": {
"first_name": {
"stringValue":"Kurt"
},
"name": {
"stringValue":"Cobain"
},
"band": {
"stringValue":"Nirvana"
}
}
"createTime": "{CREATE_TIME}",
"updateTime": "{UPDATE_TIME}"
Note that performing the above action would result in a new document being created, meaning that any fields that existed previously but have NOT been mentioned in the "fields" body will be deleted. In order to preserve fields, you'll have to add
?updateMask.fieldPaths={FIELD_NAME} --> to the end of your API call (for each individual field that you want to preserve).
For example:
PATCH https://firestore.googleapis.com/v1beta1/projects/{YOUR_PROJECT_ID}/databases/(default)/documents/{COLLECTION_NAME}/{DOCUMENT_NAME}?updateMask.fieldPaths=name&updateMask.fieldPaths=band&updateMask.fieldPaths=age. --> and so on

dialogflow python SDK: How do I parse the DetectIntentResponse

I am moving from dialogflow V1 to V2.
Using the dialogflow python SDK I receive a DetectIntentResponse struct object that should have the information inside that I need.
After some time of trying to find documentation and trying to inspect this object I need your help. This object is so far out of my league ...
For documentation, thats how I get the response object:
import dialogflow_v2 as dialogflow
session_client = dialogflow.SessionsClient()
session = session_client.session_path(project_id, session_id)
text_input = dialogflow.types.TextInput(text=text, language_code=language_code)
query_input = dialogflow.types.QueryInput(text=text_input)
response = session_client.detect_intent(session=session, query_input=query_input)
How can I parse the response?
e.g. I get some parameter struct by using response.query_result.parameters But how do I get this list?
Maybe I can transform the response into json (that would make things quite easy)?
I need dicts, lists, strings ... :)

You can use MessageToJson in google.protobuf project. (Google's Protocol Buffer)
#import the function
from google.protobuf.json_format import MessageToJson
#...after getting the response = session_client.detect_intent(...)
json_response = MessageToJson(response)
You can convert various types included in the DetectIntentResponse such as QueryResult by MessageToJson(response.query_result) etc. to get a specific response.
My answer is specific to DialogFlow however I took the hint from this answer at SO which answers a much general question related to python.
Please note that I have used google.protobuf.json_format.MessageToJson specifically because the DialogFlow API V2 returns class objects defined by Google. I cannot guarantee that this will work with other Chatbot APIs (may be I need to explore it too).

The response we get after calling detect_intent() function is of type <class 'google.cloud.dialogflow_v2.types.DetectIntentResponse'>
Here is a sample response after calling detect_intent() function:
query_result {
query_text: "testing testing 123 abc#gmail.com"
action: "test"
parameters {
fields {
key: "email"
value {
string_value: "abc#gmail.com"
}
}
fields {
key: "number-integer"
value {
list_value {
values {
number_value: 123.0
}
}
}
}
}
all_required_params_present: true
fulfillment_text: "testing invoked"
fulfillment_messages {
text {
text: "testing invoked"
}
}
output_contexts {
name: "projects/*****/agent/sessions/session-test/contexts/testing-context"
lifespan_count: 5
parameters {
fields {
key: "email"
value {
string_value: "abc#gmail.com"
}
}
fields {
key: "email.original"
value {
string_value: "abc#gmail.com"
}
}
fields {
key: "number-integer"
value {
list_value {
values {
number_value: 123.0
}
}
}
}
fields {
key: "number-integer.original"
value {
string_value: "123"
}
}
}
}
intent {
name: "projects/*****/agent/intents/*****"
display_name: "test"
}
intent_detection_confidence: 1.0
language_code: "en"
}
You can easily parse values using below code:
Query Result Type : type(response) --> <class 'google.cloud.dialogflow_v2.types.DetectIntentResponse'>
Query text : response.query_result.query_text --> testing testing 123 abc#gmail.com
Detected intent : response.query_result.intent.display_name --> test
Fulfillment text : response.query_result.fulfillment_text --> testing invoked
Parameters : response.query_result.parameters --> fields {
key: "email"
value {
string_value: "abc#gmail.com"
}
}
fields {
key: "number-integer"
value {
list_value {
values {
number_value: 123.0
}
}
}
}
Hope it helps.

I solved my problem using the property ".pb". Here is my code:
...
json_response = MessageToJson(response._pb)
...

how to change schema type for single document value

I want to change the default type from dict to string for a particular user.
DOMAIN = {
'item': {
'schema': {
'profile':{
'type': 'dict'
},
'username': {
'type': 'string'
}
}
}
}
suppose if I get a request from x user type should not change. If I get a request from y user type should change from dict to string. How to change for a particular item resource without affecting others.
TIA.

Your best approach would probably be to set up two different API endpoints, one for users of type X, and another for users of type Y. Both endpoints would consume the same underlying datasource (same DB collection being updated). You achieve that by setting the datasource for your endpoint, like so:
itemx = {
'url': 'endpoint_1',
'datasource': {
'source': 'people', # actual DB collection consumed by the endpoint
'filter': {'usertype': 'x'} # optional
'projection': {'username': 1} # optional
},
'schema': {...} # here you set username to dict, or string
}
Rinse and repeat for the second endpoint. See the docs for more info.

Adding session attributes in Python for Alexa skills

I have 3 slots (account, dollar_value, recipient_first) within my intent schema for an Alexa skill and I want to save whatever slots are provided by the speaker in the session Attributes.
I am using the following methods to set session attributes:
def create_dollar_value_attribute(dollar_value):
return {"dollar_value": dollar_value}
def create_account_attribute(account):
return {"account": account}
def create_recipient_first_attribute(recipient_first):
return {"recipient_first": recipient_first}
However, as you may guess, if I want to save more than one slot as data in sessionAttributes, the sessionAttributes is overwritten as in the following case:
session_attributes = {}
if session.get('attributes', {}) and "recipient_first" not in session.get('attributes', {}):
recipient_first = intent['slots']['recipient_first']['value']
session_attributes = create_recipient_first_attribute(recipient_first)
if session.get('attributes', {}) and "dollar_value" not in session.get('attributes', {}):
dollar_value = intent['slots']['dollar_value']['value']
session_attributes = create_dollar_value_attribute(dollar_value)
The JSON response from my lambda function for a speech input in which two slots (dollar_value and recipient_first) were provided is as follows (my guess is that the create_dollar_value_attribute method in the second if statement is overwriting the first):
{
"version": "1.0",
"response": {
"outputSpeech": {
"type": "PlainText",
"text": "Some text output"
},
"card": {
"content": "SessionSpeechlet - Some text output",
"title": "SessionSpeechlet - Send Money",
"type": "Simple"
},
"reprompt": {
"outputSpeech": {
"type": "PlainText"
}
},
"shouldEndSession": false
},
"sessionAttributes": {
"dollar_value": "30"
}
}
The correct response for sessionAttributes should be:
"sessionAttributes": {
"dollar_value": "30",
"recipient_first": "Some Name"
},
How do I create this response? Is there a better way to add values to sessionAttributes in the JSON response?

The easiest way to add sessionAttributes with Python in my opinion seems to be by using a dictionary. For example, if you want to store some of the slots for future in the session attributes:
session['attributes']['slotKey'] = intent['slots']['slotKey']['value']
Next, you can just pass it on to the build response method:
buildResponse(session['attributes'], buildSpeechletResponse(title, output, reprompt, should_end_session))
The implementation in this case:
def buildSpeechletResponse(title, output, reprompt_text, should_end_session):
return {
'outputSpeech': {
'type': 'PlainText',
'text': output
},
'card': {
'type': 'Simple',
'title': "SessionSpeechlet - " + title,
'content': "SessionSpeechlet - " + output
},
'reprompt': {
'outputSpeech': {
'type': 'PlainText',
'text': reprompt_text
}
},
'shouldEndSession': should_end_session
}
def buildResponse(session_attributes, speechlet_response):
return {
'version': '1.0',
'sessionAttributes': session_attributes,
'response': speechlet_response
}
This creates the sessionAttributes in the recommended way in the Lambda response JSON.
Also just adding a new sessionAttribute doesn't overwrite the last one if it doesn't exist. It will just create a new key-value pair.
Do note, that this may work well in the service simulator but may return a key attribute error when testing on an actual Amazon Echo. According to this post,
On Service Simulator, sessions starts with Session:{ ... Attributes:{}, ... }
When sessions start on the Echo, Session does not have an Attributes key at all.
The way I worked around this was to just manually create it in the lambda handler whenever a new session is created:
if event['session']['new']:
event['session']['attributes'] = {}
onSessionStarted( {'requestId': event['request']['requestId'] }, event['session'])
if event['request']['type'] == 'IntentRequest':
return onIntent(event['request'], event['session'])

First, you have to define the session_attributes.
session_attributes = {}
Then instead of using
session_attributes = create_recipient_first_attribute(recipient_first)
You should use
session_attributes.update(create_recipient_first_attribute(recipient_first)).
The problem you are facing is because you are reassigning the session_attributes. Instead of this, you should just update the session_attributes.
So your final code will become:
session_attributes = {}
if session.get('attributes', {}) and "recipient_first" not in session.get('attributes', {}):
recipient_first = intent['slots']['recipient_first']['value']
session_attributes.update(create_recipient_first_attribute(recipient_first))
if session.get('attributes', {}) and "dollar_value" not in session.get('attributes', {}):
dollar_value = intent['slots']['dollar_value']['value']
session_attributes.update(create_dollar_value_attribute(dollar_value))

The ASK SDK for Python provides an attribute manager, to manage request/session/persistence level attributes in the skill. You can look at the color picker sample, to see how to use these attributes in skill development.

Take a look at the below:
account = intent['slots']['account']['value']
dollar_value = intent['slots']['dollar_value']['value']
recipient_first = intent['slots']['recipient_first']['value']
# put your data in a dictionary
attributes = {
'account':account,
'dollar_value':dollar_value,
'recipient_first':recipient_first
}
Put the attributes dictionary in 'sessionAttributes' in your response. You should get it back in 'sessionAttributes' once Alexa replies to you.
Hope this helps.

The following code snippet will also prevent overwriting the session attributes:
session_attributes = session.get('attributes', {})
if "recipient_first" not in session_attributes:
session_attributes['recipient_first'] = intent['slots']['recipient_first']['value']
if "dollar_value" not in session_attributes:
session_attributes['dollar_value'] = = intent['slots']['dollar_value']['value']