Trying to append a new notification to a bucket. Couldn't find any example in the internet.
I need to have ObjectCreated events sent to SQS. I need to decide which queue the event is sent by the prefix. Thus, each notification would have a diferent queue and prefix over the same bucket.
The problem is that I cannot append new notification. I just override the previous notification configured in the bucket.
This is the code I have so far:
bucket_notifications_configuration = {
'QueueConfigurations': [{
'Events': ['s3:ObjectCreated:*'],
'Id': f"Notif_{queue_name}",
'QueueArn': queue.attributes['QueueArn'] ,
"Filter": {
"Key": {
"FilterRules": [
{
"Name": "suffix",
"Value": f"{prefix}"
}
]
}
}
}]
}
qpolicy = {
"Version": "2012-10-17",
"Id": f"{queue_arn}/SQSDefaultPolicy",
"Statement": [{
"Sid": f"allow bucket {bucket} to notify queue {queue_name}",
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": "SQS:SendMessage",
"Resource": queue_arn,
"Condition": {
"ArnLike": {
"aws:SourceArn": f"arn:aws:s3:*:*:{bucket}"
}
}
}]
}
queue_attrs = queue.attributes
queue_attrs = {"Policy": json.dumps(qpolicy), }
queue.set_attributes(Attributes=queue_attrs)
logger.debug(f"queue ready with attributes: {queue.attributes}")
previous_notif = client.get_bucket_notification_configuration(Bucket=bucket)
previous_notif.pop('ResponseMetadata')
try:
print("apendado")
previous_notif['QueueConfigurations'].append(bucket_notifications_configuration['QueueConfigurations'][0])
print(f"apendado {previous_notif} ")
except KeyError:
previous_notif['QueueConfigurations'] = bucket_notifications_configuration['QueueConfigurations'][0]
print("cread")
client.put_bucket_notification_configuration(
Bucket=bucket,
NotificationConfiguration=bucket_notifications_configuration)
I make sure the notification id is diferent from any other, also I make sure the prefix is difefent.
This code overrides previous notification with the new one, instead of appending the new one.
I managed to get it working with the code below.
It takes the existing configurations, adds a new configuration, then saves it back to the bucket. The code assumes that there is an existing configuration.
import boto3
s3_client = boto3.client('s3', region_name = 'ap-southeast-2')
queue_name = 'queue2'
queue_arn = 'arn:aws:sqs:ap-southeast-2:123456789012:queue2'
bucket = 'my-bucket'
prefix = 'folder2/'
# Get the current notification configurations
response = s3_client.get_bucket_notification_configuration(Bucket=bucket)
configurations = response['QueueConfigurations']
# New configuration to add
new_configuration = {
'Id': f"Notif_{queue_name}",
'QueueArn': queue_arn,
'Events': [
's3:ObjectCreated:*',
],
'Filter': {
'Key': {
'FilterRules': [
{
'Name': 'prefix',
'Value': prefix
},
]
}
}
}
configurations.append(new_configuration)
# Save combined configurations
response = s3_client.put_bucket_notification_configuration(
Bucket = bucket,
NotificationConfiguration = {'QueueConfigurations' : configurations}
)
Related
When executing a select query against an athena table via boto3, the response object given is in the syntax:
{
"UpdateCount":0,
"ResultSet":{
"Rows":[
{
"Data":[
{
"VarCharValue":"site_name"
},
{
"VarCharValue":"volume_out_capacity"
},
{
"VarCharValue":"region"
},
{
"VarCharValue":"site_ref"
}
]
},
{
"Data":[
{
"VarCharValue":"ASSET 12"
},
{
"VarCharValue":"10"
},
{
"VarCharValue":"NORTH"
},
{
"VarCharValue":"RHW007777000138"
}
]
}
]
}
Is there an additional argument that can be passed so that the response object will contain columns that do not contain values? Something like:
{
"VarCharValue":"xyz"
}
]
},
{
"Data":[
{
"VarCharValue":None
}
I have looked through the documentation extensively but cannot find arguments that can describe how to format the response in get_query_results() or start_query_execution()
I do not see a option to get that data back directly from Athena. Alternatively if you use S3 Select instead of Athena, you'll get back a json object with all the columns whether they have data or are empty.
Sample Data:
name,number,city,state
coin,1,driggs,
michelle,,chicago,
shaniqua,2,,
marcos,3,stlouis,
S3 Select Result:
{"name":"coin","number":"1","city":"driggs","state":""}
{"name":"michelle","number":"","city":"chicago","state":""}
{"name":"shaniqua","number":"2","city":"","state":""}
{"name":"marcos","number":"3","city":"stlouis","state":""}
Code:
import boto3
session = boto3.session.Session(profile_name="<my-profile>")
s3 = session.client('s3')
resp = s3.select_object_content(
Bucket='<my-bucket>',
Key='<my-file>',
ExpressionType='SQL',
Expression="SELECT * FROM s3object",
InputSerialization={'CSV': {"FileHeaderInfo": "Use", "RecordDelimiter": '\r\n'}, 'CompressionType': 'NONE'},
OutputSerialization={'JSON': {}},
)
for event in resp['Payload']:
if 'Records' in event:
records = event['Records']['Payload'].decode('utf-8')
print(records)
elif 'Stats' in event:
statsDetails = event['Stats']['Details']
print("Stats details bytesScanned: ")
print(statsDetails['BytesScanned'])
print("Stats details bytesProcessed: ")
print(statsDetails['BytesProcessed'])
print("Stats details bytesReturned: ")
print(statsDetails['BytesReturned'])
I am trying to get my reply to the Google assistant as a string to search it through a Google spreadsheet and return additional information about it. Specifically, I have a Google spreadsheet with two columns, one being for the name of a thing, and another for my rating of it. I'm using ngrok and Flask to get my script's response online. The two parts I'm struggling with are...
(1) Getting my requested thing into the program.
(2) Switching the url of the webhook and/or changing the webhook so that the system can output the final result.
I am a beginner so I'm probably misusing a few terms here but anyways my code is below. I put a bunch of underscores in the places where I thought sensitive information was.
action.json >>
"actions": [
{
"description": "Welcome Intent",
"name": "MAIN",
"fulfillment": {
"conversationName": "jdii ratings"
},
"intent": {
"name": "actions.intent.MAIN",
"trigger": {
"queryPatterns": [
"What did I rate $Test:text",
"Look up $Test:text on my ratings spreadsheet"
]
},
"parameters": [
{
"name": "text",
"type": "Test"
}
]
}
}
],
"conversations": {
"jdii ratings": {
"name": "jdii ratings",
"url": "<_________>",
"fulfillmentApiVersion": 2
}
}
}
program.py >>
import random
import types
import gspread
from flask import Flask
from google.cloud import dialogflow_v2
from oauth2client.service_account import ServiceAccountCredentials
app = Flask(__name__)
wanted_thingy = "Squirt"
def get_requested_thingy_rating(requested_thingy_name):
scopes = [
'https://www.googleapis.com/auth/spreadsheets',
'https://www.googleapis.com/auth/drive'
]
credentials = ServiceAccountCredentials.from_json_keyfile_name("_______.json", scopes)
file = gspread.authorize(credentials)
sheet = file.open("JDII Ratings API")
requested_thingy_rating_final = 0.0
if not isinstance(requested_thingy_name, types.NoneType):
sheet = sheet.sheet1
thingy_list = sheet.col_values(1)
ratings_list = sheet.col_values(2)
requested_thingy_rating = 0.0
if requested_thingy_name in thingy_list:
thingy_pos = thingy_list.index(requested_thingy_name)
requested_thingy_rating = ratings_list[thingy_pos]
requested_thingy_rating_final = str(requested_thingy_rating)
# split_string = ready_string.split('\'')
# requested_thingy_rating_final = split_string[1]
print(requested_thingy_rating_final)
return requested_thingy_rating_final
#app.route("/start", methods=['POST', 'GET', 'PUT'])
def main():
response = {
"expectUserResponse": True,
"expectedInputs": [
{
"inputPrompt": {
"richInitialPrompt": {
"items": [
{
"simpleResponse": {
"textToSpeech": "What would you like to look up?",
"displayText": "What would you like to look up?"
}
}
]
}
},
"possibleIntents": [
{
"intent": "actions.intent.TEXT"
}
]
}
]
}
response_text = json.dumps(response)
return response_text
#app.route("/webhook", methods=['POST', 'GET', 'PUT'])
def main2():
global wanted_thingy
wanted_thingy_final = str(wanted_thingy)
wanted_thingy_rating = get_requested_thingy_rating(wanted_thingy_final)
print(wanted_thingy)
string = f"You rated {wanted_thingy} a {wanted_thingy_rating} out of ten."
response2 = {
"expectUserResponse": False,
"finalResponse": {
"richResponse": {
"items": [
{
"simpleResponse": {
'ssml': f'<speak>{string}</speak>'
}
}
]
}
}
}
response_text = json.dumps(response2)
return response_text
app.run()
I am trying to modify the bucket policy of one of my buckets using boto3. Please refer to the following inital/existing bucket policy of the bucket:
{
"Version": "2012-10-17",
"Id": "Policy1604310539665",
"Statement": [
{
"Sid": "Stmt1604310537860",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxx:root"
},
"Action": [
"s3:ListBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::bucket",
"arn:aws:s3:::bucket*"
]
}
]
}
I am trying to modify the above policy using the following piece of code, I am trying to attach one more role to the bucket policy :
import boto3
import json
s3 = boto3.client('s3')
result = s3.get_bucket_policy(Bucket=bucket_name)
policy_statement=json.loads(result['Policy'])
store = policy_statement['Statement'][0]['Principal']['AWS']
del policy_statement['Statement'][0]['Principal']['AWS']
if(isinstance(store, str)):
role_arn_list = [role_arn] + [store]
policy_statement['Statement'][0]['Principal'].update({"AWS": role_arn_list})
else:
role_arn_list = [role_arn] + store
policy_statement['Statement'][0]['Principal'].update({"AWS": role_arn_list})
# Convert the policy from JSON dict to string
policy_new = json.dumps(policy_statement)
# Update the policy of the given bucket
s3 = boto3.client('s3')
s3.put_bucket_policy(Bucket=bucket_name, Policy=policy_new)
The above code works fine, but when I try to put the policy to the bucket I am getting a MalformedPolicy exception. when I tried to debug and find the policy that is created using the above code, I can see the following policy:
{
'Version': '2012-10-17',
'Id': 'Policy1604310539665',
'Statement': [{
'Sid': 'Stmt1604310537860',
'Effect': 'Allow',
'Principal': {
'AWS': ['arn:aws:iam::xxx:role/xx-xx-xx', 'arn:aws:iam::xx:root',
'AROAVCQ6H5MBRCO7T5NKB'
]
},
'Action': ['s3:ListBucket', 's3:PutObject'],
'Resource': ['arn:aws:s3:::bucket', 'arn:aws:s3:::bucket/*']
}]
}
Problem: I am not able to understand from where the random string AROAVCQ6H5MBRCO7T5NKB is coming and how to handle this?
An identifier starting with AROA is a unique ID for an IAM Role, much like an Access Key always starts with AKIA.
See: IAM identifiers - AWS Identity and Access Management
Below is the json file
[
{
"year": 2013,
"title": "Rush",
"actors": [
"Daniel Bruhl",
"Chris Hemsworth",
"Olivia Wilde"
]
},
{
"year": 2013,
"title": "Prisoners",
"actors": [
"Hugh Jackman",
"Jake Gyllenhaal",
"Viola Davis"
]
}
]
Below is the code to push to dynamodb. I have created testjsonbucket bucket name, moviedataten.json is the filename and saved above json.Create a dynamodb with Primary partition key as year (Number) and
Primary sort key as title (String).
import json
from decimal import Decimal
import json
import boto3
s3 = boto3.resource('s3')
obj = s3.Object('testjsonbucket', 'moviedataten.json')
body = obj.json
#def lambda_handler(event,context):
# print (body)
def load_movies(movies, dynamodb=None):
if not dynamodb:
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Movies')
for movie in movies:
year = int(movie['year'])
title = movie['title']
print("Adding movie:", year, title)
table.put_item(Item=movie)
def lambda_handler(event, context):
movie_list = json.loads(body, parse_float=Decimal)
load_movies(movie_list)
I want to push in to ElasticSearch from dynamodb.
I have created a Elastic Domain https://xx.x.x.com/testelas
I have gone through the link https://aws.amazon.com/blogs/compute/indexing-amazon-dynamodb-content-with-amazon-elasticsearch-service-using-aws-lambda/
I clicked Managestream also
My Requirement:
Any change in Dynamodb has to reflect in the Elasticsearch?
This lambda just writing the document to DynamoDb, and I will not recommend adding the code in this lambda to push the same object to Elastic search, as lambda function should perform a single task and pushing the same document to ELK should be managed as a DynamoDB stream.
What if ELK is down or not available how you will manage this in lambda?
What if you want to disable this in future? you will need to modify lambda instead of controlling this from AWS API or AWS console, all you need to just disable the stream when required no changes on above lambda side code
What if you want to move only modify or TTL item to elastic search?
So create Dyanodb Stream that pushes the document to another Lambda that is responsible to push the document to ELK, with this option you can also push old and new both items.
You can look into this article too that describe another approach data-streaming-from-dynamodb-to-elasticsearch
For above approach look into this GitHub project dynamodb-stream-elasticsearch.
const { pushStream } = require('dynamodb-stream-elasticsearch');
const { ES_ENDPOINT, INDEX, TYPE } = process.env;
function myHandler(event, context, callback) {
console.log('Received event:', JSON.stringify(event, null, 2));
pushStream({ event, endpoint: ES_ENDPOINT, index: INDEX, type: TYPE })
.then(() => {
callback(null, `Successfully processed ${event.Records.length} records.`);
})
.catch((e) => {
callback(`Error ${e}`, null);
});
}
exports.handler = myHandler;
DynamoDB has a built in feature (DynamoDB streams) that will handle the stream part of this question.
When you configure this you have the choice of the following configurations:
KEYS_ONLY — Only the key attributes of the modified item.
NEW_IMAGE — The entire item, as it appears after it was modified.
OLD_IMAGE — The entire item, as it appeared before it was modified.
NEW_AND_OLD_IMAGES — Both the new and the old images of the item.
This will produce an event that looks like the following
{
"Records":[
{
"eventID":"1",
"eventName":"INSERT",
"eventVersion":"1.0",
"eventSource":"aws:dynamodb",
"awsRegion":"us-east-1",
"dynamodb":{
"Keys":{
"Id":{
"N":"101"
}
},
"NewImage":{
"Message":{
"S":"New item!"
},
"Id":{
"N":"101"
}
},
"SequenceNumber":"111",
"SizeBytes":26,
"StreamViewType":"NEW_AND_OLD_IMAGES"
},
"eventSourceARN":"stream-ARN"
},
{
"eventID":"2",
"eventName":"MODIFY",
"eventVersion":"1.0",
"eventSource":"aws:dynamodb",
"awsRegion":"us-east-1",
"dynamodb":{
"Keys":{
"Id":{
"N":"101"
}
},
"NewImage":{
"Message":{
"S":"This item has changed"
},
"Id":{
"N":"101"
}
},
"OldImage":{
"Message":{
"S":"New item!"
},
"Id":{
"N":"101"
}
},
"SequenceNumber":"222",
"SizeBytes":59,
"StreamViewType":"NEW_AND_OLD_IMAGES"
},
"eventSourceARN":"stream-ARN"
},
{
"eventID":"3",
"eventName":"REMOVE",
"eventVersion":"1.0",
"eventSource":"aws:dynamodb",
"awsRegion":"us-east-1",
"dynamodb":{
"Keys":{
"Id":{
"N":"101"
}
},
"OldImage":{
"Message":{
"S":"This item has changed"
},
"Id":{
"N":"101"
}
},
"SequenceNumber":"333",
"SizeBytes":38,
"StreamViewType":"NEW_AND_OLD_IMAGES"
},
"eventSourceARN":"stream-ARN"
}
]
}
As you're already familiar with Lambda it makes sense to use a Lambda function to consume the records and then iterate through them to process them in the Elasticsearch format before adding them to your index.
When doing this make sure that you iterate through each record as there may be multiple depending on your configuration.
For more information on the steps required for the Lambda side of the function check out the Tutorial: Using AWS Lambda with Amazon DynamoDB streams page.
I am creating a signed url using the following:
AWS_ACCESS_KEY_ID = my_access_key
AWS_SECRET_ACCESS_KEY = my_secret_access_key
KEYPAIR_ID = my_keypair_id
KEYPAIR_FILE = path_to_keypair_file
CF_DISTRIBUTION_ID = cf_dist_id
my_connection = cloudfront.CloudFrontConnection(
AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY
)
distro_summary = my_connection.get_all_distributions()[0]
distro_info = my_connection.get_distribution_info(distro_summary.id)
distro = distro_summary.get_distribution()
SECS = 8000
signed_url = distro.create_signed_url(
"https://%s/%s" % (distro_info.domain_name, 'restaurant_1_banner.png'),
KEYPAIR_ID,
expire_time=time.time() + SECS,
valid_after_time=None,
ip_address=None,
policy_url=None,
private_key_file=KEYPAIR_FILE
#private_key_string=KEYPAIR_ID
)
return signed_url
This returns a url like: "https://d1yllqv1oc7n6x.cloudfront.net/restaurant_1_banner.png?Expires=1426681326.67&Signature=Nsvyl-EowDRGuw-MfdgS34C6bsHKKC2L88ROfPBRAnsbpoeYfpJj6NQaTj4PGiG02Z7PRqkk5F0cBWKOik738H8xrlQQf8CuS0AouisnqMvZ4FLx94fSMo8vwFDg9jKLTMB1T0AGjWvgAcDlkLo4nYxyHQ077pwp3Do8g1eP62QD-~Ys4kejtVGtPTx6O1pM4gRLsmM8Kn7HJ618Hp4XMgRWwqJaCL-2C0YQP1PdEMbSOS6ZrmGTN~U5T-s-PZX1poS6qRiY4-Ma66DVLgmOTBh5vqjCWEqsbKZKFWFufsA2mMa4ON11yBUSyIbGJPpgKdRLU0pZuo7RX3~sIe6Q9w__&Key-Pair-Id=APKAISF4B35DSGOUTGTQ"
When I click on this link, I get the message:
<Error>
<Code>AccessDenied</Code>
<Message>Access denied</Message>
</Error>
This is my bucket policy for my s3 bucket.
{
"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity E3I8A03QRR3ASO"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::menumaster/*"
}
]
}
Please let me know if any additional information is required.
This is my bucket policy.
{
"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EH238ELEGANOC"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::onur.deneme/*"
}
]
}
This is the distribution :
di53i9yykewl5.cloudfront.net
Restrict Bucket Access : Yes
Origin Access Identity : Use an Existing Identity
Restrict Viewer Access(Use Signed URLs) : Yes
Trusted Signers : Self
There should be no other ACL or policy.
Is the "Restrict Bucket Access" selected as "yes" and "origin access identity" selected?
Can you try the code below that I used before?
#!/usr/bin/python
import time,boto,rsa
from boto import cloudfront
from boto.cloudfront import distribution
AWS_ACCESS_KEY_ID="your access key"
AWS_SECRET_ACCESS_KEY="your secret access key"
conn = boto.cloudfront.CloudFrontConnection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
dist = conn.get_all_distributions()
a=dist[0].get_distribution()
#Set parameters for URL
key_pair_id = "your key pair id" #cloudfront security key
priv_key_file = "xxxxxxxxx.pem" #cloudfront private keypair file
expires = int(time.time()) + 60 #1 min
url="http://dbvvi2cumi6nj.cloudfront.net/santa.png"
signed_url = a.create_signed_url(url, key_pair_id, expires,private_key_file=priv_key_file)
print signed_url