I have the following code:
import json
import requests
import boto3
import os
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
print('Received event:' + str(event))
bucket = event['Records'][0]['s3']['bucket']['name']
key = event['Records'][0]['s3']['object']['key']
S3Client = boto3.client('s3','us-east-1')
url = "https://s3.amazonaws.com/%s/%s" % (bucket, key)
ID_Resp = S3Client.head_object(Bucket=bucket, Key=key)
print(ID_Resp)
version_id = ID_Resp['ResponseMetadata']['HTTPHeaders']['x-amz-version-id']
Where I'm receiving an S3 object from a bucket on Object Create. Ultimately I want to POST metadata about the file to a custom API. I'm trying to call the head_object method so I can get the x-amz-version-id so I know what version of the file was submitted to the API. When I run the last line on my local machine, it works, but when I run it from within the lambda environment, the key value pair for x-amz-version-id is missing altogether. Does anyone know why this might be? Cant find anything in the docs about it.
Related
Documentation
import logging
import boto3
from botocore.exceptions import ClientError
def create_bucket(bucket_name, region=None):
try:
if region is None:
s3_client = boto3.client('s3')
s3_client.create_bucket(Bucket=bucket_name)
else:
s3_client = boto3.client('s3', region_name=region)
location = {'LocationConstraint': region}
s3_client.create_bucket(Bucket=bucket_name,
CreateBucketConfiguration=location)
except ClientError as e:
logging.error(e)
return False
return True
Testing:
create_bucket('test', 'us-west-2') Works as expected -> Please select a different name and try again
create_bucket('test') The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
create_bucket('test', 'us-east-1') The us-east-1 location constraint is incompatible for the region specific endpoint this request was sent to.
What did I miss?
The problem is in the error message.
Obviously, the bucket with the name test is already taken, but for some reason, instead of saying Please select a different name and try again, we see a message that misleads us.
So the solution is simple - use a unique name for the bucket.
I hope AWS will fix the error message ASAP but it's not a blocker for us anyway
This creates a bucket in us-east-1:
s3_client = boto3.client('s3', region_name='us-east-1')
s3_client.create_bucket(Bucket='unique-name-here')
I have a list of bucket names that I need to enable logging on programmatically. I am using boto3 and lambda. I can't seem to find the correct function in boto3/s3 to do what I need although I'm sure it's possible. Here is my code so far:
import json
import boto3
s3 = boto3.client('s3')
def lambda_handler(event, context):
# TODO implement
# print("hanlder:event")
# print(event)
# bucketDump()
setBucketPolicy(bucketDump())
def bucketDump():
##This program lists all exsisting buckets within an aws account (Tommy's Personal Account)
response = s3.list_buckets()
buckets = []
for bucket in response['Buckets']:
value = bucket["Name"]
buckets.append(value)
return buckets
##setting a bucket policy
def setBucketPolicy(buckets):
for bucket in buckets:
value = s3.get_bucket_logging(bucket)
print(value)
##TODO if bucket in buckets does not have loggin enabled, enable it!
# print(bucket)
My process is I want to iterate over the list of buckets I have and enable_logging for them! Thank you in advance.
As suggested by #jordanm in the comment below your question, using a resource instead of the client would make your life much easier as it provides a higher-level interface.
If the only goal of the bucketDump in your question was to retrieve all the buckets in your account, then you could totally remove it and use the standard function s3.buckets.all() that already returns an iterable of buckets (docs).
Assuming that you want to enable logging on all your buckets that don't have it already enabled and that you want to deliver logs from all the buckets to the same bucket, you could add a parameter to the `` function to specify this bucket. The implementation suggested below will enable logging and result in logs being organized like so:
- name_of_bucket_in_wich_to_store_logs
- bucket_name_1
- logs
- bucket_name_2
- logs
If you want to organize your logs differently you have to play with the TargetBucket and TargetPrefix parameters and, if needed, you can specify other parameters for grants as detailed in the docs.
import boto3
s3 = boto3.resource('s3')
def lambda_handler(event, context):
# TODO implement
setBucketPolicy(target_bucket='name_of_bucket_in_wich_to_store_logs')
def setBucketPolicy(target_bucket: str):
for bucket in s3.buckets.all():
bucket_logging = s3.BucketLogging(bucket.name)
if not bucket_logging.logging_enabled:
bucket_logging.put(
BucketLoggingStatus={
'LoggingEnabled': {
'TargetBucket': target_bucket,
'TargetPrefix': f'{bucket.name}/'
}
}
)
filename variable is used to get the name of latest file
My aim is to monitor a folder and whenever new file is retrieved, automatically upload it to s3 bucket using boto3.
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
from subprocess
import call
import os
import boto3
session = boto3.Session(aws_access_key_id='aws_access_key_id', aws_secret_access_key='aws_secret_access_key',
region_name='region_name')
s3 = session.client('s3')
class Watcher:
def __init__(self):
self.dir = os.path.abspath('D:\\project')
self.observer = Observer()
def run(self):
event_handler = Handler()
self.observer.schedule(event_handler, self.dir, recursive=True)
self.observer.start()
try:
while True:
time.sleep(5)
except:
self.observer.stop()
print("Error")
self.observer.join()
class Handler(FileSystemEventHandler):
#staticmethod
def on_any_event(event):
if event.is_directory:
return None
elif event.event_type == 'created':
print("Received created event - %s." % event.src_path)
s3.upload_file(Filename=event.src_path, bucket='bucketname, key=test-file-1)
if __name__ == '__main__':
w = Watcher()
w.run()
FileNotFoundError: [WinError 2] The system cannot find the file specified
As #alexhall mentioned in the comment, s3.meta.client.upload_file method will upload a file. You can read about boto3 s3 client's upload method documentation here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.upload_file. However, it is a bit silly example they have there, as they are first creating an s3 resource rather than s3 client, and then because s3 resource does not actually have a method to upload file, they revert back to s3 client. You might as well directly create and use s3 client for uploads.
You are also relying on the fact that boto3 uses default session when you create the s3 resource like you did:
boto3.resource('s3')
This would work fine if you are running the code on lambda or if you are in an ec2 instance that has an IAM role configured for it to access s3 but I think you are running this outside AWS, in which case, you can have a boto3.Session() created first using your credentials, and then a client (or resource) can use that session.
aws_access_key_id = '<AWS_ACCESS_KEY_ID>'
aws_secret_access_key = '<AWS_SECRET_ACCESS_KEY>'
region_name = 'us-east-1'
session = boto3.Session(aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
region_name=region_name)
s3 = session.client('s3')
You can read about Session configuration here: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
As mentioned above, because you are trying to upload file, and you do not seem to do anything else with it, you may as well directly create an s3 client rather than s3 resource like you did and then get the s3 client using 'meta.client'.
instead of command = ... line, you simply use:
s3.upload_file(Filename, Bucket = 'aaaaa', Key='test-file-1')
You can delete the last line. You would 'call' if you you were running an OS/System command rather than a something within python.
Not sure if you are doing this to learn python (boto3). If so, congrats.
If not, AWS already provided such feature. So you keep everything in your code, but shell out to the AWS CLI.
I am accessing the Project Gutenberg API and I wrote a Python program that creates a text file out of a randomly chosen book from the API. Since the txt file is different each time the document in the S3 bucket should be blank. I want to be able to store an object I can constantly write over in S3 then pull from it in Flask and put it on the user's computer.
So far I have been using boto3 and I set up an AWS account with a specific bucket. I loaded a trial .txt file in there but when the program is accessed now it only downloads the file I put in there with the specific text, it doesn't change based on my program like it should.
Boto seems to be throwing me for a loop so if there's another way I am open to it.
My code right now is a mess. I'm throwing everything I can at it to make it work but I know I've reach the point where I need some help.
from flask import Flask, render_template
from gutenberg.acquire import load_etext
from gutenberg.cleanup import strip_headers
import random
import os
from flask import send_from_directory
import boto3
from flask import Flask, request
from flask import Flask, Response
from boto3 import client
import botocore
app = Flask(__name__, static_url_path='/static')
def get_client():
return client(
's3',
'us-east-1',
aws_access_key_id='XXXXXXX',
aws_secret_access_key='XXXXXXX'
)
#app.route('/')
def welcome():
return 'Welcome to the server'
#app.route('/favicon.ico')
def favicon():
return send_from_directory(os.path.join(app.root_path, 'static'), 'favicon.ico', mimetype='image/vnd.microsoft.icon')
#app.route('/roulette', methods=['POST', 'GET'])
def roulette():
s3 = get_client()
f = open("GutProject.txt", "w")
file = s3.get_object(Bucket='book-roulette', Key='GutProject.txt')
for x in range(1):
y = (random.randint(0, 59000))
text = strip_headers(load_etext(y)).strip()
s3_client = boto3.client('s3')
open('GutProject.txt').write(text)
s3_client.upload_file('GutProject.txt', 'book-roulette', 'Gut-remote.txt')
s3_client.download_file('book-roulette', 'hello-remote.txt', 'hello2.txt')
print(open('hello2.txt').read())s3_client.download_file('MyBucket', 'hello-remote.txt', 'hello2.txt')
print(open('hello2.txt').read())
return Response(
file['Body'].read(),
mimetype='text/plain',
headers={"Content-Disposition": "attachment;filename=GutProject.txt"}
)
if __name__ == "__main__":
app.run(debug=True)
My app should let the user click a button on the URL page and it will download a random file to their computer. The HTML works great and the Python/Flask worked before but the file wasn't downloading (I'm on Heroku).
I keep getting these errors:
botocore.errorfactory.NoSuchKey
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
If the error stems from the line: s3_client.download_file('book-roulette', 'hello-remote.txt', 'hello2.txt'), then the error NoSuchKey is trying to say it cannot find the file s3://book-roulette/hello-remote.txt in the s3 region you specify.
I would suggest checking that s3 path to make sure it exists, or that the specified bucket and key are correct.
Edit: I notice that you create the s3_client object within your loop and overwrite the one where you specify your region and credentials, so it's possible it might not be checking in the right region anymore, but that might result in an access denied error or bucket not found error instead
I am trying to upload an image to S3 through Python. My code looks like this:
import os
from PIL import Image
import boto
from boto.s3.key import Key
def upload_to_s3(aws_access_key_id, aws_secret_access_key, file, bucket, key, callback=None, md5=None, reduced_redundancy=False, content_type=None):
conn = boto.connect_s3(aws_access_key_id, aws_secret_access_key)
bucket = conn.get_bucket(bucket, validate=False)
k = Key(bucket)
k.key = key
k.set_contents_from_file(file)
AWS_ACCESS_KEY = "...."
AWS_ACCESS_SECRET_KEY = "....."
filename = "images/image_0.jpg"
file = Image.open(filename)
key = "image"
bucket = 'images'
upload_to_s3(AWS_ACCESS_KEY, AWS_ACCESS_SECRET_KEY, file, bucket, key)
I am getting this error message:
S3ResponseError: S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidRequest</Code><Message> The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message>
<RequestId>90593132BA5E6D6C</RequestId>
<HostId>...</HostId></Error>
This code is based on the tutorial from this website: http://stackabuse.com/example-upload-a-file-to-aws-s3/
I have tried k.set_contents_from_file as well as k.set_contents_from_filename, but both don't seem to work for me.
The error says something about using AWS4-HMAC-SHA256, but I am not sure how to do that. Is there another way to solve this problem besides using AWS4-HMAC-SHA256? If anyone can help me out, I would really appreciate it.
Thank you!
Just use:
import boto3
client = boto3.client('s3', region_name='us-west-2')
client.upload_file('images/image_0.jpg', 'mybucket', 'image_0.jpg')
Try to avoid putting your credentials in the code. Instead:
If you are running the code from an Amazon EC2 instance, simply assign an IAM Role to the instance with appropriate permissions. The credentials will automatically be used.
If you are running the code on your own computer, use the AWS Command-Line Interface (CLI) aws configure command to store your credentials in a file, which will be automatically used by your code.