We Keep Coding

Boto3 S3 list_objects_v2 Not Returning Any Objects

I'm using Boto3 to try to get a list of keys from an S3 bucket via an AWS Lambda Python script. No matter what I try, the bucket returns no objects.
import json, boto3, os
def getConfig():
cfg = {
"aws_key_id": os.getenv("AWS_KEY_ID", ""),
"aws_secret": os.getenv("AWS_SECRET", ""),
}
return cfg
def lambda_handler(event, context):
cfg = getConfig()
bucket_name = "zachs-taxi"
session = boto3.Session(
aws_access_key_id=cfg.get('aws_key_id'),
aws_secret_access_key=cfg.get('aws_secret')
)
s3 = session.client('s3')
I've tried both of the following but both return empty:
response = s3.list_objects_v2(
Bucket=bucket_name)
for content in response.get('Contents', []):
print(content['Key'])
And
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=bucket_name):
for content in page.get('Contents', ()):
print(content['Key'])
The S3 bucket is public and I can access it. Inside there is a folder called content and within that folder is a .png file.
Any help would be appreciated. Thanks!

Your code ran perfectly fine for me (with a different bucket name) when I ran it on my own computer:
import boto3
bucket_name = "my-bucketname"
s3 = boto3.client('s3')
response = s3.list_objects_v2(Bucket=bucket_name)
for content in response.get('Contents', []):
print(content['Key'])

Error when generating presigned url from aws s3 using boto3

When I try to return a generate presigned url using boto3 from bucket in aws s3 and the code:
import fastapi
import boto3
from botocore.exceptions import ClientError
s3 = boto3.client("s3",
aws_access_key_id="...",
aws_secret_access_key="...")
BUCKET_NAME = "tayibat-files"
app = FastAPI()
#app.get('/{file_name}')
async def method_name(file_name: str):
try:
url = s3.generate_presigned_url(
'get_object',
Params={'Bucket': BUCKET_NAME,
'Key': f"products/{file_name}"},
ExpiresIn=3600
)
except ClientError as e:
logging.error(e)
return url
the get request return an url, but when I try to open it in browsers, It generate:
This XML file does not appear to have any style information associated with it. The document
tree is shown below.
<Error>
<Code>InvalidRequest</Code>
<Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-
SHA256.</Message>
<RequestId>ZW269CV1TAYC7CWC</RequestId>
<HostId>1yozjolBbu4difnOjjopLeOk79i34WDOFwp1VQA4Nqd0RBdLNkaOkb/uJVjFtyNu78fx06JfCbI=</HostId>
</Error>

The issue is not your code but your method of authentication or region.
I ran your code sample successfully:
import boto3
session = boto3.session.Session(profile_name="<my-profile>")
client = session.client('s3')
BUCKET_NAME = "<bucket>"
file_name = "<file>"
url = client.generate_presigned_url(
'get_object',
Params={'Bucket': BUCKET_NAME,
'Key': f"products/{file_name}"},
ExpiresIn=3600
)
print(url)
It worked fine because the region of my bucket aligned with the region of my credentials. When I tried to generate a presigned url from another region I got your same error:
<Error>
<Code>InvalidRequest</Code>
<Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message>
<RequestId>JJPZYSMZTC7Z8H</RequestId>
<HostId>TsgdZIibKxZ4GVL3h28OJYIvh59yfgeZwVf+eGPXVEzIJsAxdp1VQL67vw20LR/r9uIBxro=</HostId>
</Error>

Python Boto3 there is a difference in AWS S3 presign url response for bucket in us-east-1 and us-east-2

With Python Boto3 i create post presign Url, below is sample code.
client = boto3.client('s3', region_name="us-east-1")
response = client.generate_presigned_post(Bucket="tes_bucket", Key=filename, ExpiresIn=300)
There is difference in response fields for the bucket in us-east-1 and us-east-2
With the same code, if i try on bucket with us-east-1 i get ressponse fields.
AWSAccessKeyId, key, policy, signature, and x-amz-security-token
Where as when created with bucket in us-east-2 region i get response fields
key, policy, x-amz-algorithm, x-amz-credential, x-amz-date, x-amz-security-token, x-amz-signature
There is no differecen in bucket configuraion, other than region, but still why there is such difference in response fields.
What we change to get same response across all region

As i checked this two scenario.
lambda code :
import boto3
def lambda_handler(event, context):
filename = "example.pdf"
client = boto3.client('s3', region_name="us-east-1")
response = client.generate_presigned_post(Bucket="bucket1", Key=filename, ExpiresIn=300)
print(response)
client1 = boto3.client('s3', region_name="ap-south-1")
response1 = client1.generate_presigned_post(Bucket="bucket2", Key=filename, ExpiresIn=300)
print(response1)
in response only for ap-south-1 region bucket got extra params :
'x-amz-algorithm': 'AWS4-HMAC-SHA256',
'x-amz-credential': 'xxxxxxxxxxxxxxx/xxxxxxx/ap-south-1/s3/aws4_request',
'x-amz-date': '20200928T183454Z',
Reason behind this you are using generate_presigned_post boto3 S3 function which is used for either API call or form action or CURL request. When you are using same region and hand shaking resource internally in same region this extra check are not required to validate resource access policy. Where as if two AWS resources are handshaking to each other which having different region or different AWS account then required extra params to access resources.
This all params are part of AWS signature to validate resources having proper access control to hand shake.
For getting same params here is approach :
import boto3
import datetime
def lambda_handler(event, context):
filename = "example.pdf"
date_short = datetime.datetime.utcnow().strftime('%Y%m%d')
date_long = datetime.datetime.utcnow().strftime('%Y%m%dT000000Z')
client = boto3.client('s3', region_name="us-east-1")
fields = {
'acl': 'private',
'date': date_short,
'region': "us-east-1",
'x-amz-algorithm': 'AWS4-HMAC-SHA256',
'x-amz-date': date_long
}
response = client.generate_presigned_post(Bucket="bucket1",Fields = fields, Key=filename, ExpiresIn=300)
print(response)
client1 = boto3.client('s3', region_name="ap-south-1")
fields = {
'acl': 'private',
'date': date_short,
'region': "ap-south-1",
'x-amz-algorithm': 'AWS4-HMAC-SHA256',
'x-amz-date': date_long
}
response1 = client1.generate_presigned_post(Bucket="bucket2", Fields = fields,Key=filename, ExpiresIn=300)
print(response1)
Botocore uses s3v2 while generating presigned post for us-east-1 region and uses s3v4 for other region. That's why you are not getting some parameter in fields.
So if you explicitly specify the signature version to s3v4 then you can get the same field. Something like this:
https://github.com/boto/boto3/issues/2606#issuecomment-701587119
from botocore.client import Config
s3 = boto3.client('s3', 'us-east-1', config=Config(signature_version='s3v4'))
response = s3.generate_presigned_post(Bucket="bucket2", Key=filename, ExpiresIn=300)
I tried this got same fields in both request.
Reference : https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.generate_presigned_post
Amazon AWS S3 browser-based upload using POST -

How to create a signed s3 url for requester pays bucket in python

I have a requester pays bucket that I do not control in form:
s3://bucket-name/path-to-my-file
I am attempting to generate a presigned url to send to a web app to render it in browser.
I've gone through the boto s3 documentation but can't find anything that covers this :(
My script below creates returns URL that does not have access and returns this error from s3:
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>11DCA24D8DF2E9E8</RequestId>
<HostId>SeTDlt66hPsj5/dV1pOT9GnYyWgpSGI4ezI3wC7iz8Pny9sy2sUWsuUsl4JkEQeBXXIyiE8EXCk=</HostId>
</Error>
I'm confident this is because the bucket is requester pays, becuase when I run this command in aws cli it works:
aws s3 cp s3://blackfynn-discover-use1/66/2/files/Derivatives . --request-payer requester --recursive
But this one returns Forbidden:
aws s3 cp s3://blackfynn-discover-use1/66/2/files/Derivatives .
Here's my python script which would work if it was not requester pays:
import requests
import boto3
def get_signed_url(s3_url):
# Get the service client.
s3 = boto3.client('s3')
bucket_name, key_name = split_s3_bucket_key(s3_url)
# Generate the URL to get 'key-name' from 'bucket-name'
url = s3.generate_presigned_url(
ClientMethod='get_object',
Params={
'Bucket': bucket_name,
'Key': key_name
}
)
return url
def split_s3_bucket_key(s3_path):
"""Split s3 path into bucket and key prefix.
This will also handle the s3:// prefix.
:return: Tuple of ('bucketname', 'keyname')
"""
if s3_path.startswith('s3://'):
s3_path = s3_path[5:]
return find_bucket_key(s3_path)
def find_bucket_key(s3_path):
"""
This is a helper function that given an s3 path such that the path is of
the form: bucket/key
It will return the bucket and the key represented by the s3 path
"""
s3_components = s3_path.split('/')
bucket = s3_components[0]
s3_key = ""
if len(s3_components) > 1:
s3_key = '/'.join(s3_components[1:])
return bucket, s3_key
s3_file_path = 's3://blackfynn-discover-use1/66/2/files/Derivatives/manifest.xlsx'
get_signed_url(s3_file_path)

It looks like the URL will need to include x-amz-request-payer=requester, but this might also need to be specified when creating the pre-signed URL.
Try the advice shown below, then let us know whether it worked for you!
From Downloading Objects in Requester Pays Buckets - Amazon Simple Storage Service:
For signed URLs, include x-amz-request-payer=requester in the request
From Support Requester Pays S3 buckets · Issue #346 · samtools/htslib:
OK, was able to compile htslib with good libcurl support. Confirmed that it can take a presigned URL to view files:
import boto3
client = boto3.client('s3')
url = client.generate_presigned_url("get_object", Params={"Bucket":"angel-reqpay","Key":"test.cram" , "RequestPayer":'requester'})
From AWS Developer Forums: Announcing “Requester Pays” Option for ...:
You URL would look something like:
http://somebucket.s3.amazonaws.com/key/[.....]&x-amz-request-payer=requester

How do I get an object from S3 using it's pre-signed url in boto3?

I have generated a pre-signed url for an object in one of my buckets using boto3:
s3.generate_presigned_url('get_object', Params = {'Bucket': 'https://s3.amazonaws.com/<>', 'Key': '<>.json'}, ExpiresIn = 100)
Now, how do I get_object it in boto3? The boto3's get_object reference doesn't specify any argument for a pre-signed url.
So, how do I get that object from S3 using it's pre-signed url in boto3?

If you have a pre-signed URL, you don't need boto -- you can download the object using any HTTP user agent library.
Conversely, if you have boto and the credentials, you don't need a pre-signed URL.
Pre-signed URLs are intended for allowing someone with credentials to enable someone else without credentials to access a resource, without exposing the credentials to them.
A pre-signed URL includes the access-key-id and possibly a session-token, but not the access-key-secret, and are computationally-infeasible to reverse-engineer... and in this sense, they do not expose the credentials in a way that allows the entity possessing the pre-signed URL to use the associated credentials for any other purpose.

you can use this code to get the result.
import boto3
s3_client = boto3.client('s3')
resp = s3_client.generate_presigned_url('get_object', Params = {'Bucket': 'your-s3-bucket', 'Key': 'filepath/inside-bucket/filename.json'}, ExpiresIn = 100)
print(resp)
if any doubt please let me know.