I have been given access to a subfolder of an S3 bucket, and want to access all files inside using Python and boto3. I am new to S3 and have read the docs to death, but haven't been able to figure out how to successfully access just one subfolder. I understand that s3 does not use unix-like directory structure, but I don't have access to the root bucket.
How can I configure boto3 to just connect to this subfolder?
I have successfully used this AWS CLI command to download the entire subfolder to my machine:
aws s3 cp --recursive s3://s3-bucket-name/SUB_FOLDER/ /Local/Path/Where/Files/Download/To --profile my-profile
This code:
AWS_BUCKET='s3-bucket-name'
s3 = boto3.client("s3", region_name='us-east-1', aws_access_key_id=AWS_KEY_ID, aws_secret_access_key=AWS_SECRET)
response = s3.list_objects(Bucket=AWS_BUCKET)
Returns this error:
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
I have also tried specifying the 'prefix' option in the call to list_objects, but this produces the same error.
You want to aws configure and save have your credentials and region then using boto3 is simple and easy.
Use boto3.resource and get the client like this:
s3_resource = boto3.resource('s3')
s3_client = s3_resource.meta.client
s3_client.list_objects(Bucket=AWS_BUCKET)
You should be good to go.
Related
Im trying to change S3 file's metadata from python, and test with chalice server locally (need the chalice for other things). When running (without chalice server):
# change file metadata in s3:
s3 = boto3.resource('s3')
s3_object = s3.Object(bucket, key)
s3_object.metadata.update({'metadata-key': 'something'})
s3_object.copy_from(CopySource={'Bucket': bucket, 'Key': key}, Metadata=s3_object.metadata,
MetadataDirective='REPLACE')
In python locally (with aws configured locaclly) everything is fine and the metadata is changing in S3.
The problem is when using chalice python local server, Im getting this error:
An error occurred (404) when calling the HeadObject operation: Not
Found
(I need the chalice to simulate a lambda, and the s3 metadata part is in there)
Any ideas why this happenes?
Thanks :-)
An error occurred (404) when calling the HeadObject operation: Not Found
If the object exists, this error usually means the role you are using to make the request doesn't have the required permissions.
Your local tests with Python work because they are using your default AWS profile. But that key is not passed to the Chalice APP.
You need to add read/write permissions to that bucket for that Lambda in Chalice.
I'm trying to view S3 bucket list through a python scripts using boto3. Credential file and config file is available in the C:\Users\user1.aws location. Secret access and access key available there for user "vscode". But unable to run the script which return exception message as
"botocore.exceptions.NoCredentialsError: Unable to locate credentials".
Code sample follows,
import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print(bucket.name)
Do I need to specify user mentioned above ("vscode") ?
Copied the credential and config file to folder of python script is running. But same exception occurs.
When I got this error, I replaced resource with client and also added the secrets during initialization:
client = boto3.client('s3', region_name=settings.AWS_REGION, aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
You can try with boto3.client('s3') instead of boto3.resource('s3')
Im new to Python and for my project purpose and Im using using boto3 to access AWS S3 in a pycharm IDE
I completed package installation for boto3 ,pyboto then created a Python file and successfully created bucket and transferred the files to S3 from my local using boto3
Later i created another python file in the same working directory and using the same steps but this time Im not able to connect AWS and not even API calls Im getting
So am doubtful that whether we can use boto3 packages with only one python file and we cant use it another python file in same directory?
I tried by creating both s3 client and s3 resource but no luck
Please advice is there any limitations is there for boto3 ?
Below are the Python code:-
import boto3
import OS
bucket_name='*****'
def s3_client():
s3=boto3.client('s3')
""":type:pyboto3:s3"""
return s3
def s3_resource():
s3=boto3.resource('s3')
return s3
def create_bucket(bucket_name):
val=s3_client().create_bucket(=bucket_name,
CreateBucketConfiguration={
'LocationConstraint':'ap-south-1'
})
return val
def upload_file():
s3=s3_resource().meta.client.upload_file('d:/s3_load2.csv',bucket_name,'snowflake.csv')
return s3
def upload_small_file():
s3=s3_client().upload_file('d:/s3_load2.csv',bucket_name,'snowflake.csv')
return s3
def create_bucket(bucket_name):
val=s3_client().create_bucket(
Bucket=bucket_name,
CreateBucketConfiguration={
'LocationConstraint':'ap-south-1'
})
return val
#calling
upload_small_file()
Perhaps the AWS credentials weren't set in the environment where you run the 2nd script. Or maybe the credentials you were using while running the 1st script already expired. Try getting your AWS credentials and set them when you instantiate a boto3 client or resource as documented:
import boto3
client = boto3.client(
's3',
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY,
aws_session_token=SESSION_TOKEN # This is only required for temporary credentials
)
Or you can also try setting them as environment variables.
export AWS_ACCESS_KEY_ID="some key"
export AWS_SECRET_ACCESS_KEY="some key"
export AWS_SESSION_TOKEN="some token" # This is only required for temporary credentials
Or as a configuration file. See the docs for the complete list.
I am trying to download a specific S3 file off a server using Python Boto and am getting "403 Forbidden" and "Access Denied" error messages. It says the error is occurring at line 24 (get_contents command). I have tried it with and without the "aws s3 cp" at the start of the source file path, received the same error message both time. My code is below, any advice would be helpful.
# Code to append csv:
import csv
import boto
from boto.s3.key import Key
keyId ="key"
sKeyId="secretkey"
srcFileName="aws s3 cp s3://...."
destFileName="C:\\Users...."
bucketName="bucket00001"
conn = boto.connect_s3(keyId,sKeyId)
bucket = conn.get_bucket(bucketName, validate = False)
#Get the Key object of the given key, in the bucket
k = Key(bucket, srcFileName)
#Get the contents of the key into a file
k.get_contents_to_filename(destFileName)
AWS is very vague with the errors that it outputs. This is intentional, but it definitely doesn't help with debugging. You are receiving an Access Denied error because the source file name you are using is not the correct path for the file.
aws s3 cp
This is the CLI command to copy one or more files from a source to a destination (with either being an s3 bucket). This should not be apart of the source file name.
s3://...
This prefix is appended to your bucket name that denotes that the path refers to an s3 object, however, this is not necessary in your source file path name when using boto3.
To download an s3 file using boto3, perform the following:
import boto3
BUCKET_NAME = 'my-bucket' # does not include s3://
KEY = 'image.jpg' # the file you want to download
s3 = boto3.resource('s3')
s3.Bucket(BUCKET_NAME).download_file(KEY, 'image.jpg')
Documentation for this command can be found here:
https://boto3.readthedocs.io/en/latest/guide/s3-example-download-file.html
In general, boto3 (and any other AWS SDK's) are simply wrappers around AWS api requests. You can also use the aws cli like I mentioned earlier to download a file from s3. That command would be:
aws s3 cp s3://my-bucket/my-file.jpg C:\location\my-file.jpg
srcFileName="aws s3 cp s3://...."
This has to be a key like somefolder/somekey or somekey as string.
You are providing a path or command to it.
Using this aws cli command (with access keys configured), I'm able to copy a key from S3 locally:
aws s3 cp s3://<bucketname>/test.txt test.txt
Using the following code in boto, I get S3ResponseError: 403 Forbidden, whether I allow boto to use configured credentials, or explicitly pass it keys.
import boto
c = boto.connect_s3()
b = c.get_bucket('<bucketname>')
k = b.get_key('test.txt')
d = k.get_contents_as_string() # exception thrown here
I've seen the other SO posts about not validating the key with validate=False etc, but none of these are my issue. I get similar results when copying the key to another location in the same bucket. Succeeds with the cli, but not with boto.
I've looked at the boto source to see if it's doing anything that requires extra permissions, but nothing stands out to me.
Does anyone have any suggestions? How does boto resolve its credentials?
Explicitly set your credentials so that our the same as the CLI with the ENV variables.
echo $ACCESS_KEY
echo $SECRET_KEY
import boto3
client = boto3.client(
's3',
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY
)
b = client.get_bucket('<bucketname>')
k = b.get_key('test.txt')
d = k.get_contents_as_string()
How boto resolves its credential.
The mechanism in which boto3 looks for credentials is to search through a list of possible locations and stop as soon as it finds credentials. The order in which Boto3 searches for credentials is:
Passing credentials as parameters in the boto.client() method
Passing credentials as parameters when creating a Session object
Environment variables
Shared credential file (~/.aws/credentials)
AWS config file (~/.aws/config)
Assume Role provider
Boto2 config file (/etc/boto.cfg and ~/.boto)
Instance metadata service on an Amazon EC2 instance that has an IAM role configured.
http://boto3.readthedocs.io/en/latest/guide/configuration.html#guide-configuration