We Keep Coding

CloudCube and Boto3 - list content of the Objects

I am encountering you with the request for help with listing the objects in my CloudCube bucket. I am developing a Django application hosted on Heroku. I am using the CloudCube add-on for persistent storage. CloudCube is running on AWS S3 Bucket and CloudCube provides private Key/Namespace in order to access my files. I use the boto3 library to access the bucket and everything works fine when I want to upload/download the file; however, I am struggling with the attempts to list objects in that particular bucket with the CloudCube prefix key. On any request, I receive AccessDennied Exception.
To access the bucket I use the following implementation:
s3_client = boto3.client('s3', aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
endpoint_url=settings.AWS_S3_ENDPOINT_URL, region_name='eu-west-1')
s3_result = s3_client.list_objects(Bucket=settings.AWS_STORAGE_BUCKET_NAME, Prefix=settings.CLOUD_CUBE_KEY)
if 'Contents' not in s3_result:
return []
file_list = []
for key in s3_result['Contents']:
if f"{username}/{mode.value}" in key['Key']:
file_list.append(key['Key'])
As a bucket name, I am using the prefix in the URI that aims to the CloudCube bucket on the AWS according to their documentation: https://BUCKETNAME.s3.amazonaws.com/CUBENAME. CUBENAME is used then as a Prefix Key.
Does anyone have clue what do I miss?
Thank you in advance!

According to CloudCube's documentation, you need a trailing slash on the prefix to list the directory.
So you should update your code like this to make it work:
s3_result = s3_client.list_objects(Bucket=settings.AWS_STORAGE_BUCKET_NAME, Prefix=f'{settings.CLOUD_CUBE_KEY}/')

Using boto3 to filter s3 objects so that caller is not filtering

My s3 bucket has a thousands of files. They are all in the same "folder". So the s3 prefix is the same. I want to use python boto3 to get a list of filenames that contains a certain word. I dont want the boto3 call to send back all the filenames and have the client filter out the names. I seen example from using "yield" and ".filter" but those are receiving all the files and making the client do a lot of work.
To help give a better understanding, if I use the AWS CLI:
aws --profile test s3api list-objects-v2 --bucket mybucket --prefix tmp/ --output json --query "Contents[?contains(Key, 'foobar')]"
BUT I need to send a request using boto3 and AWS just send the filenames back with "foobar" in them.

BUT I need to send a request using boto3 and AWS just send the filenames back with "foobar" in them
You can't do this with regular boto3 s3 API calls as this is not how that API works. So if you don't want to get all the names first, and then filter them out by yourself, then there is no way to achieve what you want with just a single boto3 request.
The only help would maybe be from Amazon S3 inventory. So you could request the inventory, get the resulting CSV file, and filter that. But still you would have to filter it yourself.

Google cloud get bucket - works with cli but not in python

I was asked to preform integration with an external google storage bucket, I had received a credentials json,
And while trying to do
gsutil ls gs://bucket_name (after configuring myself with the creds json) I had received a valid response, as well as when I tried to upload a file into the bucket.
When trying to do it with Python3, it does not work:
While using google-cloud-storage==1.16.0 (tried also the newer versions), I'm doing:
project_id = credentials_dict.get("project_id")
credentials = service_account.Credentials.from_service_account_info(credentials_dict)
client = storage.Client(credentials=credentials, project=project_id)
bucket = client.get_bucket(bucket_name)
But on the get_bucket line, I get:
google.api_core.exceptions.Forbidden: 403 GET https://www.googleapis.com/storage/v1/b/BUCKET_NAME?projection=noAcl: USERNAME#PROJECT_ID.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket.
The external partner which I'm integrating with, saying that the user is set correctly, and to prove it they're showing that I can preform the action with gsutil.
Can you please assist? Any idea what might be the problem?

The answer was that the creds were indeed wrong, but it did worked when I tried to preform on the client client.bucket(bucket_name) instead of client.get_bucket(bucket_name).

Please follow these steps in order to correctly set up the Cloud Storage Client Library for Python. In general, the Cloud Storage Libraries can use Application default credentials or environment variables for authentication.
Notice that the recommended method to use would be to set up authentication using environment variables (i.e if you are using Linux: export GOOGLE_APPLICATION_CREDENTIALS="/path/to/[service-account-credentials].json" should work) and avoid the use of the service_account.Credentials.from_service_account_info() method altogether:
from google.cloud import storage
storage_client = storage.Client(project='project-id-where-the-bucket-is')
bucket_name = "your-bucket"
bucket = client.get_bucket(bucket_name)
should simply work because the authentication is handled by the client library via the environment variable.
Now, if you are interested in explicitly using the service account instead of using service_account.Credentials.from_service_account_info() method you can use the from_service_account_json() method directly in the following way:
from google.cloud import storage
# Explicitly use service account credentials by specifying the private key
# file.
storage_client = storage.Client.from_service_account_json(
'/[service-account-credentials].json')
bucket_name = "your-bucket"
bucket = client.get_bucket(bucket_name)
Find all the relevant details as to how to provide credentials to your application here.

tl;dr: dont use client.get_bucket at all.
See for detailed explanation and solution https://stackoverflow.com/a/51452170/705745

Writing file to S3 with AWS Lambda errors - An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

I am trying to write a pandas dataframe to S3 bucket in AWS Lambda, my code:
import boto3
import pandas as pd
import s3fs
from io import StringIO
...
bucket = 'info' # already created on S3
csv_buffer = StringIO()
result.to_csv(csv_buffer)
s3_resource = boto3.resource('s3')
s3_resource.Object(bucket, 'testing.csv').put(Body=csv_buffer.getvalue())
I have turned off the public access blocking. When I go to the bucket -> access points there is this: Access points can be used to provide access to your bucket. The S3 console doesn't support using virtual private cloud (VPC) access points to access bucket resources. To access bucket resources from a VPC access point, you’ll need to use the AWS CLI, AWS SDK, or Amazon S3 REST API.
Does that mean I can't write files to S3 with AWS lambda the way I am trying to?

You have to grant your lambda a permission allowing it to write to your S3 bucket. In the link below it is explained how to do it.
How do I allow my Lambda execution role to access my Amazon S3 bucket?

How to access someone else's AWS S3 'bucket' with Boto3 and Username?

I'm trying to download AWS S3 content using Python/Boto3.
A third-party is uploading a data, and I need to download it.
They provided credentials like this:
Username : MYUser
aws_access_key_id : SOMEKEY
aws_secret_access_key : SOMEOTHERKEY
Using a popular Windows 10 app CyberDuck, my 'Username' is added to the application's path settings, third-party/MYUser/myfolder
Nothing I'm given here is my bucket.
my_bucket = s3.Bucket('third-party/MYUser')
ParamValidationError: Parameter validation failed:
Invalid bucket name 'third-party/MYUser': Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"
my_bucket = s3.Bucket('third-party')
ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
my_bucket = s3.Bucket('MYStuff')
NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjects operation: The specified bucket does not exist
From what I've read, third-party is the AWS S3 bucket name, but I can't find an explanation for how to access a sub-directory of someone else's bucket.
I'm see Bucket() has some user parameters. I read elsewhere about roles, and access control lists. But I'm not finding a simple example.
How do I access someone else's bucket on AWS S3 given Username?

Amazon S3 does not actually have directories. Rather, the Key (filename) of an object contains the full path of the object.
For example, consider this object:
s3://my-bucket/invoices/foo.txt
The bucket is my-bucket
The Key of the object is invoices/foo.txt
So, you could access the object with:
import boto3
s3_resource = boto3.resource('s3')
object = s3.Object('my-bucket','invoices/foo.txt')
To keep S3 compatible with systems and humans who expect to have folders and directories, it maintains a list of CommonPrefixes, which are effectively the same as directories. They are derived from the names between slashes (/). So, CyberDuck can give users the ability to navigate through directories.
However, the third-party might have only assigned you enough permission to access your own directory, but not the root directory. In this case, you will need to navigate straight to your directory without clicking through the hierarchy.
A good way to use an alternate set of credentials is to store them as a separate profile:
aws configure --profile third-party
You will then be prompted for the credentials.
Then, you can use the credentials like this:
aws s3 ls s3://third-party/MyUser --profile third-party
aws s3 cp s3://third-party/MyUser/folder/foo.txt .
The --profile at the end lets you select which credentials to use.
The boto3 equivalent is:
session = boto3.Session(profile_name='third-party')
s3_resource = session.resource('s3')
object = s3.Object('THEIR-bucket','MYinvoices/foo.txt')
See: Credentials — Boto 3 Documentation