Code in lambda:
import boto3
def lambda_handler(event, context):
s3_client = boto3.resource('s3')
mybucket = s3_client.Bucket('bucket-name')
for object in mybucket.objects.all():
print(object)
for key in s3_client.list_objects(Bucket='bucket-name')['Contents']:
print(key['Key'])'
the first 'for' block list all the keys in the bucket but the second 'for' block throws following error.
's3.Service Resource' object has no attribute 'list_objects' : AttributeError
It doesn't make sense based on http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_buckets . any hint on what could be the problem? i used python 2.7 as well as python 3.6
A boto3 service resource is not the same as the older boto library's service client. You are apparently mixing the documentation of the two.
A client is low level client, and just wraps the AWS APIs to python basic datatypes. All services have a client available in boto3.
Do check the documentation for a Resource and a Client.
Resources
Resources represent an object-oriented interface to Amazon Web
Services (AWS). They provide a higher-level abstraction than the raw,
low-level calls made by service clients.
resource = boto3.resource('s3')
Clients
Clients provide a low-level interface to AWS whose methods map close
to 1:1 with service APIs. All service operations are supported by
clients. Clients are generated from a JSON service definition file.
client = boto3.client('s3')
Related
I am making calls to API using python request library, and I am receiving the response in JSON. Currently I am saving JSON response on local computer, what I would like to do is to load JSON response directly to s3 bucket. The reason for loading to s3 bucket is my s3 bucket is acting as source to parse the JSON response for relational output. I was wondering how can I load JSON file directly to s3 bucket without using Access key or secret key ?
Most of my research on this topic lead to usingboto3 in python. Unfortunately, this library also requires key and ID. The reason for not using secret key and ID is because my organization has separate department which takes care of giving access to s3 bucket, and the department can only create IAM role with read and write access. I am curious what is the common industry practice of loading JSON in your organization ?
You can make unsigned requests for S3 through VPC Endpoint (VPCE), and don't need any AWS credentials this way.
# https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-privatelink.html
s3 = boto3.client('s3', config=Config(signature_version=UNSIGNED), endpoint_url="https://bucket.vpce-xxx-xxx.s3.ap-northeast-1.vpce.amazonaws.com")
You can restrict source ip by setting security group in VPC Endpoint to protect your S3 Bucket. Note that the owner of s3 objects uploaded by unsigned requests is anonymous, and may cause some side effects. In my case, Lifecycle rules cannot apply to those s3 objects.
I was asked to preform integration with an external google storage bucket, I had received a credentials json,
And while trying to do
gsutil ls gs://bucket_name (after configuring myself with the creds json) I had received a valid response, as well as when I tried to upload a file into the bucket.
When trying to do it with Python3, it does not work:
While using google-cloud-storage==1.16.0 (tried also the newer versions), I'm doing:
project_id = credentials_dict.get("project_id")
credentials = service_account.Credentials.from_service_account_info(credentials_dict)
client = storage.Client(credentials=credentials, project=project_id)
bucket = client.get_bucket(bucket_name)
But on the get_bucket line, I get:
google.api_core.exceptions.Forbidden: 403 GET https://www.googleapis.com/storage/v1/b/BUCKET_NAME?projection=noAcl: USERNAME#PROJECT_ID.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket.
The external partner which I'm integrating with, saying that the user is set correctly, and to prove it they're showing that I can preform the action with gsutil.
Can you please assist? Any idea what might be the problem?
The answer was that the creds were indeed wrong, but it did worked when I tried to preform on the client client.bucket(bucket_name) instead of client.get_bucket(bucket_name).
Please follow these steps in order to correctly set up the Cloud Storage Client Library for Python. In general, the Cloud Storage Libraries can use Application default credentials or environment variables for authentication.
Notice that the recommended method to use would be to set up authentication using environment variables (i.e if you are using Linux: export GOOGLE_APPLICATION_CREDENTIALS="/path/to/[service-account-credentials].json" should work) and avoid the use of the service_account.Credentials.from_service_account_info() method altogether:
from google.cloud import storage
storage_client = storage.Client(project='project-id-where-the-bucket-is')
bucket_name = "your-bucket"
bucket = client.get_bucket(bucket_name)
should simply work because the authentication is handled by the client library via the environment variable.
Now, if you are interested in explicitly using the service account instead of using service_account.Credentials.from_service_account_info() method you can use the from_service_account_json() method directly in the following way:
from google.cloud import storage
# Explicitly use service account credentials by specifying the private key
# file.
storage_client = storage.Client.from_service_account_json(
'/[service-account-credentials].json')
bucket_name = "your-bucket"
bucket = client.get_bucket(bucket_name)
Find all the relevant details as to how to provide credentials to your application here.
tl;dr: dont use client.get_bucket at all.
See for detailed explanation and solution https://stackoverflow.com/a/51452170/705745
I have been tasked with converting some bash scripting used by my team that performs various cloudformation tasks into Python using the boto3 library. I am currently stuck on one item. I cannot seem to determine how to do a wildcard type search where a cloud formation stack name contains a string.
My bash version using the AWS CLI is as follows:
aws cloudformation --region us-east-1 describe-stacks --query "Stacks[?contains(StackName,'myString')].StackName" --output json > stacks.out
This works on the cli, outputting the results to a json file, but I cannot find any examples online to do a similar search for contains using boto3 with Python. Is it possible?
Thanks!
Yes, it is possible. What you are looking for is the following:
import boto3
# create a boto3 client first
cloudformation = boto3.client('cloudformation', region_name='us-east-1')
# use client to make a particular API call
response = cloudformation.describe_stacks(StackName='myString')
print(response)
# as an aside, you'd need a different client to communicate
# with a different service
# ec2 = boto3.client('ec2', region_name='us-east-1')
# regions = ec2.describe_regions()
where, response is a Python dictionary, which, among other things, will contain the description of the stack, "myString".
I've been playing around with AWS using boto3 for a while now and interacting with multiple services such as ec2, s3, RDS etc. So, is there a way I can list all the private and public services available on an AWS account using boto3 code?
As far as I understand your question, you can use -
service_list = boto3.Session.get_available_services()
from https://boto3.readthedocs.io/en/latest/reference/core/session.html#boto3.session.Session.get_available_services based on the Boto 3 documentation version 1.7.62
In the boto3 document, it shows an example of how to migrate the connection from boto 2.x to boto3
# Boto 2.x
import boto
s3_connection = boto.connect_s3()
# Boto 3
import boto3
s3 = boto3.resource('s3')
However, in boto, it is possible to pass a parameter https_connection_factory. What is the equivalent in boto3?
There's not a direct equivalent. When creating a client or resource, you can make some very broad choices about SSL (use_ssl, verify). Both those can also take a botocore.config.Config object which can let you control timeouts and http pooling behavior among other options.
However, if you want full control of the ssl context, there doesn't appear to be any official support. Internally, boto is using a requests.Session to do all its work. You can see where the session is setup here. If you're okay with digging into botocore's internal implementation, you could reach into your resources/clients to mount a new custom Adapter for https:// paths as described in the requests user guide. The path to the http session object is <client>._endpoint.http_session or <resource>.meta.client._endpoint.http_session.