In the boto3 document, it shows an example of how to migrate the connection from boto 2.x to boto3
# Boto 2.x
import boto
s3_connection = boto.connect_s3()
# Boto 3
import boto3
s3 = boto3.resource('s3')
However, in boto, it is possible to pass a parameter https_connection_factory. What is the equivalent in boto3?
There's not a direct equivalent. When creating a client or resource, you can make some very broad choices about SSL (use_ssl, verify). Both those can also take a botocore.config.Config object which can let you control timeouts and http pooling behavior among other options.
However, if you want full control of the ssl context, there doesn't appear to be any official support. Internally, boto is using a requests.Session to do all its work. You can see where the session is setup here. If you're okay with digging into botocore's internal implementation, you could reach into your resources/clients to mount a new custom Adapter for https:// paths as described in the requests user guide. The path to the http session object is <client>._endpoint.http_session or <resource>.meta.client._endpoint.http_session.
Related
By default boto3 (AWS Python SDK) implements an incremental back-oof retry strategy for (all?) clients. That can be customized via the retries.max_attempts entry at the Botocore Config. That works pretty well for me in many scenarios. But I have no traces about how many attempts have been actually required, besides when you can notice them in the client latencies.
So, is there any way to consistently get the number of retries used after a successful request to a boto3 client?
Inspecting the code it looks like handler.context.attempt_number stored that, but I have no idea how to reach that after a successful invocation to a client.
It looks like the ResponseMetada from every client comes with that information :-)
Minimal code example:
import logging
import boto3
client = boto3.client('s3')
response = client.list_buckets()
logging.info("Retry Attempts: %d", response['ResponseMetadata']['RetryAttempts'])
I can create a bucket using these parameters. But none of them is a custom header. It's also said that boto3 will not support it because S3 does not currently allow setting arbitrary headers on buckets or objects.
But in my case. I am using Cloudian as storage. It supports x-gmt-policyid this policy determines how data in the bucket will be distributed and protected through either replication or erasure coding.
Any idea how to inject custom header to boto bucket creation?
s3_resource.create_bucket(Bucket='foo-1')
My last two options:
1) to fork botocore and add this functionality, but I saw they use loaders.py that read everything from json file, and it seems a bit complicated for a beginner.
2) or maybe I need to use pure python implementation using request module to create s3 bucket.
Thanks for suggestions.
My current solution is to fetch S3 compatible cloudian API directly. Signing the request is very complicated, so I use the help of requests-aws4auth library. I tried other libs but failed.
example to create bucket with clodian x-gmt-policyid value:
import requests
from requests_aws4auth import AWS4Auth
endpoint = "http://awesome-bucket.my-s3.net"
auth = AWS4Auth(
"00ac60d1a669fakekey",
"S2/x9sRvb1Jys9n+fakekey",
"eu-west-1",
"s3",
)
headers = {
"x-gmt-policyid": "9f934425b7f5de611c32fakeid",
"x-amz-acl": "public-read",
}
response = requests.put(endpoint, auth=auth, headers=headers)
print(response.text)
Code in lambda:
import boto3
def lambda_handler(event, context):
s3_client = boto3.resource('s3')
mybucket = s3_client.Bucket('bucket-name')
for object in mybucket.objects.all():
print(object)
for key in s3_client.list_objects(Bucket='bucket-name')['Contents']:
print(key['Key'])'
the first 'for' block list all the keys in the bucket but the second 'for' block throws following error.
's3.Service Resource' object has no attribute 'list_objects' : AttributeError
It doesn't make sense based on http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_buckets . any hint on what could be the problem? i used python 2.7 as well as python 3.6
A boto3 service resource is not the same as the older boto library's service client. You are apparently mixing the documentation of the two.
A client is low level client, and just wraps the AWS APIs to python basic datatypes. All services have a client available in boto3.
Do check the documentation for a Resource and a Client.
Resources
Resources represent an object-oriented interface to Amazon Web
Services (AWS). They provide a higher-level abstraction than the raw,
low-level calls made by service clients.
resource = boto3.resource('s3')
Clients
Clients provide a low-level interface to AWS whose methods map close
to 1:1 with service APIs. All service operations are supported by
clients. Clients are generated from a JSON service definition file.
client = boto3.client('s3')
When I'm using the boto3 SDK for Python, is it better to have a single client object, like this:
client = boto3.client('s3')
# use client through the file
and then use that everywhere, or should I instantiate it as needed, like this:
size = client('s3').head_object(Bucket=bucket, Key=key)['ContentLength']
Which is better? Does it make a different?
I don't see any harm using single client object through the file for a particular AWS service. As boto is widely used standard SDK, it won't be changed drastically and even if they make backwards incompatible change, they announce it. So it won't affect your application while running some process.
Can anyone help me with connecting to amazon s3. I have next problem:
I want to check whether provided credential are good to connect on step when just creating connection. For example:
import boto
boto.connect_s3(access_key,wrong_secret_key)
Raise error on this step when providing bad key.
I know we can catch Error on step when connecting to specific bucket, but I want to catch on previous step.
Thanks.
The connect_s3 function does not actually make any requests to S3. It simply creates and configures the S3Connection object and returns it. If you want to validate the credentials you will have to perform some S3 operation with the connection. For example, you could try to do a get_bucket:
import boto
s3 = boto.connect_s3(...)
s3.get_bucket('mybucket')
This will actually make a round trip to the S3 service and, if there is a problem with your credentials, you will see an error.