I am trying to detect faces in an image using AWS Image Rekognition API. But getting the following Error:
Error1:
ClientError: An error occurred (InvalidS3ObjectException) when calling the DetectFaces operation: Unable to get image metadata from S3. Check object key, region and/or access permissions.
Python Code1:
def detect_faces(object_name="path/to/image/001.jpg"):
client = get_aws_client('rekognition')
response = client.detect_faces(
Image={
# 'Bytes': source_bytes,
'S3Object': {
'Bucket': "bucket-name",
'Name': object_name,
'Version': 'string'
}
},
Attributes=[
'ALL',
]
)
return response
The Object "path/to/image/001.jpg" exists in the AWS S3 Bucket "bucket-name". And the region Name is also correct.
The Permissions for this object '001.jpg' is: Everyone is granted Open/Download/view Permission.
MetaData for the Object: Content-Type: image/jpeg
Not sure how to debug this. Any Suggestion to resolve this please ?
Thanks,
You appear to be asking the service to fetch the object with a version id of string.
Version
If the bucket is versioning enabled, you can specify the object version.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 1024.
Required: No
http://docs.aws.amazon.com/rekognition/latest/dg/API_S3Object.html#rekognition-Type-S3Object-Version
Remove 'Version': 'string' from your request parameters, unless you really intend to fetch a specific version of the object from a versioned bucket, in which case, provide the actual version id of the object in question.
I had the same issue and avoid using '-' or whitespaces in bucket and uploaded file names solved it for me.
Maybe removing underscores can also help.
Old thread, new solution:
I got the same error message.
My mistake was due to a region mismatch; My S3 bucket was residing in us-east-2, but my recognition client defaulted to us-west-1.
I changed the line
client = get_aws_client('rekognition')
to
client = get_aws_client('rekognition', region_name='us-east-2')
and it worked.
Related
I am trying to change the metadata of an image in an s3 bucket through lambda, this lambda triggers when an object is uploaded. But for some reason when I update the metadata through copy_from it adds user metadata instead of the System Metadata like this:
Is there a special way to edit the system metadata? My code is:
import json
import boto3
import urllib
s3 = boto3.resource('s3')
def lambda_handler(event, context):
# TODO implement
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])
s3_object = s3.Object(bucket, key)
s3_object.metadata.update({'Content-Type':'image/png'})
s3_object.copy_from(CopySource={'Bucket':bucket, 'Key':key}, Metadata=s3_object.metadata, MetadataDirective='REPLACE')
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
Content-Type is special metadata categorised as system defined metadata, and there are other means to update it. It is derived based on the contents of the object when it is created/uploaded.
Let's say you want to update System defined Content-Type metadata. Try with this code, which updates System defined metadata and also adds a user defined metadata:
s3_object.metadata.update({'My-Metadata':'abc'})
s3_object.copy_from(CopySource={'Bucket':BUCKET_NAME, 'Key':OBJECT_KEY}, ContentType='image/png', Metadata=s3_object.metadata, MetadataDirective='REPLACE')
As you see here, copy_from takes parameter ContentType explicitly to update content-type. One does not need to use metadata json to update this parameter. Use metadata json to update other user defined parameters.
The are any solutions to the problem but m not able to find the root cause and also to mention no of the solution is working in my case.
What I am trying to do is to upload a file to Django rest API from a angular Http Client Service.
I am taking up the input form the user passing to the service at the end I have declared a type specific for the type of model I am creating an object for but I am not getting the same error again and again.
I read some where that Django file uploader doesn't understand some char-set format but I still can't understand whats wrong with it.
var report = <CompleteReport> {
title:'heelo',
description:'sdfsdfs',
author: 'report',
article_upload_images: [uuid4(),],
presentation_upload_images: [uuid4(),],
report_article: article_object,
report_image: image_object,
report_podcast: podcast_object,
report_presentation: presentation_object,
report_video: video_object
};
let headers = new HttpHeaders({
'Accept': 'application/json'
});
let options = {headers: headers};
return this.http.post<any>(url, report, options)
I have been trying to get my Python notebook in Watson Studio to connect to and retrieve data from my cloud storage for hours and read many tutorials but no idea why it is not working, code below:
credentials = {
'BUCKET': 'openai-data',
'URL': 'https://s3.private.us-east.cloud-object-storage.appdomain.cloud',
'SECRET_KEY': '',
'API_KEY': '*********************', #this key has been *d out but the value is there in my code
'RESOURCE_INSTANCE_ID': 'crn:v1:bluemix:public:cloud-object-storage:global:a/e448d36ef93f4d3ca55077db903d3461:51ce6e50-4e92-41d0-b002-5023e815cadc::',
'FILE': 'test_Y_smallsample.h5',
'ACCESS_KEY': ''
}
from ibm_botocore.client import Config
import ibm_boto3
cos = ibm_boto3.resource(service_name='s3',
ibm_api_key_id=credentials['API_KEY'],
ibm_service_instance_id=credentials['RESOURCE_INSTANCE_ID'],
ibm_auth_endpoint='https://iam.bluemix.net/oidc/token',
config=Config(signature_version='oauth'),
endpoint_url=credentials['URL'])
files = cos.Bucket('openai-data').objects.all()
for file in files:
print("Item: {0} ({1} bytes).".format(file.key, file.size))
this produces the error:
CredentialRetrievalError: Error when retrieving credentials from https://iam.bluemix.net/oidc/token: HttpCode(400) - Retrieval of tokens from server failed.
same result if I use "https://iam.cloud.ibm.com/identity/token" for the ibm_auth_endpoint
I've also tried a separate connection with HMAC credentials as well, but can't find any tutorials that show you how to incorporate those either...
Please help!
Thanks
This can be caused due to the use of invalid apikey. To get an apikey value, go to the 'Service credentials' menu of your storage instance and then click on "View credentials" of the "WDP-Project-Management-..." and see the value in the 'apikey' field.
This issue does not seem to be caused by invalid endpoint_url, bu anyway, to get the endpoint_url, go to Buckets > 'your bucket' > Configuration, and then choose an endpoint according to you needs (Private/Public/Direct). When filling the field, use "https://".
See below a code snippet to get data from a cvs file directly:
import pandas as pd
import ibm_boto3
from ibm_botocore.client import Config
cos = ibm_boto3.client(service_name='s3',
ibm_api_key_id='<apikey>',
ibm_auth_endpoint="https://iam.ng.bluemix.net/oidc/token",
config=Config(signature_version='oauth'),
endpoint_url="<endpoint>")
obj = cos.get_object(Bucket='<bucket_name>', Key='<csv_file>')
df = pd.read_csv(obj['Body'])
I have a s3 link provided to me by a third-party with the following structure: http://s3.amazonaws.com/bucket_name_possibly/path/to/file_possibly/filename?AWSAccessKeyId=SomeKey&Expires=888888&Signature=SomeCharactersPossiblyHTMLencoded
Clicking on the link downloads the file for me. However, in python when I try to use urllib.request.urlretrieve(link_string) on the link I get the error HTTP Error 403: Forbidden
I have also tried using boto3 and manually parsing out the bucket_name, key, AWSAccessKeyID as well as the signature(treating it as the AWSSecretAccessKey - I know that this is probably wrong). I setup a client with the credentials and try to run a get_object method. Something similar to below:
client= boto3.client(
's3',
aws_access_key_id='AWSACCESSKEY',
aws_secret_access_key='SomeCharactersPossiblyHTMLencoded',
config=Config(signature_version='s3v4') # tried with/without this option
)
client.get_object(
Bucket='bucket_name_possibly',
Key='path/to/file_possibly/filename'
)
The resulting error is An error occurred (SignatureDoesNotMatch) when calling the GetObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I am stuck, how can I get python to programmatically download the link?
You can use boto to download file as follows.
import boto3
import botocore
BUCKET_NAME = 'my-bucket' # replace with your bucket name
KEY = 'my_image_in_s3.jpg' # replace with your object key
s3 = boto3.resource('s3')
try:
s3.Bucket(BUCKET_NAME).download_file(KEY, 'my_local_image.jpg')
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == "404":
print("The object does not exist.")
else:
raise
for more info you can refer this
Running a line like:
s3_obj = boto3.resource('s3').Object(bucket, key)
s3_obj.meta.client.generate_presigned_url('get_object', ExpiresIn=0, Params={'Bucket':bucket,'Key':key})
Yields a result like:
https://my-bucket.s3.amazonaws.com/my-key/my-object-name?AWSAccessKeyId=SOMEKEY&Expires=SOMENUMBER&x-amz-security-token=SOMETOKEN
For an s3 object with public-read ACL, all the GET params are unnecessary.
I could cheat and use rewrite the URL without the GET params but that feels unclean and hacky.
How do I use boto3 to provide me with just the public link, e.g. https://my-bucket.s3.amazonaws.com/my-key/my-object-name? In other words, how do I skip the signing step in generate_presigned_url? I don't see anything like a generated_unsigned_url function.
The best solution I found is still to use the generate_presigned_url, just that the Client.Config.signature_version needs to be set to botocore.UNSIGNED.
The following returns the public link without the signing stuff.
config.signature_version = botocore.UNSIGNED
boto3.client('s3', config=config).generate_presigned_url('get_object', ExpiresIn=0, Params={'Bucket': bucket, 'Key': key})
The relevant discussions on the boto3 repository are:
https://github.com/boto/boto3/issues/110
https://github.com/boto/boto3/issues/169
https://github.com/boto/boto3/issues/1415