My code is successfully uploading documents into the correct bucket. I can login and see the docs in the buckets on AWS S3. When I try to use the generate_signed_url method in boto3 to obtain a URL for these documents that can then be sent to users for accessing the doc, I'm getting a SignatureDoesNotMatch with the message saying "The request signature we calculated does not match the signature you provided. Check your key and signing method."
When i save the object (verified that it is working correctly by logging into AWS and downloading the files manually), I use:
s3 = boto3.client('s3', region_name='us-east-2', aws_access_key_id='XXXX', aws_secret_access_key='XXXX', config=Config(signature_version='s3v4'))
s3.put_object(Bucket=self.bucket_name,Key=path, Body=temp_file.getvalue(),ACL='public-read')
Then, when I try to get the URL, I'm using:
s3 = boto3.client('s3', region_name='us-east-2', aws_access_key_id='XXXX', aws_secret_access_key='XXXX', config=Config(signature_version='s3v4'))
url = s3.generate_presigned_url(
'put_object', Params={
'Bucket':self.pdffile_storage_bucket_name,
'Key':self.pdffile_url
},
ExpiresIn=604799,
)
I saw quite a bit of users on the web talking about making sure your AWS access key doesn't include any special characters. I did this, and I still get the same issue.
You are generating a pre-signed URL for an upload, not a download. You don't want put_object here... it's get_object.
Also, as #ThijsvanDien pointed out, ACL='public-read' may not be what you want, when uploading -- this makes the object accessible to anyone with an unsigned URL.
I tried many solutions from many different places. None of the worked for me. Then i got a solution on GitHub, that worked for me. Here is the original solution, just copy pasting his solution.
s3 = boto3.client('s3', region_name='us-east-2',
aws_access_key_id='XXXX', aws_secret_access_key='XXXX',
config=Config(signature_version='s3v4'),
endpoint_url='https://s3.us-east-2.amazonaws.com')
(after 3+ hours of debugging and almost smashing the keyboard....)
in the response, it's telling you which header is missing:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<!-- .... -->
<CanonicalRequest>PUT (your pre-signed url)
content-type:image/jpeg
host:s3.eu-west-2.amazonaws.com
x-amz-acl:
content-type;host;x-amz-acl
UNSIGNED-PAYLOAD</CanonicalRequest>
Needed a x-amz-acl header matching the ACL set when generating the pre-signed URL
def python_presign_url():
return s3.generate_presigned_url('put_object', Params={
'Bucket': bucket_name,
'Key': filename,
'ContentType': type,
'ACL':'public-read' # your x-amz-acl
})
curl -X PUT \
-H "content-type: image/jpeg" \
-H "Host: s3.eu-west-2.amazonaws.com" \
-H "x-amz-acl: public-read" \
-d #/path/to/upload/file.jpg "$PRE_SIGNED_URL"
I had a trailing space on my secret access key variable. That was causing the issue. E.g:
AWS_SECRET_KEY = "xyz "
Related
I am working with a s3 presigned url.
OK: links works well to download.
NOT OK: using the presigned url to read a file in the bucket
I am getting the following error in console:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationQueryParametersError</Code><Message>Query-string authentication version 4 requires the X-Amz-Algorithm, X-Amz-Credential, X-Amz-Signature, X-Amz-Date, X-Amz-SignedHeaders, and X-Amz-Expires parameters.</Message>
and here is how generate the url with boto3
s3_client = boto3.client('s3', config=boto3.session.Config(signature_version='s3v4'), region_name='eu-west-3')
bucket_name = config("AWS_STORAGE_BUCKET_NAME")
file_name = '{}/{}/{}/{}'.format(user, 'projects', building.building_id, file.file)
ifc_url = s3_client.generate_presigned_url(
'get_object',
Params={
'Bucket': bucket_name,
'Key': file_name,
},
ExpiresIn=1799
)
I am using IFC.js which allows to load ifc formated models from their urls . Basically the url of the bucket acts as the path to the file. Accessing files in a public bucket has been working well however it won't work with private buckets.
Something to note as well is that using the presigned url copied from clipboard from the aws s3 interface works.
it looks like this:
"https://bucket.s3.eu-west-3.amazonaws.com/om1/projects/1/v1.ifc?response-content-disposition=inline&X-Amz-Security-Token=qqqqzdfffrA%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230105T224241Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=5%2Feu-west-3%2Fs3%2Faws4_request&X-Amz-Signature=c470c72b3abfb99"
the one I obtain with boto3 is the following:
"https://bucket.s3.amazonaws.com/om1/projects/1/v1.ifc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKI230105%2Feu-west-3%2Fs3%2Faws4_request&X-Amz-Date=20230105T223404Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=1b4f277b85639b408a9ee16e"
I am fairly new to use s3 buckets so I am not sure what is wrong here, and searching around on SO and online has not been very helpful so far. Could anyone point me in the right direction?
I have this existing piece of code that is used to upload files to my s3 bucket.
def get_user_upload_url(customer_id, filename, content_type):
s3_client = boto3.client('s3')
object_name = "userfiles/uploads/{}/{}".format(customer_id, filename)
try:
url = s3_client.generate_presigned_url('put_object',
Params={'Bucket': BUCKET,
'Key': object_name,
"ContentType": content_type # set to "image/png"
},
ExpiresIn=100)
except Exception as e:
print(e)
return None
return url
This returns to my client a presigned URL that I use to upload my files without a issue. I have added a new use of it where I'm uploading a png and I have behave test that uploads to the presigned url just fine. The problem is if i go look at the file in s3 i cant preview it. If I download it, it wont open either. The s3 web client shows it has Content-Type image/png. I visual compared the binary of the original file and the downloaded file and i can see differences. A file type tool detects that its is an octet-stream.
signature_file_name = "signature.png"
with open("features/steps/{}".format(signature_file_name), 'rb') as f:
files = {'file': (signature_file_name, f)}
headers = {
'Content-Type': "image/png" # without this or with a different value the presigned url will error with a signatureDoesNotMatch
}
context.upload_signature_response = requests.put(response, files=files, headers=headers)
I would have expected to have been returned a PNG instead of an octet stream however I'm not sure what I have done wrong . Googling this generally results in people having a problem with the signature because there not properly setting or passing the content type and I feel like I've effectively done that here proven by the fact that if I change the content type everything fails . I'm guessing that there's something wrong with the way I'm uploading the file or maybe reading the file for the upload?
So it is todo with how im uploading. So instead it works if i upload like this.
context.upload_signature_response = requests.put(response, data=open("features/steps/{}".format(signature_file_name), 'rb'), headers=headers)
So this must have to do with the use of put_object. It must be expecting the body to be the file of the defined content type. This method accomplishes that where the prior one would make it a multi part upload. So I think it's safe to say the multipart upload is not compatible with a presigned URL for put_object.
Im still piecing it altogether, so feel free to fill in the blanks.
I want to return an image from a Chalice/python application. My entire application code is pasted below:
from chalice import Chalice, Response
import base64
app = Chalice(app_name='hello')
#app.route('/makeImage', methods=['GET'])
def makeImage():
return Response(
base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg=="
),
headers={
'Content-Type': 'image/jpeg'
},
status_code=200)
The result...
{"Code":"BadRequest","Message":"Request did not specify an Accept
header with image/jpeg, The response has a Content-Type of image/jpeg.
If a response has a binary Content-Type then the request must specify
an Accept header that matches."}
Why does this happen?
I have poured through a ton of documentation already and most of it is outdated as binary support was added to Chalice very recently:
https://github.com/aws/chalice/pull/352
https://github.com/aws/chalice/issues/592
https://github.com/aws/chalice/issues/348
AWS Chalice Return an Image File from S3 (Warning: the sole answer to this question is COMPLETELY WRONG)
https://chalice.readthedocs.io/en/latest/api.html
https://github.com/aws/chalice/issues/391 (issue WRONGLY CLOSED in 2017 without a resolution)
https://github.com/aws/chalice/issues/1095 is a re-open of 391 above
Just for troubleshooting purposes I'm able to obtain a response by using curl -H "accept: image/jpeg", but this is useless since browsers to not work this way, and I need to use the response in a browser (HTML IMG TAG).
UPDATE
I also tried #app.route('/makeImage', methods=['GET'], content_types=['image/jpeg'])
And result became {"Code":"UnsupportedMediaType","Message":"Unsupported media type: application/json"}
I had the same issue.
If no header accept is present, AWS set it to default application/json and I receive a base64 response. If I set accept to images/jpeg or any binary content type in header, then I got the images. Great but web browser to not set the accept header.
But if I add
app.api.binary_types =['*/*']
then ok my images apis now works. Great but now the json ones fail.
Currently I do not see any solution except having two API gateway : one for json and one for images. If you really want only one API Gateway, I think you have to use gzip conpression on all you json response to convert them to binaries.
It is more how AWS API Gateway works with lambda proxy than a Chalice issue. But I agree, it is a big limitation
There was a bug in Chalice that was fixed on 14-May-2019 and documented here:
https://github.com/aws/chalice/issues/1095
In addition to installing the latest Chalice directly from GitHub, I also had to add:
app.api.binary_types =['*/*']
in app.py.
The final working code looks like this:
from chalice import Chalice, Response
import base64
app = Chalice(app_name='hello')
app.api.binary_types =['*/*']
#app.route('/makeImage', methods=['GET'])
def makeImage():
return Response(
base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg=="
),
headers={
'Content-Type': 'image/jpeg'
},
status_code=200)
I am trying to use AWS DynamoDB in a Flutter app, and given the lack of an official AWS SDK for Dart I am forced to use the low level HTTP REST API.
The method for signing an AWS HTTP request is quite tedious, but using an AWS supplied sample as a guide, I was able to convert the Python to Dart pretty much line-for-line relatively easily. The end result was both sets of code producing the same auth signatures.
My issue came when I actually went to sent the request. The Python works as expected but sending a POST with Dart's HTTP package gives the error
The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method. Consult
the service documentation for details.
I'll spare you the actual code for generating the auth signature, as the issue can be replicated simply by sending the same request hard-coded. See the Python and Dart code below.
Note: A valid response will return
Signature expired: 20190307T214900Z is now earlier than
20190307T215809Z (20190307T221309Z - 15 min.)
as the request signature uses current date and is only valid for 15 mins.
*****PYTHON CODE*****
import requests
headers = {'Content-Type':'application/json',
'X-Amz-Date':'20190307T214900Z',
'X-Amz-Target':'DynamoDB_20120810.GetItem',
'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'}
endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/'
request_parameters = '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}'
r = requests.post(endpoint, data=request_parameters, headers=headers)
print('Response status: %d\n' % r.status_code)
print('Response body: %s\n' % r.text)
*****DART CODE*****
import 'package:http/http.dart' as http;
void main(List<String> arguments) async {
var headers = {'Content-Type':'application/json',
'X-Amz-Date':'20190307T214900Z',
'X-Amz-Target':'DynamoDB_20120810.GetItem',
'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'};
var endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/';
var request_parameters = '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}';
http.post(endpoint, body: request_parameters, headers: headers).then((response) {
print("Response status: ${response.statusCode}");
print("Response body: ${response.body}");
});
}
The endpoint, headers and body are literally copy and pasted between the two sets of code.
Is there some nuance to how Dart HTTP works that I am missing here? Is there some map/string/json conversion of the headers or request_paramaters happening?
One thing I did note is that in the AWS provided example it states
For DynamoDB, the request can include any headers, but MUST include
"host", "x-amz-date", "x-amz-target", "content-type", and
"Authorization". Except for the authorization header, the headers must
be included in the canonical_headers and signed_headers values, as
noted earlier. Order here is not significant. Python note: The 'host'
header is added automatically by the Python 'requests' library.
But
a) When I add 'Host':'dynamodb.ap-southeast-2.amazonaws.com' to the headers in the Dart code I get the same result
and
b) If I look at r.request.headers after the Python requests returns, I can see that it has added a few new headers (Content-Length etc) automatically, but "Host" isn't one of them.
Any ideas why the seemingly same HTTP request works for Python Requests but not Dart HTTP?
Ok this is resolved now. My issue was in part a massive user-error. I was using a new IDE and when I generated the hardcoded example I provided I was actually still executing the previous file. Stupid, stupid, stupid.
But...
I was able to sort out the actual issue that caused me raise the question in the first place. I found that if you set the content type to "application/json" in the headers, the dart HTTP package automatically appends "; charset=utf-8". Because this value is part of the auth signature, when AWS encodes the values from the header to compare to the user-generated signature, they don't match.
The fix is simply to ensure that when you are setting the header content-type, make sure that you manually set it to "application/json; charset=utf-8" and not "application/json".
Found a bit more discussion about this "bug" after the fact here.
Following instructions on this link using Lambda and API Gateway: https://sookocheff.com/post/api/uploading-large-payloads-through-api-gateway/ I have a setup that allows me to get a pre-signed URL and upload files. I've tested using CURL and it has worked.
But when I try to send raw string (csv format or json format) it fails!
Example of what works
curl --request PUT --upload-file Testing.csv "**pre signed upload url**"
Example of what doesn't work
curl --request PUT -H "Content-Type: text/plain" --data "this is raw data" "**pre signed upload url**"
curl --request PUT --data "this is raw data" "**pre signed upload url**"
Am I making the call incorrectly? Should I be switching to POST and what would the call look like then?
It is not becoz of self signed url, it is becoz of content type with the API Gateway set to,
consumes:
- application/json
produces:
- application/json
You add additional content types, it should make it through.
Hope it helps.
So the solution was specifying the content-type during the pre-signed url generation and then the same one in the CURL put command. Figured out thanks to answer here: S3 PUT doesn't work with pre-signed URL in javascript and pointer from #Kannaiyan in the right direction regarding content-types