We Keep Coding

S3 presigned url: 403 error when reading a file but OK when downloading

I am working with a s3 presigned url.
OK: links works well to download.
NOT OK: using the presigned url to read a file in the bucket
I am getting the following error in console:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationQueryParametersError</Code><Message>Query-string authentication version 4 requires the X-Amz-Algorithm, X-Amz-Credential, X-Amz-Signature, X-Amz-Date, X-Amz-SignedHeaders, and X-Amz-Expires parameters.</Message>
and here is how generate the url with boto3
s3_client = boto3.client('s3', config=boto3.session.Config(signature_version='s3v4'), region_name='eu-west-3')
bucket_name = config("AWS_STORAGE_BUCKET_NAME")
file_name = '{}/{}/{}/{}'.format(user, 'projects', building.building_id, file.file)
ifc_url = s3_client.generate_presigned_url(
'get_object',
Params={
'Bucket': bucket_name,
'Key': file_name,
},
ExpiresIn=1799
)
I am using IFC.js which allows to load ifc formated models from their urls . Basically the url of the bucket acts as the path to the file. Accessing files in a public bucket has been working well however it won't work with private buckets.
Something to note as well is that using the presigned url copied from clipboard from the aws s3 interface works.
it looks like this:
"https://bucket.s3.eu-west-3.amazonaws.com/om1/projects/1/v1.ifc?response-content-disposition=inline&X-Amz-Security-Token=qqqqzdfffrA%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230105T224241Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=5%2Feu-west-3%2Fs3%2Faws4_request&X-Amz-Signature=c470c72b3abfb99"
the one I obtain with boto3 is the following:
"https://bucket.s3.amazonaws.com/om1/projects/1/v1.ifc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKI230105%2Feu-west-3%2Fs3%2Faws4_request&X-Amz-Date=20230105T223404Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=1b4f277b85639b408a9ee16e"
I am fairly new to use s3 buckets so I am not sure what is wrong here, and searching around on SO and online has not been very helpful so far. Could anyone point me in the right direction?

Signed URL for video upload causing problem in GCP

I am working on a project in which I need to upload the video files to GCS bucket using V4 Signed URL. Currently I am generating the signed url using Python script which is a part of Flask API. Here is the method signature I am using to generate url.
def GenerateURL(self,bucket_name,blob_name,method,timeout,content_type=None):
bucket = StoreCon.get_con(bucket_name)
blob = bucket.blob(blob_name)
url = blob.generate_signed_url(
version="v4",
expiration=datetime.timedelta(minutes=timeout),
method=method,
content_type=content_type,
)
resp = jsonify({'message':{'%s URL'%method:url}})
resp.status_code = 200
return resp
Now this is being called inside a blueprint route. Here is the snippet:
#CloudStoreEnd.route('/uploadMedia',methods=['POST'])
def uploadMedia():
blob_name = request.get_json()['FILE_NAME']
return StoreOperator.postMediaURL(blob_name)
When I make the call to this API route using Client side code, the video files are getting uploaded successfully to GCS bucket. But when I download the same video file from GCS bucket. The file becomes corrupted. Mentioning "0xc00d36c4" error.
Here is a sample function for client side:
def upload_file(path):
file_name = path.split('\\')[-1]
data = {'FILE_NAME':file_name}
#GET SIGNED URL FOR MEDIA UPLOAD
get_signed_url = 'https://CLOUD-RUN-SERVICE/uploadMedia'
headers = {'Content-Type':'application/json'}
resp = requests.post(url=get_signed_url,data=json.dumps(data),headers=headers)
upload_url = json.loads(resp.content)['message']['PUT URL']
#SEND A PUT REQUEST WITH MEDIA FILE
headers = {'Content-Type':MimeTypes().guess_type(file_name)[0]}
file = {'file':open(path,'rb')}
resp = requests.put(url=upload_url,headers=headers,files=file)
return resp
I am not sure why the Media(.mp4,.mov) are getting corrupted when I retrieve the same files, whereas for other files like (.pdf,.png) the files are fine. Is there an extra request parameter I need to add to get proper signed url? Or from client application I am sending the files wrong way to the signed url?

Boto3 not generating correct signed-url

I've a use case where I use lambda function to generate signed URL to upload to an S3 bucket, I also set the metadata values when generating signed URL, my boto3 version is boto3==1.18.35. Previously when I generate the signed-url to upload to the bucket the URL looks like this:
https://bucket-name.s3.amazonaws.com/scanned-file-list/cf389880-09ff-4301-8fa7-b4054941685b/6919de95-b795-4cac-a2d3-f88ed87a0d08.zip?AWSAccessKeyId=ASIAVK6XU35LOIUAABGC&Signature=xxxx%3D&content-type=application%2Fx-zip-compressed&x-amz-meta-scan_id=6919de95-b795-4cac-a2d3-f88ed87a0d08&x-amz-meta-collector_id=2e8672a1-72fd-41cc-99df-1ae3c581d31a&x-amz-security-token=xxxx&Expires=1641318176
But now the URL looks like this:
https://bucket-name.s3.amazonaws.com/scanned-file-list/f479e304-a2e4-47e7-b1c8-058e3012edac/3d349bab-c814-4aa7-b227-6ef86dd4b0a7.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA2BIILAZ55MATXAGA%2F20220105%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20220105T001950Z&X-Amz-Expires=36000&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-meta-collector_id%3Bx-amz-meta-scan_id&X-Amz-Security-Token=xxxxx&X-Amz-Signature=xxxx
Notice the URL it generates now does not have the correct value for metadata information i.e. x-amz-meta-collector_id and x-amz-meta-scan_id.
The I'm using to generate signed-url is:
bucket_name = os.environ['S3_UPLOADS_BUCKET_NAME']
metadata = {
'scan_id': scan_id,
'collector_id': collector_id
}
params = {
'Bucket': bucket_name,
'Key': path + file_obj['fileName'],
'ContentType': file_obj.get('contentType') or '',
'Metadata': metadata
}
logger.info('metadata used for generating URL: ' + str(metadata))
s3 = boto3.client('s3')
presigned_url = s3.generate_presigned_url('put_object', Params=params, ExpiresIn=36000)
logger.info(f'Presigned URL: {presigned_url}')
return presigned_url
Because of the change in the URL, I'm getting a SignatureDidNotMatch error, Thanks for the help in advance!

The problem is on the AWS servers, the URL generated from us-west-2 is different from the URL generated in ap-south-1.
More:
The signed-url generated from a lambda deployed in the ap-south-1 region, and the X-Amz-Signature-Version was automatically being added to the URL, but when I deploy the same lambda in a different region i.e. us-west-2, I get a different format of signed-url which in my case was the correct one!

python AWS boto3 create presigned url for file upload

I'm writing a django backend for an application in which the client will upload a video file to s3. I want to use presigned urls, so the django server will sign a url and pass it back to the client, who will then upload their video to s3. The problem is, the generate_presigned_url method does not seem to know about the s3 client upload_file method...
Following this example, I use the following code to generate the url for upload:
s3_client = boto3.client('s3')
try:
s3_object_name = str(uuid4()) + file_extension
params = {
"file_name": local_filename,
"bucket": settings.VIDEO_UPLOAD_BUCKET_NAME,
"object_name": s3_object_name,
}
response = s3_client.generate_presigned_url(ClientMethod="upload_file",
Params=params,
ExpiresIn=500)
except ClientError as e:
logging.error(e)
return HttpResponse(503, reason="Could not retrieve upload url.")
When running it I get the error:
File "/Users/bridgedudley/.local/share/virtualenvs/ShoMe/lib/python3.6/site-packages/botocore/signers.py", line 574, in generate_presigned_url
operation_name = self._PY_TO_OP_NAME[client_method]
KeyError: 'upload_file'
which triggers the exception:
botocore.exceptions.UnknownClientMethodError: Client does not have method: upload_file
Afer debugging I found that the self._PY_TO_OP_NAME dictionary only contains a subset of the s3 client commands offered here:
scrolling down to "upload"...
No upload_file method! I tried the same code using "list_buckets" and it worked perfectly, giving me a presigned url that listed the buckets under the signer's credentials.
So without the upload_file method available in the generate_presigned_url function, how can I achieve my desired functionality?
Thanks!

In addition to the already mentioned usage of:
boto3.client('s3').generate_presigned_url('put_object', Params={'Bucket':'your-bucket-name', 'Key':'your-object-name'})
You can also use:
boto3.client('s3').generate_presigned_post('your-bucket_name', 'your-object_name')
Reference: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html#generating-a-presigned-url-to-upload-a-file
Sample generation of URL:
import boto3
bucket_name = 'my-bucket'
key_name = 'any-name.txt'
s3_client = boto3.client('s3')
upload_details = s3_client.generate_presigned_post(bucket_name, key_name)
print(upload_details)
Output:
{'url': 'https://my-bucket.s3.amazonaws.com/', 'fields': {'key': 'any-name.txt', 'AWSAccessKeyId': 'QWERTYUOP123', 'x-amz-security-token': 'a1s2d3f4g5h6j7k8l9', 'policy': 'z0x9c8v7b6n5m4', 'signature': 'qaz123wsx456edc'}}
Sample uploading of file:
import requests
filename_to_upload = './some-file.txt'
with open(filename_to_upload, 'rb') as file_to_upload:
files = {'file': (filename_to_upload, file_to_upload)}
upload_response = requests.post(upload_details['url'], data=upload_details['fields'], files=files)
print(f"Upload response: {upload_response.status_code}")
Output:
Upload response: 204
Additional notes:
As documented:
The credentials used by the presigned URL are those of the AWS user
who generated the URL.
Thus, make sure that the entity that would execute this generation of a presigned URL allows the policy s3:PutObject to be able to upload a file to S3 using the signed URL. Once created, it can be configured through different ways. Some of them are:
As an allowed policy for a Lambda function
Or through boto3:
s3_client = boto3.client('s3',
aws_access_key_id="your-access-key-id",
aws_secret_access_key="your-secret-access-key",
aws_session_token="your-session-token", # Only for credentials that has it
)
Or on the working environment:
# Run in the Linux environment
export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_SESSION_TOKEN="your-session-token", # Only for credentials that has it
Or through libraries e.g. django-storages for Django

You should be able to use the put_object method here. It is a pure client object, rather than a meta client object like upload_file. That is the reason that upload_file is not appearing in client._PY_TO_OP_NAME. The two functions do take different inputs, which may necessitate a slight refactor in your code.
put_object: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.put_object

The accepted answer doesn't let you post your data to S3 from your client. This will:
import boto3
s3_client = boto3.client('s3',
aws_access_key_id="AKIA....",
aws_secret_access_key="G789...",
)
s3_client.generate_presigned_url('put_object', Params={
'Bucket': 'cat-pictures',
'Key': 'whiskers.png',
'ContentType': 'image/png', # required!
})
Send that to your front end, then in JavaScript on the frontend:
fetch(url, {
method: "PUT",
body: file,
})
Where file is a File object.

How to upload files >10MB with resumable uploads to Google Bucket using Python

I have an app that successfully uploads videos less than 10MB to a specified Google Cloud Storage Bucket, however, it does not do so is the file is greater than 10MB.
I would like to know how to do this with resumable uploads. I do not want any links to documentation, I have read all of the documentation. It does not help.
I am doing this in python. I understand for me to do a resumable upload I need my authorization token and I have no idea how to do that. So could some one help me with some code?
Here is the code that post Videos < 10MB
class GetData(webapp2.RequestHandler):
def post(self):
data = self.request.get('file')
bucketname = self.request.get('bucketname')
typeOf = self.request.get('content_type')
sender = self.request.get('sender')
profileVideo = self.request.get('isProfileVideo')
file_path = ''
now = time.time()
objectname = now
try:
keytext = open(conf.PRIVATE_KEY_PATH, 'rb').read()
except IOError as e:
sys.exit('Error while reading private key: %s' % e)
private_key = RSA.importKey(keytext)
signer = CloudStorageURLSigner(private_key, conf.SERVICE_ACCOUNT_EMAIL,
GCS_API_ENDPOINT)
subDirectory = 'videoMesseagesFrom' + sender
if profileVideo == 'true':
file_path = '/%s/Profile_Videos/%s' % (bucketname, objectname)
r = signer.Put(file_path, typeOf, data)
else:
file_path = '/%s/%s/%s' % (bucketname, subDirectory, objectname)
r = signer.Put(file_path, typeOf, data)
self.response.headers['Content-Type'] = 'application/json'
obj = {
'Completion' : 'Video Successfully uploaded'
}
self.response.out.write(json.dumps(obj))
I pass in the file and the bucket name. This grabs my credentials from my Google generated key that is in the project and signs a PUT url, then PUTs it for me.
def Put(self, path, content_type, data):
"""Performs a PUT request.
Args:
path: The relative API path to access, e.g. '/bucket/object'.
content_type: The content type to assign to the upload.
data: The file data to upload to the new file.
Returns:
An instance of requests.Response containing the HTTP response.
"""
md5_digest = base64.b64encode(md5.new(data).digest())
base_url, query_params = self._MakeUrl('PUT', path, content_type,
md5_digest)
headers = {}
headers['Content-Type'] = content_type
headers['Content-Length'] = str(len(data))
headers['Content-MD5'] = md5_digest
return self.session.put(base_url, params=query_params, headers=headers,
data=data)
It is already making a put request with Content-Type headers and all that. Maybe I could put a check on this, so if the file is larger than 10MB somehow edit that PUT method to be a resumable upload, or upload it in chunks. However, I have no idea how to do that.