I tried to write a python lambda function that returns a pre-signed url to put an object.
import os
import boto3
import json
import boto3
session = boto3.Session(region_name=os.environ['AWS_REGION'])
s3 = session.client('s3', region_name=os.environ['AWS_REGION'])
upload_bucket = 'BUCKER_NAME' # Replace this value with your bucket name!
URL_EXPIRATION_SECONDS = 30000 # Specify how long the pre-signed URL will be valid for
# Main Lambda entry point
def lambda_handler(event, context):
return get_upload_url(event)
def get_upload_url(event):
key = 'testimage.jpg' # Random filename we will use when uploading files
# Get signed URL from S3
s3_params = {
'Bucket': upload_bucket,
'Key': key,
'Expires': URL_EXPIRATION_SECONDS,
'ContentType': 'image/jpeg' # Change this to the media type of the files you want to upload
}
# Get signed URL
upload_url = s3.generate_presigned_url(
'put_object',
Params=s3_params,
ExpiresIn=URL_EXPIRATION_SECONDS
)
return {
'statusCode': 200,
'isBase64Encoded': False,
'headers': {
'Access-Control-Allow-Origin': '*'
},
'body': json.dumps(upload_url)
}
The code itself works and returns a signed URL in the format "https://BUCKET_NAME.s3.amazonaws.com/testimage.jpg?[...]"
However when using POSTMAN to try to put an object, it loads without ending.
Originally I thought it was because of my code, and after a while I wrote a NodeJS function that does the same thing:
const AWS = require('aws-sdk')
AWS.config.update({ region: process.env.AWS_REGION })
const s3 = new AWS.S3()
const uploadBucket = 'BUCKET_NAME' // Replace this value with your bucket name!
const URL_EXPIRATION_SECONDS = 30000 // Specify how long the pre-signed URL will be valid for
// Main Lambda entry point
exports.handler = async (event) => {
return await getUploadURL(event)
}
const getUploadURL = async function(event) {
const randomID = parseInt(Math.random() * 10000000)
const Key = 'testimage.jpg' // Random filename we will use when uploading files
// Get signed URL from S3
const s3Params = {
Bucket: uploadBucket,
Key,
Expires: URL_EXPIRATION_SECONDS,
ContentType: 'image/jpeg' // Change this to the media type of the files you want to upload
}
return new Promise((resolve, reject) => {
// Get signed URL
let uploadURL = s3.getSignedUrl('putObject', s3Params)
resolve({
"statusCode": 200,
"isBase64Encoded": false,
"headers": {
"Access-Control-Allow-Origin": "*"
},
"body": JSON.stringify(uploadURL)
})
})
}
The NodeJs version gives me a url in the format of "https://BUCKET_NAME.s3.eu-west-1.amazonaws.com/testimage.jpg?"
The main difference between the two is the aws sub domain. When using NodeJS it gives me "BUCKET_NAME.s3.eu-west-1.amazonaws.com" and when using Python "https://BUCKET_NAME.s3.amazonaws.com"
When using python the region does not appear.
I tried, using the signed url generated in python to add the "s3.eu-west-1" manually and IT Works!!
Is this a bug in the AWS Boto3 python library?
as you can see, in the python code I tried to specify the region but it does not do anything.?
Any idea guys ?
I wanna solve this mystery :)
Thanks a lot in advance,
Léo
I was able to reproduce the issue in us-east-1. There are a few bugs in Github (e.g., this and this), but the proposed resolutions are inconsistent.
The workaround is to create an Internet-facing access point for the bucket and then assign the full ARN of the access point to your upload_bucket variable.
Please note that the Lambda will create a pre-signed URL, but it will only work if the Lambda has an appropriate permissions policy attached to its execution role.
I created a S3 bucket to store incoming data.
The flow looked like this.
Whenever a message came to messengerpeople or a heyflow was submitted by a customer a webhook was triggered and the data was sent to my Lambda-Url. There I had the following code
const AWS = require( 'aws-sdk' );
const firehose = new AWS.Firehose();
exports.handler = async (event) => {
console.log(JSON.stringify(event, null, 4));
try {
await firehose
.putRecord({
DeliveryStreamName: 'delivery-stream-name',
Record: {
Data:JSON.stringify(event.body)
}
})
.promise();
} catch (error) {
console.error(error);
return {
statusCode: 400,
body: `Cannot process event: ${error}`
};
}
return {
statusCode: 200,
body: JSON.stringify({
ack: new Date().getTime()
})
};
};
so the lambda was putting the data to my firehose delivery stream which I used to get the partition of yyyy/mm/dd/h.
Then when I wanted to crawl the data, the crawler didn't recognize what is inside the json file. So when a table was created it didn't give me the columns with their type.
e.g. id: string
created_at: timestamp
etc.
after reviewing it a friend of mine noticed that two things needed to be changed.
First: the code should look like this
const AWS = require( 'aws-sdk' );
const firehose = new AWS.Firehose();
exports.handler = async (event) => {
console.log(JSON.stringify(event, null, 4));
try {
await firehose
.putRecord({
DeliveryStreamName: 'delivery-stream-name',
Record: {
Data:event.body
}
})
.promise();
} catch (error) {
console.error(error);
return {
statusCode: 400,
body: `Cannot process event: ${error}`
};
}
return {
statusCode: 200,
body: JSON.stringify({
ack: new Date().getTime()
})
};
};
so we changed on part
from
Data:JSON.stringify(event.body)
to
Data:event.body
Next I set in my delivery stream UI the compression to gzip.
Since I changed these thing when I crawl my data it now recognizes the columns and their type.
So in order to make the old data, which was send in before these changes, usable, I need to implement these two changes.
So one idea was to list all objects before the change, as they are in a different bucket and apply these two changes to it in vscode.
When i was listing objects to compare I noticed this difference.
So this is how the json file in wrong format is read.
I used the following code in vscode
s3_client = boto3.client("s3")
response = s3_client.get_object(Bucket = 'bucketname', Key = 'key')
data = response['Body'].read()
print(data)
and get the following result of the read json file
b'"{\\"event_id\\":\\"123456789\\", etc.}\\n"'
when i read the correct saved json with the same code as before, so with this
s3_client = boto3.client("s3")
response = s3_client.get_object(Bucket = 'bucketname', Key = 'key')
data = response['Body'].read()
print(data)
i get the following output
b'\x1f\x8b\x'
So I need a way to make the data which was send in before the two changes to be in the same format.
I was thinking of doing it in to ways
1.
Make everything in VSCode
So get all objects
Solve this problem with the json.
Gzipping it either in vscode and putting the data back into the bucket or sending it to a delivery stream and compressing it there.
Writing a lambda which will be triggered by new uploaded files in a S3 bucket.
Within the lambda solving the issue with json.
Next put it to a delivery stream, compressing it there and from there putting it to the correct bucket.
I am trying to download a file stored in Azure Blob Storage with my react app using SAS and am running into issues. I have a working version that relies on having the flask app download the file, then sending the blob to the react app to be downloaded again (obviously not ideal). Here's the current implementation:
flask endpoint:
from azure.storage.blob import BlobServiceClient
blobService = BlobServiceClient(account_url="https://<account_name>.blob.core.windows.net/", credential=<blob_key>)
#app.route('/download')
def downloadFile():
filename = request.args.get('filename')
blob_client = blobService.get_blob_client(container='<container_name>', blob=filename)
blobObject = blob_client.download_blob()
fileObject = io.BytesIO(blobObject.readall())
return send_file(fileObject, attachment_filename=filename, as_attachment=True)
get request in react:
const getFile = (e) => {
e.preventDefault();
const filename = e.currentTarget.getAttribute('name');
axios({
url: `${serverUrl}/download?filename=${filename}`,
method: 'GET',
responseType: 'blob',
})
.then(({ data }) => {
const link = document.createElement('a');
const url = URL.createObjectURL(new Blob([data]));
link.href = url;
link.setAttribute('download', filename);
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
})
.catch(() => {
_isMounted.current && setDisplayError(true);
});
};
I would like to be able to just have my react app download the file direct from blob storage, but am running into authentication issues, along with another issue where clicking the download button navigates the browser to the url rather than just downloading the file at the location while staying on the current page. Here is the new code with the issues.
new flask endpoint:
from azure.storage.blob._shared_access_signature import BlobSharedAccessSignature
signatureService = BlobSharedAccessSignature('<account_name>', account_key='<azure_key>')
#app.route('/download')
def downloadFile():
filename = request.args.get('filename')
expiration = datetime.datetime.today() + datetime.timedelta(minutes=5)
container = '<container_name>'
key = signatureService.generate_blob(container, filename, permission='read', expiry=expiration)
data = {
'container': container,
'key': key
}
return app.response_class(
response=jsonifyWithNumPy(data),
status=200,
mimetype='application/json'
)
get request in react:
const getFile = (e) => {
e.preventDefault();
const filename = e.currentTarget.getAttribute('name');
axios
.get(`${serverUrl}/download?filename=${filename}`)
.then(({data}) => {
const url = `https://<account_name>.blob.core.windows.net/${data.container}/${filename}?${data.key}`
const link = document.createElement('a');
link.href = url;
link.setAttribute('download', filename);
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
})
.catch(() => {
_isMounted.current && setDisplayError(true);
});
};
This is the error I get when I follow the URL generated by the above react code:
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:48c49456-001e-008b-263c-3625fd000000 Time:2021-04-20T23:23:26.1909093Z</Message>
<AuthenticationErrorDetail>Signature fields not well formed.</AuthenticationErrorDetail>
</Error>
I tried your code to create a blob SAS key and get the same error, just try the code below that works for me to create a blob SAS key and access a blob successfully:
from azure.storage.blob import generate_blob_sas,BlobSasPermissions
from datetime import datetime, timedelta
account = ''
container = ''
blob = ''
account_key = ''
permission=BlobSasPermissions(read=True)
exp = datetime.utcnow() + timedelta(hours=1)
key = generate_blob_sas(account_name=account,container_name=container,blob_name=blob,account_key=account_key,permission=permission,expiry=exp)
I get an error AWS::S3::Errors::InvalidRequest The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256. when I try upload file to S3 bucket in new Frankfurt region. All works properly with US Standard region.
Script:
backup_file = '/media/db-backup_for_dev/2014-10-23_02-00-07/slave_dump.sql.gz'
s3 = AWS::S3.new(
access_key_id: AMAZONS3['access_key_id'],
secret_access_key: AMAZONS3['secret_access_key']
)
s3_bucket = s3.buckets['test-frankfurt']
# Folder and file name
s3_name = "database-backups-last20days/#{File.basename(File.dirname(backup_file))}_#{File.basename(backup_file)}"
file_obj = s3_bucket.objects[s3_name]
file_obj.write(file: backup_file)
aws-sdk (1.56.0)
How to fix it?
Thank you.
AWS4-HMAC-SHA256, also known as Signature Version 4, ("V4") is one of two authentication schemes supported by S3.
All regions support V4, but US-Standard¹, and many -- but not all -- other regions, also support the other, older scheme, Signature Version 2 ("V2").
According to http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html ... new S3 regions deployed after January, 2014 will only support V4.
Since Frankfurt was introduced late in 2014, it does not support V2, which is what this error suggests you are using.
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html explains how to enable V4 in the various SDKs, assuming you are using an SDK that has that capability.
I would speculate that some older versions of the SDKs might not support this option, so if the above doesn't help, you may need a newer release of the SDK you are using.
¹US Standard is the former name for the S3 regional deployment that is based in the us-east-1 region. Since the time this answer was originally written,
"Amazon S3 renamed the US Standard Region to the US East (N. Virginia) Region to be consistent with AWS regional naming conventions." For all practical purposes, it's only a change in naming.
With node, try
var s3 = new AWS.S3( {
endpoint: 's3-eu-central-1.amazonaws.com',
signatureVersion: 'v4',
region: 'eu-central-1'
} );
You should set signatureVersion: 'v4' in config to use new sign version:
AWS.config.update({
signatureVersion: 'v4'
});
Works for JS sdk.
For people using boto3 (Python SDK) use the below code
from botocore.client import Config
s3 = boto3.resource(
's3',
aws_access_key_id='xxxxxx',
aws_secret_access_key='xxxxxx',
config=Config(signature_version='s3v4')
)
I have been using Django, and I had to add these extra config variables to make this work. (in addition to settings mentioned in https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html).
AWS_S3_REGION_NAME = "ap-south-1"
Or previous to boto3 version 1.4.4:
AWS_S3_REGION_NAME = "ap-south-1"
AWS_S3_SIGNATURE_VERSION = "s3v4"
Similar issue with the PHP SDK, this works:
$s3Client = S3Client::factory(array('key'=>YOUR_AWS_KEY, 'secret'=>YOUR_AWS_SECRET, 'signature' => 'v4', 'region'=>'eu-central-1'));
The important bit is the signature and the region
AWS_S3_REGION_NAME = "ap-south-1"
AWS_S3_SIGNATURE_VERSION = "s3v4"
this also saved my time after surfing for 24Hours..
Code for Flask (boto3)
Don't forget to import Config. Also If you have your own config class, then change its name.
from botocore.client import Config
s3 = boto3.client('s3',config=Config(signature_version='s3v4'),region_name=app.config["AWS_REGION"],aws_access_key_id=app.config['AWS_ACCESS_KEY'], aws_secret_access_key=app.config['AWS_SECRET_KEY'])
s3.upload_fileobj(file,app.config["AWS_BUCKET_NAME"],file.filename)
url = s3.generate_presigned_url('get_object', Params = {'Bucket':app.config["AWS_BUCKET_NAME"] , 'Key': file.filename}, ExpiresIn = 10000)
In Java I had to set a property
System.setProperty(SDKGlobalConfiguration.ENFORCE_S3_SIGV4_SYSTEM_PROPERTY, "true")
and add the region to the s3Client instance.
s3Client.setRegion(Region.getRegion(Regions.EU_CENTRAL_1))
With boto3, this is the code :
s3_client = boto3.resource('s3', region_name='eu-central-1')
or
s3_client = boto3.client('s3', region_name='eu-central-1')
For thumbor-aws, that used boto config, i needed to put this to the $AWS_CONFIG_FILE
[default]
aws_access_key_id = (your ID)
aws_secret_access_key = (your secret key)
s3 =
signature_version = s3
So anything that used boto directly without changes, this may be useful
Supernova answer for django/boto3/django-storages worked with me:
AWS_S3_REGION_NAME = "ap-south-1"
Or previous to boto3 version 1.4.4:
AWS_S3_REGION_NAME = "ap-south-1"
AWS_S3_SIGNATURE_VERSION = "s3v4"
just add them to your settings.py and change region code accordingly
you can check aws regions from:
enter link description here
For Android SDK, setEndpoint solves the problem, although it's been deprecated.
CognitoCachingCredentialsProvider credentialsProvider = new CognitoCachingCredentialsProvider(
context, "identityPoolId", Regions.US_EAST_1);
AmazonS3 s3 = new AmazonS3Client(credentialsProvider);
s3.setEndpoint("s3.us-east-2.amazonaws.com");
Basically the error was because I was using old version of aws-sdk and I updated the version so this error occured.
in my case with node js i was using signatureVersion in parmas object like this :
const AWS_S3 = new AWS.S3({
params: {
Bucket: process.env.AWS_S3_BUCKET,
signatureVersion: 'v4',
region: process.env.AWS_S3_REGION
}
});
Then I put signature out of params object and worked like charm :
const AWS_S3 = new AWS.S3({
params: {
Bucket: process.env.AWS_S3_BUCKET,
region: process.env.AWS_S3_REGION
},
signatureVersion: 'v4'
});
Check your AWS S3 Bucket Region and Pass proper Region in Connection Request.
In My Senario I have set 'APSouth1' for Asia Pacific (Mumbai)
using (var client = new AmazonS3Client(awsAccessKeyId, awsSecretAccessKey, RegionEndpoint.APSouth1))
{
GetPreSignedUrlRequest request1 = new GetPreSignedUrlRequest
{
BucketName = bucketName,
Key = keyName,
Expires = DateTime.Now.AddMinutes(50),
};
urlString = client.GetPreSignedURL(request1);
}
In my case, the request type was wrong. I was using GET(dumb) It must be PUT.
Here is the function I used with Python
def uploadFileToS3(filePath, s3FileName):
s3 = boto3.client('s3',
endpoint_url=settings.BUCKET_ENDPOINT_URL,
aws_access_key_id=settings.BUCKET_ACCESS_KEY_ID,
aws_secret_access_key=settings.BUCKET_SECRET_KEY,
region_name=settings.BUCKET_REGION_NAME
)
try:
s3.upload_file(
filePath,
settings.BUCKET_NAME,
s3FileName
)
# remove file from local to free up space
os.remove(filePath)
return True
except Exception as e:
logger.error('uploadFileToS3#Error')
logger.error(e)
return False
Sometime the default version will not update. Add this command
AWS_S3_SIGNATURE_VERSION = "s3v4"
in settings.py
For Boto3 , use this code.
import boto3
from botocore.client import Config
s3 = boto3.resource('s3',
aws_access_key_id='xxxxxx',
aws_secret_access_key='xxxxxx',
region_name='us-south-1',
config=Config(signature_version='s3v4')
)
Try this combination.
const s3 = new AWS.S3({
endpoint: 's3-ap-south-1.amazonaws.com', // Bucket region
accessKeyId: 'A-----------------U',
secretAccessKey: 'k------ja----------------soGp',
Bucket: 'bucket_name',
useAccelerateEndpoint: true,
signatureVersion: 'v4',
region: 'ap-south-1' // Bucket region
});
I was stuck for 3 days and finally, after reading a ton of blogs and answers I was able to configure Amazon AWS S3 Bucket.
On the AWS Side
I am assuming you have already
Created an s3-bucket
Created a user in IAM
Steps
Configure CORS settings
you bucket > permissions > CORS configuration
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>```
Generate A bucket policy
your bucket > permissions > bucket policy
It should be similar to this one
{
"Version": "2012-10-17",
"Id": "Policy1602480700663",
"Statement": [
{
"Sid": "Stmt1602480694902",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::harshit-portfolio-bucket/*"
}
]
}
PS: Bucket policy should say `public` after this
Configure Access Control List
your bucket > permissions > acces control list
give public access
PS: Access Control List should say public after this
Unblock public Access
your bucket > permissions > Block Public Access
Edit and turn all options Off
**On a side note if you are working on django
add the following lines to you settings.py file of your project
**
#S3 BUCKETS CONFIG
AWS_ACCESS_KEY_ID = '****not to be shared*****'
AWS_SECRET_ACCESS_KEY = '*****not to be shared******'
AWS_STORAGE_BUCKET_NAME = 'your-bucket-name'
AWS_S3_FILE_OVERWRITE = False
AWS_DEFAULT_ACL = None
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# look for files first in aws
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
# In India these settings work
AWS_S3_REGION_NAME = "ap-south-1"
AWS_S3_SIGNATURE_VERSION = "s3v4"
Also coming from: https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html
For me this was the solution:
AWS_S3_REGION_NAME = "eu-central-1"
AWS_S3_ADDRESSING_STYLE = 'virtual'
This needs to be added to settings.py in your Django project
Using PHP SDK Follow Below.
require 'vendor/autoload.php';
use Aws\S3\S3Client;
use Aws\S3\Exception\S3Exception;
$client = S3Client::factory(
array(
'signature' => 'v4',
'region' => 'me-south-1',
'key' => YOUR_AWS_KEY,
'secret' => YOUR_AWS_SECRET
)
);
Nodejs
var aws = require("aws-sdk");
aws.config.update({
region: process.env.AWS_REGION,
secretAccessKey: process.env.AWS_S3_SECRET_ACCESS_KEY,
accessKeyId: process.env.AWS_S3_ACCESS_KEY_ID,
});
var s3 = new aws.S3({
signatureVersion: "v4",
});
let data = await s3.getSignedUrl("putObject", {
ContentType: mimeType, //image mime type from request
Bucket: "MybucketName",
Key: folder_name + "/" + uuidv4() + "." + mime.extension(mimeType),
Expires: 300,
});
console.log(data);
AWS S3 Bucket Permission Configuration
Deselect Block All Public Access
Add Below Policy
{
"Version":"2012-10-17",
"Statement":[{
"Sid":"PublicReadGetObject",
"Effect":"Allow",
"Principal": "*",
"Action":["s3:GetObject"],
"Resource":["arn:aws:s3:::MybucketName/*"
]
}
]
}
Then Paste the returned URL and make PUT request on the URL with binary file of image
Full working nodejs version:
const AWS = require('aws-sdk');
var s3 = new AWS.S3( {
endpoint: 's3.eu-west-2.amazonaws.com',
signatureVersion: 'v4',
region: 'eu-west-2'
} );
const getPreSignedUrl = async () => {
const params = {
Bucket: 'some-bucket-name/some-folder',
Key: 'some-filename.json',
Expires: 60 * 60 * 24 * 7
};
try {
const presignedUrl = await new Promise((resolve, reject) => {
s3.getSignedUrl('getObject', params, (err, url) => {
err ? reject(err) : resolve(url);
});
});
console.log(presignedUrl);
} catch (err) {
if (err) {
console.log(err);
}
}
};
getPreSignedUrl();