We Keep Coding

Bug in Boto3 AWS S3 generate_presigned_url in Lambda Python 3.X with specified region?

I tried to write a python lambda function that returns a pre-signed url to put an object.
import os
import boto3
import json
import boto3
session = boto3.Session(region_name=os.environ['AWS_REGION'])
s3 = session.client('s3', region_name=os.environ['AWS_REGION'])
upload_bucket = 'BUCKER_NAME' # Replace this value with your bucket name!
URL_EXPIRATION_SECONDS = 30000 # Specify how long the pre-signed URL will be valid for
# Main Lambda entry point
def lambda_handler(event, context):
return get_upload_url(event)
def get_upload_url(event):
key = 'testimage.jpg' # Random filename we will use when uploading files
# Get signed URL from S3
s3_params = {
'Bucket': upload_bucket,
'Key': key,
'Expires': URL_EXPIRATION_SECONDS,
'ContentType': 'image/jpeg' # Change this to the media type of the files you want to upload
}
# Get signed URL
upload_url = s3.generate_presigned_url(
'put_object',
Params=s3_params,
ExpiresIn=URL_EXPIRATION_SECONDS
)
return {
'statusCode': 200,
'isBase64Encoded': False,
'headers': {
'Access-Control-Allow-Origin': '*'
},
'body': json.dumps(upload_url)
}
The code itself works and returns a signed URL in the format "https://BUCKET_NAME.s3.amazonaws.com/testimage.jpg?[...]"
However when using POSTMAN to try to put an object, it loads without ending.
Originally I thought it was because of my code, and after a while I wrote a NodeJS function that does the same thing:
const AWS = require('aws-sdk')
AWS.config.update({ region: process.env.AWS_REGION })
const s3 = new AWS.S3()
const uploadBucket = 'BUCKET_NAME' // Replace this value with your bucket name!
const URL_EXPIRATION_SECONDS = 30000 // Specify how long the pre-signed URL will be valid for
// Main Lambda entry point
exports.handler = async (event) => {
return await getUploadURL(event)
}
const getUploadURL = async function(event) {
const randomID = parseInt(Math.random() * 10000000)
const Key = 'testimage.jpg' // Random filename we will use when uploading files
// Get signed URL from S3
const s3Params = {
Bucket: uploadBucket,
Key,
Expires: URL_EXPIRATION_SECONDS,
ContentType: 'image/jpeg' // Change this to the media type of the files you want to upload
}
return new Promise((resolve, reject) => {
// Get signed URL
let uploadURL = s3.getSignedUrl('putObject', s3Params)
resolve({
"statusCode": 200,
"isBase64Encoded": false,
"headers": {
"Access-Control-Allow-Origin": "*"
},
"body": JSON.stringify(uploadURL)
})
})
}
The NodeJs version gives me a url in the format of "https://BUCKET_NAME.s3.eu-west-1.amazonaws.com/testimage.jpg?"
The main difference between the two is the aws sub domain. When using NodeJS it gives me "BUCKET_NAME.s3.eu-west-1.amazonaws.com" and when using Python "https://BUCKET_NAME.s3.amazonaws.com"
When using python the region does not appear.
I tried, using the signed url generated in python to add the "s3.eu-west-1" manually and IT Works!!
Is this a bug in the AWS Boto3 python library?
as you can see, in the python code I tried to specify the region but it does not do anything.?
Any idea guys ?
I wanna solve this mystery :)
Thanks a lot in advance,
Léo

I was able to reproduce the issue in us-east-1. There are a few bugs in Github (e.g., this and this), but the proposed resolutions are inconsistent.
The workaround is to create an Internet-facing access point for the bucket and then assign the full ARN of the access point to your upload_bucket variable.
Please note that the Lambda will create a pre-signed URL, but it will only work if the Lambda has an appropriate permissions policy attached to its execution role.

SignatureDoesNotMatch while uploading file from React.js using boto3 generate_presigned_url

Currently the presigned url is generated from Python Lambda function and testing it on postman to upload the file works perfectly.
When uploading file from React.js using axios it fails with 403 status code and below error.
Code: SignatureDoesNotMatch
Message: The request signature we calculated does not match the signature you provided. Check your key and signing method
Python Code
import boto3
s3_client = boto3.client('s3')
params = {
'Bucket': 'bucket_name',
'Key': 'unique_id.pdf',
'ContentType': "application/pdf"
}
s3_response = s3_client.generate_presigned_url(ClientMethod='put_object', Params=params, ExpiresIn=300)
React Code
const readFileDataAsBuffer = (file) =>
new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onload = (event) => {
resolve(event.target.result);
};
reader.onerror = (err) => {
reject(err);
};
reader.readAsArrayBuffer(file);
});
const onFileUploadChange = async (e) => {
const file = e.target.files[0];
const tempdata = await readFileDataAsBuffer(file);
return axios({
method: 'put',
url: presigned_url_link,
data: tempdata
})
.then(() => {})
.catch(() => {});
};

passing Content-Type worked for s3 link while uploading

How do i authenticate to and get catalog from Azure Data Catalog using Rest API in Python

I would like to get the name of the catalog from Azure Data Catalog using API. When I tried using the following command to get catalog from Azure Data Catalog
requests.get("https://management.azure.com/subscriptions/{id}/resourceGroups/{group_name}/providers/Microsoft.DataCatalog/catalogs/{catalogname}")
as mentioned in the link https://learn.microsoft.com/en-us/rest/api/datacatalog/data-catalog-data-catalog
It throws the following error
Response [400]
Looks like I have to authenticate first. How do I authenticate prior to getting catalog?

Adding the new answer in python
for getting the auth context in python you could do the following
here is the settings for the parameters we need it while calling graph api.
RESOURCE = "https://graph.microsoft.com" # Add the resource you want the access token for
TENANT = "Your tenant" # Enter tenant name, e.g. contoso.onmicrosoft.com
AUTHORITY_HOST_URL = "https://login.microsoftonline.com"
CLIENT_ID = "Your client id " # copy the Application ID of your app from your Azure portal
CLIENT_SECRET = "Your client secret" # copy the value of key you generated when setting up the application
# These settings are for the Microsoft Graph API Call
API_VERSION = 'v1.0'
here is the code for logging in
AUTHORITY_URL = config.AUTHORITY_HOST_URL + '/' + config.TENANT
REDIRECT_URI = 'http://localhost:{}/getAToken'.format(PORT)
TEMPLATE_AUTHZ_URL = ('https://login.microsoftonline.com/{}/oauth2/authorize?' +
'response_type=code&client_id={}&redirect_uri={}&' +
'state={}&resource={}')
def login():
auth_state = str(uuid.uuid4())
flask.session['state'] = auth_state
authorization_url = TEMPLATE_AUTHZ_URL.format(
config.TENANT,
config.CLIENT_ID,
REDIRECT_URI,
auth_state,
config.RESOURCE)
resp = flask.Response(status=307)
resp.headers['location'] = authorization_url
return resp
here is how you can retrieve the token
auth_context = adal.AuthenticationContext(AUTHORITY_URL)
token_response = auth_context.acquire_token_with_authorization_code(code, REDIRECT_URI, config.RESOURCE,
config.CLIENT_ID, config.CLIENT_SECRET)
and then you can create a endpoint for your azure data catalog api. Here is the http header for the same-
http_headers = {'Authorization': 'Bearer ' + token_response['accessToken'],
'User-Agent': 'adal-python-sample',
'Accept': 'application/json',
'Content-Type': 'application/json',
'client-request-id': str(uuid.uuid4())}
and then finally you can call the api. Here endpoint being the data catalog API URL.
data = requests.get(endpoint, headers=http_headers, stream=False).json()
Hope it helps.

To call a Data Catalog REST operation, create an instance of AuthenticationContext and call AcquireToken. AuthenticationContext is part of the Active Directory Authentication Library NuGet package. To install the Active Directory Authentication Library NuGet package in Visual Studio,run
"Install-Package Microsoft.IdentityModel.Clients.ActiveDirectory"
from the NuGet Package Manager Console.
Here is the code to get the token for the same.
static async Task<AuthenticationResult> AccessToken()
{
if (authResult == null)
{
//Resource Uri for Data Catalog API
string resourceUri = "https://api.azuredatacatalog.com";
//To learn how to register a client app and get a Client ID, see https://msdn.microsoft.com/en-us/library/azure/mt403303.aspx#clientID
string clientId = clientIDFromAzureAppRegistration;
//A redirect uri gives AAD more details about the specific application that it will authenticate.
//Since a client app does not have an external service to redirect to, this Uri is the standard placeholder for a client app.
string redirectUri = "https://login.live.com/oauth20_desktop.srf";
// Create an instance of AuthenticationContext to acquire an Azure access token
// OAuth2 authority Uri
string authorityUri = "https://login.windows.net/common/oauth2/authorize";
AuthenticationContext authContext = new AuthenticationContext(authorityUri);
// Call AcquireToken to get an Azure token from Azure Active Directory token issuance endpoint
// AcquireToken takes a Client Id that Azure AD creates when you register your client app.
authResult = await authContext.AcquireTokenAsync(resourceUri, clientId, new Uri(redirectUri), new PlatformParameters(PromptBehavior.Always));
}
return authResult;
}
Here is the sample code to get the data asset base on id
// The Get Data Asset operation retrieves data asset by Id
static JObject GetDataAsset(string assetUrl)
{
string fullUri = string.Format("{0}?api-version=2016-03-30", assetUrl);
//Create a GET WebRequest as a Json content type
HttpWebRequest request = WebRequest.Create(fullUri) as HttpWebRequest;
request.KeepAlive = true;
request.Method = "GET";
request.Accept = "application/json;adc.metadata=full";
try
{
var response = SetRequestAndGetResponse(request);
using (var reader = new StreamReader(response.GetResponseStream()))
{
var itemPayload = reader.ReadToEnd();
Console.WriteLine(itemPayload);
return JObject.Parse(itemPayload);
}
}
catch (WebException ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.Status);
if (ex.Response != null)
{
// can use ex.Response.Status, .StatusDescription
if (ex.Response.ContentLength != 0)
{
using (var stream = ex.Response.GetResponseStream())
{
using (var reader = new StreamReader(stream))
{
Console.WriteLine(reader.ReadToEnd());
}
}
}
}
}
return null;
}
Here is how you can set the request, token and get the response.
static HttpWebResponse SetRequestAndGetResponse(HttpWebRequest request, string payload = null)
{
while (true)
{
//To authorize the operation call, you need an access token which is part of the Authorization header
request.Headers.Add("Authorization", AccessToken().Result.CreateAuthorizationHeader());
//Set to false to be able to intercept redirects
request.AllowAutoRedirect = false;
if (!string.IsNullOrEmpty(payload))
{
byte[] byteArray = Encoding.UTF8.GetBytes(payload);
request.ContentLength = byteArray.Length;
request.ContentType = "application/json";
//Write JSON byte[] into a Stream
request.GetRequestStream().Write(byteArray, 0, byteArray.Length);
}
else
{
request.ContentLength = 0;
}
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
// Requests to **Azure Data Catalog (ADC)** may return an HTTP 302 response to indicate
// redirection to a different endpoint. In response to a 302, the caller must re-issue
// the request to the URL specified by the Location response header.
if (response.StatusCode == HttpStatusCode.Redirect)
{
string redirectedUrl = response.Headers["Location"];
HttpWebRequest nextRequest = WebRequest.Create(redirectedUrl) as HttpWebRequest;
nextRequest.Method = request.Method;
request = nextRequest;
}
else
{
return response;
}
}
}
Basically you need to get the bearer token and pass it as a request parameter to get the catalog using azure data catalog api.
for further code sample, please browse below code repository.
https://github.com/Azure-Samples/data-catalog-dotnet-get-started
Hope it helps.

Sending a group of large buffers from express to flask

I'm trying to send a group of .mp4 files first from a web browser client, then to express, then to flask. This code might look simple, but it has taken literally days to get this far, because nobody on the internet has posted an example of how to do this with large files (.mp4 video files), and to be honest, there are a lot of inconsistencies in using these libraries.
I think it might be an encoding error from express to Flask, however. The actual data seems to be being transferred, but ffmpeg doesn't recognize the received .mp4 files as a valid video file.
From the client, I send an XHR request like this:
var formData = new FormData();
for(var x=0; x< datas.videoDatas.length; x++){
// data.videoDatas[x] is a blob of a .mp4 file. it can be played
formData.append(x+"_video.mp4",datas.videoDatas[x]);
}
var xhr = new XMLHttpRequest();
xhr.open("POST", 'http://localhost:3000/uploadVid', true);
xhr.onreadystatechange = function() {
if(this.readyState == XMLHttpRequest.DONE && this.status == 200) {
console.log(xhr.response);
}
}
xhr.onload=console.log;
xhr.send(formData);
The express server at port 3000 receives the post request using the "multer" library. Data is stored into memory as a buffer.
var express = require('express');
var multer=require("multer");
var request=require("request")
var app = express();
var storage = multer.memoryStorage();
const upload = multer({
storage: storage
}).any();
app.post('/uploadVid',function(req,res){
var originalResponse=res;
console.log("begin upload");
upload(req, res, function(err) {
console.log("begin upload to flask server");
var requestStruct={
url: 'http://localhost:3001/receiveUpload2?numFiles='+req.files.length,
method: 'POST',
form: {
}
}
for(var x=0; x < req.files.length; x++){
var testBuffer=req.files[x].buffer; // should be a buffer of the .mp4 file
//testBuffer = new Buffer(10);
//testBuffer=testBuffer.toString();
requestStruct.form[x+'_customBufferFile']= {
value: testBuffer,
options: {
filename: req.files[x].fieldname
}
}
}
request(requestStruct, function(err,res,body){
originalResponse.send(body)
});
});
});
http.createServer(app).listen(3000)
In the python flask server, the .mp4 files are received as a buffer and written to a .mp4 file. However, the .mp4 file is unplayable by ffmpeg "Unknown EBML doctype '(none)'0/0 0_video.mp4: End of file"
from flask import Flask
from flask import request;
app = Flask(__name__);
#app.route('/receiveUpload2', methods=["POST"])
def receiveUpload2():
print("uploaded received....");
numFiles=int(request.args.get("numFiles").encode('ascii', 'ignore'));
print("numFiles",int(numFiles));
for x in range(0,numFiles):
name=request.form.getlist(str(x)+'_customBufferFile[options][filename]');
name=name[0];
print("writing",name);
filebytes=request.form.getlist(str(x)+'_customBufferFile[value]');
filebytes=filebytes[0];
#print(filebytes);
newFileByteArray = bytearray(filebytes,'utf8')
with open("./uploads/"+name,"wb") as output:
output.write(newFileByteArray);
app.run(host="localhost", port=3001);

Uploading file and data from Python to Node.js in POST

How do I send a file to Node.js AND parameter data in POST. I am happy to use any framework. I have attempted it with formidable but I am happy to change.
In my attempt, the file sends, but req.body is empty
Python code to upload:
with open('fileName.txt', 'rb') as f:
payLoad = dict()
payLoad["data"] = "my_data"
r = requests.post('http://xx.xx.xx.xx:8080/sendFile',json = payLoad, files={'fileName.txt': f})
Server Side Node.js:
var express = require('express');
var formidable = require('formidable');
var app = express();
var bodyParser = require('body-parser');
app.use( bodyParser.json() );
app.use(bodyParser.urlencoded({ extended: false }));
app.post('/sendFile', function (req, res){
console.log(req.body )
// req.body is empty

I don't know how to correctly send the file using python, but to receive file with node.js you can use express-fileupload
var fileUpload = require('express-fileupload');
app.use(fileUpload());
app.post('/upload', function(req, res) {
if (!req.files)
return res.status(400).send('No files were uploaded.');
// The name of the input field (i.e. "sampleFile") is used to retrieve the uploaded file
let sampleFile = req.files.sampleFile;
// Use the mv() method to place the file somewhere on your server
sampleFile.mv('/somewhere/on/your/server/filename.jpg', function(err) {
if (err)
return res.status(500).send(err);
res.send('File uploaded!');
});
});
https://www.npmjs.com/package/express-fileupload