We Keep Coding

Signed URL for video upload causing problem in GCP

I am working on a project in which I need to upload the video files to GCS bucket using V4 Signed URL. Currently I am generating the signed url using Python script which is a part of Flask API. Here is the method signature I am using to generate url.
def GenerateURL(self,bucket_name,blob_name,method,timeout,content_type=None):
bucket = StoreCon.get_con(bucket_name)
blob = bucket.blob(blob_name)
url = blob.generate_signed_url(
version="v4",
expiration=datetime.timedelta(minutes=timeout),
method=method,
content_type=content_type,
)
resp = jsonify({'message':{'%s URL'%method:url}})
resp.status_code = 200
return resp
Now this is being called inside a blueprint route. Here is the snippet:
#CloudStoreEnd.route('/uploadMedia',methods=['POST'])
def uploadMedia():
blob_name = request.get_json()['FILE_NAME']
return StoreOperator.postMediaURL(blob_name)
When I make the call to this API route using Client side code, the video files are getting uploaded successfully to GCS bucket. But when I download the same video file from GCS bucket. The file becomes corrupted. Mentioning "0xc00d36c4" error.
Here is a sample function for client side:
def upload_file(path):
file_name = path.split('\\')[-1]
data = {'FILE_NAME':file_name}
#GET SIGNED URL FOR MEDIA UPLOAD
get_signed_url = 'https://CLOUD-RUN-SERVICE/uploadMedia'
headers = {'Content-Type':'application/json'}
resp = requests.post(url=get_signed_url,data=json.dumps(data),headers=headers)
upload_url = json.loads(resp.content)['message']['PUT URL']
#SEND A PUT REQUEST WITH MEDIA FILE
headers = {'Content-Type':MimeTypes().guess_type(file_name)[0]}
file = {'file':open(path,'rb')}
resp = requests.put(url=upload_url,headers=headers,files=file)
return resp
I am not sure why the Media(.mp4,.mov) are getting corrupted when I retrieve the same files, whereas for other files like (.pdf,.png) the files are fine. Is there an extra request parameter I need to add to get proper signed url? Or from client application I am sending the files wrong way to the signed url?

Python/GCP sanity check: is this correct means of referencing image stored in GCP Storage in a POST call

Scenario: an image file stored in a GCP bucket need to be sent to a third-party REST endpoint via a POST
Question: Is this really the best pattern? Is there a more efficient less verbose way?
We have images being uploaded by a mobile app to a GCP Storage bucket. When the finalize event for the image upload fires we have a GCP Cloud Function (Python 3) that reacts to this by getting ref to uploaded image, downloads it to a temp file, and then uses that temp file as the image source for the POST. This is our current code and it works, but to my eye seems convoluted with the multiple open commands. More specifically: is there a better way to simply get the image blob from GCP Storage and simply attach it to the POST call without first saving it as a local file and then opening it so it can be attached to the POST?
def third_party_upload(data, context):
# get image from bucket
storage_client = storage.Client()
bucket = storage_client.bucket(data['bucket'])
image_blob = bucket.get_blob(data['name'])
download_path = '/tmp/{}.jpg'.format(str(uuid.uuid4())) #temp image file download location
# save GCP Storage blob as a temp file
with open(download_path, 'wb') as file_obj:
image_blob.download_to_file(file_obj)
# open temp file and send to 3rd-party via rest post call
with open(download_path, 'rb') as img:
files = {'image': (data['name'], img, 'multipart/form-data', {'Expires': '0'}) }
headers = {
'X-Auth-Token': api_key,
'Content-Type': 'image/jpg',
'Accept': 'application/json'
}
# make POST call
response = requests.post(third_party_endpoint, headers=headers, files=files)
print('POST response:', response)
Update: a couple of commenters have mentioned that Signed URLs are a possibility and I agree they are an excellent choice. However we are stuck with a requirement to include the image binary as the POST body. Signed-URLs won't work in this case.

The HTTP method POST requires data. You must provide that data in the HTTP request. There is no magic method to obtain Cloud Storage data except to read it. The process is to read the data from Cloud Storage and then provide that data to the POST request.

If you're able to send a URL to the third-party endpoint instead of the actual image contents, you could use Signed URLs give time-limited access to the image without needing to provide the 3rd party access to the bucket or make the bucket public.
More information here: https://cloud.google.com/storage/docs/access-control/signed-urls

Possible to request github JSON file without token

So I have been having some issues solving how I can read my repo file, which is in JSON format, with requests. (Python)
Basically I have created something simple like:
r = requests.get('https://raw.githubusercontent.com/Test/testrepo/master/token.json?token=ADAJKFAHFAKNQ3RKVSUQ5T12333777777')
which works, however, every time I make a new commit/changes on that file, it gives me a new token and then I need to recode all over again.
So my question is, is it possible to access the JSON file without the token? (I do need to keep the repo in private as well), but the point is that I want to be able to do changes on the file without the URL being changed.

The easiest solution is probably to use the GitHub API, rather than trying to use the "raw" link you see in the browser.
First, acquire a personal access token
Now issue an API request to /repos using that access token:
import requests
token = "MY_SECRET_TOKEN"
owner = 'Test'
repo = 'testrepo'
path = 'token.json'
r = requests.get(
'https://api.github.com/repos/{owner}/{repo}/contents/{path}'.format(
owner=owner, repo=repo, path=path),
headers={
'accept': 'application/vnd.github.v3.raw',
'authorization': 'token {}'.format(token),
}
)
print(r.text)

You can use the Github python library to get any file in your repository. Since you mentioned keeping the repo in private, you have to login to github using one of the methods described here. Here is an example of getting the file using the github username and password
from github import Github
user_name = <YOUR_USERNAME>
password = <YOUR_PASSWORD>
g = Github(user_name, password)
file_name='test.json' #Choose your required file name location
repo_name = 'repo_name'
repo_location = '{}/{}'.format(user_name, repo_name)
repo = g.get_repo(repo_location)
file = repo.get_contents(file_name)
#if you want the download url for the file (this comes along with the token that you talked about earlier)
download_url = file.download_url
#if you simply want the content inside the file
content = file.decoded_content

#larsks provides solution are great, and I want to supplement.
I choose a public repositoryawesome-python as an example
suppose you want to access master/docs/CNAME contents
import requests
token = "MY_SECRET_TOKEN"
owner = 'vinta'
repo = 'awesome-python'
path = 'docs/CNAME'
branch = 'master' # or sha1, for example: 6831740
url = f'https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={branch}'
print(url)
r = requests.get(url,
headers={
'accept': 'application/vnd.github.v3.raw',
# 'authorization': f'token {token}', # If you are want to read "public" only, then you can ignore this line.
}
)
print(r.text)
"""
Type
r.text: str
r.content: bytes
"""
# If you want to save it as a file, then you can try as below.
# f=open('temp.ico','wb')
# f.write(r.content)
But I think many people may want to access a private repository.
then go to
github.com/settings/tokens
Generate a new token
click repo (Full control of private repositories)
add header of authorizationcancel comment

How to download files that were created by a slackbot?

I created a bot which collects info from users in a workspace. It stores this info in a csv file on the local server. How do I download said file? I got this bit of code from Stack Overflow, attempted to contact the author but didn't get any response.
import requests
url = 'https://slack-files.com/T0JU09BGC-F0UD6SJ21-a762ad74d3'
token = 'xoxp-TOKEN'
requests.get(url, headers={'Authorization': 'Bearer %s' % token})
How do I obtain the URL & token of the file? What is the token? Is it the OAuth token of the bot?
Say I wished to download the file named stats.csv from the server that was created by the slackbot and I don't have it's URL, how would I download it?

I would not recommend to patch together the URL for downloading the file yourself, because Slack might change it and then your code breaks.
Instead, first get the current URL of the file by calling the API method files.info with the file ID. Then use property url_private as URL for download. Alternatively you can also call files.list to get the list of all files with IDs and their URLs.
To ensure you have access to the file its best to use the token from it's creator, e.g. your slackbot.
I also included the code to save the downloaded data to file and some rudimentary error handling. Note that the token is excepted to be set as environment variable names SLACK_TOKEN. This is much safer than putting it directly into the code.
Here is a complete example:
import os
import requests
token = os.environ['SLACK_TOKEN']
file_id = "F12345678"
# call file info to get url
url = "https://slack.com/api/files.info"
r = requests.get(url, {"token": token, "file": file_id})
r.raise_for_status
response = r.json()
assert response["ok"]
file_name = response["file"]["name"]
file_url = response["file"]["url_private"]
print("Downloaded " + file_name)
# download file
r = requests.get(file_url, headers={'Authorization': 'Bearer %s' % token})
r.raise_for_status
file_data = r.content # get binary content
# save file to disk
with open(file_name , 'w+b') as f:
f.write(bytearray(file_data))
print("Saved " + file_name + " in current folder")

Uploading Item Image using Square Connect API

I've reviewed the examples posted in the Square Connect API documentation and the examples on GitHub, however, I can't seem to adapt these examples to the guidance on uploading images: http://docs.connect.squareup.com/#post-image
Part of the challenge is working with the Content-Type: multipart/form-data which only the image upload requires so the documentation is non-existent (with the connect-api docs).
My ultimate question is, can Square please post an example of how to upload images? Most relevant would be an example that shows how to update multiple items with images versus just one item. Any help is appreciated.

Thanks for pointing out this gap in the documentation. The function below uses the Requests Python library to upload an image for an item (this library makes multipart/form-data requests significantly simpler). Note that you'll need to install Requests first if you haven't.
import requests
def upload_item_image(item_id, image_path, access_token):
endpoint_path = 'https://connect.squareup.com/v1/' + your location + '/items/' + item_id + '/image'
# Don't include a Content-Type header, because the Requests library adds its own
upload_request_headers = {'Authorization': 'Bearer ' + access_token,
'Accept': 'application/json'}
# Be sure to set the correct MIME type for the image
files = [('image_data', (image_path, open(image_path, 'rb'), "image/jpeg"))]
response = requests.post(endpoint_path, files=files, headers=upload_request_headers)
# Print the response body
print response.text
item_id is the ID of the item you're uploading an image for.
image_path is the relative path to the image you're uploading.
access_token is the access token for the merchant you're acting on behalf of.
It isn't possible to upload images for multiple items in a single request to this endpoint. Instead, send a separate request for each item.