We Keep Coding

Updating Binary File on Github using Contents API

After successfully updating a plain text file using the GitHub Repository Contents API, I tried to do the same thing with an Excel file. I understand that git isn't really designed to store binaries; however, this is what my client needs.
Here are the relevant lines of Python code:
# Get the XLSX file from the repo to get its SHA
g = GitHub(my_admin_token)
repo = g.get_repo("theowner/therepo")
contents = repo.get_contents("myfile.xlsx", ref="main")
sha = contents.sha
# So far, so good. We have the SHA.
# Read the bytes we want to use to replace the contents of the file
data = open('my_new_file.xlsx', 'rb').read()
base64_encoded_data = base64.b64encode(data)
# Update the XLSX file in the repo with the new bytes
result = repo.update_file(contents.path, "auto-committed", base64_encoded_data,
sha, branch="main")
print("Result of update_file:")
print(result)
# Result: {'commit': Commit(sha="88f46eb99ce6c1d7d7d287fb8913a7f92f6faeb2"), 'content': ContentFile(path="myfile.xlsx")}
Now, you'd think everything went well; however, when I go to GitHub and look at the file, it's a mass of Base64 encoded data. It somehow "loses the fact that it's an Excel file" in the translation. When I click on the file in the GitHub user interface, and I have the option to Download the file, I get the "big blob" of Base64 text vs. having the XLSX file download.
There doesn't seem to be a way to tell the API what encoding I want to use, e.g., there doesn't seem to be a way to set HTTP headers on the call.
I also tried using the Python requests library to PUT (per doc) to the GitHub API:
result = requests.put('https://api.github.com/repos/myname/myrepo/contents/myfile.xlsx', {
"headers": {
"Accept": "application/vnd.github.VERSION.raw",
"Authorization": "token my_admin_token"
},
"committer": {'name':'My Name', 'email':'me#mymail.com'},
"message": "Did it work?",
"branch": "main",
"content": base64_encoded_data})
and I get an HTTP 404.
I tried playing with the Accept header types as well. No dice.
Various other issues trying this with curl.
If you have a working sample of updating/replacing an XLSX file on GitHub using curl, python, etc. I'd love to see it! Thanks.

Uploading a binary file is very much possible to GitHub. Both via git and via the GitHub API.
The following python snippet works as expected and uploads an excel file to a test repository at https://github.com/recycle-bin/github-playground/tree/main/hello . And I'm able to download the excel file as expected too.
import base64
import datetime
import os
import requests
github_token = os.environ["GITHUB_API_TOKEN"]
repository = "recycle-bin/github-playground"
xlsx_file_path = "workbook.xlsx"
def upload_file_to_github(source_file_path: str, destination_path: str):
headers = {
"content-type": "application/json",
"authorization": f"token {github_token}",
"accept": "application/vnd.github+json",
}
with open(source_file_path, "rb") as source_file:
encoded_string = base64.b64encode(source_file.read()).decode("utf-8")
payload = {
"message": f"Uploaded file at {datetime.datetime.utcnow().isoformat()}",
"content": encoded_string,
}
requests.put(
f"https://api.github.com/repos/{repository}/contents/{destination_path}",
json=payload,
headers=headers,
)
def main():
upload_file_to_github(xlsx_file_path, "hello/workbook.xlsx")
if __name__ == "__main__":
main()
Your 404 could possibly be due to one of the following
The repository does not exist
The branch does not exist

How to upload a binary/video file using Python http.client PUT method?

I am communicating with an API using HTTP.client in Python 3.6.2.
In order to upload a file it requires a three stage process.
I have managed to talk successfully using POST methods and the server returns data as I expect.
However, the stage that requires the actual file to be uploaded is a PUT method - and I cannot figure out how to syntax the code to include a pointer to the actual file on my storage - the file is an mp4 video file.
Here is a snippet of the code with my noob annotations :)
#define connection as HTTPS and define URL
uploadstep2 = http.client.HTTPSConnection("grabyo-prod.s3-accelerate.amazonaws.com")
#define headers
headers = {
'accept': "application/json",
'content-type': "application/x-www-form-urlencoded"
}
#define the structure of the request and send it.
#Here it is a PUT request to the unique URL as defined above with the correct file and headers.
uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
#get the response from the server
uploadstep2response = uploadstep2.getresponse()
#read the data from the response and put to a usable variable
step2responsedata = uploadstep2response.read()
The response I am getting back at this stage is an
"Error 400 Bad Request - Could not obtain the file information."
I am certain this relates to the body="C:\Test.mp4" section of the code.
Can you please advise how I can correctly reference a file within the PUT method?
Thanks in advance

uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
will put the actual string "C:\Test.mp4" in the body of your request, not the content of the file named "C:\Test.mp4" as you expect.
You need to open the file, read it's content then pass it as body. Or to stream it, but AFAIK http.client does not support that, and since your file seems to be a video, it is potentially huge and will use plenty of RAM for no good reason.
My suggestion would be to use requests, which is a way better lib to do this kind of things:
import requests
with open(r'C:\Test.mp4'), 'rb') as finput:
response = requests.put('https://grabyo-prod.s3-accelerate.amazonaws.com/youruploadpath', data=finput)
print(response.json())

I do not know if it is useful for you, but you can try to send a POST request with requests module :
import requests
url = ""
data = {'title':'metadata','timeDuration':120}
mp3_f = open('/path/your_file.mp3', 'rb')
files = {'messageFile': mp3_f}
req = requests.post(url, files=files, json=data)
print (req.status_code)
print (req.content)
Hope it helps .

Python - Outputting to .JSON with results from Microsoft's Computer Vision API

Trying to output my response from Microsoft's Computer Vision API to a .json file, it works with all of the other APIs I've been using so far. With the code below, directly from Microsoft's documentation, I get an error:
Error: the JSON object must be str, not 'bytes'
Removing the parsed = json.loads(data) and using print(json.dumps(data, sort_keys=True, indent=2)) prints out the information for the image that I want, but also says Error and is prefixed with
b
denoting it's in bytes and ending with
is not JSON serializable
I'm just trying to find out how I can get the response into a .json file like i'm able to do with other APIs and am at a loss for how I can possible convert this in a way that will work.
import http.client, urllib.request, urllib.parse, urllib.error, base64, json
API_KEY = '{API_KEY}'
uri_base = 'westus.api.cognitive.microsoft.com'
headers = {
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': API_KEY,
}
params = urllib.parse.urlencode(
{
'visualFeatures': 'Categories, Description, Color',
'language': 'en',
}
)
body = "{'url': 'http://i.imgur.com/WgPtc53.jpg'}"
try:
conn = http.client.HTTPSConnection(uri_base)
conn.request('POST', '/vision/v1.0/analyze?%s' % params, body, headers)
response = conn.getresponse()
data = response.read()
# 'data' contains the JSON data. The following formats the JSON data for display.
parsed = json.loads(data)
print ("Response:")
print (json.dumps(parsed, sort_keys=True, indent=2))
conn.close()
except Exception as e:
print('Error:')
print(e)
Shortly after posting the question, I realized I had missed looking for something: just converting bytes to a string.
found this Convert bytes to a Python string
and was able to modify my code to:
parsed = json.loads(data.decode('utf-8'))
And it seems to have resolved my issue. Now error-free and able to export to .json file like I needed.

Encode file as base64 and send to API

I'm currently having issues sending a file to an API. I've manually tested my scripts base64 output by printing to the screen and copying and pasting this directly into the API's sandbox which works correctly but when I package it up in JSON ready to send, it no longer works.
What I need is this to send to the API:
{
"content": "mybase64encodedfilestuff"
}
and my python code is:
with open(filename, "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
encoded_string = encoded_string.decode("utf-8")
payload = {}
payload['content'] = encoded_string
json_payload = json.dumps(payload)
I then send this to the API as:
r = requests.post(url='https://api.example.com/uploads', data=payload,
headers={'Content-Type': 'application/json',
'Authorization': 'Basic '+api_string}, timeout=5)
I feel like I've missed something simple but can't figure it out as I just get a error 400, please provide valid content first. If I make the payload a copy and paste of the print output it works.

I'd converted my string to a JSON but they used the original string, not the JSONified one.

Importing Qualtrics Responses using Python Requests library

I am trying to import a csv of responses into Qualtrics using the API shown here: https://api.qualtrics.com/docs/import-responses. But, since I'm a noob at Python and (by extension) at Requests, I'm having trouble figuring out why I keep getting a 413. I've gotten this far:
formTest = {
'surveyId': 'my_id',
'file': {
'value': open('dataFiles/myFile.csv', 'rb'),
'options': {
'contentType': 'text/csv'
}
}
}
headersTest = {
"X-API-TOKEN": "my_token",
'content-type': "multipart/form-data"
}
r = requests.request("POST", url, data=formTest, headers=headersTest)
print(r.text)
The format for the formTest variable is something I found when looking through other code bases for an angular implementation of this, which may not apply to a python version of the code. I can successfully use cUrl, but Python Requests, in my current situation is the way to go (for various reasons).
In a fit of desperation, I tried directly translating the cUrl request to python requests, but that didn't seem to help much either.
Has anyone done something like this before? I took a look at posts for importing contacts and the like, but there was no luck there either (since the data that needs to be sent is formatted differently). Is there something I am missing?

It's best not to mix post data and files but use two separate dictionaries. For the files you should use the files= parameter, because it encodes the POST data as a Multipart Form data and creates the required Content-Type headers.
import requests
url = 'Qualtrics API'
file_path = 'path/to/file'
file_name = 'file.name'
data = {'surveyId':'my_id'}
files = {'file' : (file_name, open(file_path, 'rb'), 'text/csv')}
headers = {'X-API-TOKEN': 'my_token'}
r = requests.post(url, data=data, files=files, headers=headers)
print(r.text)
The first value in files['file'] is the file name (optional), followed by the file object, followed by the file content type (optional).
You will find more info in the docs: Requests, POST a Multipart-Encoded File.