I'm currently having issues sending a file to an API. I've manually tested my scripts base64 output by printing to the screen and copying and pasting this directly into the API's sandbox which works correctly but when I package it up in JSON ready to send, it no longer works.
What I need is this to send to the API:
{
"content": "mybase64encodedfilestuff"
}
and my python code is:
with open(filename, "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
encoded_string = encoded_string.decode("utf-8")
payload = {}
payload['content'] = encoded_string
json_payload = json.dumps(payload)
I then send this to the API as:
r = requests.post(url='https://api.example.com/uploads', data=payload,
headers={'Content-Type': 'application/json',
'Authorization': 'Basic '+api_string}, timeout=5)
I feel like I've missed something simple but can't figure it out as I just get a error 400, please provide valid content first. If I make the payload a copy and paste of the print output it works.
I'd converted my string to a JSON but they used the original string, not the JSONified one.
Related
I had a script to send a file to anonfiles.com for sharing using requests. It emulates the suggested curl command:
curl -F "file=#test.txt" https://api.anonfiles.com/upload
with open(filename, 'rb') as file:
req = requests.post(
'https://api.anonfiles.com/upload',
files={'file': file},
)
res = json.loads(req.text)
This worked properly for files smaller than 2GB, but when the file grew bigger, it started printing a error message about the filesize.
I tried to using requests-toolbelt to send the file with no success.
with open(filename, 'rb') as file:
encoder = MultipartEncoder({'file': file})
req = requests.post(
'https://api.anonfiles.com/upload',
data=encoder,
headers={'Content-Type': encoder.content_type}
)
res = json.loads(req.text)
Which returns a the following error from anonfiles.com:
{
"status": false,
"error": {
"message": "No file chosen.",
"type": "ERROR_FILE_NOT_PROVIDED",
"code": 10
}
}
I'm probably missing something obvious, but can't figure it out right now.
P.S.: the use of requests-toolbelt.MultipartEncoder is not mandatory, I tried it and didn't manage to make it work. I will accept any answer even if it doesn't include this. A solution without request is not optimal but would also work.
Using a tuple seems to work:
with open(filename, 'rb') as file:
encoder = MultipartEncoder({'file': (filename, file)})
req = requests.post(
'https://api.anonfiles.com/upload',
data=encoder,
headers={'Content-Type': encoder.content_type}
)
res = json.loads(req.text)
EDIT:
Upon further examination, I really am not sure what's going on. I originally thought you could fix it by sending json=json_file with the request, but then I realized you were using the encoder. I do not know if the MultipartEncoder will work the same way with the json object.
Anyway...good luck!
I think this is a simple case of json.load vs json.loads:
json.load is for files
json.loads is for strings
In your original code you never loaded the file because you used json.loads, which is for strings.
Although I can't explain why it worked for smaller files so maybe I'm just off base!! you might try some sort of streaming for the larger file as well.
I haven't seen the syntax you used for the response before so I rewrote it a tiny bit to simplify it to understand it a little better, but mostly I think the issue is what I wrote above:
url = 'www.url.com'
headers = 'header stuff'
with open(filename, 'rb') as file:
json_file = json.load(file)
encoder = MultipartEncoder({'file': json_file})
response = requests.post(url, data=encoder, headers=headers).text
# or instead of appending .text you could create
# data = response.text for easier use in the rest of your code.
I am currently able to send OpenCV image frames to my Flask Server using the following code
def sendtoserver(frame):
imencoded = cv2.imencode(".jpg", frame)[1]
headers = {"Content-type": "text/plain"}
try:
conn.request("POST", "/", imencoded.tostring(), headers)
response = conn.getresponse()
except conn.timeout as e:
print("timeout")
return response
But I want to send a unique_id along with the frame I tried combining the frame and the id using JSON but getting following error TypeError: Object of type 'bytes' is not JSON serializable does anybody have any idea how I can send some additional data along with the frame to the server.
UPDATED:
json format code
def sendtoserver(frame):
imencoded = cv2.imencode(".jpg", frame)[1]
data = {"uid" : "23", "frame" : imencoded.tostring()}
headers = {"Content-type": "application/json"}
try:
conn.request("POST", "/", json.dumps(data), headers)
response = conn.getresponse()
except conn.timeout as e:
print("timeout")
return response
I have actually solved the query by using the Python requests module instead of the http.client module and have done the following changes to my above code.
import requests
def sendtoserver(frame):
imencoded = cv2.imencode(".jpg", frame)[1]
file = {'file': ('image.jpg', imencoded.tostring(), 'image/jpeg', {'Expires': '0'})}
data = {"id" : "2345AB"}
response = requests.post("http://127.0.0.1/my-script/", files=file, data=data, timeout=5)
return response
As I was trying to send a multipart/form-data and requests module has the ability to send both files and data in a single request.
You can try encoding your image in base64 string
import base64
with open("image.jpg", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
And send it as a normal string.
As others suggested base64 encoding might be a good solution, however if you can't or don't want to, you could add a custom header to the request, such as
headers = {"X-my-custom-header": "uniquevalue"}
Then on the flask side:
unique_value = request.headers.get('X-my-custom-header')
or
unique_value = request.headers['X-my-custom-header']
That way you avoid the overhead of processing your image data again (if that matters) and you can generate a unique id for each frame with something like the python uuid module.
Hope that helps
Trying to output my response from Microsoft's Computer Vision API to a .json file, it works with all of the other APIs I've been using so far. With the code below, directly from Microsoft's documentation, I get an error:
Error: the JSON object must be str, not 'bytes'
Removing the parsed = json.loads(data) and using print(json.dumps(data, sort_keys=True, indent=2)) prints out the information for the image that I want, but also says Error and is prefixed with
b
denoting it's in bytes and ending with
is not JSON serializable
I'm just trying to find out how I can get the response into a .json file like i'm able to do with other APIs and am at a loss for how I can possible convert this in a way that will work.
import http.client, urllib.request, urllib.parse, urllib.error, base64, json
API_KEY = '{API_KEY}'
uri_base = 'westus.api.cognitive.microsoft.com'
headers = {
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': API_KEY,
}
params = urllib.parse.urlencode(
{
'visualFeatures': 'Categories, Description, Color',
'language': 'en',
}
)
body = "{'url': 'http://i.imgur.com/WgPtc53.jpg'}"
try:
conn = http.client.HTTPSConnection(uri_base)
conn.request('POST', '/vision/v1.0/analyze?%s' % params, body, headers)
response = conn.getresponse()
data = response.read()
# 'data' contains the JSON data. The following formats the JSON data for display.
parsed = json.loads(data)
print ("Response:")
print (json.dumps(parsed, sort_keys=True, indent=2))
conn.close()
except Exception as e:
print('Error:')
print(e)
Shortly after posting the question, I realized I had missed looking for something: just converting bytes to a string.
found this Convert bytes to a Python string
and was able to modify my code to:
parsed = json.loads(data.decode('utf-8'))
And it seems to have resolved my issue. Now error-free and able to export to .json file like I needed.
I am trying to do the following with requests:
data = {'hello', 'goodbye'}
json_data = json.dumps(data)
headers = {
'Access-Key': self.api_key,
'Access-Signature': signature,
'Access-Nonce': nonce,
'Content-Type': 'application/json',
'Accept': 'text/plain'
}
r = requests.post(url, headers=headers, data=json_data,
files={'file': open('/Users/david/Desktop/a.png', 'rb')})
However, I get the following error:
ValueError: Data must not be a string.
Note that if I remove the files parameter, it works as needed. Why won't requests allow me to send a json-encoded string for data if files is included?
Note that if I change data to be just the normal python dictionary (and not a json-encoded string), the above works. So it seems that the issue is that if files is not json-encoded, then data cannot be json-encoded. However, I need to have my data encoded to match a hash signature that's being created by the API.
When you specify your body to a JSON string, you can no longer attach a file since file uploading requires the MIME type multipart/form-data.
You have two options:
Encapsulate your JSON string as part as the form data (something like json => json.dumps(data))
Encode your file in Base64 and transmit it in the JSON request body. This looks like a lot of work though.
1.Just remove the line
json_data = json.dumps(data)
and change in request as data=data.
2.Remove 'Content-Type': 'application/json' inside headers.
This worked for me.
Alternative solution to this problem is to post data as file.
You can post strings as files. Read more here:
http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
Here is explained how to post multiple files:
http://docs.python-requests.org/en/latest/user/advanced/#post-multiple-multipart-encoded-files
removing the following helped me in my case:
'Content-Type': 'application/json'
then the data should be passed as dictionary
If your files are small, you could simply convert the binary (image or anything) to base64 string and send that as JSON to the API. That is much simpler and more straight forward than the suggested solutions. The currently accepted answer claims that is a lot of work, but it's really simple.
Client:
with open('/Users/houmie/Downloads/log.zip','rb') as f:
bytes = f.read()
tb = b64encode(bytes)
tb_str = tb.decode('utf-8')
body = {'logfile': tb_str}
r = requests.post('https://prod/feedback', data=json.dumps(body), headers=headers)
API:
def create(event, context):
data = json.loads(event["body"])
if "logfile" in data:
tb_back = data["logfile"].encode('utf-8')
zip_data = base64.b64decode(tb_back)
I am trying to do the following with requests:
data = {'hello', 'goodbye'}
json_data = json.dumps(data)
headers = {
'Access-Key': self.api_key,
'Access-Signature': signature,
'Access-Nonce': nonce,
'Content-Type': 'application/json',
'Accept': 'text/plain'
}
r = requests.post(url, headers=headers, data=json_data,
files={'file': open('/Users/david/Desktop/a.png', 'rb')})
However, I get the following error:
ValueError: Data must not be a string.
Note that if I remove the files parameter, it works as needed. Why won't requests allow me to send a json-encoded string for data if files is included?
Note that if I change data to be just the normal python dictionary (and not a json-encoded string), the above works. So it seems that the issue is that if files is not json-encoded, then data cannot be json-encoded. However, I need to have my data encoded to match a hash signature that's being created by the API.
When you specify your body to a JSON string, you can no longer attach a file since file uploading requires the MIME type multipart/form-data.
You have two options:
Encapsulate your JSON string as part as the form data (something like json => json.dumps(data))
Encode your file in Base64 and transmit it in the JSON request body. This looks like a lot of work though.
1.Just remove the line
json_data = json.dumps(data)
and change in request as data=data.
2.Remove 'Content-Type': 'application/json' inside headers.
This worked for me.
Alternative solution to this problem is to post data as file.
You can post strings as files. Read more here:
http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
Here is explained how to post multiple files:
http://docs.python-requests.org/en/latest/user/advanced/#post-multiple-multipart-encoded-files
removing the following helped me in my case:
'Content-Type': 'application/json'
then the data should be passed as dictionary
If your files are small, you could simply convert the binary (image or anything) to base64 string and send that as JSON to the API. That is much simpler and more straight forward than the suggested solutions. The currently accepted answer claims that is a lot of work, but it's really simple.
Client:
with open('/Users/houmie/Downloads/log.zip','rb') as f:
bytes = f.read()
tb = b64encode(bytes)
tb_str = tb.decode('utf-8')
body = {'logfile': tb_str}
r = requests.post('https://prod/feedback', data=json.dumps(body), headers=headers)
API:
def create(event, context):
data = json.loads(event["body"])
if "logfile" in data:
tb_back = data["logfile"].encode('utf-8')
zip_data = base64.b64decode(tb_back)