We Keep Coding

How to send base64 image using Python requests and FastAPI?

I am trying to make a code for image style transfer based on FastAPI. I found it effective to convert the byte of the image into base64 and transmit it.
So, I designed my client codeto encode the image into a base64 string and send it to the server, which received it succesfully. However, I face some difficulties in restoring the image bytes to ndarray.
I get the following this errors:
image_array = np.frombuffer(base64.b64decode(image_byte)).reshape(image_shape)
ValueError: cannot reshape array of size 524288 into shape (512,512,4)
This is my client code :
import base64
import requests
import numpy as np
import json
from matplotlib.pyplot import imread
from skimage.transform import resize
if __name__ == '__main__':
path_to_img = "my image path"
image = imread(path_to_img)
image = resize(image, (512, 512))
image_byte = base64.b64encode(image.tobytes())
data = {"shape": image.shape, "image": image_byte.decode()}
response = requests.get('http://127.0.0.1:8000/myapp/v1/filter/a', data=json.dumps(data))
and this is my server code:
import json
import base64
import uvicorn
import model_loader
import numpy as np
from fastapi import FastAPI
from typing import Optional
app = FastAPI()
#app.get("/")
def read_root():
return {"Hello": "World"}
#app.get("/myapp/v1/filter/a")
async def style_transfer(data: dict):
image_byte = data.get('image').encode()
image_shape = tuple(data.get('shape'))
image_array = np.frombuffer(base64.b64decode(image_byte)).reshape(image_shape)
if __name__ == '__main__':
uvicorn.run(app, port='8000', host="127.0.0.1")

Option 1
As previously mentioned here, as well as here and here, one should use UploadFile, in order to upload files from client apps (for async read/write have a look at this answer). For example:
server side:
#app.post("/upload")
def upload(file: UploadFile = File(...)):
try:
contents = file.file.read()
with open(file.filename, 'wb') as f:
f.write(contents)
except Exception:
return {"message": "There was an error uploading the file"}
finally:
file.file.close()
return {"message": f"Successfuly uploaded {file.filename}"}
client side:
import requests
url = 'http://127.0.0.1:8000/upload'
file = {'file': open('images/1.png', 'rb')}
resp = requests.post(url=url, files=file)
print(resp.json())
Option 2
If, however, you still need to send a base64 encoded image, you can do it as previously described here (Option 2). On client side, you can encode the image to base64 and send it using a POST request as follows:
client side:
import base64
import requests
url = 'http://127.0.0.1:8000/upload'
with open("photo.png", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
payload ={"filename": "photo.png", "filedata": encoded_string}
resp = requests.post(url=url, data=payload)
On server side you can receive the image using a Form field, and decode the image as follows:
server side:
#app.post("/upload")
def upload(filename: str = Form(...), filedata: str = Form(...)):
image_as_bytes = str.encode(filedata) # convert string to bytes
img_recovered = base64.b64decode(image_as_bytes) # decode base64string
try:
with open("uploaded_" + filename, "wb") as f:
f.write(img_recovered)
except Exception:
return {"message": "There was an error uploading the file"}
return {"message": f"Successfuly uploaded {filename}"}

Is it possible to make a POST request to Slack with AWS Lambda using only native libraries?

I am trying to make a POST request to Slack using webhooks. I can send a curl to my Slack instance locally but when trying to do so in lambda I run into trouble trying to send the payload.
Everything I've seen says I must use and zip custom libraries but for the purposes of what I'm doing I need to use native python code. Is there a way to send this POST request?
import json
import urllib.request
#import botocore.requests as requests
def lambda_handler(event, context):
message=event['message']
response = urllib.request.urlopen(message)
print(response)
This code gives me a 400 error which is how I know I'm hitting the URL I want (URL is in the message variable) but every attempt at sending a payload by adding headers and a text body seems to fail.

You may try as below:
SLACK_URL = 'https://hooks.slack.com/services/....'
req = urllib.request.Request(SLACK_URL, json.dumps(message).encode('utf-8'))
response = urllib.request.urlopen(req)

Please find attached lambda_handler code, hope this helps you.
All the messages to be posted on slack are put on a SNS topic which in turn is read by a lambda and posted to the slack channel using the slack webhook url.
import os
import json
from urllib2 import Request, urlopen, URLError, HTTPError
# Get the environment variables
SLACK_WEBHOOK_URL = os.environ['SLACK_WEBHOOK_URL']
SLACK_CHANNEL = os.environ['SLACK_CHANNEL']
SLACK_USER = os.environ['SLACK_USER']
def lambda_handler(event, context):
# Read message posted on SNS Topic
message = json.loads(event['Records'][0]['Sns']['Message'])
# New slack message is created
slack_message = {
'channel': SLACK_CHANNEL,
'username': SLACK_USER,
'text': "%s" % (message)
}
# Post message on SLACK_WEBHOOK_URL
req = Request(SLACK_WEBHOOK_URL, json.dumps(slack_message))
try:
response = urlopen(req)
response.read()
print(slack_message['channel'])
except HTTPError as e:
print(e)
except URLError as e:
print(e)

Sending opencv image along with additional data to Flask Server

I am currently able to send OpenCV image frames to my Flask Server using the following code
def sendtoserver(frame):
imencoded = cv2.imencode(".jpg", frame)[1]
headers = {"Content-type": "text/plain"}
try:
conn.request("POST", "/", imencoded.tostring(), headers)
response = conn.getresponse()
except conn.timeout as e:
print("timeout")
return response
But I want to send a unique_id along with the frame I tried combining the frame and the id using JSON but getting following error TypeError: Object of type 'bytes' is not JSON serializable does anybody have any idea how I can send some additional data along with the frame to the server.
UPDATED:
json format code
def sendtoserver(frame):
imencoded = cv2.imencode(".jpg", frame)[1]
data = {"uid" : "23", "frame" : imencoded.tostring()}
headers = {"Content-type": "application/json"}
try:
conn.request("POST", "/", json.dumps(data), headers)
response = conn.getresponse()
except conn.timeout as e:
print("timeout")
return response

I have actually solved the query by using the Python requests module instead of the http.client module and have done the following changes to my above code.
import requests
def sendtoserver(frame):
imencoded = cv2.imencode(".jpg", frame)[1]
file = {'file': ('image.jpg', imencoded.tostring(), 'image/jpeg', {'Expires': '0'})}
data = {"id" : "2345AB"}
response = requests.post("http://127.0.0.1/my-script/", files=file, data=data, timeout=5)
return response
As I was trying to send a multipart/form-data and requests module has the ability to send both files and data in a single request.

You can try encoding your image in base64 string
import base64
with open("image.jpg", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
And send it as a normal string.

As others suggested base64 encoding might be a good solution, however if you can't or don't want to, you could add a custom header to the request, such as
headers = {"X-my-custom-header": "uniquevalue"}
Then on the flask side:
unique_value = request.headers.get('X-my-custom-header')
or
unique_value = request.headers['X-my-custom-header']
That way you avoid the overhead of processing your image data again (if that matters) and you can generate a unique id for each frame with something like the python uuid module.
Hope that helps

POST request results in "internal server error", most of the times but not all (!?)

EDIT:
tl;dr:
How do I post an image using python request to a flask app where I can use openCV to manipulate it?
Long version of the question and stuff I've tried out:
I am trying to POST an image from one Python Flask app to another. The problem I am facing is that it seems to work sometimes, but usually not.
App that receives the data:
#app.route("/post", methods=['GET', 'POST'])
def post():
print "in post"
req = request.files['file']
# req = open('pug.jpg', 'r+b')
arr = np.asarray(bytearray(req.read()), dtype=np.uint8)
img = cv2.imdecode(arr,-1) # 'load it as it is'
print img
cv2.imwrite('result.png',img)
return "Hello"
App that sends data:
def hello():
url = 'http://IP_ADDRESS/post'
#This image will not work
image = open('PATH/TO/IMAGE/pug.jpg', 'r+b')
#But with this image it will work!?
#image = open('PATH/TO/IMAGE/res_tile.png', 'r+b')
files = {'file': image}
r = requests.post(url, files=files)
image.close()
return 'Hello World!'
What I have tried:
First, I thought there were something wrong with that specific image
but I have already tried a bunch.
I have eliminated the POST and put it all
in one module. Then it works.
So, there has to be something wrong in the way I am doing the POST; but it works for that other image. Really strange.
Any help would be appreciated!
EDIT:
Ive now tried a couple of additional ways, both in receiving and sending of the POST request but still no luck. Ive found more images that work and more that don't work. Puzzling and very irritating. See below for more code.
App that receives the data:
def post():
print "in post"
photo = request.files['file']
in_memory_file = io.BytesIO()
photo.save(in_memory_file)
data = np.fromstring(in_memory_file.getvalue(), dtype=np.uint8)
color_image_flag = 1
img = cv2.imdecode(data, color_image_flag)
print img
cv2.imwrite('result.png',img)
return "Hello"
App that sends the data:
def hello():
url = 'http://IPADDRESS/post'
image = open('../PATH/TO/IMAGE/pug2.png', 'r+b')
files = {'file': ('hej.png', image, 'image/png')}
r = requests.post(url, files=files, stream=True)
image.close()
print r.content
return 'Hello World!'
Still no luck.
EDIT:
Now I've also tried "Multipartencoder" from requests_toolbelt and Im still facing the same issue. On some images it works. On others not. Does anyone out there have any idea what Im doing wrong?
App that sends request:
def hello():
m = MultipartEncoder(
fields={'field0': 'value', 'field1': 'value',
'field2': ('filename', open('../PATH/TO/IMG', 'rb'), 'image/png')}
)
print m.fields['field0']
r = requests.post('http://192.168.0.3/post', data=m,
headers={'Content-Type': m.content_type})
App that receives the request is same as before, just changed so I access the correct field for the file (with "req = request.files['field2']")
Any tips/hits/ideas are welcome!

Its "solved". Or rather, I found a workaround.
Both apps (both receiving and sending the image) were running locally on my machine. Pushing my receiving app to AWS and keeping my sending app on my machine seams to have solved the issue (at least from what I can see in initial tests). I wanted to have both locally to easier develop but now it seams I need to have one app running non-locally.
If any of you have any idea why this now works...Im all ears.

How to verify POST file transfer with python requests library?

I am using the python requests library's Session feature to request dynamically generated images from a remote server and write them to a file. The remote server is often unreliable and will respond with an html document, or pieces of the image. What is the best way to verify that the content is indeed the right format (not html), and has completely loaded? (my formats are png and csv) An example of my code is as follows:
import requests
ses = requests.Session()
data = ses.get("http://url")
localDest = os.path.join("local/file/path")
with open(localDest,'wb') as f:
for chunk in data.iter_content()
f.write(chunk)
How would I modify this code to check that it is the right format, and is a complete file?

You have two options:
If the server gave correct information in the headers about the content, check that for an invalid content type or an invalid content length.
If the server is lying about the content type or sets a content length to the size of the incomplete image, validate the content afterwards.
The following does both:
import imghdr
import os
import os.path
import requests
import shutil
ses = requests.Session()
r = ses.get("http://url", stream=True)
localDest = os.path.join("local/file/path")
if r.status_code == 200:
ctype = r.headers.get('content-type', '')
if ctype.partition('/')[0].lower() != 'image':
raise ValueError('Not served an image')
clength = r.headers.get('content-length')
clength = clength and int(clength)
with open(localDest, 'wb') as f:
r.raw.decode_content = True
shutil.copyfileobj(r.raw, f)
if clength and os.path.getsize(localDest) != clength:
os.remove(localDest)
raise ValueError('Served incomplete response')
image_type = imghdr.test(localDest)
if image_type is None:
os.remove(localDest)
raise ValueError('Not served an image')
You can also install Pillow and validate the image further with that.