I want to process an image uploaded to my SimpleHTTPServer.
I tried directly feeding rfile to Image.open(), but this does not work.
import SimpleHTTPServer
import SocketServer
from PIL import Image
class Handler(SimpleHTTPServer.SimpleHTTPRequestHandler):
def do_POST(self):
img = Image.open(self.rfile)
# resize, crop, etc.
httpd = SocketServer.TCPServer(("", PORT), Handler)
httpd.serve_forever()
I could save the image to the disk and open it normally with PIL, but it doesn't sound like the quickest/cleanest way.
self.rfile is a simple file-like wrapper around the socket object (see the socket.makefile() function which produces this file object). The wrapper doesn't support seeking, as there only is a stream of data feeding this object, not a random-access region on a disk.
PIL on the other hand, requires random access to the whole file (through seeking) as most image formats use different sections in the file to store different information that the PIL objects need access to at different times.
Your only choice is to copy the data from self.rfile to a file object that does support seeking. I recommend you use tempfile.SpooledTemporaryFile() for this, as it'll store data in memory until a threshold is reached before moving the data to disk.
You'll need to be careful you only copy only up to the Content-Length header bytes into a local file; it is an error to send fewer or more bytes than that. If you don't your server could easily be brought to its knees by sending way bigger POST requests than your disk space can handle.
Perhaps use a while loop to copy a buffer across, until Content-Length bytes have been reached, or the socket no longer returns data:
from tempfile import SpooledTemporaryFile
def do_POST(self):
try:
length = int(self.headers.get('content-length'))
except (TypeError, ValueError):
# no Content-Length or not a number
# return error
if length > SOME_MAXIMUM_LENGTH:
# return error
with SpooledTemporaryFile() as imgfile:
read = 0
while read < length:
buffer = self.rfile.read(1024)
if not buffer:
# too short, return error
imgfile.write(buffer)
read += len(buffer)
if read > length or self.rfile.read(1):
# too long, return error
img_file.seek(0)
img = Image.open(img_file)
If you are accepting a multipart/form-data request with this handler, you'll actually have to parse out that specific request type differently. Use the cgi.FieldStorage() class to handle the parsing, it'll put files into TemporaryFile objects for you, directly to disk:
from cgi import FieldStorage
def do_POST(self):
if self.headers.get('content-type', '').lower() == 'multipart/form-data':
fields = FieldStorage(self.rfile, self.headers, environ={'METHOD': 'POST'})
imgfile = fields['image_file'] # or whatever exact field name you expect
Related
I am trying to learn python socket programming (Networks) and I have a function that sends a file but I am not fully understanding each line of this function (can't get my head around it). Could someone please explain line by line what this is doing. I am also unsure why it needs the length of the data to send the file. Also if you can see any improvements to this function as well, thanks.
def send_file(socket, filename):
with open(filename, "rb") as x:
data = x.read()
length_data = len(data)
try:
socket.sendall(length_data.to_bytes(20, 'big'))
socket.sendall(data)
except Exception as error:
print("Error", error)
This protocol is written to first send the size of the file and then its data. The size is 20 octets (bytes) serialized as "big endian", meaning the most significant byte is sent first. Suppose you had a 1M file,
>>> val = 1000000
>>> val.to_bytes(20, 'big')
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0fB#'
The program sends those 20 bytes and then the file payload itself. The receiver would read the 20 octets, convert that back into a file size and would know exactly how many more octets to receive for the file. TCP is a streaming protocol. It has no idea of message boundaries. Protocols need some way of telling the other side how much data makes up a message and this is a common way to do it.
As an aside, this code has the serious problem that it reads the entire file in one go. Suppose it was huge, this code would crash.
A receiver would look something like the following. This is a rudimentary.
import io
def recv_exact(skt, count):
buf = io.BytesIO()
while count:
data = skt.recv(count)
if not data:
return b"" # socket closed
buf.write(data)
count -= len(data)
return buf.getvalue()
def recv_file(skt):
data = recv_exact(skt, 20)
if not data:
return b""
file_size = int.from_bytes(data, "big")
file_bytes = recv_exact(skt, file_size)
return file_bytes
I know I can save an RGB array to a file using
from matplotlib.pyplot import imsave
rgb = numpy.zeros([height,width,3], dtype=numpy.uint8)
paint_picture(rgb)
imsave("/tmp/whatever.png", rgb)
but now I want to write the PNG to a byte buffer instead of a file so I can later transmit those PNG bytes via HTTP. There must be no temporary files involved.
Bonus points if the answer has variations that support formats other than PNG.
Evidently imsave supports "file-like objects" of which io.BytesIO is the one I need:
buffer = BytesIO()
imsave(buffer, rgb)
encoded_png = buffer.getbuffer()
#and then my class derived from BaseHTTPRequestHandler can transmit it as the response to a GET
self.send_response(200)
self.send_header("Content-Type", "image/png")
self.end_headers()
self.wfile.write(encoded_png)
return
If you want to do a file upload (any type of picture)
Send file using POST from a Python script
But if you want to send raw png data you can reread the file and encode it in base64. Your server just have to decode base64 and write the file.
import base64
from urllib.parse import urlencode
from urllib.request import Request, urlopen
array_encoded = ""
with open("/tmp/whatever.png") as f:
array_encoded = base64.encode(f)
url = 'https://URL.COM' # Set destination URL here
post_fields = {'image': array_encoded} # Set POST fields here
request = Request(url, urlencode(post_fields).encode())
responce = urlopen(request).read()
print(responce)
Code not tested!!!!!!!
I am using a Flask app to receive a mutipart/form-data request with an uploaded file (a video, in this example).
I don't want to save the file in the local directory because this app will be running on a server, and saving it will slow things down.
I am trying to use the file object created by the Flask request.files[''] method, but it doesn't seem to be working.
Here is that portion of the code:
#bp.route('/video_upload', methods=['POST'])
def VideoUploadHandler():
form = request.form
video_file = request.files['video_data']
if video_file:
s3 = boto3.client('s3')
s3.upload_file(video_file.read(), S3_BUCKET, 'video.mp4')
return json.dumps('DynamoDB failure')
This returns an error:
TypeError: must be encoded string without NULL bytes, not str
on the line:
s3.upload_file(video_file.read(), S3_BUCKET, 'video.mp4')
I did get this to work by first saving the file and then accessing that saved file, so it's not an issue with catching the request file. This works:
video_file.save(form['video_id']+".mp4")
s3.upload_file(form['video_id']+".mp4", S3_BUCKET, form['video_id']+".mp4")
What would be the best method to handle this file data in memory and pass it to the s3.upload_file() method? I am using the boto3 methods here, and I am only finding examples with the filename used in the first parameter, so I'm not sure how to process this correctly using the file in memory. Thanks!
First you need to be able to access the raw data sent to Flask. This is not as easy as it seems, since you're reading a form. To be able to read the raw stream you can use flask.request.stream, which behaves similarly to StringIO. The trick here is, you cannot call request.form or request.file because accessing those attributes will load the whole stream into memory or into a file.
You'll need some extra work to extract the right part of the stream (which unfortunately I cannot help you with because it depends on how your form is made, but I'll let you experiment with this).
Finally you can use the set_contents_from_file function from boto, since upload_file does not seem to deal with file-like objects (StringIO and such).
Example code:
from boto.s3.key import Key
#bp.route('/video_upload', methods=['POST'])
def VideoUploadHandler():
# form = request.form <- Don't do that
# video_file = request.files['video_data'] <- Don't do that either
video_file_and_metadata = request.stream # This is a file-like object which does not only contain your video file
# This is what you need to implement
video_title, video_stream = extract_title_stream(video_file_and_metadata)
# Then, upload to the bucket
s3 = boto3.client('s3')
bucket = s3.create_bucket(bucket_name, location=boto.s3.connection.Location.DEFAULT)
k = Key(bucket)
k.key = video_title
k.set_contents_from_filename(video_stream)
I have a simple server app written in Python using Bottle framework. At one route I create an image and write it to a stream and I want to return it as a response. I know how to return an image file using static_file function but this is costly for me since I need to write the image to a file first. I want to serve the image directly using the stream object. How can I do this?
My current code is something like this (file version):
#route('/image')
def video_image():
pi_camera.capture("image.jpg", format='jpeg')
return static_file("image.jpg",
root=".",
mimetype='image/jpg')
Instead of this, I want to do something like this:
#route('/image')
def video_image():
image_buffer = BytesIO()
pi_camera.capture(image_buffer, format='jpeg') # This works without a problem
# What to write here?
Just return the bytes. (You should also set the Content-Type header.)
#route('/image')
def video_image():
image_buffer = BytesIO()
pi_camera.capture(image_buffer, format='jpeg') # This works without a problem
image_buffer.seek(0) # this may not be needed
bytes = image_buffer.read()
response.set_header('Content-type', 'image/jpeg')
return bytes
The Problem:
I have been playing around with CherryPy for the past couple of days but I'm still having some trouble with getting images to work how I could expect them to. I can save an uploaded image as a jpg without issue but I can't convert it to a base64 image properly. Here's the simple server I wrote:
server.py
#server.py
import os
import cherrypy #Import framework
frameNumber = 1
lastFrame = ''
lastFrameBase64 = ''
class Root (object):
def upload(self, myFile, username, password):
global frameNumber
global lastFrameBase64
global lastFrame
size = 0
lastFrameBase64 = ''
lastFrame = ''
while True:
data = myFile.file.read(8192)
if not data:
break
size += len(data)
lastFrame += data
lastFrameBase64 += data.encode('base64').replace('\n','')
f = open('/Users/brian/Files/git-repos/learning-cherrypy/tmp_image/lastframe.jpg','w')
f.write(lastFrame)
f.close()
f = open('/Users/brian/Files/git-repos/learning-cherrypy/tmp_image/lastframe.txt','w')
f.write(lastFrameBase64)
f.close()
cherrypy.response.headers['Content-Type'] = 'application/json'
print "Image received!"
frameNumber = frameNumber + 1
out = "{\"status\":\"%s\"}"
return out % ( "ok" )
upload.exposed = True
cherrypy.config.update({'server.socket_host': '192.168.1.14',
'server.socket_port': 8080,
})
if __name__ == '__main__':
# CherryPy always starts with app.root when trying to map request URIs
# to objects, so we need to mount a request handler root. A request
# to '/' will be mapped to HelloWorld().index().
cherrypy.quickstart(Root())
When I view the lastframe.jpg file, the image renders perfectly. However, when I take the text string found in lastframe.txt and prepend the proper data-uri identifier data:image/jpeg;base64, to the base64 string, I get a broken image icon in the webpage I'm trying to show the image in.
<!DOCTYPE>
<html>
<head>
<title>Title</title>
</head>
<body>
<img src="data:image/jpeg;base64,/9....." >
</body>
</html>
I have tried using another script to convert my already-saved jpg image into a data-uri and it works. I'm not sure what I'm doing wrong in the server example b/c this code gives me a string that works as a data-uri:
Working Conversion
jpgtxt = open('tmp_image/lastframe.jpg','rb').read().encode('base64').replace('\n','')
f = open("jpg1_b64.txt", "w")
f.write(jpgtxt)
f.close()
So basically it comes down to how is the data variable taken from myFile.file.read(8192) is different from the data variable taken from open('tmp_image/lastframe.jpg','rb') I read that the rb mode in the open method tells python to read the file as a binary file rather than a string. Here's where I got that.
Summary
In summary, I don't know enough about python or the cherrypy framework to see how the actual data is stored when reading from the myFile variable and how the data is store when reading from the output of the open() method. Thanks for taking the time to look at this problem.
Base64 works by taking every 3 bytes of input and producing 4 characters. But what happens when the input isn't a multiple of 3 bytes? There's special processing for that, appending = signs to the end. But that's only supposed to happen at the end of the file, not in the middle. Since you're reading 8192 bytes at a time and encoding them, and 8192 is not a multiple of 3, you're generating corrupt output.
Try reading 8190 bytes instead, or read and encode the entire file at once.