The Problem:
I have been playing around with CherryPy for the past couple of days but I'm still having some trouble with getting images to work how I could expect them to. I can save an uploaded image as a jpg without issue but I can't convert it to a base64 image properly. Here's the simple server I wrote:
server.py
#server.py
import os
import cherrypy #Import framework
frameNumber = 1
lastFrame = ''
lastFrameBase64 = ''
class Root (object):
def upload(self, myFile, username, password):
global frameNumber
global lastFrameBase64
global lastFrame
size = 0
lastFrameBase64 = ''
lastFrame = ''
while True:
data = myFile.file.read(8192)
if not data:
break
size += len(data)
lastFrame += data
lastFrameBase64 += data.encode('base64').replace('\n','')
f = open('/Users/brian/Files/git-repos/learning-cherrypy/tmp_image/lastframe.jpg','w')
f.write(lastFrame)
f.close()
f = open('/Users/brian/Files/git-repos/learning-cherrypy/tmp_image/lastframe.txt','w')
f.write(lastFrameBase64)
f.close()
cherrypy.response.headers['Content-Type'] = 'application/json'
print "Image received!"
frameNumber = frameNumber + 1
out = "{\"status\":\"%s\"}"
return out % ( "ok" )
upload.exposed = True
cherrypy.config.update({'server.socket_host': '192.168.1.14',
'server.socket_port': 8080,
})
if __name__ == '__main__':
# CherryPy always starts with app.root when trying to map request URIs
# to objects, so we need to mount a request handler root. A request
# to '/' will be mapped to HelloWorld().index().
cherrypy.quickstart(Root())
When I view the lastframe.jpg file, the image renders perfectly. However, when I take the text string found in lastframe.txt and prepend the proper data-uri identifier data:image/jpeg;base64, to the base64 string, I get a broken image icon in the webpage I'm trying to show the image in.
<!DOCTYPE>
<html>
<head>
<title>Title</title>
</head>
<body>
<img src="data:image/jpeg;base64,/9....." >
</body>
</html>
I have tried using another script to convert my already-saved jpg image into a data-uri and it works. I'm not sure what I'm doing wrong in the server example b/c this code gives me a string that works as a data-uri:
Working Conversion
jpgtxt = open('tmp_image/lastframe.jpg','rb').read().encode('base64').replace('\n','')
f = open("jpg1_b64.txt", "w")
f.write(jpgtxt)
f.close()
So basically it comes down to how is the data variable taken from myFile.file.read(8192) is different from the data variable taken from open('tmp_image/lastframe.jpg','rb') I read that the rb mode in the open method tells python to read the file as a binary file rather than a string. Here's where I got that.
Summary
In summary, I don't know enough about python or the cherrypy framework to see how the actual data is stored when reading from the myFile variable and how the data is store when reading from the output of the open() method. Thanks for taking the time to look at this problem.
Base64 works by taking every 3 bytes of input and producing 4 characters. But what happens when the input isn't a multiple of 3 bytes? There's special processing for that, appending = signs to the end. But that's only supposed to happen at the end of the file, not in the middle. Since you're reading 8192 bytes at a time and encoding them, and 8192 is not a multiple of 3, you're generating corrupt output.
Try reading 8190 bytes instead, or read and encode the entire file at once.
Related
I am trying to get the contents of an XML file stored in an S3 bucket to show as text in the browser. However, it just displays as numbers (bytes) rather than a legible string.
My code (views.py):
def file_content(request):
file_buffer = io.BytesIO()
s3_client = boto3.client("s3", "eu-west-1")
s3_client.download_fileobj("mybucket", "myfolder/example.xml", file_buffer)
file_content = file_buffer.getvalue()
return HttpResponse(file_content)
What I've tried
I've tried changing this line:
file_content = file_buffer.getvalue()
To:
file_content = file_buffer.getvalue().decode("utf-8")
Or:
file_content = str(file_buffer.getvalue())
But it is still displaying as numbers/ bytes in the browser. The file content displays correctly as a string when using print() or if I check type(file_content) in the console but this is not happening in the browser.
I'm not sure whats going wrong, if somebody could point me in the right direction it would be much appreciated. Thank you.
Doh, I fixed this by adding a content_type to the HttpResponse so my function now looks like this:
def file_content(request):
file_buffer = io.BytesIO()
s3_client = boto3.client("s3", "eu-west-1")
s3_client.download_fileobj("mybucket", "myfolder/example.xml", file_buffer)
file_content = file_buffer.getvalue().decode('UTF-8')
response = HttpResponse(content_type="text/plain")
response.write(file_content)
return response
Following multiple suggestions from other StackOverflow questions and the mutagen documentation, I was able to come up with code to get and set every ID3 tag in both MP3 and MP4 files. The issue I have is with setting the cover art for M4B files.
I have reproduced the code exactly like it is laid out in this answer:
Embedding album cover in MP4 file using Mutagen
But I am still receiving errors when I attempt to run the code. If I run the code with the 'albumart' value by itself I receive the error:
MP4file.tags['covr'] = albumart
Exception has occurred: TypeError
can't concat int to bytes
However, if I surround the albumart variable with brackets like is shown in the aforementioned StackOverflow question I get this output:
MP4file.tags['covr'] = [albumart]
Exception has occurred: struct.error
required argument is not an integer
Here is the function in it's entirety. The MP3 section works without any problems.
from mutagen.mp3 import MP3
from mutagen.mp4 import MP4, MP4Cover
def set_cover(filename, cover):
r = requests.get(cover)
with open('C:/temp/cover.jpg', 'wb') as q:
q.write(r.content)
if(filename.endswith(".mp3")):
MP3file = MP3(filename, ID3=ID3)
if cover.endswith('.jpg') or cover.endswith('.jpeg'):
mime = 'image/jpg'
else:
mime = 'image/png'
with open('C:/temp/cover.jpg', 'rb') as albumart:
MP3file.tags.add(APIC(encoding=3, mime=mime, type=3, desc=u'Cover', data=albumart.read()))
MP3file.save(filename)
else:
MP4file = MP4(filename)
if cover.endswith('.jpg') or cover.endswith('.jpeg'):
cover_format = 'MP4Cover.FORMAT_JPEG'
else:
cover_format = 'MP4Cover.FORMAT_PNG'
with open('C:/temp/cover.jpg', 'rb') as f:
albumart = MP4Cover(f.read(), imageformat=cover_format)
MP4file.tags['covr'] = [albumart]
I have been trying to figure out what I am doing wrong for two days now. If anyone can help me spot the problem I would be in your debt.
Thanks!
In the source code of mutagen at the location where the exception is being raised I've found the following lines:
def __render_cover(self, key, value):
...
for cover in value:
try:
imageformat = cover.imageformat
except AttributeError:
imageformat = MP4Cover.FORMAT_JPEG
...
Atom.render(b"data", struct.pack(">2I", imageformat, 0) + cover))
...
There key is the name for the cover tag and value is the data read from the image, wrapped into an MP4Cover object. Well, it turns out that if you iterates over an MP4Cover object, as the above code does, the iteration yields one byte of the image per iteration as int.
Moreover, in Python version 3+, struct.pack returns an object of type bytes. I think the cover argument was intended to be the collection of bytes taken from the cover image.
In the code you've given above the bytes of the cover image are wrapped inside an object of type MP4Cover that cannot be added to bytes as done in the second argument of Atom.render.
To avoid having to edit or patch the mutagen library source code, the trick is converting the 'MP4Cover' object to bytes and wrapping the result inside a collection as shown below.
import requests
from mutagen.mp3 import MP3
from mutagen.mp4 import MP4, MP4Cover
def set_cover(filename, cover):
r = requests.get(cover)
with open('C:/temp/cover.jpg', 'wb') as q:
q.write(r.content)
if(filename.endswith(".mp3")):
MP3file = MP3(filename, ID3=ID3)
if cover.endswith('.jpg') or cover.endswith('.jpeg'):
mime = 'image/jpg'
else:
mime = 'image/png'
with open('C:/temp/cover.jpg', 'rb') as albumart:
MP3file.tags.add(APIC(encoding=3, mime=mime, type=3, desc=u'Cover', data=albumart.read()))
MP3file.save(filename)
else:
MP4file = MP4(filename)
if cover.endswith('.jpg') or cover.endswith('.jpeg'):
cover_format = 'MP4Cover.FORMAT_JPEG'
else:
cover_format = 'MP4Cover.FORMAT_PNG'
with open('C:/temp/cover.jpg', 'rb') as f:
albumart = MP4Cover(f.read(), imageformat=cover_format)
MP4file.tags['covr'] = [bytes(albumart)]
MP4file.save(filename)
I've also added MP4file.save(filename) as the last line of the code to persists the changes done to the file.
This was a suggestion from another stack thread that I'm finally getting back to. It was part of a discussion about how to embed tools into a maya file.
You can write the whole thing as a python package, zip it, then stuff the binary contents of the zip file into a fileInfo. When you need to code, look for it in the user's $MAYA_APP_DIR; if there's no zip, write the contents of the fileInfo to disk as a zip and then insert the zip into sys.path
Source discussions were:
python copy scripts into script
and Maya Python Create and Use Zipped Package?
So far the programming is going okay, but I think I hit a snag. When I attempt this:
with open("..directory/myZip.zip","rb") as file:
cmds.fileInfo("myZip", file.read())
..and then I..
print cmds.fileInfo("myZip",q=1)
I get
[u'PK\003\004\024']
which is a bad translation of the first line of gibberish when reading the zip file as a text document.
How can I embed my zip file into my maya file as binary?
====================
Update:
Maya doesn't like writing to the file as a direct read of the utf-8 encoded zip file. I found various methods of making it into an acceptable string that could be written, but the decoding back to the file didn't appear to work. I see now that Theodox's suggestion was to write it to binary and put that in the fileInfo node.
How can one encode, store and then decode to write to file later?
If I were to convert to binary using for instance:
' '.join(format(ord(x), 'b') for x in line)
What code would I need to turn that back into the original utf-8 zip information?
you can find related code here:
http://tech-artists.org/forum/showthread.php?4161-Maya-API-Singleton-Nodes&highlight=mayapersist
the relevant bit is
import base64
encoded = base64.b64encode(value)
decoded = base64.b64decode(encoded)
basically it's the same idea, except using the base64 module instead of binascii. Any method of converting an arbitary character stream to an ascii-safe representation will work fine, as long as you use a reversable method: the potential problem you need to watch out for is a character in the data block that looks to maya like a close quote - an open quote int he fileInfo will be messy in an MA file.
This example uses YAML to do arbitrary key-value pairs but that part is irrelevant to storing the binary stuff. I've used this technique for fairly large data (up to 640k if i recall) but I don't know if it has an upper limit in terms of what you can stash in Maya
Found the answer. Great script on stack overflow. I had to encode to 'string_escape', which is something I found while trying to figure out the whole characters situation. But anyways, you open the zip, encode to 'string_escape', write it to fileInfo and then before fetching to write it back to zip, you decode it back.
Convert binary to ASCII and vice versa
import maya.cmds as cmds
import binascii
def text_to_bits(text, encoding='utf-8', errors='surrogatepass'):
bits = bin(int(binascii.hexlify(text.encode(encoding, errors)), 16))[2:]
return bits.zfill(8 * ((len(bits) + 7) // 8))
def text_from_bits(bits, encoding='utf-8', errors='surrogatepass'):
n = int(bits, 2)
return int2bytes(n).decode(encoding, errors)
def int2bytes(i):
hex_string = '%x' % i
n = len(hex_string)
return binascii.unhexlify(hex_string.zfill(n + (n & 1)))
And then you can
with open("..\maya\scripts/test.zip","rb") as thing:
texty = text_to_bits(thing.read().encode('string_escape'))
cmds.fileInfo("binaryZip",texty)
...later
with open("..\maya\scripts/test_2.zip","wb") as thing:
texty = cmds.fileInfo("binaryZip",q=1)
thing.write( text_from_bits( texty ).decode('string_escape') )
and this appears to work.. so far..
Figured I'd post the final product for anyone wanting to undertake this approach. I tried to do some corruption checking, so that a bad zip doesn't get passed between machines. That's what all the check hashing is for.
def writeTimeFull(tl):
import TimeFull
#reload(TimeFull)
with open(TimeFull.__file__.replace(".pyc",".py"),"r") as file:
cmds.scriptNode( tl.scriptConnection[1][0], e=1, bs=file.read() )
cmds.expression("spark_timeliner_activator",
e=1,s='if (Spark_Timeliner.ShowTimeliner == 1)\n'
'{\n'
'\tsetAttr Spark_Timeliner.ShowTimeliner 0;\n'
'\tpython \"Timeliner.InitTimeliner()\";\n'
'}',
o="Spark_Timeliner",ae=1,uc=all)
def checkHash(zipPath,hash1,hash2,hash3):
check = False
hashes = [hash1,hash2,hash3]
for ii, hash in enumerate(hashes):
if hash == hashes[(ii+1)%3]:
hashes[(ii+2)%3] = hashes[ii]
check = True
if check:
if md5(zipPath) == hashes[0]:
return [zipPath,hashes[0],hashes[1],hashes[2]]
else:
cmds.warning("Hash checks and/or zip are corrupted. Attain toolbox_fix.zip, put it in scripts folder and restart.")
return []
#this writes the zip file to the local users space
def saveOutZip(filename):
if os.path.isfile(filename):
if not os.path.isfile(filename.replace('_pkg','_'+__version__)):
os.rename(filename,filename.replace('_pkg','_'+__version__))
with open(filename,"w") as zipFile:
zipInfo = cmds.fileInfo("zipInfo1",q=1)[0]
zipHash_1 = cmds.fileInfo("zipHash1",q=1)[0]
zipHash_2 = cmds.fileInfo("zipHash2",q=1)[0]
zipHash_3 = cmds.fileInfo("zipHash3",q=1)[0]
zipFile.write( base64.b64decode(zipInfo) )
if checkHash(filename,zipHash_1,zipHash_2,zipHash_3):
cmds.fileInfo("zipInfo2",zipInfo)
return filename
with open(filename,"w") as zipFile:
zipInfo = cmds.fileInfo("zipInfo2",q=1)[0]
zipHash_1 = cmds.fileInfo("zipHash1",q=1)[0]
zipHash_2 = cmds.fileInfo("zipHash2",q=1)[0]
zipHash_3 = cmds.fileInfo("zipHash3",q=1)[0]
zipFile.write( base64.b64decode(zipInfo) )
if checkHash(filename,zipHash_1,zipHash_2,zipHash_3):
cmds.fileInfo("zipInfo1",zipInfo)
return filename
return False
#this writes the local zip to this file
def loadInZip(filename):
zipResults = []
for ii in range(0,10):
with open(filename,"r") as theRead:
zipResults.append([base64.b64encode(theRead.read())]+checkHash(filename,md5(filename),md5(filename),md5(filename)))
if ii>0 and zipResults[ii]==zipResults[ii-1]:
cmds.fileInfo("zipInfo1",zipResults[ii][0])
cmds.fileInfo("zipInfo2",zipResults[ii-1][0])
cmds.fileInfo("zipHash1",zipResults[ii][2])
cmds.fileInfo("zipHash2",zipResults[ii][3])
cmds.fileInfo("zipHash3",zipResults[ii][4])
return True
#file check
#http://stackoverflow.com/questions/3431825/generating-a-md5-checksum-of-a-file
def md5(fname):
import hashlib
hash = hashlib.md5()
with open(fname, "rb") as f:
for chunk in iter(lambda: f.read(4096), b""):
hash.update(chunk)
return hash.hexdigest()
filename = path+'/toolbox_pkg.zip'
zipPaths = [path+'/toolbox_update.zip',
path+'/toolbox_fix.zip',
path+'/toolbox_'+__version__+'.zip',
filename]
zipPaths_exist = [os.path.isfile(zipPath) for zipPath in zipPaths ]
if any(zipPaths_exist[:2]):
if zipPaths_exist[0]:
cmds.warning('Timeliner update present. Forcing file to update version')
if zipPaths_exist[2]:
os.remove(zipPaths[3])
elif os.path.isfile(zipPaths[3]):
os.rename(zipPaths[3], zipPaths[2])
os.rename(zipPaths[0],zipPaths[3])
if zipPaths_exist[1]:
os.remove(zipPaths[1])
else:
cmds.warning('Timeliner fix present. Replacing file to the fix version.')
if os.path.isfile(zipPaths[3]):
os.remove(zipPaths[3])
os.rename(zipPaths[1],zipPaths[3])
loadInZip(filename)
if not cmds.fileInfo("zipInfo1",q=1) and not cmds.fileInfo("zipInfo2",q=1):
loadInZip(filename)
if not os.path.isfile(filename):
saveOutZip(filename)
sys.path.append(filename)
import Timeliner
Timeliner.InitTimeliner(theVers=__version__)
if not any(zipPaths[:2]):
if __version__ > Timeliner.__version__:
cmds.warning('Saving out newer version of timeliner to local machine. Restart Maya to access latest version.')
saveOutZip(filename)
elif __version__ < Timeliner.__version__:
cmds.warning('Timeliner on machine is newer than file version. Saving machine version over timeliner in file.')
loadInZip(filename)
__version__ = Timeliner.__version__
if __name__ != "__main__":
tl = getTimeliner()
writeTimeFull(tl)
We are importing an screen capture from a web page direct into a variable in Python; and then producing a Numpy array using the following code :
To capture is a PNG image (note - the device url has an embedded cgi to do the capture work) :
response = requests.get(url.format(ip, device), auth=credentials)
Once screen is captured, covert to a Numpy array called image :
image = imread(BytesIO(response.content))
After analysis of image, we would like to FTP the captured PNG to a server for reference at a later date. The best solution we can find right now involves using imsave to create a file locally and then FTP with storbinary to take the local image and put it on the server.
Is it possible to FTP response.content; or a conversion of the numpy array back into a PNG (using imsave?) direct to the server and skip the local storage step?
Update
As per MattDMo comment, we tried:
def ftp_artifact (ftp_ip, ftp_dir, tid, artifact_name, artifact_path, imgdata) :
ftp = FTP(ftp_ip)
ftp.login("autoftp","autoftp")
ftp.mkd ("FTP/" + ftp_dir)
ftp.cwd("FTP/" + ftp_dir)
filepath = artifact_path
filename = artifact_name
f = BytesIO(imgdata)
ftp.storbinary ('STOR ' + filename, f)
ftp.quit()
Where imgdata is the result of io.imread. The result file is 5x bigger and not an image. The BytesIO object is the numpy array I presume?
In the ftplib module, the FTP.storbinary() method takes an open file object as its second argument. Since your BytesIO object can act as a file object, all you'd need to do is pass that - no need for a temporary file on the server.
EDIT
Without seeing your full code, what I suspect is happening is that you are passing the NumPy array to storbinary(), not the BytesIO object. You also need to make sure the object's read pointer is at the beginning by calling bytesio_object.seek(0) before uploading. The following code demonstrates how to do everything:
from ftplib import FTP
from io import BytesIO
import requests
r = requests.get("http://example.com/foo.png")
png = BytesIO(r.content)
# do image analysis
png.seek(0)
ftp = FTP("ftp.server.com")
ftp.login(user="username", passwd="password")
# change to desired upload directory
ftp.storbinary("STOR " + file_name, png)
try:
ftp.quit()
except:
ftp.close()
Took a bit of research but our student figured it out :
def ftp_image_to(ftp_ip, ftp_dir, filename, data):
ftp = FTP(ftp_ip)
print("logging in")
ftp.login('autoftp', 'autoftp')
print("making dir")
ftp.mkd('FTP/' + ftp_dir)
ftp.cwd('FTP/' + ftp_dir)
print("formatting image")
bytes = BytesIO()
plt.imsave(bytes, data, format='png')
bytes.seek(0)
print("storing binary")
ftp.storbinary('STOR ' + filename, bytes)
ftp.quit()
Thanks IH!
I want to process an image uploaded to my SimpleHTTPServer.
I tried directly feeding rfile to Image.open(), but this does not work.
import SimpleHTTPServer
import SocketServer
from PIL import Image
class Handler(SimpleHTTPServer.SimpleHTTPRequestHandler):
def do_POST(self):
img = Image.open(self.rfile)
# resize, crop, etc.
httpd = SocketServer.TCPServer(("", PORT), Handler)
httpd.serve_forever()
I could save the image to the disk and open it normally with PIL, but it doesn't sound like the quickest/cleanest way.
self.rfile is a simple file-like wrapper around the socket object (see the socket.makefile() function which produces this file object). The wrapper doesn't support seeking, as there only is a stream of data feeding this object, not a random-access region on a disk.
PIL on the other hand, requires random access to the whole file (through seeking) as most image formats use different sections in the file to store different information that the PIL objects need access to at different times.
Your only choice is to copy the data from self.rfile to a file object that does support seeking. I recommend you use tempfile.SpooledTemporaryFile() for this, as it'll store data in memory until a threshold is reached before moving the data to disk.
You'll need to be careful you only copy only up to the Content-Length header bytes into a local file; it is an error to send fewer or more bytes than that. If you don't your server could easily be brought to its knees by sending way bigger POST requests than your disk space can handle.
Perhaps use a while loop to copy a buffer across, until Content-Length bytes have been reached, or the socket no longer returns data:
from tempfile import SpooledTemporaryFile
def do_POST(self):
try:
length = int(self.headers.get('content-length'))
except (TypeError, ValueError):
# no Content-Length or not a number
# return error
if length > SOME_MAXIMUM_LENGTH:
# return error
with SpooledTemporaryFile() as imgfile:
read = 0
while read < length:
buffer = self.rfile.read(1024)
if not buffer:
# too short, return error
imgfile.write(buffer)
read += len(buffer)
if read > length or self.rfile.read(1):
# too long, return error
img_file.seek(0)
img = Image.open(img_file)
If you are accepting a multipart/form-data request with this handler, you'll actually have to parse out that specific request type differently. Use the cgi.FieldStorage() class to handle the parsing, it'll put files into TemporaryFile objects for you, directly to disk:
from cgi import FieldStorage
def do_POST(self):
if self.headers.get('content-type', '').lower() == 'multipart/form-data':
fields = FieldStorage(self.rfile, self.headers, environ={'METHOD': 'POST'})
imgfile = fields['image_file'] # or whatever exact field name you expect