We Keep Coding

Extracting a zipfile to memory?

How do I extract a zip to memory?
My attempt (returning None on .getvalue()):
from zipfile import ZipFile
from StringIO import StringIO
def extract_zip(input_zip):
return StringIO(ZipFile(input_zip).extractall())

extractall extracts to the file system, so you won't get what you want. To extract a file in memory, use the ZipFile.read() method.
If you really need the full content in memory, you could do something like:
def extract_zip(input_zip):
input_zip=ZipFile(input_zip)
return {name: input_zip.read(name) for name in input_zip.namelist()}

If you work with in-memory archives frequently, I would recommend making a tool. Something like this:
# Works in Python 2 and 3.
try:
import BytesIO
except ImportError:
from io import BytesIO # Python 3
import zipfile
class InMemoryZip(object):
def __init__(self):
# Create the in-memory file-like object for working w/IMZ
self.in_memory_zip = BytesIO()
# Just zip it, zip it
def append(self, filename_in_zip, file_contents):
# Appends a file with name filename_in_zip and contents of
# file_contents to the in-memory zip.
# Get a handle to the in-memory zip in append mode
zf = zipfile.ZipFile(self.in_memory_zip, "a", zipfile.ZIP_DEFLATED, False)
# Write the file to the in-memory zip
zf.writestr(filename_in_zip, file_contents)
# Mark the files as having been created on Windows so that
# Unix permissions are not inferred as 0000
for zfile in zf.filelist:
zfile.create_system = 0
return self
def read(self):
# Returns a string with the contents of the in-memory zip.
self.in_memory_zip.seek(0)
return self.in_memory_zip.read()
# Zip it, zip it, zip it
def writetofile(self, filename):
# Writes the in-memory zip to a physical file.
with open(filename, "wb") as file:
file.write(self.read())
if __name__ == "__main__":
# Run a test
imz = InMemoryZip()
imz.append("testfile.txt", "Make a test").append("testfile2.txt", "And another one")
imz.writetofile("testfile.zip")
print("testfile.zip created")

Probable reasons:
1.This module does not currently handle multi-disk ZIP files.
(OR)
2.Check with StringIO.getvalue() weather Unicode Error is coming up.

function for loading both strings and files on disk?

I have a design question. I have a function loadImage() for loading an image file. Now it accepts a string which is a file path. But I also want to be able to load files which are not on physical disk, eg. generated procedurally. I could have it accept a string, but then how could it know the string is not a file path but file data? I could add an extra boolean argument to specify that, but that doesn't sound very clean. Any ideas?
It's something like this now:
def loadImage(filepath):
file = open(filepath, 'rb')
data = file.read()
# do stuff with data
The other version would be
def loadImage(data):
# do stuff with data
How to have this function accept both 'filepath' or 'data' and guess what it is?

You can change your loadImage function to expect an opened file-like object, such as:
def load_image(f):
data = file.read()
... and then have that called from two functions, one of which expects a path and the other a string that contains the data:
from StringIO import StringIO
def load_image_from_path(path):
with open(path, 'rb') as f:
load_image(f)
def load_image_from_string(s):
sio = StringIO(s)
try:
load_image(sio)
finally:
sio.close()

How about just creating two functions, loadImageFromString and loadImageFromFile?

This being Python, you can easily distinguish between a filename and a data string. I would do something like this:
import os.path as P
from StringIO import StringIO
def load_image(im):
fin = None
if P.isfile(im):
fin = open(im, 'rb')
else:
fin = StringIO(im)
# Read from fin like you would from any open file object
Other ways to do it would be a try block instead of using os.path, but the essence of the approach remains the same.

Django "Unable to determine the file's size" error with tempfile.TemporaryFile

I'm having problems with the standard Django FileField and tempfile.TemporaryFile. Whenever I try to save a FileField with the TemporaryFile, I get the "Unable to determine the file's size" error.
For example, given a model named Model, a filefield named FileField, and a temporaryfile named TempFile:
Model.FileField.save('foobar', django.core.files.File(TempFile), save=True)
This will give me the aforementioned error. Any thoughts?

I had this problem with tempfile.TemporaryFile. When I switched to tempfile.NamedTemporaryFile it went away. I believe that TemporaryFile just simulates being a file (on some operating system at least), whereas NamedTemporaryFile really is a file.

I was having the same problem and was able to solve it for my case. This is the code that django uses to determine the size of a file:
def _get_size(self):
if not hasattr(self, '_size'):
if hasattr(self.file, 'size'):
self._size = self.file.size
elif os.path.exists(self.file.name):
self._size = os.path.getsize(self.file.name)
else:
raise AttributeError("Unable to determine the file's size.")
return self._size
Therefore, django will raise an AttributeError if the file does not exist on disk (or have a size attribute already defined). Since the TemporaryFile class attempts to create a file in memory instead of actually on disk, this _get_size method doesn't work. In order to get it to work, I had to do something like this:
import tempfile, os
# Use tempfile.mkstemp, since it will actually create the file on disk.
(temp_filedescriptor, temp_filepath) = tempfile.mkstemp()
# Close the open file using the file descriptor, since file objects
# returned by os.fdopen don't work, either
os.close(temp_filedescriptor)
# Open the file on disk
temp_file = open(temp_filepath, "w+b")
# Do operations on your file here . . .
modelObj.fileField.save("filename.txt", File(temp_file))
temp_file.close()
# Remove the created file from disk.
os.remove(temp_filepath)
Alternatively (and preferably), if you can calculate the size of the temporary file you're creating, you could set a size attribute on the TemporaryFile object directly. Due to the libraries I was using, this was not a possibility for me.

I had this issue on Heroku even with tempfile.NamedTemporaryFile and was quite disappointed ...
I solved it using Steven's tips by setting arbitrary size manually (yes, dirty, but work for me):
from django.core.files import File
from django.core.files.temp import NamedTemporaryFile
img_temp = NamedTemporaryFile()
# Do your stuffs ...
img_temp.flush()
img_temp.size = 1024
media.thumbnail.save('dummy', File(img_temp))
Thanks !

I know this is a bit old but I've managed to save a base64 file (without having the actual file saved on the disk) by using the ContentFile class provided by Django.
According to the docs:
The ContentFile class inherits from File, but unlike File it operates on string content (bytes also supported), rather than an actual file.
The snippet below receives a base64 string, extract it's data and file extension and save it to an ImageField using the ContentFile class
import uuid
from django.core.files.base import ContentFile
def convert_b64data(b64data, filename):
file_format, imgstr = b64data.split(';base64,')
ext = file_format.split('/')[-1]
return {
'obj': base64.b64decode(imgstr),
'extension': ext,
}
b64data = request.data['b64file']
filename = str(uuid.uuid4())
file_data = convert_b64data(b64data, filename)
file_path = 'media/{}/{}.{}'.format(
user.code,
filename,
file_data['extension']
)
user.banner.save(file_path, ContentFile(file_data['obj']))

In newer versions of Django (I checked on 3.2), you just may need to wrap the file in a ContentFile.
from django.core.files.base import ContentFile
Model.FileField.save('foobar', ContentFile(file))
https://docs.djangoproject.com/en/3.2/ref/files/file/

NameError: global name is not defined

Hello
My error is produced in generating a zip file. Can you inform what I should do?
main.py", line 2289, in get
buf=zipf.read(2048)
NameError: global name 'zipf' is not defined
The complete code is as follows:
def addFile(self,zipstream,url,fname):
# get the contents
result = urlfetch.fetch(url)
# store the contents in a stream
f=StringIO.StringIO(result.content)
length = result.headers['Content-Length']
f.seek(0)
# write the contents to the zip file
while True:
buff = f.read(int(length))
if buff=="":break
zipstream.writestr(fname,buff)
return zipstream
def get(self):
self.response.headers["Cache-Control"] = "public,max-age=%s" % 86400
start=datetime.datetime.now()-timedelta(days=20)
count = int(self.request.get('count')) if not self.request.get('count')=='' else 1000
from google.appengine.api import memcache
memcache_key = "ads"
data = memcache.get(memcache_key)
if data is None:
a= Ad.all().filter("modified >", start).filter("url IN", ['www.koolbusiness.com']).filter("published =", True).order("-modified").fetch(count)
memcache.set("ads", a)
else:
a = data
dispatch='templates/kml.html'
template_values = {'a': a , 'request':self.request,}
path = os.path.join(os.path.dirname(__file__), dispatch)
output = template.render(path, template_values)
self.response.headers['Content-Length'] = len(output)
zipstream=StringIO.StringIO()
file = zipfile.ZipFile(zipstream,"w")
url = 'http://www.koolbusiness.com/list.kml'
# repeat this for every URL that should be added to the zipfile
file =self.addFile(file,url,"list.kml")
# we have finished with the zip so package it up and write the directory
file.close()
zipstream.seek(0)
# create and return the output stream
self.response.headers['Content-Type'] ='application/zip'
self.response.headers['Content-Disposition'] = 'attachment; filename="list.kmz"'
while True:
buf=zipf.read(2048)
if buf=="": break
self.response.out.write(buf)

That is probably zipstream and not zipf. So replace that with zipstream and it might work.

i don't see where you declare zipf?
zipfile? Senthil Kumaran is probably right with zipstream since you seek(0) on zipstream before the while loop to read chunks of the mystery variable.
edit:
Almost certainly the variable is zipstream.
zipfile docs:
class zipfile.ZipFile(file[, mode[, compression[, allowZip64]]])
Open a ZIP file, where file can be either a path to a file (a string) or
a file-like object. The mode parameter
should be 'r' to read an existing
file, 'w' to truncate and write a new
file, or 'a' to append to an existing
file. If mode is 'a' and file refers
to an existing ZIP file, then
additional files are added to it. If
file does not refer to a ZIP file,
then a new ZIP archive is appended to
the file. This is meant for adding a
ZIP archive to another file (such as
python.exe).
your code:
zipsteam=StringIO.StringIO()
create a file-like object using StringIO which is essentially a "memory file" read more in docs
file = zipfile.ZipFile(zipstream,w)
opens the zipfile with the zipstream file-like object in 'w' mode
url = 'http://www.koolbusiness.com/list.kml'
# repeat this for every URL that should be added to the zipfile
file =self.addFile(file,url,"list.kml")
# we have finished with the zip so package it up and write the directory
file.close()
uses the addFile method to retrieve and write the retrieved data to the file-like object and returns it. The variables are slightly confusing because you pass a zipfile to the addFile method which aliases as zipstream (confusing because we are using zipstream as a StringIO file-like object). Anyways, the zipfile is returned, and closed to make sure everything is "written".
It was written to our "memory file", which we now seek to index 0
zipstream.seek(0)
and after doing some header stuff, we finally reach the while loop that will read our "memory-file" in chunks
while True:
buf=zipstream.read(2048)
if buf=="": break
self.response.out.write(buf)

You need to declare:
global zipf
right after your
def get(self):
line. you are modifying a global variable, and this is the only way python knows what you are doing.

Create zip archive for instant download

In a web app I am working on, the user can create a zip archive of a folder full of files. Here here's the code:
files = torrent[0].files
zipfile = z.ZipFile(zipname, 'w')
output = ""
for f in files:
zipfile.write(settings.PYRAT_TRANSMISSION_DOWNLOAD_DIR + "/" + f.name, f.name)
downloadurl = settings.PYRAT_DOWNLOAD_BASE_URL + "/" + settings.PYRAT_ARCHIVE_DIR + "/" + filename
output = "Download " + torrent_name + ""
return HttpResponse(output)
But this has the nasty side effect of a long wait (10+ seconds) while the zip archive is being downloaded. Is it possible to skip this? Instead of saving the archive to a file, is it possible to send it straight to the user?
I do beleive that torrentflux provides this excat feature I am talking about. Being able to zip GBs of data and download it within a second.

Check this Serving dynamically generated ZIP archives in Django

As mandrake says, constructor of HttpResponse accepts iterable objects.
Luckily, ZIP format is such that archive can be created in single pass, central directory record is located at the very end of file:
(Picture from Wikipedia)
And luckily, zipfile indeed doesn't do any seeks as long as you only add files.
Here is the code I came up with. Some notes:
I'm using this code for zipping up a bunch of JPEG pictures. There is no point compressing them, I'm using ZIP only as container.
Memory usage is O(size_of_largest_file) not O(size_of_archive). And this is good enough for me: many relatively small files that add up to potentially huge archive
This code doesn't set Content-Length header, so user doesn't get nice progress indication. It should be possible to calculate this in advance if sizes of all files are known.
Serving the ZIP straight to user like this means that resume on downloads won't work.
So, here goes:
import zipfile
class ZipBuffer(object):
""" A file-like object for zipfile.ZipFile to write into. """
def __init__(self):
self.data = []
self.pos = 0
def write(self, data):
self.data.append(data)
self.pos += len(data)
def tell(self):
# zipfile calls this so we need it
return self.pos
def flush(self):
# zipfile calls this so we need it
pass
def get_and_clear(self):
result = self.data
self.data = []
return result
def generate_zipped_stream():
sink = ZipBuffer()
archive = zipfile.ZipFile(sink, "w")
for filename in ["file1.txt", "file2.txt"]:
archive.writestr(filename, "contents of file here")
for chunk in sink.get_and_clear():
yield chunk
archive.close()
# close() generates some more data, so we yield that too
for chunk in sink.get_and_clear():
yield chunk
def my_django_view(request):
response = HttpResponse(generate_zipped_stream(), mimetype="application/zip")
response['Content-Disposition'] = 'attachment; filename=archive.zip'
return response

Here's a simple Django view function which zips up (as an example) any readable files in /tmp and returns the zip file.
from django.http import HttpResponse
import zipfile
import os
from cStringIO import StringIO # caveats for Python 3.0 apply
def somezip(request):
file = StringIO()
zf = zipfile.ZipFile(file, mode='w', compression=zipfile.ZIP_DEFLATED)
for fn in os.listdir("/tmp"):
path = os.path.join("/tmp", fn)
if os.path.isfile(path):
try:
zf.write(path)
except IOError:
pass
zf.close()
response = HttpResponse(file.getvalue(), mimetype="application/zip")
response['Content-Disposition'] = 'attachment; filename=yourfiles.zip'
return response
Of course this approach will only work if the zip files will conveniently fit into memory - if not, you'll have to use a disk file (which you're trying to avoid). In that case, you just replace the file = StringIO() with file = open('/path/to/yourfiles.zip', 'wb') and replace the file.getvalue() with code to read the contents of the disk file.

Does the zip library you are using allow for output to a stream. You could stream directly to the user instead of temporarily writing to a zip file THEN streaming to the user.

It is possible to pass an iterator to the constructor of a HttpResponse (see docs). That would allow you to create a custom iterator that generates data as it is being requested. However I don't think that will work with a zip (you would have to send partial zip as it is being created).
The proper way, I think, would be to create the files offline, in a separate process. The user could then monitor the progress and then download the file when its ready (possibly by using the iterator method described above). This would be similar what sites like youtube use when you upload a file and wait for it to be processed.