I have a Python/Django web application deployed to Heroku that writes information to a .csv file.
Once the file has been written I want to pull it down from the Heroku server to the users local drive.
I don't need to persist the file anywhere and so I am avoiding using S3 or storing in the database.
I have used the Heroku "ps:copy" command which works but surely this would mean the user would need the Heroku CLI installed on their machine for this to work?
Is there any other way?
I've pasted the code below that currently generates the .csv using the djqscsv library that works with Django QuerySets:
# Generate report filename
filename = djqscsv.generate_filename(qs, append_datestamp=True)
# Generate report
try:
with open(filename, 'ab') as csv_file:
print(filename)
write_csv(qs, csv_file)
messages.success(request, 'Consultation added to report successfully!')
messages.warning(request, 'Note: Certain needs may not appear in report, \
this is a result of filtering process.')
So once "csv_file" has been written I would then redirect to the "csv_view" you have described above, obviously without writing any further rows?
This should do the trick. When sent to the csv_view, Django generates a CSV and has it automatically download to the client's browser.
Your provided code:
# Generate report filename
filename = djqscsv.generate_filename(qs, append_datestamp=True)
# Generate report
try:
with open(filename, 'ab') as csv_file:
print(filename)
write_csv(qs, csv_file)
messages.success(request, 'Consultation added to report successfully!')
messages.warning(request, 'Note: Certain needs may not appear in report, \
this is a result of filtering process.')
You need to merge this code with my code into the same view.
def csv_view(request):
filename = djqscsv.generate_filename(qs, append_datestamp=True)
response = HttpResponse(content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename="{}.csv"',format(filename)
writer = csv.writer(response)
writer.writerow(qs) #use a for loop if you have multiple rows
messages.success(request, 'Consultation added to report successfully!')
messages.warning(request, 'Note: Certain needs may not appear in report, \
this is a result of filtering process.')
return response
Just to be clear, csv_view is where the CSV is generated, not merely a link to the CSV generated in another view.
This method does not save the CSV to the Dyno either. I thought that it did and just deleted it after, but I don't think it saves it to the server ever.
Discovered that the djqscsv library has the render_to_csv_response included which solves the problem:
# Generate file name from QuerySet
filename = djqscsv.generate_filename(qs, append_datestamp=True)
# Autodownload csv file in browser
return render_to_csv_response(qs, filename=filename)
Related
I have a flask web app and after a long time run, I want to send an email with a download link to the zip file created and stored after the long time run. The zip file should be htpasswd protected. My ideas up to now:
Create a zip file with the results and store it inside a folder in the flask app root
Question: How to set a htpasswd the zip file?
Sent an email with flask-mail with the link and the password
Delete the zip file after some time
How to check when a file needs to be deleted? My idea was to check with every newly submitted job, delete all jobs older than xy weeks.
with zipfile.ZipFile(memory_file, 'w') as zf:
files = result['files']
for individualFile in files:
data = zipfile.ZipInfo(individualFile['fileName'])
data.date_time = time.localtime(time.time())[:6]
data.compress_type = zipfile.ZIP_DEFLATED
zf.writestr(data, individualFile['fileData'])
memory_file.seek(0)
return send_file(memory_file, attachment_filename='capsule.zip', as_attachment=True)
I want to track how many times file has been downloaded. I see it this way:
1) Instead of ...
file.id and file.name needed for proper work of function bellow.
2) In my download view I need to registrate download of specific file, let's say I have registrate_dl function that does its job. Also, I need to take that {{ file.url }} value like in first link from first paragraph.
3) Finally, I registered download for specific file and got {{ file.url }} as file_url variable.
But, if, at the end of view function I place return redirect(file_url), it just redirects me to file, without starting download.
So, how should i return this file_url, and trigger download?
You can just return the file as part of the response. How you go about it depends on the exact file type, but here is a CSV example I used from another answer.
def csv_view(request):
filename = "Your filename"
response = HttpResponse(content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename="{}.csv"',format(filename)
writer = csv.writer(response)
writer.writerow("Some stuff")
messages.success(request, 'Consultation added to report successfully!')
messages.warning(request, 'Note: Certain needs may not appear in report, \
this is a result of filtering process.')
return response
I'm using the following django/python code to stream a file to the browser:
wrapper = FileWrapper(file(path))
response = HttpResponse(wrapper, content_type='text/plain')
response['Content-Length'] = os.path.getsize(path)
return response
Is there a way to delete the file after the reponse is returned? Using a callback function or something?
I could just make a cron to delete all tmp files, but it would be neater if I could stream files and delete them as well from the same request.
You can use a NamedTemporaryFile:
from django.core.files.temp import NamedTemporaryFile
def send_file(request):
newfile = NamedTemporaryFile(suffix='.txt')
# save your data to newfile.name
wrapper = FileWrapper(newfile)
response = HttpResponse(wrapper, content_type=mime_type)
response['Content-Disposition'] = 'attachment; filename=%s' % os.path.basename(modelfile.name)
response['Content-Length'] = os.path.getsize(modelfile.name)
return response
temporary file should be deleted once the newfile object is evicted.
For future references:
I just had the case in which I couldn't use temp files for downloads.
But I still needed to delete them after it; so here is how I did it (I really didn't want to rely on cron jobs or celery or wossnames, its a very small system and I wanted it to stay that way).
def plug_cleaning_into_stream(stream, filename):
try:
closer = getattr(stream, 'close')
#define a new function that still uses the old one
def new_closer():
closer()
os.remove(filename)
#any cleaning you need added as well
#substitute it to the old close() function
setattr(stream, 'close', new_closer)
except:
raise
and then I just took the stream used for the response and plugged into it.
def send_file(request, filename):
with io.open(filename, 'rb') as ready_file:
plug_cleaning_into_stream(ready_file, filename)
response = HttpResponse(ready_file.read(), content_type='application/force-download')
# here all the rest of the heards settings
# ...
return response
I know this is quick and dirty but it works. I doubt it would be productive for a server with thousands of requests a second, but that's not my case here (max a few dozens a minute).
EDIT: Forgot to precise that I was dealing with very very big files that could not fit in memory during the download. So that is why I am using a BufferedReader (which is what is underneath io.open())
Mostly, we use periodic cron jobs for this.
Django already has one cron job to clean up lost sessions. And you're already running it, right?
See http://docs.djangoproject.com/en/dev/topics/http/sessions/#clearing-the-session-table
You want another command just like this one, in your application, that cleans up old files.
See this http://docs.djangoproject.com/en/dev/howto/custom-management-commands/
Also, you may not really be sending this file from Django. Sometimes you can get better performance by creating the file in a directory used by Apache and redirecting to a URL so the file can be served by Apache for you. Sometimes this is faster. It doesn't handle the cleanup any better, however.
One way would be to add a view to delete this file and call it from the client side using an asynchronous call (XMLHttpRequest). A variant of this would involve reporting back from the client on success so that the server can mark this file for deletion and have a periodic job clean it up.
This is just using the regular python approach (very simple example):
# something generates a file at filepath
from subprocess import Popen
# open file
with open(filepath, "rb") as fid:
filedata = fid.read()
# remove the file
p = Popen("rm %s" % filepath, shell=True)
# make response
response = HttpResponse(filedata, content-type="text/plain")
return response
Python 3.7 , Django 2.2.5
from tempfile import NamedTemporaryFile
from django.http import HttpResponse
with NamedTemporaryFile(suffix='.csv', mode='r+', encoding='utf8') as f:
f.write('\uFEFF') # BOM
f.write('sth you want')
# ref: https://docs.python.org/3/library/tempfile.html#examples
f.seek(0)
data=f.read()
response = HttpResponse(data, content_type="text/plain")
response['Content-Disposition'] = 'inline; filename=export.csv'
I have a Flask view that generates data and saves it as a CSV file with Pandas, then displays the data. A second view serves the generated file. I want to remove the file after it is downloaded. My current code raises a permission error, maybe because after_request deletes the file before it is served with send_from_directory. How can I delete a file after serving it?
def process_data(data)
tempname = str(uuid4()) + '.csv'
data['text'].to_csv('samo/static/temp/{}'.format(tempname))
return file
#projects.route('/getcsv/<file>')
def getcsv(file):
#after_this_request
def cleanup(response):
os.remove('samo/static/temp/' + file)
return response
return send_from_directory(directory=cwd + '/samo/static/temp/', filename=file, as_attachment=True)
after_request runs after the view returns but before the response is sent. Sending a file may use a streaming response; if you delete it before it's read fully you can run into errors.
This is mostly an issue on Windows, other platforms can mark a file deleted and keep it around until it not being accessed. However, it may still be useful to only delete the file once you're sure it's been sent, regardless of platform.
Read the file into memory and serve it, so that's it's not being read when you delete it later. In case the file is too big to read into memory, use a generator to serve it then delete it.
#app.route('/download_and_remove/<filename>')
def download_and_remove(filename):
path = os.path.join(current_app.instance_path, filename)
def generate():
with open(path) as f:
yield from f
os.remove(path)
r = current_app.response_class(generate(), mimetype='text/csv')
r.headers.set('Content-Disposition', 'attachment', filename='data.csv')
return r
I have a Flask view that generates data and saves it as a CSV file with Pandas, then displays the data. A second view serves the generated file. I want to remove the file after it is downloaded. My current code raises a permission error, maybe because after_request deletes the file before it is served with send_from_directory. How can I delete a file after serving it?
def process_data(data)
tempname = str(uuid4()) + '.csv'
data['text'].to_csv('samo/static/temp/{}'.format(tempname))
return file
#projects.route('/getcsv/<file>')
def getcsv(file):
#after_this_request
def cleanup(response):
os.remove('samo/static/temp/' + file)
return response
return send_from_directory(directory=cwd + '/samo/static/temp/', filename=file, as_attachment=True)
after_request runs after the view returns but before the response is sent. Sending a file may use a streaming response; if you delete it before it's read fully you can run into errors.
This is mostly an issue on Windows, other platforms can mark a file deleted and keep it around until it not being accessed. However, it may still be useful to only delete the file once you're sure it's been sent, regardless of platform.
Read the file into memory and serve it, so that's it's not being read when you delete it later. In case the file is too big to read into memory, use a generator to serve it then delete it.
#app.route('/download_and_remove/<filename>')
def download_and_remove(filename):
path = os.path.join(current_app.instance_path, filename)
def generate():
with open(path) as f:
yield from f
os.remove(path)
r = current_app.response_class(generate(), mimetype='text/csv')
r.headers.set('Content-Disposition', 'attachment', filename='data.csv')
return r