We Keep Coding

save .xlsx file writen by pandas and export as a django model

I am going to have a web app that does some processing on user data and is going to give him/her back an excel file of processed data in Django.
here is my app/views.py function:
#login_required(login_url='/signin/')
def processing(request):
...
# read imported file (uploaded by user) and do the process.
# finally have 3 uf, vf, wf array.
...
# save the file in "excel_file = models.FileField"
from pandas import DataFrame, ExcelWriter
data = {
'uf': uf,
'vf': vf,
'wf': wf,
}
excel_data = DataFrame(excel_data)
despiked = Despiking(
#other models
excel_file=excel_data.to_excel(???),
)
despiked.save()
return render(request, 'filteration/despiking.html', context=context)
how can I save the excel_data into its model (excel_file) and create its media into Media directory?
I already have read django pandas dataframe download as excel file question, but wasn't able to fix the problem with HttpResponse. any help would be greatly appreciated

One option can be to create an in memory file.
import io
from django.core.files import File
# Create an in-memory buffer to store the Excel file
buffer = io.BytesIO()
writer = ExcelWriter(buffer)
excel_data.to_excel(writer)
# Save the Excel file to the model's excel_file field
despiked = Despiking(
...
excel_file=File(buffer, name='processed_data.xlsx'),
)
despiked.save()

Testing the data sent by Flask's send_file()

I have a Flask view that generates an Excel file (using openpyxl) from some data and it's returned to the user using send_file(). A very simplified version:
import io
from flask import send_file
from openpyxl.workbook import Workbook
#app.route("/download/<int:id>")
def file_download(id):
wb = Workbook()
# Add sheets and data to the workbook here.
file = io.BytesIO()
wb.save(file)
file.seek(0)
return send_file(file, attachment_filename=f"{id}.xlsx", as_attachment=True)
This works fine -- the file downloads and is a valid Excel file. But I'm not sure how to test the file download. So far I have something like this (using pytest):
def test_file_download(test_client):
response = test_client.get("/download/123")
assert response.status_code == 200
assert response.content_type == "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
Which passes, but I'd like to test that the (a) the filename used is as expected and (b) that the file... exists? Is an Excel file?
I can access response.get_data(), which is a bytes object, but I'm not sure what to do with it.

To check that the filename used is as expected you could check that the Content-Disposition header is as expected. For example:
assert response.headers['Content-Disposition'] == 'attachment; filename=123.xlsx'
To check "the existance of the file" you could for example check that for some test data it lies within an expected range of size. For example:
assert 3000 <= response.content_length <= 5000
assert 3000 <= len(response.data) <= 5000
Another level of verifying that the Excel file works would be attempting to load the data back into openpyxl and checking if it reports any problems. For example:
from io import BytesIO
from openpyxl import load_workbook
load_workbook(filename=BytesIO(response.data))
Here you risk running into some sort of exception like:
zipfile.BadZipFile: File is not a zip file
Which would indicate that the data contents of the file are invalid as a Excel file.

Download CSV File in Flask Best Practice

Right now I have a flask app in which part of the functionality allows me to select a date range and see data from a sql database from that selected date range. I then can click a button and it exports this to a csv file which is just saved in the flask project directory. I want the user to be able to download this csv file. I want to know what the best practice for a user to download a dynamic csv file. Should I send_file() and then delete the file after user has downloaded since this data shouldn't be saved and the user won't be using that file again. Should the file be saved to the database and then deleted out of the db? Or can I just keep it within the flask directory? Please provide insight if possible, thank you so much.

#brunns pointed it in very right direction.
You don't have to save the file in your database or in your file structure or anywhere. It will get created in memory on user request.
I've done this with django for pdf and for csv files it'll work in the same way with flask too. Basics are all same.
for python3 use io.StringIO, for python2 use StringIO
from io import StringIO
import csv
from flask import make_response
#app.route('/download')
def post(self):
si = StringIO.StringIO()
cw = csv.writer(si)
cw.writerows(csvList)
output = make_response(si.getvalue())
output.headers["Content-Disposition"] = "attachment; filename=export.csv"
output.headers["Content-type"] = "text/csv"
return output
Courtesy: vectorfrog

Based on #xxbinxx's answer, used with pandas
from io import StringIO
import csv
from flask import make_response
#app.route('/download')
def download_csv(self, df: pd.DataFrame):
si = StringIO()
cw = csv.writer(si)
cw.writerows(df.columns.tolist())
cw.writerows(df.values.tolist())
output = make_response(si.getvalue())
output.headers["Content-Disposition"] = "attachment; filename=export.csv"
output.headers["Content-type"] = "text/csv"
return output

Django Pandas to http response (download file)

Python: 2.7.11
Django: 1.9
Pandas: 0.17.1
How should I go about creating a potentially large xlsx file download? I'm creating a xlsx file with pandas from a list of dictionaries and now need to give the user possibility to download it. The list is in a variable and is not allowed to be saved locally (on server).
Example:
df = pandas.DataFrame(self.csvdict)
writer = pandas.ExcelWriter('pandas_simple.xlsx', engine='xlsxwriter')
df.to_excel(writer, sheet_name='Sheet1')
writer.save()
This example would just create the file and save it where the executing script is located. What I need is to create it to a http response so that the user would get a download prompt.
I have found a few posts about doing this for a xlsxwriter but non for pandas. I also think that I should be using 'StreamingHttpResponse' for this and not a 'HttpResponse'.

I will elaborate on what #jmcnamara wrote. This if for the latest versions of Excel, Pandas and Django. The import statements would be at the top of your views.py and the remaining code could be in a view:
import pandas as pd
from django.http import HttpResponse
try:
from io import BytesIO as IO # for modern python
except ImportError:
from io import StringIO as IO # for legacy python
# this is my output data a list of lists
output = some_function()
df_output = pd.DataFrame(output)
# my "Excel" file, which is an in-memory output file (buffer)
# for the new workbook
excel_file = IO()
xlwriter = pd.ExcelWriter(excel_file, engine='xlsxwriter')
df_output.to_excel(xlwriter, 'sheetname')
xlwriter.save()
xlwriter.close()
# important step, rewind the buffer or when it is read() you'll get nothing
# but an error message when you try to open your zero length file in Excel
excel_file.seek(0)
# set the mime type so that the browser knows what to do with the file
response = HttpResponse(excel_file.read(), content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
# set the file name in the Content-Disposition header
response['Content-Disposition'] = 'attachment; filename=myfile.xlsx'
return response

Jmcnamara is pointing you in the rigth direction. Translated to your question you are looking for the following code:
sio = StringIO()
PandasDataFrame = pandas.DataFrame(self.csvdict)
PandasWriter = pandas.ExcelWriter(sio, engine='xlsxwriter')
PandasDataFrame.to_excel(PandasWriter, sheet_name=sheetname)
PandasWriter.save()
sio.seek(0)
workbook = sio.getvalue()
response = StreamingHttpResponse(workbook, content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = 'attachment; filename=%s' % filename
Notice the fact that you are saving the data to the StringIO variable and not to a file location. This way you prevent the file being saved before you generate the response.

Maybe a bit off-topic, but it's worth pointing out that the to_csv method is generally faster than to_excel, since excel contains format information of the sheets. If you only have data and not formatting information, consider to_csv. Microsoft Excel can view and edit csv files with no problem.
One gain by using to_csv is that to_csv function can take any file-like object as the first argument, not only a filename string. Since Django response object is file-like, to_csv function can directly write to it. Some codes in your view function will look like:
df = <your dataframe to be downloaded>
response = HttpResponse(content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename=<default filename you wanted to give to the downloaded file>'
df.to_csv(response, index=False)
return response
Reference:
https://gist.github.com/jonperron/733c3ead188f72f0a8a6f39e3d89295d
https://docs.djangoproject.com/en/2.1/howto/outputting-csv/

With Pandas 0.17+ you can use a StringIO/BytesIO object as a filehandle to pd.ExcelWriter. For example:
import pandas as pd
import StringIO
output = StringIO.StringIO()
# Use the StringIO object as the filehandle.
writer = pd.ExcelWriter(output, engine='xlsxwriter')
# Write the data frame to the StringIO object.
pd.DataFrame().to_excel(writer, sheet_name='Sheet1')
writer.save()
xlsx_data = output.getvalue()
print len(xlsx_data)
After that follow the XlsxWriter Python 2/3 HTTP examples.
For older versions of Pandas you can use this workaround.

Just wanted to share a class-based view approach to this, using elements from the answers above. Just override the get method of a Django View. My model has a JSON field which contains the results of dumping a dataframe to JSON with the to_json method.
Python version is 3.6 with Django 1.11.
# models.py
from django.db import models
from django.contrib.postgres.fields import JSONField
class myModel(models.Model):
json_field = JSONField(verbose_name="JSON data")
# views.py
import pandas as pd
from io import BytesIO as IO
from django.http import HttpResponse
from django.views import View
from .models import myModel
class ExcelFileDownloadView(View):
"""
Allows the user to download records in an Excel file
"""
def get(self, request, *args, **kwargs):
obj = myModel.objects.get(pk=self.kwargs['pk'])
excel_file = IO()
xlwriter = pd.ExcelWriter(excel_file, engine='xlsxwriter')
pd.read_json(obj.json_field).to_excel(xlwriter, "Summary")
xlwriter.save()
xlwriter.close()
excel_file.seek(0)
response = HttpResponse(excel_file.read(),
content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
response['Content-Disposition'] = 'attachment; filename="excel_file.xlsx"'
return response
# urls.py
from django.conf.urls import url
from .views import ExcelFileDownloadView
urlpatterns = [
url(r'^mymodel/(?P<pk>\d+)/download/$', ExcelFileDownloadView.as_view(), name="excel-download"),]

You're mixing two requirements that should be separate:
Creating a .xlsx file using python or pandas--it looks like you're good on this part.
Serving a downloadable file (django); see this post or maybe this one

How do I return a CSV from a Pylons app?

I'm trying to return a CSV from an action in my webapp, and give the user a prompt to download the file or open it from a spreadsheet app. I can get the CSV to spit out onto the screen, but how do I change the type of the file so that the browser recognizes that this isn't supposed to be displayed as HTML? Can I use the csv module for this?
import csv
def results_csv(self):
data = ['895', '898', '897']
return data

To tell the browser the type of content you're giving it, you need to set the Content-type header to 'text/csv'. In your Pylons function, the following should do the job:
response.headers['Content-type'] = 'text/csv'

PAG is correct, but furthermore if you want to suggest a name for the downloaded file you can also set response.headers['Content-disposition'] = 'attachment; filename=suggest.csv'

Yes, you can use the csv module for this:
import csv
from cStringIO import StringIO
...
def results_csv(self):
response.headers['Content-Type'] = 'text/csv'
s = StringIO()
writer = csv.writer(s)
writer.writerow(['header', 'header', 'header'])
writer.writerow([123, 456, 789])
return s.getvalue()