We Keep Coding

Django, pandas, excel: uploading files, parsing them with pandas in Django

I have a big command-line script for parsing data in Excel (with pandas) and I want to wrap it with Django. I've tried both uploading files thru request.FILES and pandas, but get stuck on uploading file and, for example, saving it (not necessarily but just to check the upload for now).
Haven't had any problems with other apps on Django which didn't require uploading and parsing anything external and thought that would be much easier..:)
I've also tried Redirecting, doesn't really work, the only redirect which is actually happening is action in the form tag..
Here are the code snippets:
views.py:
def uploads(request):
if request.method == 'POST':
form = DocumentForm(request.POST, request.FILES)
if form.is_valid():
excel_file = request.FILES['document']
excel_file.save()
return render(request, 'index.html')
else:
form = DocumentForm()
return render(request, 'index.html', {'form': form})
models.py
class Document(models.Model):
document = models.FileField(upload_to="files/")
forms.py:
class DocumentForm(forms.ModelForm):
class Meta:
model = Document
fields = ('document', )
index.html:
<form
action="{% url 'reports'%}"
method="post"
enctype="multipart/form-data"
>
{% csrf_token %}
<span>
Upload .xlsx file <input type="file" name="document" />
</span>
<button type="submit">
SUBMIT
</button>
</form>

I think you must save the form's content actually:
form.save()

The reason was that I had another function in views.py leading to the same url route.

Django handle photo upload

I have model whiuch has photo attachment.
class Forum_message(models.Model):
text = models.TextField()
photo = models.ImageField(upload_to= 'forum_attachments')
I have my form(I prefer writing forms in pure html so I can have full access to modifying them)
<form action="/forum_new" method="post" enctype="multipart/form-data">
{% csrf_token %}
<input id="img_msg" type="file">
<textarea name="new_msg"></textarea>
<input type="submit" value="Submit"/>
</form>
What do I write in my function
def forum_new(request): in views.py to handle file upload and save new forum message to database?

I needed something like this:
def forum_new(request):
msg = Forum_message()
msg.user = request.user
if 'image' in request.FILES:
msg.photo = request.FILES['image']
msg.text = request.GET.get('text')
msg.save()

In views.py, you can use what is propose on the django website (https://docs.djangoproject.com/en/1.11/topics/http/file-uploads/)
If the picture is not so large, you can directly handle it as follow :
def forum_new(request):
if request.method == 'POST':
form = your_form_here(request.POST, request.FILES)
if form.is_valid():
form.save()
return redirect('/bar.html/')
else:
form = your_form_here()
return render(request, 'url.html', {'form': form})
The file should be saved in your media files in the folder forum_attachments

Django upload only csv file

I am trying to import csv file i am able to import without any problem but the present functionality accepts all file types, i want the functionality to accept only csv file. below is the view.py and template file.
myapp/views.py
def list(request):
# Handle file upload
if request.method == 'POST':
form = DocumentForm(request.POST, request.FILES)
if form.is_valid():
importing_file(request.FILES['docfile'])
myapp/templates/myapp/index.html
<form action="{% url 'ml:list' %}" method="post" enctype="multipart/form-data">
{% csrf_token %}
<p>{{ form.non_field_errors }}</p>
<p>{{ form.docfile.label_tag }} {{ form.docfile.help_text }}</p>
<p>
{{ form.docfile.errors }}
{{ form.docfile }}
</p>
<p><input type="submit" value="Upload"/></p>
</form>
EDIT
I could find a workaround by adding validate_file_extension as per the
django documentation
myapp/forms.py
def validate_file_extension(value):
if not value.name.endswith('.csv'):
raise forms.ValidationError("Only CSV file is accepted")
class DocumentForm(forms.Form):
docfile = forms.FileField(label='Select a file',validators=[validate_file_extension])

form widget to validate file extension
csv_file = forms.FileField(widget=forms.FileInput(attrs={'accept': ".csv"}))

added code snippet to the forms.py file to validate file extension and now it is working fine.
def validate_file_extension(value):
if not value.name.endswith('.csv'):
raise forms.ValidationError("Only CSV file is accepted")
class DocumentForm(forms.Form):
docfile = forms.FileField(label='Select a file',validators=[validate_file_extension])

Uploading excel data into django without saving the file

I am a newbie to django and i'm desperate for help uploading and reading excel data without actually saving the data on the machine. I have written some code and taken some from the research i've done online.
These are my questions:
1. How do I Upload an excel file (without it being saved on the machine). I just want the excel file to populate some django fields and not save it.
How do I make django read the columns in the excel file and feeds into some other fields on another page. (How do I link them up?)
Most docs I've seen require me to hard-code the name of the excel file and its locations.IS there a way around this as I don't know where the user might be uploading from.
pls advice.
My views.py:
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
from credit.models import Document
from credit.forms import DocumentForm
def list(request):
if request.method == 'POST':
form = DocumentForm(request.POST, request.FILES)
if form.is_valid():
newdoc = Document(docfile = request.FILES['docfile'])
newdoc.save()
return HttpResponseRedirect(reverse('credit.views.list'))
else:
form = DocumentForm()
documents = Document.objects.all()
return render_to_response('credit/list.html',
{'documents': documents, 'form': form},
context_instance=RequestContext(request)
)
My models.py is:
class Document(models.Model):
docfile = models.FileField(upload_to='documents/')
#these are the models I want the excel columns to feed into
policies = DecimalNumberField()
capital = DecimalNumberField()
inflation = DecimalNumberField()
My forms.py is:
import os
import xlrd
IMPORT_FILE_TYPES = ['.xls', ]
class DocumentForm(forms.Form):
docfile = forms.FileField(label='Select a file')
def clean(self):
data = super(DocumentForm, self).clean()
if 'docfile' not in data:
raise forms.ValidationError(_('The Excel file is required to proceed'))
docfile = data['docfile']
extension = os.path.splitext(docfile.name)[1]
if not (extension in IMPORT_FILE_TYPES):
raise forms.ValidationError(u'%s is not a valid Excel file. Please make sure your input file is an Excel file )' % docfile.name)
file_data = StringIO.StringIO()
for chunk in docfile.chunks():
file_data.write(chunk)
data['file_data'] = file_data.getvalue()
file_data.close()
try:
xlrd.open_workbook(file_contents=data['file_data'])
except xlrd.XLRDError, e:
raise forms.ValidationError(_('Unable to open XLS file: %s' % e))
return data
#i do not want to do this (specify the exact file name). Need an alternative
sh = xlrd.open_workbook('documents\june.xls').sheet_by_index(1)
inflation = open("inflation.txt", 'w')
policies= open("policies.txt", 'w')
capital= open("access_to_finance.txt", 'w')
try:
for rownum in range(sh.nrows):
inflation.write(str(rownum)+ " = " +str(sh.cell(rownum, 1).value)+"\n")
policies.write(str(rownum)+ " = " +str(sh.cell(rownum, 2).value)+"\n")
capital.write(str(rownum)+ " = " +str(sh.cell(rownum, 3).value)+"\n")
finally:
inflation.close()
policies.close()
capital.close()
Then I have a list.html file:
{% if documents %}
<ul class="nav nav-tabs">
{% for document in documents %}
<li>{{ document.docfile.name }}</li>
{% endfor %}
</ul>
{% else %}
<p>Click Upload to go to Upload page</p>
{% endif %}
<form action="{% url list %}" method="post" enctype="multipart/form-data">
{% csrf_token %}
<p>{{ form.non_field_errors }}</p>
<p>{{ form.docfile.label_tag }} {{ form.docfile.help_text }}</p>
<p>
{{ form.docfile.errors }}
{{ form.docfile }}
</p>
<p><input type="submit" value="Upload" /></p>
</form>

Answer for question 1:
If your upload file is less then FILE_UPLOAD_MAX_MEMORY_SIZE(2.5MB), django puts the uploaded file in memory. If your file is bigger than 2.5MB, you can change FILE_UPLOAD_MAX_MEMORY_SIZE in your settings file

What am I not doing right in this django file upload form?

This is my form:
from django import forms
class UploadFileForm(forms.Form):
titl = forms.CharField(max_length=50)
ffile = forms.FileField()
This is my views.py file:
def handle_uploaded_file(file_path):
print "handle_uploaded_file"
dest = open(file_path.name,"wb")
for chunk in file_path.chunks():
dest.write(chunk)
dest.close()
def handle_upload(request):
c = {}
c.update(csrf(request))
if request.method == "POST":
form = UploadFileForm(request.POST)
if form.is_valid():
handle_uploaded_file(request.FILES["ffile"])
return HttpResponseRedirect("/thanks")
else:
form = UploadFileForm()
c.update({"form":form})
return render_to_response("upload.html",c)
And this is the content of upload.html:
<form enctype="multipart/form-data" method="post" action="/handle_upload/">
{% csrf_token %}
{{ form.as_p }}
<input type="submit" value="Upload it"/>
</form>
Whenever I try to submit the form, I get a "This field is required" for the ffile field. What am I doing wrong? Just to mention, I am uploading a file each time.

Just for future reference. I had the same error, though I included request.FILES in form initialization. The problem was in the template: I forgot to add enctype="multipart/form-data" attribute to the <form> tag.

form = UploadFileForm(request.POST, request.FILES)

If you have included request.FILES and added the enctype="multipart/form-data", but are still seeing this error, it could be you are not declaring the <input> correctly.
For example, if explicitly declare the input html in your template like:
<input type="file" value="Upload CSV File" />
You may not be passing the expected input id or name attributes of the input form element.
Be sure that your template is using the form element tag, i.e. {{ form.file }},
which django will then render as: <input id="id_file" name="file" type="file" required=""> on the page.