I am a newbie to django and i'm desperate for help uploading and reading excel data without actually saving the data on the machine. I have written some code and taken some from the research i've done online.
These are my questions:
1. How do I Upload an excel file (without it being saved on the machine). I just want the excel file to populate some django fields and not save it.
How do I make django read the columns in the excel file and feeds into some other fields on another page. (How do I link them up?)
Most docs I've seen require me to hard-code the name of the excel file and its locations.IS there a way around this as I don't know where the user might be uploading from.
pls advice.
My views.py:
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
from credit.models import Document
from credit.forms import DocumentForm
def list(request):
if request.method == 'POST':
form = DocumentForm(request.POST, request.FILES)
if form.is_valid():
newdoc = Document(docfile = request.FILES['docfile'])
newdoc.save()
return HttpResponseRedirect(reverse('credit.views.list'))
else:
form = DocumentForm()
documents = Document.objects.all()
return render_to_response('credit/list.html',
{'documents': documents, 'form': form},
context_instance=RequestContext(request)
)
My models.py is:
class Document(models.Model):
docfile = models.FileField(upload_to='documents/')
#these are the models I want the excel columns to feed into
policies = DecimalNumberField()
capital = DecimalNumberField()
inflation = DecimalNumberField()
My forms.py is:
import os
import xlrd
IMPORT_FILE_TYPES = ['.xls', ]
class DocumentForm(forms.Form):
docfile = forms.FileField(label='Select a file')
def clean(self):
data = super(DocumentForm, self).clean()
if 'docfile' not in data:
raise forms.ValidationError(_('The Excel file is required to proceed'))
docfile = data['docfile']
extension = os.path.splitext(docfile.name)[1]
if not (extension in IMPORT_FILE_TYPES):
raise forms.ValidationError(u'%s is not a valid Excel file. Please make sure your input file is an Excel file )' % docfile.name)
file_data = StringIO.StringIO()
for chunk in docfile.chunks():
file_data.write(chunk)
data['file_data'] = file_data.getvalue()
file_data.close()
try:
xlrd.open_workbook(file_contents=data['file_data'])
except xlrd.XLRDError, e:
raise forms.ValidationError(_('Unable to open XLS file: %s' % e))
return data
#i do not want to do this (specify the exact file name). Need an alternative
sh = xlrd.open_workbook('documents\june.xls').sheet_by_index(1)
inflation = open("inflation.txt", 'w')
policies= open("policies.txt", 'w')
capital= open("access_to_finance.txt", 'w')
try:
for rownum in range(sh.nrows):
inflation.write(str(rownum)+ " = " +str(sh.cell(rownum, 1).value)+"\n")
policies.write(str(rownum)+ " = " +str(sh.cell(rownum, 2).value)+"\n")
capital.write(str(rownum)+ " = " +str(sh.cell(rownum, 3).value)+"\n")
finally:
inflation.close()
policies.close()
capital.close()
Then I have a list.html file:
{% if documents %}
<ul class="nav nav-tabs">
{% for document in documents %}
<li>{{ document.docfile.name }}</li>
{% endfor %}
</ul>
{% else %}
<p>Click Upload to go to Upload page</p>
{% endif %}
<form action="{% url list %}" method="post" enctype="multipart/form-data">
{% csrf_token %}
<p>{{ form.non_field_errors }}</p>
<p>{{ form.docfile.label_tag }} {{ form.docfile.help_text }}</p>
<p>
{{ form.docfile.errors }}
{{ form.docfile }}
</p>
<p><input type="submit" value="Upload" /></p>
</form>
Answer for question 1:
If your upload file is less then FILE_UPLOAD_MAX_MEMORY_SIZE(2.5MB), django puts the uploaded file in memory. If your file is bigger than 2.5MB, you can change FILE_UPLOAD_MAX_MEMORY_SIZE in your settings file
Related
I am developing a simple Data Visualization App, where a user can register, login upload a file and visualize its content.
I am using default User model, and a Detail Model having OneToOne relation with User Model. The Detail has 3 fields, which are:
OneToOne(User)
FileField()
TextField()
Now, I want to access the file that is saved in FileField, in views.pyand read it's content, then visualize it using Python'sNumPyandMatplotlib`, then finally show visualization results on the frontend.
I need a guidance that how should I start and approach this task? Basically, I need deep guidance in how to access and read the file in views.py?
My models.py is:
class Detail(models.Model):
user = models.OneToOneField(User, on_delete=models.CASCADE)
file = models.FileField(verbose_name="CSV File", upload_to='csv_files/')
file_desc = models.TextField("CSV File Description")
def __str__(self):
return ("{} ({} {})".format(self.user.email, self.user.first_name, self.user.last_name))
and in views.py, I am approaching it this way:
class VisualizeAPI(View):
template_name = 'accounts/visualize.html'
def get(self, request):
msg = {'message': "Please login to view this page."}
if request.user.is_authenticated:
detail, _ = Detail.objects.get_or_create(user=request.user)
context = {'detail': detail}
return render(request, self.template_name, context)
return render(request, self.template_name, msg)
and in template, I am approaching it this way:
<body>
<h1>Visualized Details</h1>
{% if request.user.is_authenticated %}
{{ detail }}
{% else %}
<h2>{{ message }}</h2>
{% endif %}
</body>
But it is not printing the content of the file on the frontend.
I will be glad for proper approach and guidance provided.
You need to parse the CSV file in your view and then pass it to your template.
import csv
from io import StringIO
class VisualizeAPI(View):
template_name = "accounts/visualize.html"
def get(self, request):
msg = {"message": "Please login to view this page."}
if request.user.is_authenticated:
detail, _ = Detail.objects.get_or_create(user=request.user)
if detail.file:
# Read file from detail object
file = detail.file.read().decode("utf-8")
# Parse file as csv
csv_data = csv.reader(StringIO(file), delimiter=",")
else:
csv_data = None
context = {
"detail": detail,
"csv_data": csv_data,
}
return render(request, self.template_name, context)
return render(request, self.template_name, msg)
Then you can print each row of your csv file in your template.
<body>
<h1>Visualized Details</h1>
{% if request.user.is_authenticated %}
{{ detail }}
{% for row in csv_data %}
{{ row }}
{% endfor %}
{% else %}
<h2>{{ message }}</h2>
{% endif %}
</body>
I want to save age_s, age_e , gender and filename in SQLite database using codes below, but I have a problem.
When I upload the file, my file name is stored in the SQLite database, but the file itself is not stored in my system directory. How can I save a file in my system directory ?
I know that the problem is from the html file but I can not save the file in the directory and save the file name in the database, both actions at the same time.
app.py
#app.route("/new_contact", methods=('GET', 'POST'))
def new_contact():
'''
Create new contact
'''
form = ContactForm()
if form.validate_on_submit():
my_contact = Contact()
form.populate_obj(my_contact)
db.session.add(my_contact)
try:
db.session.commit()
# User info
flash('Contact created correctly', 'success')
return redirect(url_for('contacts'))
except:
db.session.rollback()
flash('Error generating contact.', 'danger')
return render_template('web/new_contact.html', form=form)
new_contact.html
{% extends 'layouts/master.html' %}
{% block title %}New contact{% endblock %}
{% block body %}
<h1>New contact</h1>
<form action="{{ url_for('new_contact') }}" method="post">
{{ generate_fields(form) }}
<input type="submit" class="btn btn-success" value="Add">
</form>
{% endblock %}
forms.py
from flask_wtf import FlaskForm
from wtforms import StringField,SelectField,IntegerField,FileField
from wtforms.ext.sqlalchemy.fields import QuerySelectField
from wtforms.validators import DataRequired, Email, Length
class ContactForm(FlaskForm):
age_s = SelectField(u'age_s', choices=age_types)
age_e = SelectField(u'age_e', choices=age_types1)
#age_e = IntegerField('age_e', validators=[Length(min=-1, max=100, message='You cannot have more than 100 characters')])
gender = SelectField('gender', choices=gender_types)
#filename = StringField('filename', validators=[Length(min=-1, max=20, message='You cannot have more than 20 characters')])
filename = FileField()
Try saving it to your preferred path, per Flask docs
I just have a upload file form in my django project (I'm using 1.10). Everything is working pretty fine actually, but as people will be uploading a csv file into the server I need to limit my logic into just request the file with a specific name and format.
This is my code:
views.py
def list(request):
# Handle file upload
if request.method == 'POST':
form = DocumentForm(request.POST, request.FILES)
if form.is_valid():
newdoc = Document(docfile = request.FILES['docfile'])
newdoc.save()
# Redirect to the document list after POST
return HttpResponseRedirect(reverse('action'))
else:
messages.add_message(request, messages.ERROR,
'Something went wrong with files! Please contact the system administrator')
return render(request, 'upaction-report.html')
# Load documents for the list page
# Render list page with the documents and the form
return render(request, 'upaction-report.html', {'form': form})
forms.py
class DocumentForm(forms.Form):
docfile = forms.FileField(
label='File format should be under the following format:',
help_text='- .csv format and under the name "Action_Report"',
)
template html
<form action="{% url "list" %}" method="post" enctype="multipart/form-data">
{% csrf_token %}
<p>{{ form.non_field_errors }}</p>
<p>{{ form.docfile.label_tag }}</p>
<p> {{ form.docfile.help_text }}</p>
<p>
{{ form.docfile }}
</p>
<p><input type="submit" value="Upload File"/></p>
</form>
Any guess on how applying that limit? In case the file is not the right just to pop up a message telling to try again until it has the right format and name. Thanks for yout time, if you need more details just let me know.
You can validate filename inside DocumentForm clean method like this:
class DocumentForm(forms.Form):
docfile = forms.FileField(
label='File format should be under the following format:',
help_text='- .csv format and under the name "Action_Report"',
)
def clean(self):
docfile = self.cleaned_data.get('docfile')
if not docfile:
raise forms.ValidationError('file is required')
if not docfile.name.startswith('filename'):
raise forms.ValidationError('incorrect file name')
if not docfile.name.endswith('fileformat'):
raise forms.ValidationError('incorrect file format')
return super(DocumentForm, self).clean()
I am trying to import csv file i am able to import without any problem but the present functionality accepts all file types, i want the functionality to accept only csv file. below is the view.py and template file.
myapp/views.py
def list(request):
# Handle file upload
if request.method == 'POST':
form = DocumentForm(request.POST, request.FILES)
if form.is_valid():
importing_file(request.FILES['docfile'])
myapp/templates/myapp/index.html
<form action="{% url 'ml:list' %}" method="post" enctype="multipart/form-data">
{% csrf_token %}
<p>{{ form.non_field_errors }}</p>
<p>{{ form.docfile.label_tag }} {{ form.docfile.help_text }}</p>
<p>
{{ form.docfile.errors }}
{{ form.docfile }}
</p>
<p><input type="submit" value="Upload"/></p>
</form>
EDIT
I could find a workaround by adding validate_file_extension as per the
django documentation
myapp/forms.py
def validate_file_extension(value):
if not value.name.endswith('.csv'):
raise forms.ValidationError("Only CSV file is accepted")
class DocumentForm(forms.Form):
docfile = forms.FileField(label='Select a file',validators=[validate_file_extension])
form widget to validate file extension
csv_file = forms.FileField(widget=forms.FileInput(attrs={'accept': ".csv"}))
added code snippet to the forms.py file to validate file extension and now it is working fine.
def validate_file_extension(value):
if not value.name.endswith('.csv'):
raise forms.ValidationError("Only CSV file is accepted")
class DocumentForm(forms.Form):
docfile = forms.FileField(label='Select a file',validators=[validate_file_extension])
Following code work fine to store thumbnails images in /media/ directory. i want to store all images in following directory but django also save images path in database. Please let me know, how i can stop django to save their path and address in database ?
model.py : Code
from django.db import models
class Document(models.Model):
thumbnail = models.ImageField()
views.py : Code
from .models import Document
from .forms import DocumentForm
if (request.FILES['thumbnail']):
newdoc = Document(thumbnail = request.FILES['thumbnail'])
newdoc.save()
forms.py : Code
from django import forms
class DocumentForm(forms.Form):
thumbnail = forms.ImageField(
label='Select a file',
help_text='max. 42 megabytes'
)
template.html : Code
<form action="" method="post" id="imgUpload" enctype="multipart/form-data">
<p>{{ form.non_field_errors }}</p>
<p>{{ form.thumbnail.label_tag }} {{ form.thumbnail.help_text }}</p>
<p>
{{ form.thumbnail.errors }}
{{ form.thumbnail }}
<button type="submit" id="thumbUpload" class="btn btn-success"><i class="glyphicon glyphicon-picture"></i> Upload Thumbnails</button>
</form>
you have to override your form method , before save your validform , or you can handle with your code , you don't need to have model or form you can handle with
image = request.FILES['myfile']
now you have your image you can save it directly without model
example reproduced from Django Documentation.
from django.http import HttpResponseRedirect
from django.shortcuts import render_to_response
def upload_file(request):
if request.method == 'POST':
form = UploadFileForm(request.POST, request.FILES)
if form.is_valid():
handle_uploaded_file(request.FILES['file'])
return HttpResponseRedirect('/success/url/')
else:
form = UploadFileForm()
return render_to_response('upload.html', {'form': form})
def handle_uploaded_file(f):
destination = open('file path', 'wb+')
for chunk in f.chunks():
destination.write(chunk)
destination.close()
you can change 'file path ' with your path where you want to store your image
I'm not clear about what you are trying to achieve but the solution is barely simple. Just get the file object and save...
oh.. one more thing just remove all the thumbnail in models.py and other files...
def handle_uploaded_file(f):
destination = open('your_media_path/media/file.png', 'wb+')
for chunk in f.chunks():
destination.write(chunk)
destination.close()
in your POST view
f = request.FILES['thumbnail']
handle_uploaded_file(f)