We Keep Coding

Django input CSV file to create a new table. (CSV file convert into a table in the database)

I wanted some help with a project I'm working on:
So I have to take a CSV as input from the user and convert the CSV save it as a Table in the Database.
So I was able to take the input from the user and save it in the MEDIA_ROOT.
But now I'm unsure how to create a model without knowing the columns and so on. (I know I can get the columns from pandas(in views.py) but how to send that column details to models.py)
I'm new to Django and very much confused with so many files. Please help.
Note: I don't want to save the CSV file I want to convert it into a table in the database.
views.py
import pandas as pd
from django.shortcuts import render
from django.http import HttpResponse
from django.views.generic import TemplateView, ListView, CreateView
from django.core.files.storage import FileSystemStorage
def home(request):
return render(request, 'index.html')
def upload(request):
if request.method == 'POST':
upload_file = request.FILES['csvfile']
fs = FileSystemStorage()
fs.save(upload_file.name, upload_file)
dataset = pd.read_csv('media/'+upload_file.name)
colums = list(dataset.columns)
return render(request, 'upload.html')
models.py
from assignment.settings import MEDIA_ROOT
from django.db import models
from django.db.models.fields import CharField
class CSV(models.Model):
csvfile = models.FileField(upload_to='CSV')

The best scenario here is to use a single model called Files and store CSV in a JSON field. You can easily read CSV as a Dict using csv.DictReader and save it in database.

Saving PDFs to disk as they are generated with django-wkhtmltopdf

What I'm trying to implement is this:
User sends query parameters from React FE microservice to the Django BE microservice.
URI is something like /api/reports?startingPage=12&dataView=Region
These PDFs are way too big to be generated in FE, so doing it server side
Request makes its way into the view.py where the data related to dataView=Region is queried from the database, each row is iterated through and a PDF report is generated for each item
Each dataView=Region can consist of a few hundred items and each of those items is its own report that can be a page long or several pages long
As the reports are generated, they should be saved to the server persistent volume claim and not be sent back to FE until they have all run.
When they have all run, I plan to use pypdf2 to combine all of the PDFs into one large file.
At that point, the file is sent back to the FE to download.
I'm only working on 1. and 3. at this point and I'm unable to:
Get the files to save to storage
Prevent the default behavior of the PDF being sent back to the FE after it has been generated
The PDFs are being generated, so that is good.
I'm trying to implement the suggestions as found here, but I'm not getting the desired results:
Save pdf from django-wkhtmltopdf to server (instead of returning as a response)
This is what I currently have on the Django side:
# urls.py
from django.urls import path
from .views import GeneratePDFView
app_name = 'Reports'
urlpatterns = [
path('/api/reports',
GeneratePDFView.as_view(), name='generate_pdf'),
]
# views.py
from django.conf import settings
from django.views.generic.base import TemplateView
from rest_framework.permissions import IsAuthenticated
from wkhtmltopdf.views import PDFTemplateResponse
# Create your views here.
class GeneratePDFView(TemplateView):
permission_classes = [IsAuthenticated]
template_name = 'test.html'
filename = 'test.pdf'
def generate_pdf(self, request, **kwargs):
context = {'key': 'value'}
# generate response
response = PDFTemplateResponse(
request=self.request,
template=self.template_name,
filename=self.filename,
context=context,
cmd_options={'load-error-handling': 'ignore'})
self.save_pdf(response.rendered_content, self.filename)
# Handle saving the document
# This is what I'm using elsewhere where files are saved and it works there
def save_pdf(self, file, filename):
with open(settings.PDF_DIR + '/' + filename, 'wb+') as destination:
for chunk in file.chunks():
destination.write(chunk)
# settings.py
...
DOWNLOAD_ROOT = '/mnt/files/client-downloads/'
MEDIA_ROOT = '/mnt/files/client-submissions/'
PDF_DIR = '/mnt/files/pdf-sections/'
...
I should note the other DOWNLOAD_ROOT and MEDIA_ROOT are working fine where the app uses them. I've even tried using settings.MEDIA_ROOT because I know it works, but still nothing is saved there. But as you can see, I'm starting out super basic and haven't added a query, loops, etc.
My save_pdf() is different than the SO question I linked to because that is what I'm using in other parts of my application and it is saving files fine there. I did try what they provided in the SO question, but had the same results with it not saving. That being:
with open("file.pdf", "wb") as f:
f.write(response.rendered_content)
So what do I need to do to get these PDFs to save to disk?
Perhaps I need to be using a different library for my needs as django-wkhtmltopdf seems to do a number of things out of the box that I don't want that I'm not clear I can override.

OK, my smooth brain gained a few ripples overnight and figured it out this morning:
# views.py
class GeneratePDFView(TemplateView):
permission_classes = [IsAuthenticated]
def get(self, request, *args, **kwargs):
template_name = 'test.html'
filename = 'test.pdf'
context = {'key': 'value'}
# generate response
response = PDFTemplateResponse(
request=request,
template=template_name,
filename=filename,
context=context,
cmd_options={'load-error-handling': 'ignore'})
# write the rendered content to a file
with open(settings.PDF_DIR + '/' + filename, "wb") as f:
f.write(response.rendered_content)
return HttpResponse('Hello, World!')
This saved the PDF to disk and also did not respond with the PDF. Obviously a minimally functioning example that I can expand on, but at least got those two issues figured out.

How do I assign images stored locally to an ImageField field and return its URL in Django?

I'm new with Django and I have a problem I hope you can help me to solve.
First of all, I'm working with Django 1.8.3 and Python 2.7.6 on a Virtual Env using Apache and Postgres as my DBMS.
I have the following model on my models.py file:
class Hotel(models.Model):
# ...
name = models.TextField(blank=True, null=True)
img = models.ImageField(upload_to='images/',blank=True, null=True)
stars = models.IntegerField(blank=True, null=True)
# ...
class Meta:
managed = False
db_table = 'hotel'
My views.py file looks like this
from django.http import HttpResponse, Http404
from webServices.models import Hotels
from django.core import serializers
from django.core.exceptions import ObjectDoesNotExist
import json
# ...
def all_hotels(request):
hotels = Hotel.objects.values('name', 'img', 'stars')
data = json.dumps(list(hotels), indent = 4)
return HttpResponse(data, content_type='application/json')
And my MEDIA_URL and MEDIA_ROOT variables on settings.py file are defined as follows:
MEDIA_URL = '/media/'
MEDIA_ROOT = '/path_to_my_projects_directory/media/'
As you can see, I'm working on a server that returns a json with the data for every hotel in the DB. It certainly works fine.
Json's img field has been 'null' until now that I need to associate every image stored locally to its hotel and return its URL in it.
The problem is I don't know how.
It's good to know as well that the DB is updated via python script since the data reside in a Google SpreadSheet (I'll show you if asked).
I've been searching all day long and I've found this post
how to manually assign imagefield in Django
and this article
http://www.revsys.com/blog/2014/dec/03/loading-django-files-from-code/, but I need them to be a little bit more specific.

Hi try this one in your views.py
from django.conf import settings
url = request.scheme
url += "://"
url += request.META['HTTP_HOST']
url += settings.MEDIA_URL
_len = len(hotels)
_range = range(0, _len)
hotels = Hotel.objects.values('name', 'img', 'stars')
for x in _range:
hotels[x]['img'] = "%s%s" % (url, hotels[x]['img'])
data = json.dumps(list(hotels), indent = 4)
return HttpResponse(data, content_type='application/json')

python script to handle uploaded file via http post

I am working on a django project where i try to upload a file via http post request.
my upload script is :
url=r'http://MYSITEURL:8000/upload'
files={'file':open('1.png','rb')}
r=requests.post(url,files=files)
my receiving side is in my django site , in views.py:
def upload_image(request):
from py_utils import open_py_shell;open_py_shell.open_py_shell()
when i do request.FILES i can see all the post details,
what i want to know is how to save it in the server side once i got the post request

What you have in request.FILES is InMemoryUploadedFile. You just need to save it somewhere in file system.
This is example method taken from Django docs:
def handle_uploaded_file(f):
with open('some/file/name.txt', 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)

I think you can work with models well. It will be the right way for Django. Here is an example, models.py file:
from django.db import models
from django.conf import settings
import os
import hashlib
def instanced_file(instance, filename):
splitted_name = filename.split('.')
extension = splitted_name[-1]
return os.path.join('files', hashlib.md5(str(instance.id).encode('UTF-8')).hexdigest() + '.' + extension)
class File(models.Model):
name = models.FileField('File', upload_to = instanced_file)
def get_file_url(self):
return '%s%s' % (settings.MEDIA_URL, self.name)
def __str__(self):
return self.name
def __unicode__(self):
return self.name
After the creating models create forms and go on.

protecting user uploaded files django

How can I allow users to upload files to their own, user designated folder, and only see files that they have uploaded? I am using django file-transfer. Currently it gives me a choice of what file to put the media in, but I can put it in any user's file and view every user's media. Here is my uploads/models.py:
from django.db import models
from django.contrib.auth.models import User, UserManager
def uploadmodel_file_upload_to(instance, filename):
print 'instance.user.username = '+ str(instance.user.username)
return 'uploads/%s/%s' % (instance.user.username, filename)
class UploadModel(models.Model):
user = models.ForeignKey('auth.user')
file = models.FileField(upload_to=uploadmodel_file_upload_to)

uploadmodel_file_upload_to returns a relative path. To build the full path, django prepends settings.MEDIA_ROOT. MEDIA_ROOT is supposed to be public readable.
So we want to save the file outside MEDIA_ROOT. Add something like this to settings.py:
import os.path
PROJECT_ROOT=os.path.abspath(os.path.dirname(__file__))
PROTECTED_MEDIA_ROOT=os.path.join(PROJECT_ROOT, 'protected_uploads')
Now you can update uploadmodel_file_upload_to to return an absolute path:
def uploadmodel_file_upload_to(instance, filename):
return '%s/%s/%s' % (settings.PROTECTED_MEDIA_ROOT, instance.user.username,
filename)
Now that the files are saved in /project/path/protected_uploads, we need to add a view to serve it, for example:
import os
import mimetypes
from django import shortcuts
from django import http
from django.conf import settings
from django.views.static import was_modified_since
from django.utils.http import http_date
from .models import *
def serve_upload(request, upload_id):
upload = shortcuts.get_object_or_404(UploadModel, pk=upload_id)
fullpath = upload.file.path
if request.user != upload.user:
return http.HttpResponseForbidden()
statobj = os.stat(fullpath)
mimetype, encoding = mimetypes.guess_type(fullpath)
mimetype = mimetype or 'application/octet-stream'
if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'),
statobj.st_mtime, statobj.st_size):
return http.HttpResponseNotModified(mimetype=mimetype)
response = http.HttpResponse(open(fullpath, 'rb').read(), mimetype=mimetype)
response["Last-Modified"] = http_date(statobj.st_mtime)
response["Content-Length"] = statobj.st_size
if encoding:
response["Content-Encoding"] = encoding
return response
And a URL:
url(r'serve_upload/(?P<upload_id>\d+)/$', 'serve_upload'),