I'm writing a Django app, where users can upload CSV files. Therefore I've created an upload model with three validators:
One that checks the file extension (FileExtensionValidator),
one for the MIME type validation (ValidateFileType),
and a third one for parsing the CSV file and checking for data types, right number of columns and so on (ValidateCsv).
It'd be reasonable to check the upload only with the next validator if the preceding validation didn't raise a ValidationError.
For instance, the user could upload a .py file. This would raise an error in all three validators, but I want to avoid, that Django checks the MIME type or even tries to treat and parse a .py file as a CSV file, although the file extension wasn't correct right from the beginning.
So here is my model for the user's upload:
models.py
from django.db import models
from .utils import unique_file_path
from django.core.validators import FileExtensionValidator
from .validators import ValidateFileType, ValidateCsv
class Upload(models.Model):
date_uploaded = models.DateTimeField(auto_now_add=True)
file = models.FileField(upload_to=unique_file_path, validators=[FileExtensionValidator(['csv']), ValidateFileType, ValidateCsv], max_length=255)
With this validators list all three validations are always performed and I can see all the error messages in upload_form.errors. For example:
File extension 'py' is not allowed. Allowed extensions are: 'csv'.
File type text/x-python not supported.
Some data is invalid. Please check your CSV file.
forms.py
from django import forms
from .models import Upload
class UploadForm(forms.ModelForm):
class Meta:
model = Upload
view.py
from .forms import UploadForm
def someView(request):
upload_form = UploadForm()
...
context = {'upload_form': upload_form}
return render(request, 'someTemplate.html', context)
Do you have an idea, what's the best approach to make such a hierarchical chain of validators? Of course I could write some big all-in-one validator function, but since I use a Django core validator, I dont't want to rewrite this one, but combine existing django validators with my own ones.
You don't need to rewrite the django core validator.
In your validators file:
from django.core.validators import FileExtensionValidator
def ValidateFileType(value):
....your code....
def ValidateCsv(value):
....your code....
def csv_validator(value):
'''Your all in one function'''
extension = FileExtensionValidator(['csv'])
extension(value) #FileExtensionValidator is a callable class. See docs for that.
ValidateFileType(value)
ValidateCsv(value)
Don't know if that's the best way, but it should do the trick.
Related
I have the following model for my students to upload their tasks to an application that I am creating, but I have a problem, I need to pass an instance of the model between views, but since it is not serializable, I can not save it in a session attribute. Keep in mind that in one view I create the object without saving it in the database and in the other I perform operations with the object and finally I save it. Any idea how I can do this?
from gdstorage.storage import GoogleDriveStorage
gd_storage = GoogleDriveStorage()
class Homework(models.Model):
code = models.AutoField(primary_key=True)
student = models.ForeignKey('Student', on_delete=models.PROTECT)
title = models.CharField(unique=True, max_length=100)
attached_file = models.FileField(upload_to='files/homeworks/', validators=[validate_file_size], storage=gd_storage)
As #dirkgroten says, you can add an additional field to your model that is called status and by default assign it the value of temporary. In addition to this you can review the package code.
Finally to delete a file in Google Drive as a storage backend is very simple. Use the following
gd_storage.delete(name_file)
So change in the code of #dirkgroten
from django.core.files.storage import default_storage
#receiver (post_delete, sender=Homework)
def remove_file (sender, instance, **kwargs):
if instance.attached_file is not None:
gd_storage.delete(instance.attached_file.name)
The only way to keep "state" between views is to save to the database (or other permanent storage). That's what the session does for you.
If you can't serialise to save in the session, then you have no alternative but to save a temporary object to the database. You could mark it as temporary and add a timestamp. And in the next view mark it as committed. And if needed clean up once in a while, removing old temporary objects.
To remove the associated file with old temporary objects, you can add a signal handler for the post_delete signal:
from django.core.files.storage import default_storage
#receiver(post_delete, sender=Homework)
def remove_file(sender, instance, **kwargs)
path = instance.attached_file.name
if path:
default_storage.delete(path)
I am using django-import-export 1.0.1 with admin integration in Django 2.1.1. I have two models
from django.db import models
class Sector(models.Model):
code = models.CharField(max_length=30, primary_key=True)
class Location(models.Model):
code = models.CharField(max_length=30, primary_key=True)
sector = ForeignKey(Sector, on_delete=models.CASCADE, related_name='locations')
and they can be imported/exported just fine using model resources
from import_export import resources
from import_export.fields import Field
from import_export.widgets import ForeignKeyWidget
class SectorResource(resources.ModelResource):
code = Field(attribute='code', column_name='Sector')
class Meta:
model = Sector
import_id_fields = ('code',)
class LocationResource(resources.ModelResource):
code = Field(attribute='code', column_name='Location')
sector = Field(attribute='sector', column_name='Sector',
widget=ForeignKeyWidget(Sector, 'code'))
class Meta:
model = Location
import_id_fields = ('code',)
and import/export actions can be integrated into the admin by
from django.contrib import admin
from import_export.admin import ImportExportModelAdmin
class SectorAdmin(ImportExportModelAdmin):
resource_class = SectorResource
class LocationAdmin(ImportExportModelAdmin):
resource_class = LocationResource
admin.site.register(Sector, SectorAdmin)
admin.site.register(Location, LocationAdmin)
For Reasons™, I would like to change this set-up so that a spreadsheet of Locations which does not contain a Sector column can be imported; the value of sector (for each imported row) should be taken from an extra field on the ImportForm in the admin.
Such a field can indeed be added by overriding import_action on the ModelAdmin as described in Extending the admin import form for django import_export. The next step, to use this value for all imported rows, is missing there, and I have not been able to figure out how to do it.
EDIT(2): Solved through the use of sessions. Having a get_confirm_import_form hook would still really help here, but even better would be having the existing ConfirmImportForm carry across all the submitted fields & values from the initial import form.
EDIT: I'm sorry, I thought I had this nailed, but my own code wasn't working as well as I thought it was. This doesn't solve the problem of passing along the sector form field in the ConfirmImportForm, which is necessary for the import to complete. Currently looking for a solution which doesn't involve pasting the whole of import_action() into an ImportMixin subclass. Having a get_confirm_import_form() hook would help a lot here.
Still working on a solution for myself, and when I have one I'll update this too.
Don't override import_action. It's a big complicated method that you don't want to replicate. More importantly, as I discovered today: there are easier ways of doing this.
First (as you mentioned), make a custom import form for Location that allows the user to choose a Sector:
class LocationImportForm(ImportForm):
sector = forms.ModelChoiceField(required=True, queryset=Sector.objects.all())
In the Resource API, there's a before_import_row() hook that is called once per row. So, implement that in your LocationResource class, and use it to add the Sector column:
def before_import_row(self, row, **kwargs):
sector = self.request.POST.get('sector', None)
if contract:
self.request.session['import_context_sector'] = sector
else:
# if this raises a KeyError, we want to know about it.
# It means that we got to a point of importing data without
# contract context, and we don't want to continue.
try:
sector = self.request.session['import_context_sector']
except KeyError as e:
raise Exception("Sector context failure on row import, " +
f"check resources.py for more info: {e}")
row['sector'] = sector
(Note: This code uses Django sessions to carry the sector value from the import form to the import confirmation screen. If you're not using sessions, you'll need to find another way to do it.)
This is all you need to get the extra data in, and it works for both the dry-run preview and the actual import.
Note that self.request doesn't exist in the default ModelResource - we have to install it by giving LocationResource a custom constructor:
def __init__(self, request=None):
super()
self.request = request
(Don't worry about self.request sticking around. Each LocationResource instance doesn't persist beyond a single request.)
The request isn't usually passed to the ModelResource constructor, so we need to add it to the kwargs dict for that call. Fortunately, Django Import/Export has a dedicated hook for that. Override ImportExportModelAdmin's get_resource_kwargs method in LocationAdmin:
def get_resource_kwargs(self, request, *args, **kwargs):
rk = super().get_resource_kwargs(request, *args, **kwargs)
rk['request'] = request
return rk
And that's all you need.
I have a ML model saved in a pkl (pickel file), I have no problem loading this model and using it for prediction, even I have a rest service that expose it, the only problem is that I load the model in every request, something like this:
https://www.analyticsvidhya.com/blog/2017/09/machine-learning-models-as-apis-using-flask/
I really want that my model just load one time like a global variable and every request useing this variable without the necessity of load the model every request
is it possible?
You can assign the model variable in settings.py. Whenever the server will start/restart django will store the model variable globally. It can be accessed like
from django.conf import settings
print settings.my_ml_model_variable
Based on the comment of Kaushal, I solved my problem using django rest framework as the following:
first I saved my model as :
> joblib.dump(<your scikit model here> , <"yourfilename.pkl">, compress
> = 1)
Once I had my model saved with the pkl extension I needed to create a variable in the settings.py file(this file is created automatically by django)
YOURMODEL = joblib.load(<"yourfilename.pkl">)
The django process call this file when you start your server, so it is called just one time
Now we just need to call our model in whatever place we want, usually in a views.py file since we are using django and/or django-rest-framework
myModel = getattr(settings, 'YOURMODEL', 'the_default_value')
res = myModel.predict_proba(s).tolist()
A simple example of the rest service:
from django.conf import settings
class myClass(APIView):
permission_classes = (permissions.AllowAny,)
'''Httpverb post method'''
def post(self, request,format=None):
myModel = getattr(settings, '../mymodel.pkl', 'the_default_value')
data = preparePostData(request.data)
res = myModel.predict_proba(data).tolist()
message = prepareMessage(res)
return Response(message, status=status.HTTP_200_OK)
Here preparePostData and prepareMessage are just function that I developed for prepare the object to my model and my response
Regards
You can do it in Django by creating yourfile.py config.
Just you should redfine the django AppConfig there, and instanciate your models .
It will load all your models when django is starting for just once .
After that you can use it in views simpely by importing your class models from yourfile.py .
To do that you should define Yourfile.py
It should look like that :
from django.apps import AppConfig
class YourModelsConfig(AppConfig):
#write what you need
And in your views you can do:
from yourfile import YourModelsConfig
YourModelsConfig.your_method_or_your_variable
I think you can get more details here .
https://medium.com/saarthi-ai/deploying-a-machine-learning-model-using-django-part-1-6c7de05c8d7v
I want to make a button named 'count' to count the number of page of the uploaded pdf file. However, there is TypeError invalid file:
How can I improve my code to tackle this error...?
PS. I'm newbie in coding using django 1.10, and just want to make some little tool to make my life easiler :)
Thanks in advance
My Models
from django.db import models
from PyPDF2 import PdfFileReader
class PdfFile(models.Model):
file = models.FileField(upload_to='document/')
num_of_pages = models.IntegerField(null=True)
def page_count(self):
pdf = PdfFileReader(open(self.file, 'rb'))
self.num_of_pages = pdf.getNumPages()
self.save()
My Views
def count(request, pk):
pdf = get_object_or_404(PdfFile, pk=pk)
pdf.page_count()
return redirect('img_preview', pk=pk)
You can't pass a filefield value directly to open. However, as the documentation shows, that object itself has an open method:
pdf = PdfFileReader(self.file.open())
Because of the way that class also defines read and __iter__, you might actually be able to pass the field directly:
pdf = PdfFileReader(self.file)
I've been using django-modeltranslation to translate models in django for a while. It is really straightforward and it works really well on apps I've been developing, where all model translated content gets inserted with forms by the final user.
eg: inputs: content, content_en, content_pt, ...
I have to build an application where I need to translate 'built-in' model strings that are generated by django, like 'auth.permission.name' or 'contenttypes.contenttype.name' and add them to translation django.po files.
I came up with a solution that works fine,
which uses post_migration signals that create a file with lists of ugettext_lazy elements, so new strings, like a new contenttype.name for example, are added to 'django.po' dynamically and loaded to the database.
Yet, is a bit weird having to create a file with ugettext calls
in order to register the strings, but I didn't find another way of registering and adding them dynamically to the django.po file, so I need your help
Here's what I have done:
1. I created an app named 'tools', that is the last one on INSTALLED_APPS, so its migrations are naturally the last ones to be called. This app does not have any models, it just runs migrations, has the django-modeltranslation translation.py file and an application config with a post_migration signal call.
# translations.py
from modeltranslation.translator import translator, TranslationOptions
from django.contrib.auth.models import Permission
from django.contrib.contenttypes.models import ContentType
class PermissionTranslationOptions(TranslationOptions):
fields = ('name',)
class ContentTypeTranslationOptions(TranslationOptions):
fields = ('name',)
translator.register(Permission, PermissionTranslationOptions)
translator.register(ContentType, ContentTypeTranslationOptions)
2. Running 'manage.py makemigrations' creates the migrations on the 'auth' and 'contenttypes' applications with the extra 'name_*' fields.
3. the app has an application config that has a post_migrate signal
# __init__.py
default_app_config = 'apps.tools.config.SystemConfig'
# config.py
from django.apps import AppConfig
from django.db.models.signals import post_migrate
from apps.tools.translations.exporter import make_translations
from apps.tools.translations.importer import load_translations
def run_translations(sender, **kwargs):
# This creates the translations
make_translations()
# This loads the the translations to the db
load_translations()
class SystemConfig(AppConfig):
name = 'apps.tools'
verbose_name = 'Tools'
def ready(self):
# Call post migration operations
post_migrate.connect(run_translations, sender=self)
4. make_translations() is called after migrations and generates a file with lists of uggettext_lazy calls.
This is the bit I would like to change. Do I really need to create a file?
# exporter
import os
from django.contrib.auth.models import Permission
from django.contrib.contenttypes.models import ContentType
from django.utils import translation
from django.contrib.contenttypes.management import update_all_contenttypes
# TODO
# It has got to be another way
def make_translations():
# lets go default
translation.activate("en")
update_all_contenttypes()
try:
f = open(os.path.join(os.path.realpath(os.path.dirname(__file__)), 'translations.py'), 'w')
# Write file
f.write("from django.utils.translation import ugettext_lazy as _\n\n")
# All Permissions to lazy text
f.write('permissions = {\n')
for perm in Permission.objects.all().order_by('id'):
f.write(' "'+str(perm.id)+'": _("'+perm.name+'"),\n')
f.write('}\n\n')
# All Content types to lazy text
f.write('content_types = {\n')
for content in ContentType.objects.all().order_by('id'):
f.write(' "'+str(content.id)+'": _("'+content.name+'"),\n')
f.write('}\n\n')
# Closing file
f.close()
# Importing file to get it registered with ugettext_lazy
try:
from apps.tools.translations import translations
except:
print('Could not import file')
pass
except:
print('Could not create file')
pass
The above results in a file like this:
from django.utils.translation import ugettext_lazy as _
permissions = {
"1": _("Can add permission"),
"2": _("Can change permission"),
"3": _("Can delete permission"),
"4": _("Can add group"),
...
}
content_types = {
"1": _("group"),
"2": _("user"),
"3": _("permission"),
"4": _("content type"),
"5": _("session"),
...
}
5. Running 'makemessages' would add this strings to 'django.po' files, yet, the post_migration signal does not stop here, and loads the existing compiled strings in the database
# importer
from django.contrib.auth.models import Permission
from django.contrib.contenttypes.models import ContentType
from django.conf import settings
from django.utils import translation
def load_translations():
try:
from apps.tools.translations.translations import permissions, content_types
except:
# File does not exists
print('Translations could not be loaded')
return
# For each language
for lang in settings.LANGUAGES:
# Activate language
translation.activate(lang[0])
# Loading translated permissions
all_permissions = Permission.objects.all()
for permission in all_permissions:
permission.name = unicode(permissions[str(permission.id)])
permission.save()
# Loading translated content_types
all_contenttypes = ContentType.objects.all()
for contenttype in all_contenttypes:
contenttype.name = unicode(content_types[str(contenttype.id)])
contenttype.save()
How can I replace 'make_translations()' without creating a file and register those strings with ugettext_lazy?
Thanks for your help
I've read your post and also somehow I had the same problem with translation for permissions, I've found a very short way to solve the problem:
I wouldn't recommend to do this way neither regret that.
but the solution is:
edit the app_labeled_name function decorated as property of this path: .pyenv/Lib/site-packages/django/contrib/contenttypes/models.py of ContentType class, to become like this:
#property
def app_labeled_name(self):
model = self.model_class()
if not model:
return self.model
return '%s | %s' % (apps.get_app_config(model._meta.app_label).verbose_name,
model._meta.verbose_name)
the trick is to use apps.get_app_config(model._meta.app_label).verbose_name instead of model._meta.app_label, so it would use the same verbose_name as whatever use set for your app in the AppConfig subclass of your app.