We Keep Coding

how to create dynamic database table using csv file in django or DRF

I am going to create a database table using csv file without model in django. Steps are:
after sending csv file by post request, one database table will be created according to csv headers (name, university, score, total_score etc). And it will be populated using csv file data. Database table name should be derived from csv file name.
I searched but couldn't find good solution.
Any help is appreciated.
Below is my code to read csv file
class UploadProductApiView(generics.CreateAPIView):
serializer_class = FileUploadSerializer
def post(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
file = serializer.validated_data['file']
decoded_file = file.read().decode()
# upload_products_csv.delay(decoded_file, request.user.pk)
io_string = io.StringIO(decoded_file)
reader = csv.reader(io_string)
for row in reader:
print(row)

You could always create a dynamic Django model: https://code.djangoproject.com/wiki/DynamicModels
With this approach you could create models on the fly and by running this snippet
from django.core.management import call_command
call_command('makemigrations')
call_command('migrate')
you could migrate the model to the database and use it for accessing and storing the csv data. After that you should create the model code with inspectdb:
python manage.py inspectdb TableName > output.py
Set the path to the where you want your model file to be generated. I'm not certain if this command would overwrite or append to the current file, so you can try to append to current file, if it doesn't work make a temp file and write the output to it and append it to your desired models.py file: https://www.geeksforgeeks.org/python-append-content-of-one-text-file-to-another/
After the whole process you will have migrations generated, executed and the model in models.py for use on server restart.

Uploading a file in django database in a specific root with diferrent folders everytime

I´m trying to upload file into database, but when i want put the function uoload to and i want to that function store the file in a root with the data the user submit in the for, for example year, course, section, I get that information and the file they uploaded I want to store in that subfolders, the code that I´m using only store the data in the Media_root, but not in the carpets how can I do in order to store the files in the subfolder I want. I'n my model document I store the information of the document, like the autor, the title,etc.
class File(models.Model):
titulo_file=models.ForeignKey(Document,on_delete=models.CASCADE,null=True,verbose_ name='Título de la Tesis')
file = models.FileField(null=True,blank=True, upload_to=generate_path)
Here the function I use to upload the file, Int only works to store the file into a folder with it´s same name.
def generate_path(instance, filename):
path =os.path.join(instance.file.name,filename)
return path
In my view, after the user submit the information, I use a os.path.join, that do the path with the information submited like this:
year/course/section
I want to send that path to my generate_path and store in that location the file, I tried with session but it doesn't work, what can I do?
models.py
class Year(models.Model):
year_number=models.CharField(max_length=10,primary_key=True)
class Course(models.Model):
name=models.CharField(max_length=50,primary_key=True)
class Section(models.Model):
numberSection=models.IntegerField(null=True,blank=True)
year = models.ForeignKey(Year, on_delete=models.CASCADE, null=True)
course = models.ForeignKey(Course, on_delete=models.CASCADE)
thopic_documents=(
('CS','CS'),
('SE','SE'),
('IS','IS'),
('IT','IT'),
)
class Document(models.Model):
title=models.CharField(max_length=200,primary_key=True)
thopic=models.CharField(max_length=50,choices=thopic_documents, default=None,null=True)
class Historial(models.Model):
id_section=models.ForeignKey(Section,on_delete=models.CASCADE)
title_document=models.ForeignKey(Document,on_delete=models.CASCADE,)

To get the year/course/section you need to be able to navigate from a File to a Section. However, the many-many relationship that exists between Document and Section through Historical makes it unclear how you would do that. Which Historical would you choose?
It may be better if File was linked directly to Historical:
class File(models.Model):
titulo_file=models.ForeignKey(Historical, on_delete=models.CASCADE,null=True,verbose_name='Título de la Tesis')
file = models.FileField(null=True,blank=True, upload_to=generate_path)
If that was the case, you could then implement your generate_path() such as:
def generate_path(instance, filename):
section = instance.titulo_file.id_section
year = section.year.year_number
course = section.course.name
return os.path.join(str(year), course, str(section.numberSection), filename)
To do the same thing with the model as it currently stands, you would have to do something like this:
def generate_path(instance, filename):
section = instance.titulo_file.historical_set.first().id_section
year = section.year.year_number
course = section.course.name
return os.path.join(str(year), course, str(section.numberSection), filename)
That example uses historical_set.first() to get the first Historical linked the Document. Maybe that would be ok, but otherwise you'd need to know which Historical to use.
Where a file with the same name is uploaded and you don't want to overwrite it, you could implement your own storage:
from django.core.files.storage import FileSystemStorage
class UseExistingStorage(FileSystemStorage):
def save(self, name, content, max_length=None):
if not self.exists(name):
return super().save(name, content, max_length)
return name # Don't save when the file exists, just return the name
Then reference the UseExistingStorage from your File model.
file = models.FileField(null=True,blank=True, upload_to=generate_path, storage=UseExistingStorage())

How to save a file to a model using upload_to for dynamic paths

I am trying to create a folder for each users to put their project in. So their file will have the path ..\project\id\filename, id is the user id and filename is the name of the file. Now using the arguments allowed for upload_to (instance and filename) in the Filefield, I realize that instance.id will be None and the path to the file will be ..\project\None\filename instead of ..\project\id\filename.
Now reading the Django documentation upload_to I saw this:
In most cases, this object will not have been saved to the database
yet, so if it uses the default AutoField, it might not yet have a
value for its primary key field.
My interpretation is that creating a new record and user_directory_path are not instantiated at the same time, that is, when I call create on Project model, instance.id will be None. My question is now, is there a way to get around this? While I see upload_to convenient, it is not necessarily convenient for dynamic path such as the one I am doing. I was thinking of creating the record, then adding the file path in an update, but I am in search of a way that can save everything in one step.
models.py
def user_directory_path(instance, filename):
# file will be uploaded to MEDIA_ROOT/user_<id>/<filename>
return 'project/{0}/{1}'.format(instance.user.id, filename)
class Project(models.Model):
email = models.ForeignKey(User,
to_field="email",
max_length=50
)
title = models.CharField(max_length=100)
date_created = models.DateTimeField(auto_now_add=True)
updated = models.DateTimeField(auto_now=True)
file = models.FileField(upload_to=user_directory_path, validators=[validate_file_type], null=True)
This is views.py when the form passes validation. Notice user_directory_path is called just before the create.
email = request.user.email
title = request.POST.get('title', '')
file = request.FILES['file']
filename = file.name
instance = Usermie.objects.get(email=request.user.email)
# Save to model
user_directory_path(instance=instance, filename=filename)
Project.objects.create(
title=title, file=file,
)

If, as you say, the id that you want to use in the file path is the id of the User, not the id of the Project.. then there's no problem because the User already exists when you are saving the Project. Since email is a foreign key to User, you would just do:
def user_directory_path(instance, filename):
# file will be uploaded to MEDIA_ROOT/user_<id>/<filename>
return 'project/{0}/{1}'.format(instance.email.id, filename)
But I will point out that, in the Django way of doing things, making a field called email that is a foreign key to User is actually pretty confusing. The field in the database will be called email_id.. and the value of the model field will return an instance of User.. not the actual email address, even though the email address is what's stored in the column. To get the email address you'd need to do one of:
myproject.email.email
myproject.email_id
Neither one is very clear. So unless you have a really good reason for doing it like that, you should call the field user and eliminate the to_field='email'. Allow Django to join the tables via id, which is the default behavior.
Then if you need the user email address you can get it any time via
myproject.user.email
And the bonus is that if the user changes their email address it will change everywhere, you don't have to rely on cascaded updates to fix all the foreign keys.
Trust me, when using Django you want to do ForeignKey by id (the default) unless there's a reason...

One simple solution can be saving object without file and then saving file like this
email = request.user.email
title = request.POST.get('title', '')
file = request.FILES['file']
filename = file.name
instance = Usermie.objects.get(email=request.user.email)
# Save to model
user_directory_path(instance=instance, filename=filename)
project = Project.objects.create(title=title)
project.file = file
project.save()

Django (Models): how to get instance from foreign key

i'm new to django and sqlite. Is that possible i can get the instance.name value from its foreign key that is related to modeldb? I need to save my pdf file under the same directory as the modeldb class. From the above experiment, im got nothing with modelDB.instance / modelDB.instance.name.

move that upload function outside the class.
instance refer as the your class instance. So you can do like this
instance.modelDB.name
citatiions = models.FileField(...., upload_to=upload) #pass your function name here
check this answerlink

You could do,
def upload(instance, file_name):
return "documents/" + instance.ModelDB.name + "/pdf/" + filename

Update record when the data already exist in model

I have a view where the user can upload several files.
#login_required(login_url='/login/')
def FileUploadView(request):
if request.method == 'POST':
form = CallgateUploadForm(request.POST,request.FILES)
fecha = datetime.date.today()
if form.is_valid():
for each in form.cleaned_data['archivo']:
Callgate_Syni.objects.create(fecha=fecha,archivo=each)
os.system('/home/pyc/DjangoProjects/tap_app/media/docs/callgate/envia.sh')
return HttpResponseRedirect('/lista/')
else:
form = CallgateUploadForm()
return render_to_response('callgateupload.html',
{'form':form},
context_instance=RequestContext(request))
Form:
class CallgateUploadForm(forms.ModelForm):
archivo = MultiFileField(min_num=1, max_num=20, max_file_size=1024*1024*5)
def __init__(self,*args,**kwargs):
super(CallgateUploadForm,self).__init__(*args,**kwargs)
self.helper = FormHelper(self)
class Meta:
model = Callgate_Syni
In the same day the user can upload the same file several times, this create a duplicated records for the same file ("archivo"). How can avoid to insert the information of the already uploaded files.
Or if the file already exist update the records in the model and if the file is not exist insert the record.
Thanks in advance.

What you need is the Django QuerySet update_or_create() method.
The usage is simple as:
updated_values = {'archivo': archivo}
obj, created = Callgate_Syni.objects.update_or_create(fecha=fecha, defaults=updated_values)
if created:
print('The object was created')
else:
print('The object was updated')
The keyword args are the filters to specify if the record exists or not, and the defaults parameter is a filedname: fieldvalue dictionary with the fields to update.
The return tuple have the object that was created or updated, and a flag to indicate if the object was created.

From the description and for each in form.cleaned_data['archivo']: I understand what your want is:
If all of the files are new, just save and create;
If some of the files already exist, update them and insert the new files.
So you might want to give a date flag to the filenames so that you can check if the file already exists for a specific date and filename. For example, sample_file_20160125.csv will indicate a file named 'sample_file.csv' has already been uploaded by the user on 01/25/2016. Then you can match such a file in database, if found, update the file object; if not found, create a new record.
If you just want to save the file and not touch the database when the file is already there, use exists() to check if the record in the database.
...
if not Callgate_Syni.objects.filter(fecha=fecha,archivo=each).exists():
Callgate_Syni.objects.create(fecha=fecha,archivo=each)
os.system('/home/pyc/DjangoProjects/tap_app/media/docs/callgate/envia.sh')
...

You can't rely on filenames, but on its content, so one idea is using hash function. Therefore one solution is:
Add hash field to your file model.
When a file is uploaded calculate hash of its content, without the name, so that it won't affect the resulting hash.
Save file to disk, and its hash to DB.
When user uploads another file calculate its hash and see if there's a record with the same hash in DB.
a. If no go to step 3.
b. If yes do nothing.
The drawback calculating hash requires resources, but it's more reliable then just filename.