We Keep Coding

Dynamic forms to parse CSV in Flask

I am working on a Flask app where the main function is taking an uploaded CSV file, turning it into a Pandas dataframe, and then running a series of analyses on it.
The application works, but is somewhat rigid in the upload process requiring it to exactly match the column names and expected values.
I want to provide a fallback function that will dynamically generate forms that will ask the user questions in order to figure out column mappings back to the original template.
So for example if the template that worked was laid out like this:
['Name', 'Phone', 'Email']
and the uploaded version was:
['Customer Name', 'Phone Number', 'Email Address
]
I would want it to ask:
Which of ['Customer Name', 'Phone Number', 'Email Address'] correspond to Name.
Preferably I would like to use Flask-wtf to maintain consistency, unless there is a clearly better way to do this.
I imagine the way to do this would be to do something like:
import pandas as pd
from flask_wtf import FlaskForm
from wtforms import SelectField
minimum_viable_columns = ['Name', 'Phone', 'Email']
data = pd.read_csv(uploaded_file)
potential_columns = []
for x in data.columns.values.tolist():
if x not in minimum_viable_columns:
potential_columns.append(x)
missing_columns = []
for x in minimum_viable_columns:
if x not in claims.columns.values.tolist():
missing_columns.append(x)
class MatchingForm(FlaskForm):
field_name = SelectField('Corresponding field name', choices=potential_columns)
I would have to generate these forms for each column in missing_columns, and I am trying to figure out a good way to do that and hopefully tie them all to a single submit button.

So the solution I ended up landing one involved a loop to create attributes for the form and a loop on the route for flask to capture the values. The code looks like this:
def map_csv():
possible_columns = session['potential_columns']
missing_columns = session['missing_columns']
class MappingForm(FlaskForm):
submit = SubmitField('Submit new mapping')
pass
possible_columns = [(x, x) for x in possible_columns]
if missing_columns is not None:
for name in missing_columns:
setattr(MappingForm, name, SelectField(name, choices=possible_columns))
form = MappingForm()
if request.method == 'POST':
if form.validate_on_submit:
mapping = {}
for x in missing_columns:
column = getattr(form, x)
mapping[column.data] = x
session['mapping'] = mapping
return redirect(url_for('main.upload'))
return render_template('map_csv.html', form=form)

Django CursorPagination for ordering by several fields

I am using Django DRF's CursorPagination for lazy loading of my data, and currently my goal is to sort the data by more than one field.
This is how my code looks like now:
class EndlessPagination(CursorPagination):
ordering_param = ''
def set_ordering_param(self, request):
self.ordering = request.query_params.get(self.ordering_param, None)
if not self.ordering:
raise ValueError('Url must contain a parameter named ' +
self.ordering_param)
if self.ordering.startswith("\"") or self.ordering.endswith("\""):
raise ValueError('Ordering parameter should not include quotation marks'
def paginate_queryset(self, queryset, request, view=None):
# This function is designed to set sorting param right in the URL
self.set_ordering_param(request)
return super(EndlessPagination, self).paginate_queryset(queryset, request, view)
This code works fine for urls like my_url/sms/270380?order_by=-timestamp, but what if I want to sort by several fields ?

Use str.split() to split the url params
class EndlessPagination(CursorPagination):
ordering_param = 'order_by'
def set_ordering_param(self, request):
ordering_param_list = request.query_params.get(self.ordering_param, None)
self.ordering = ordering_param_list.split(',')
# here, "self.ordering" will be a "list", so, you should update the validation logic
"""
if not self.ordering:
raise ValueError('Url must contain a parameter named ' +
self.ordering_param)
if self.ordering.startswith("\"") or self.ordering.endswith("\""):
raise ValueError('Ordering parameter should not include quotation marks'
"""
def paginate_queryset(self, queryset, request, view=None):
# This function is designed to set sorting param right in the URL
self.set_ordering_param(request)
return super(EndlessPagination, self).paginate_queryset(queryset, request, view)
Example URLs
1. my_url/sms/270380?order_by=-timestamp
2. my_url/sms/270380?order_by=-timestamp,name
3. my_url/sms/270380?order_by=-name,foo,-bar
UPDATE-1
First of all thanks to you for giving a chance to dig deep :)
As you said, me too didn't see comma seperated query_params in popular APIs. So, Change the url format to something like,my_url/sms/270380??order_by=-name&order_by=foo&order_by=-bar
At this time, the request.query_params['order_by'] will be a list equal to ['-name','foo','-bar']. So, you don't want to use the split() function, hence your set_ordering_param() method become,
def set_ordering_param(self, request):
self.ordering = request.query_params.get(self.ordering_param, None)
#...... your other validations

django-datatables-view load filtered model

Here we have the basic code for getting django-datatables-view to display
from django_datatables_view.base_datatable_view import BaseDatatableView
class OrderListJson(BaseDatatableView):
# The model we're going to show
model = MyModel
# define the columns that will be returned
columns = ['number', 'user', 'state', 'created', 'modified']
# define column names that will be used in sorting
# order is important and should be same as order of columns
# displayed by datatables. For non sortable columns use empty
# value like ''
order_columns = ['number', 'user', 'state', '', '']
# set max limit of records returned, this is used to protect our site if someone tries to attack our site
# and make it return huge amount of data
max_display_length = 500
def render_column(self, row, column):
# We want to render user as a custom column
if column == 'user':
return '{0} {1}'.format(row.customer_firstname, row.customer_lastname)
else:
return super(OrderListJson, self).render_column(row, column)
def filter_queryset(self, qs):
# use parameters passed in GET request to filter queryset
# simple example:
search = self.request.GET.get(u'search[value]', None)
if search:
qs = qs.filter(name__istartswith=search)
# more advanced example using extra parameters
filter_customer = self.request.GET.get(u'customer', None)
if filter_customer:
customer_parts = filter_customer.split(' ')
qs_params = None
for part in customer_parts:
q = Q(customer_firstname__istartswith=part)|Q(customer_lastname__istartswith=part)
qs_params = qs_params | q if qs_params else q
qs = qs.filter(qs_params)
return qs
This code works fine, however, how do I get it to not display the whole model, but only filtered content from the model? I've tried setting it to model = MyModel.objects.filter(name="example") but this returns an error.

def get_initial_queryset(self):
return MyModel.objects.filter(name="example")
Add this in your Class OrderListJson

You can use the get_initial_queryset method that returns the queryset used to populate the datatable.

How to save a non ModelForm form to database in Django

I'm a newbie Django user, struggling with submitting form data to the database. So that I can generate dynamic forms I am using a non-ModelForm form to capture field data.
I'm commented out validation for now but I am trying to capture the POST data from the form to the database. The latest 'draft' of my views.py code is as follows - most interested in format from form_to_save = Scenario(...):
def scenario_add(request, mode_from_url):
urlmap = {
'aviation': 'Aviation',
'maritime': 'Maritime',
'international_rail': 'International Rail',
'uk_transport_outside_london': 'UK Transport (Outside London)',
'transport_in_london': 'Transport in London',
}
target_mode = urlmap[mode_from_url]
m = Mode.objects.filter(mode=target_mode)
tl = m[0].current_tl.threat_l
scenario_form = ScenarioForm(request.POST or None, current_tl=tl, mode=target_mode)
if request.method == 'POST':
#if scenario_form.is_valid():
form_to_save = Scenario(
target = Target(descriptor = scenario_form.fields['target']),
t_type = ThreatType(t_type = scenario_form.fields['t_type']),
nra_reference = NraReference(nra_code = scenario_form.fields['nra_reference']),
subt = scenario_form.fields['subt'],
ship_t = ShipType(ship_t = scenario_form.fields['ship_t']),
likelihood = scenario_form.fields['likelihood'],
impact = scenario_form.fields['impact'],
mitigation = scenario_form.fields['mitigation'],
compliance_score = scenario_form.fields['compliance_score'],
notes = scenario_form.fields['notes']
)
form_to_save.save()
# This needs to be changed to a proper redirect or taken to
# a scenario summary page (which doesn't yet exit.)
return render(request, "ram/new_scenario_redirect.html", {
'scenario_form': scenario_form,
'mode': mode_from_url,
'mode_proper': target_mode
})
else:
# if there is no completed form then user is presented with a blank
# form
return render(request, 'ram/scenario_add.html', {
'scenario_form': scenario_form,
'current_tl': tl,
'mode': mode_from_url,
'mode_proper': target_mode
})
Any advice gratefully received. Many thanks.

You've commented out the is_valid check, for some reason. You need that: as well as checking for validity, it also populates the form's cleaned_data dictionary which is what you should be getting the data from to create your object. (And don't call that object "form_to_save": it's a model instance, not a form).
if request.method == 'POST':
if scenario_form.is_valid():
scenario = Scenario(
target = Target(descriptor=scenario_form.cleaned_data['target']),
t_type = ThreatType(t_type = scenario_form.cleaned_data['t_type'])
...etc
Plus, you should move the final "return render" call out of the else block, so that it is caught when the form is not valid, to show any errors.
However, as petkostas says, you would almost certainly be better off using an actual ModelForm in the first place.

You can add custom options by overriding the init function in your form. For example:
class SomeForm(forms.Form):
department = forms.ChoiceField(widget=forms.Select, required=True)
...
def __init__(self, *args, **kwargs):
super(SomeForm, self).__init__(*args, **kwargs)
self.fields['department'].choices = Department.objects.all().order_by('department_name).values_list('pk', 'department_name')
You can also change the queryset in the init function:
where Department is a foreign key for example
queryset = Department.objects.filter(your logic)
self.fields['department'].queryset = queryset

Django - Restrict a drop down options depending on another drop down

I have 2 drop downs.
First drop down asks for the type.
I have a second drop down that shows values from another Model.
What I need is that, if the first drop down as a type: qualif, I only want to show the pk=1 of the second drop down.
This is what I have so far:
name = models.CharField(max_length=40,verbose_name="nom")
type = models.CharField(max_length=6,choices=TYPE_CHOICES)
division = models.ForeignKey(Division,verbose_name="division")
class TournamentForm(forms.ModelForm):
def clean(self):
super(TournamentForm, self).clean() #if necessary
if 'division' in self._errors:
"""
reset the value (something like this i
think to set the value b/c it doesnt get set
b/c the field fails validation initially)
"""
if self.data['type'] == 'qualif':
division = Division.objects.get(pk=1)
self.division = division
# remove the error
del self._errors['division']
return self.cleaned_data
# Register your models here.
class TournamentAdmin(reversion.VersionAdmin):
form = TournamentForm
list_display = ('name', 'date', 'division', 'gender')
ordering = ('date', 'name')
list_filter = ['date', 'season', 'division', 'gender']
admin.site.register(Tournament, TournamentAdmin)
I read from another stack question to use clean...but sadly it's not working...
EDIT:
After looking at #Mardo's link, I tried loading up a static file.
Here is my folder setup:
myproject/static/admin/js/myfile.js
And this in my settings.py
STATIC_URL = '/static/'
But it keeps saying file not found...
Thanks,
Ara

Make two forms. First have only first dropdown, second is disabled. On selecting value commit form and render full version based on first.