We Keep Coding

CSV row jump function

I am trying to make a form with Flask and Python, I want to save the elements I get from the templates in a CSV file.
I made a function that should save the data of the templates in a row and when it arrives to the last template save in the CSV row and make a row jump.
However, instead of saving the information in the same row of the csv, it writes the last element obtained from the template in a new row, and it should be saved in the same row:
The code of the function is as follows:
def write(test,final):
lista=[]
for x in test:
print(test[x])
lista.append(test[x])
with open('covid.csv', 'a') as f:
if final == '0':
print("Dentro de if 0")
for line in lista:
name=io.StringIO(line)
coma=io.StringIO(",")
f.write(str(name.read()))
f.write(str(coma.read()))
if final == '1':
print("Dentro de if 1")
for line in lista:
name=io.StringIO(line)
coma=io.StringIO(",")
f.write(str(name.read()))
f.write(str(coma.read()))
f.write(str("\n")
This function is called as follows:
#app.route('/sintomas', methods=['GET','POST'])
def sintomas():
print("Prueba de datos:")
if request.method == 'POST':
test = request.form.to_dict()
print(test)
write(test,'0')
return render_template('nuevaplantilla6.html')
#app.route('/signos', methods=['GET','POST'])
def sintomas2():
print("Prueba de datos:")
if request.method == 'POST':
test = request.form.to_dict()
print(test)
write(test,'1')
return render_template('plantilla7.html')

Flask Select Field Form Registers as None

I am immensely baffled by what should be very simple in my forms as I have this same construct working in other areas of my Flask web application. Below is a route I have:
#data_tools.route('/dataTools', methods=['POST', 'GET'])
def data_tools_upload():
table = '<table></table>'
result = '<table></table>'
message = ''
var2use = None
vars = ''
dt_name = ''
dtRadios = dtFormRadio(request.form)
bins = 5
foo=''
if request.method == 'POST':
tmp_filename = tempfile.gettempdir()+'\\input.csv'
if request.files:
to_upload = request.files.get('file')
dt_name = str(to_upload).split()[1]
if to_upload:
f = request.files['file']
f.save(tmp_filename)
if os.path.exists(tmp_filename):
orig_df = pd.read_csv(tmp_filename)
vars = list(orig_df.columns)
## Testing new form part
#dtRadios.varlist.choices = vars
foo = dtRadios.varlist.data
## end test
var2use = request.form.get("var2use")
if var2use != None:
indx = vars.index(var2use)
if dtRadios.dtRadioList.data == 'descriptives':
result = dt.get_descriptives(orig_df[vars[indx]])
if dtRadios.dtRadioList.data == 'percentiles':
result = dt.get_percentiles(orig_df[vars[indx]])
if dtRadios.dtRadioList.data == 'frequency':
if dtRadios.binSize.data != None:
bins = dtRadios.binSize.data
result = dt.get_frequency(orig_df[vars[indx]], bins = bins)
result = pd.DataFrame.from_dict(result, orient='index', columns = [var2use]).to_html(classes='table table-striped table-hover', header = "true", justify = "center")
dims = orig_df.shape
message = 'Data file %s with %s rows and %s columns loaded.' % (dt_name, dims[0],dims[1])
table = orig_df.head(10).to_html(classes='data', header = "true")
return render_template('dataTools.html', results = [result], message = message, vars = vars, var_name = var2use, dtForm=dtRadios, foo=foo)
and then here is a form I have built to go along with this
class dtFormRadio(FlaskForm):
dtRadioList = RadioField('Choose Option:', choices=[('descriptives','Get Descriptives'),
('percentiles','Get Percentiles'), ('frequency','Get Frequency')])
#varlist = SelectField('Choose a variable', [DataRequired()], coerce=str)
varlist = SelectField('Choose a Variable', choices=[('clean', 'Clean/Processed Text'),('original', 'Original Text String')])
binSize = IntegerField('Bin Size', [DataRequired()], default = 10)
The relevant part of my question surrounds this line foo = dtRadios.varlist.data. It always evaluates to None. I'm printing it to my HTML output as foo just so I can see what python sees. When I replace this with foo = dtRadios.binSize.data or with foo = dtRadios.dtRadioList.data then whatever option is chosen in that part of the form is printed to screen (and it also works in the context of my app).
But, something about foo = dtRadios.varlist.data is not being evaluated. Does anyone see an obvious error in my code or my thinking?

Grr, solved problem. My dtRadios.varlist was in my HTML, but outside of the form whereas the other parts were properly inside the form. So, the submit button wasn't updating this part of the form.

Django - Check if a variable set exists in the Database and process it if it does?

So I have a Django App, where a CSV-File can be uploaded. My CSV-File has 9 columns that can be divided into two "datasets" where the first 5 columns need to be handled as one information and the other 4 need to be handled as another information. I cannot put the first 5 in one cell and the other ones in another cell. I would like to check whether or not the first dataset exists and if it does, process it. The same applies to the other dataset. And if both datasets do not exist already it should just update the Database with get_or_create.
Here is my views.py idea
def import_csv(request):
if request.method == "POST":
with open('C:/Users/admin/Desktop/djangoexcel/b.csv') as file:
reader = csv.reader(file)
for row in reader:
var = CSV_File4.objects.filter(
attr1=row[0], attr2=row[1], attr3=row[2], attr4=row[3], attr5=row[4],
)
if var.exists():
TemplateResponse(request, "documents/replace_entry.html", {'var' : var})
else:
for row in reader:
switch = CSV_File4.objects.filter(
attr6=row[5], attr7=row[6], attr8=row[7], attr9=row[8]
)
if var2.exists():
TemplateResponse(request, "documents/replace_entry.html", {'var2' : var2})
else:
for row in reader:
_, p = CSV_File4.objects.get_or_create(
attr1=row[0], attr2=row[1], attr3=row[2], attr4=row[3], attr5=row[4],
attr6=row[5], attr7=row[6], attr8=row[7], attr9=row[8]
)
return redirect('documents:index')
form = UploadFileForm()
return render(
request, "documents/csv_upload.html", {"form": form}
)
It should look something like this. How can I make this work. It was just an idea with filter() and exists() but is there a Python way to do something like this? Any help would be appreciated.

Currently, you are trying to recursively iterate through reader three times. That's not possible, because it is an Iterator, not a list. Anyway, you only need to do it once and then work on that particular line, before skipping to the next.
def import_csv(request):
if request.method == "POST":
with open('C:/Users/admin/Desktop/djangoexcel/b.csv') as file:
reader = csv.reader(file)
for row in reader:
ds1 = CSV_File4.objects.filter(
attr1=row[0], attr2=row[1], attr3=row[2], attr4=row[3], attr5=row[4],
).exists()
ds2 = CSV_File4.objects.filter(
attr6=row[5], attr7=row[6], attr8=row[7], attr9=row[8]
).exists()
if ds1:
pass # Process first dataset
if ds2:
pass # Process second dataset
if not (ds1 and ds2):
_, p = CSV_File4.objects.get_or_create(
attr1=row[0], attr2=row[1], attr3=row[2], attr4=row[3], attr5=row[4],
attr6=row[5], attr7=row[6], attr8=row[7], attr9=row[8]
)
return redirect('documents:index')
return render(
request, "documents/csv_upload.html", {"form": UploadFileForm()}
)

django1.8- how to append information manually when uploading Excel and inserting into database

The excel is like the left picture with 3 columns.
When inserting into database, I need to add 2 columns more manually like right picture showed and insert altogether 5 columns in database finally. These 2 additional columns information is fetched from other databases.
And another function is if there is already existing file, the newly uploaded file will override the existing one.[snippets in views.py below]
I have already tried two 3 party tools but not works, so maybe it is still better just to use the one embedded in django.
Version: Python 2.7. Excel 2013. Django1.8.
Any help is highly appreciated. Hope could provide the detail snippet for how to append these 2 columns :
uploader = request.session['uploader']
Date=request.session['date']
forms.py
from django.core.files.storage import FileSystemStorage
from financialdata.storage import OverwriteStorage
class XXXXDataForm(forms.Form):
XXXXfile=forms.FileField(label='Select a file')
views.py
from django.core.files.storage import FileSystemStorage
def import_data(request):
if request.method == "POST":
form = XXXXForm(request.POST,request.FILES)
if form.is_valid():
newdoc= XXXX(docfile=request.FILES['docfile'])
newdoc.save()
return HttpResponseRedirect(reverse('homepage'))
else:
return HttpResponseBadRequest()
else:
form = XXXXForm()
return render_to_response(
'dataentry.html',
{
'form': form,
'title': 'Import excel data into database example',
'header': 'Please upload XXXX.xlsx:',
'message': 'File Saved!'
},
context_instance=RequestContext(request))
<!--How can I embed the following part to previous part?-->
class OverwriteStorage(FileSystemStorage):
def _save(self, name, content):
if self.exists(name):
self.delete(name)
return super(OverwriteStorage, self)._save(name, content)
def get_available_name(self, name):
return name

May be like that:
I don't run my code, it's an example
if request.method == "POST":
form = XXXXForm(request.POST,request.FILES)
if form.is_valid():
docfile=request.FILES['docfile']
if isinstance(docfile, (InMemoryUploadedFile, TemporaryUploadedFile)):
book = xlrd.open_workbook(file_contents=docfile.read(), formatting_info=True, on_demand=True)
else:
book = xlrd.open_workbook(filename=docfile, formatting_info=True, on_demand=True)
sheet = book.sheet_by_index(0)
new_csv_file = csv.writer(open('new_filename', 'w'))
# read file
for line in range(1, sheet.nrows):
new_row = [request.session.get('uploader'), request.session.get('date')]
new_row.extend(sheet.row_values(line))
new_csv_file.writerow(new_row)
...

Write reusable view

I have this function that takes a QuerySet and renders a CSV. I would like to write a view that renders a template with options to download different CSV files (Basically for anything defined in models.py)
# Exports CSV file using a QuerySet
def export(qs, fields=None, file_name='data'):
model = qs.model
response = HttpResponse(mimetype='text/csv')
response['Content-Disposition'] = 'attachment; filename={0}-{1}.csv'.format(file_name, str(datetime.date.today()))
writer = csv.writer(response)
# Write headers to CSV file
if fields:
headers = fields
else:
headers = []
for field in model._meta.fields:
headers.append(field.name)
writer.writerow(headers)
# Write data to CSV file
for obj in qs:
row = []
for field in headers:
if field in headers:
val = getattr(obj, field)
if callable(val):
val = val()
row.append(val)
writer.writerow(row)
# Return CSV file to browser as download
return response
Currently I am writing a NON-reusable view:
def csv_of_surveys(request):
r = export(Survey.objects.all(), file_name='surveys')
return r
What can I do? My only idea was to send a code over and write a switch statement, so
{% url "csv_of" 0 %}
{% url "csv_of" 1 %}
{% url "csv_of" 2 %}
{% url "csv_of" 3 %}
Where 0, 1, 2 and 3 would correspond to downloading different things.
The new view would look something like:
def csv_of(request, code):
if code == 0:
r = export(Survey.objects.all(), file_name='surveys')
return r
elif code == 1:
r = export(User.objects.all(), file_name='teachers')
return r
elif code == 2:
r = export(Student.objects.all(), file_name='students')
return r
# ...
else:
return HttpResponseRedirect('/')
Is there a better way?

Create a dictionary that maps the given code to the associated object, then reduce all the if statements you have into one if. For the file name, it looks like you're doing the same thing each time, which is pluralizing and lowercasing it, in which case you should set it in model._meta.verbose_name_plural, then access that when you need it:
file_codes = {0:Survey,1:User...}
def csv_of(request, code):
if int(code) in file_codes.keys():
obj = file_codes[int(code)]
return export(obj.objects.all(), file_name = obj._meta.verbose_name_plural.title())
else:
return HttpResponseRedirect('/')