We Keep Coding

Python Flask SqlAlchemy Add Database Model Name dynamically in For Loop

I am not that familiar with Python and SQLAlchemy so please be patient :)
I need to capture if, within a FORM that holds multiple ICONS(files), one or more ICONS have been changed when editing the record.
To see which ICONS have been changed I created an Object holding the changes with "Database Model Name" as the "Key" and its "Value"
{'icon': <FileStorage: 'fire.png' ('image/png')>}
key = used as database model name
value = file.filename
now when I try the get the data within a for loop and add this data to the Database model, nothing happens and it looks like I am not really accessing variable "k" in the loop.
for k, v in notequalat.items():
responseteamdata.k = v.filename
My question is, how can I combine the Database model class "responseteamdata" and the variable "k" so that I can add the changes to the database model dynamically.
here is the full code:
if not notequalat:
try:
responseteamdata.title = title
responseteamdata.abbreviation = abbreviation
responseteamdata.isfireteam = booleanisfireteam
responseteamdata.iconposition = newlatlng
db.session.commit()
except IntegrityError:
db.session.rollback()
db.session.close()
res = make_response(jsonify("message ", "Error Updating the Team"), 500)
return res
else:
responseteamdata.title = title
responseteamdata.abbreviation = abbreviation
responseteamdata.isfireteam = booleanisfireteam
responseteamdata.iconposition = newlatlng
for k, v in notequalat.items():
responseteamdata.k = v.filename
db.session.commit()
dbevent = "updated"
db.session.close()

To be able to dynamically assign the Table Column Name the following command has been working for me:
setattr(DB-Object, ColumnName, Value)
which means in my case:
setattr(responseteamdata, k, v.filename)

I have an Error with python flask cause of an API result (probably cause of my list) and my Database

I use flask, an api and SQLAlchemy with SQLite.
I begin in python and flask and i have problem with the list.
My application work, now i try a news functions.
I need to know if my json informations are in my db.
The function find_current_project_team() get information in the API.
def find_current_project_team():
headers = {"Authorization" : "bearer "+session['token_info']['access_token']}
user = requests.get("https://my.api.com/users/xxxx/", headers = headers)
user = user.json()
ids = [x['id'] for x in user]
return(ids)
I use ids = [x['id'] for x in user] (is the same that) :
ids = []
for x in user:
ids.append(x['id'])
To get ids information. Ids information are id in the api, and i need it.
I have this result :
[2766233, 2766237, 2766256]
I want to check the values ONE by One in my database.
If the values doesn't exist, i want to add it.
If one or all values exists, I want to check and return "impossible sorry, the ids already exists".
For that I write a new function:
def test():
test = find_current_project_team()
for find_team in test:
find_team_db = User.query.filter_by(
login=session['login'], project_session=test
).first()
I have absolutely no idea to how check values one by one.
If someone can help me, thanks you :)
Actually I have this error :
sqlalchemy.exc.InterfaceError: (InterfaceError) Error binding
parameter 1 - probably unsupported type. 'SELECT user.id AS user_id,
user.login AS user_login, user.project_session AS user_project_session
\nFROM user \nWHERE user.login = ? AND user.project_session = ?\n
LIMIT ? OFFSET ?' ('my_tab_login', [2766233, 2766237, 2766256], 1, 0)

It looks to me like you are passing the list directly into the database query:
def test():
test = find_current_project_team()
for find_team in test:
find_team_db = User.query.filter_by(login=session['login'], project_session=test).first()
Instead, you should pass in the ID only:
def test():
test = find_current_project_team()
for find_team in test:
find_team_db = User.query.filter_by(login=session['login'], project_session=find_team).first()
Asides that, I think you can do better with the naming conventions though:
def test():
project_teams = find_current_project_team()
for project_team in project_teams:
project_team_result = User.query.filter_by(login=session['login'], project_session=project_team).first()

All works thanks
My code :
project_teams = find_current_project_team()
for project_team in project_teams:
project_team_result = User.query.filter_by(project_session=project_team).first()
print(project_team_result)
if project_team_result is not None:
print("not none")
else:
project_team_result = User(login=session['login'], project_session=project_team)
db.session.add(project_team_result)
db.session.commit()

Updating a specific row with Flask SQLAlchemy

I have this app with a profile and I want to update only a specific row based on its account id. Inserting the data works, but updating a specific row doesn't and I'm not sure which part is wrong with my code.
#app.route('/edit_parent/<int:acc_id>', methods=['GET','POST'])
def edit_parent(acc_id):
myParent = Parent.query.filter_by(acc_id=int(acc_id)).first()
if request.method == "POST":
myParent.fname_p = request.form['fname_p']
myParent.lname_p = request.form['lname_p']
myParent.bday_p = request.form['bday_p']
myParent.add_p = request.form['add_p']
db.session.commit()
print "hello success"
return redirect(url_for('parent', acc_id=int(acc_id)))
if request.method == "GET":
return render_template('edit_p.html', acc_id=int(acc_id))
It prints the "hello success" and redirects to the parent url but returns an error 302 and still no changes in the db.

I don't think you are updating a specific row at all, but instead you are just inserting new one each time with:
myParent = Parent(request.form['fname_p'], request.form['lname_p'],
request.form['bday_p'], request.form['add_p']).where(acc_id=acc_id)
db.session.add(myParent)`
So, what you are supposed to do instead is:
myParent = Parent.query.filter_by(acc_id=acc_id)
assuming your Parent db has the following attributes:
myParent.fname = request.form['fname_p']
myParent.lname = request.form['lname_p']
myParent.bday = request.form['bday_p']
myParent.add = request.form['add_p']
db.session.commit()

solved it by adding:
myParent = db.session.merge(myParent)
this way it merges the current session with the previous one. It still returns a 302 but the data on the db has been successfully updated.

Do I need to use transactions in google appengine

update 0
My def post() code has changed dramatically because originally it was base on a digital form which included both checkboxes and text entry fields, not just text entry fields, which is the current design to be more paper-like. However, as a result I have other problems which may be solved by one of the proposed solutions, but I cannot exactly follow that proposed solution, so let me try to explain new design and the problems.
The smaller problem is the inefficiency of my implementation because in the def post() I create a distinct name for each input timeslot which is a long string <courtname><timeslotstarthour><timeslotstartminute>. In my code this name is read in a nested for loop with the following snippet [very inefficient, I imagine].
tempreservation=courtname+str(time[0])+str(time[1])
name = self.request.get('tempreservation',None)
The more serious immediate problem is that my def post() code is never read and I cannot figure out why (and maybe it wasn't being read before, either, but I had not tested that far). I wonder if the problem is that for now I want both the post and the get to "finish" the same way. The first line below is for the post() and the second is for the get().
return webapp2.redirect("/read/%s" % location_id)
self.render_template('read.html', {'courts': courts,'location': location, ... etc ...}
My new post() is as follows. Notice I have left in the code the logging.info to see if I ever get there.
class MainPageCourt(BaseHandler):
def post(self, location_id):
logging.info("in MainPageCourt post ")
startTime = self.request.get('startTime')
endTime = self.request.get('endTime')
day = self.request.get('day')
weekday = self.request.get('weekday')
nowweekday = self.request.get('nowweekday')
year = self.request.get('year')
month = self.request.get('month')
nowmonth = self.request.get('nowmonth')
courtnames = self.request.get_all('court')
for c in courtnames:
logging.info("courtname: %s " % c)
times=intervals(startTime,endTime)
for courtname in courtnames:
for time in times:
tempreservation=courtname+str(time[0])+str(time[1])
name = self.request.get('tempreservation',None)
if name:
iden = courtname
court = db.Key.from_path('Locations',location_id,'Courts', iden)
reservation = Reservations(parent=court)
reservation.name = name
reservation.starttime = time
reservation.year = year
reservation.nowmonth = int(nowmonth)
reservation.day = int(day)
reservation.nowweekday = int(nowweekday)
reservation.put()
return webapp2.redirect("/read/%s" % location_id)
Eventually I want to add checking/validating to the above get() code by comparing the existing Reservations data in the datastore with the implied new reservations, and kick out to an alert which tells the user of any potential problems which she can address.
I would also appreciate any comments on these two problems.
end of update 0
My app is for a community tennis court. I want to replace the paper sign up sheet with an online digital sheet that mimics a paper sheet. As unlikely as it seems there may be "transactional" conflicts where two tennis appointments collide. So how do I give the second appointment maker a heads up to the conflict but also give the successful party the opportunity to alter her appointment like she would on paper (with an eraser).
Each half hour is a time slot on the form. People normally sign up for multiple half hours at one time before "submitting".
So in my code within a loop I do a get_all. If any get succeeds I want to give the user control over whether to accept the put() or not. I am still thinking the put() would be an all or nothing, not selective.
So my question is, do I need to make part of the code use an explicit "transaction"?
class MainPageCourt(BaseHandler):
def post(self, location_id):
reservations = self.request.get_all('reservations')
day = self.request.get('day')
weekday = self.request.get('weekday')
nowweekday = self.request.get('nowweekday')
year = self.request.get('year')
month = self.request.get('month')
nowmonth = self.request.get('nowmonth')
if not reservations:
for r in reservations:
r=r.split()
iden = r[0]
temp = iden+' '+r[1]+' '+r[2]
court = db.Key.from_path('Locations',location_id,'Courts', iden)
reservation = Reservations(parent=court)
reservation.starttime = [int(r[1]),int(r[2])]
reservation.year = int(r[3])
reservation.nowmonth = int(r[4])
reservation.day = int(r[5])
reservation.nowweekday = int(nowweekday)
reservation.name = self.request.get(temp)
reservation.put()
return webapp2.redirect("/read/%s" % location_id)
else:
... this important code is not written, pending ...
return webapp2.redirect("/adjust/%s" % location_id)

Have a look at optimistic concurrency control:
http://en.wikipedia.org/wiki/Optimistic_concurrency_control

You can check for the availability of the time slots in a given Court, and write the corresponding Reservations child entities only if their stat_time don't conflict.
Here is how you would do it for 1 single reservation using a ancestor Query:
#ndb.transactional
def make_reservation(court_id, start_time):
court = Court(id=court_id)
existing = Reservation.query(Reservation.start_time == start_time,
ancestor=court.key).fetch(2, keys_only=True)
if len(existing):
return False, existing[0]
return True, Reservation(start_time=start_time, parent=court.key).put()
Alternativly, if you make the slot part of the Reservation id, you can remove the query and construct the Reservation entity keys to check if they already exists:
#ndb.transactional
def make_reservations(court_id, slots):
court = Court(id=court_id)
rs = [Reservation(id=s, parent=court.key) for s in slots]
existing = ndb.get_multi(r.key for r in rs)
if any(existing):
return False, existing
return True, ndb.put_multi(rs)

I think you should always use transactions, but I don't think your concerns are best addressed by transactions.
I think you should implement a two-stage reservation system - which is what you see on most shopping bags and ticketing companies.
Posting the form creates a "reservation request" , which blocks out the time(s) as "in someone else's shopping bag" for 5-15 minutes
Users must submit again on an approval screen to confirm the times. You can give them the ability to update the conflicts on that screen too, and reset the 'reservation lock' on the timeslots as long as possible.
A cronjob - or a faked one that is triggered by a request coming in at a certain window - clears out expired reservation locks and returns the times back to the pool of available slots.

What is an efficient way of inserting thousands of records into an SQLite table using Django?

I have to insert 8000+ records into a SQLite database using Django's ORM. This operation needs to be run as a cronjob about once per minute.
At the moment I'm using a for loop to iterate through all the items and then insert them one by one.
Example:
for item in items:
entry = Entry(a1=item.a1, a2=item.a2)
entry.save()
What is an efficient way of doing this?
Edit: A little comparison between the two insertion methods.
Without commit_manually decorator (11245 records):
nox#noxdevel marinetraffic]$ time python manage.py insrec
real 1m50.288s
user 0m6.710s
sys 0m23.445s
Using commit_manually decorator (11245 records):
[nox#noxdevel marinetraffic]$ time python manage.py insrec
real 0m18.464s
user 0m5.433s
sys 0m10.163s
Note: The test script also does some other operations besides inserting into the database (downloads a ZIP file, extracts an XML file from the ZIP archive, parses the XML file) so the time needed for execution does not necessarily represent the time needed to insert the records.

You want to check out django.db.transaction.commit_manually.
http://docs.djangoproject.com/en/dev/topics/db/transactions/#django-db-transaction-commit-manually
So it would be something like:
from django.db import transaction
#transaction.commit_manually
def viewfunc(request):
...
for item in items:
entry = Entry(a1=item.a1, a2=item.a2)
entry.save()
transaction.commit()
Which will only commit once, instead at each save().
In django 1.3 context managers were introduced.
So now you can use transaction.commit_on_success() in a similar way:
from django.db import transaction
def viewfunc(request):
...
with transaction.commit_on_success():
for item in items:
entry = Entry(a1=item.a1, a2=item.a2)
entry.save()
In django 1.4, bulk_create was added, allowing you to create lists of your model objects and then commit them all at once.
NOTE the save method will not be called when using bulk create.
>>> Entry.objects.bulk_create([
... Entry(headline="Django 1.0 Released"),
... Entry(headline="Django 1.1 Announced"),
... Entry(headline="Breaking: Django is awesome")
... ])
In django 1.6, transaction.atomic was introduced, intended to replace now legacy functions commit_on_success and commit_manually.
from the django documentation on atomic:
atomic is usable both as a decorator:
from django.db import transaction
#transaction.atomic
def viewfunc(request):
# This code executes inside a transaction.
do_stuff()
and as a context manager:
from django.db import transaction
def viewfunc(request):
# This code executes in autocommit mode (Django's default).
do_stuff()
with transaction.atomic():
# This code executes inside a transaction.
do_more_stuff()

Bulk creation is available in Django 1.4:
https://django.readthedocs.io/en/1.4/ref/models/querysets.html#bulk-create

Have a look at this. It's meant for use out-of-the-box with MySQL only, but there are pointers on what to do for other databases.

You might be better off bulk-loading the items - prepare a file and use a bulk load tool. This will be vastly more efficient than 8000 individual inserts.

To answer the question particularly with regard to SQLite, as asked, while I have just now confirmed that bulk_create does provide a tremendous speedup there is a limitation with SQLite: "The default is to create all objects in one batch, except for SQLite where the default is such that at maximum 999 variables per query is used."
The quoted stuff is from the docs--- A-IV provided a link.
What I have to add is that this djangosnippets entry by alpar also seems to be working for me. It's a little wrapper that breaks the big batch that you want to process into smaller batches, managing the 999 variables limit.

You should check out DSE. I wrote DSE to solve these kinds of problems ( massive insert or updates ). Using the django orm is a dead-end, you got to do it in plain SQL and DSE takes care of much of that for you.
Thomas

def order(request):
if request.method=="GET":
cust_name = request.GET.get('cust_name', '')
cust_cont = request.GET.get('cust_cont', '')
pincode = request.GET.get('pincode', '')
city_name = request.GET.get('city_name', '')
state = request.GET.get('state', '')
contry = request.GET.get('contry', '')
gender = request.GET.get('gender', '')
paid_amt = request.GET.get('paid_amt', '')
due_amt = request.GET.get('due_amt', '')
order_date = request.GET.get('order_date', '')
print(order_date)
prod_name = request.GET.getlist('prod_name[]', '')
prod_qty = request.GET.getlist('prod_qty[]', '')
prod_price = request.GET.getlist('prod_price[]', '')
print(prod_name)
print(prod_qty)
print(prod_price)
# insert customer information into customer table
try:
# Insert Data into customer table
cust_tab = Customer(customer_name=cust_name, customer_contact=cust_cont, gender=gender, city_name=city_name, pincode=pincode, state_name=state, contry_name=contry)
cust_tab.save()
# Retrive Id from customer table
custo_id = Customer.objects.values_list('customer_id').last() #It is return
Tuple as result from Queryset
custo_id = int(custo_id[0]) #It is convert the Tuple in INT
# Insert Data into Order table
order_tab = Orders(order_date=order_date, paid_amt=paid_amt, due_amt=due_amt, customer_id=custo_id)
order_tab.save()
# Insert Data into Products table
# insert multiple data at a one time from djanog using while loop
i=0
while(i<len(prod_name)):
p_n = prod_name[i]
p_q = prod_qty[i]
p_p = prod_price[i]
# this is checking the variable, if variable is null so fill the varable value in database
if p_n != "" and p_q != "" and p_p != "":
prod_tab = Products(product_name=p_n, product_qty=p_q, product_price=p_p, customer_id=custo_id)
prod_tab.save()
i=i+1

I recommend using plain SQL (not ORM) you can insert multiple rows with a single insert:
insert into A select from B;
The select from B portion of your sql could be as complicated as you want it to get as long as the results match the columns in table A and there are no constraint conflicts.

def order(request):
if request.method=="GET":
# get the value from html page
cust_name = request.GET.get('cust_name', '')
cust_cont = request.GET.get('cust_cont', '')
pincode = request.GET.get('pincode', '')
city_name = request.GET.get('city_name', '')
state = request.GET.get('state', '')
contry = request.GET.get('contry', '')
gender = request.GET.get('gender', '')
paid_amt = request.GET.get('paid_amt', '')
due_amt = request.GET.get('due_amt', '')
order_date = request.GET.get('order_date', '')
prod_name = request.GET.getlist('prod_name[]', '')
prod_qty = request.GET.getlist('prod_qty[]', '')
prod_price = request.GET.getlist('prod_price[]', '')
# insert customer information into customer table
try:
# Insert Data into customer table
cust_tab = Customer(customer_name=cust_name, customer_contact=cust_cont, gender=gender, city_name=city_name, pincode=pincode, state_name=state, contry_name=contry)
cust_tab.save()
# Retrive Id from customer table
custo_id = Customer.objects.values_list('customer_id').last() #It is return Tuple as result from Queryset
custo_id = int(custo_id[0]) #It is convert the Tuple in INT
# Insert Data into Order table
order_tab = Orders(order_date=order_date, paid_amt=paid_amt, due_amt=due_amt, customer_id=custo_id)
order_tab.save()
# Insert Data into Products table
# insert multiple data at a one time from djanog using while loop
i=0
while(i<len(prod_name)):
p_n = prod_name[i]
p_q = prod_qty[i]
p_p = prod_price[i]
# this is checking the variable, if variable is null so fill the varable value in database
if p_n != "" and p_q != "" and p_p != "":
prod_tab = Products(product_name=p_n, product_qty=p_q, product_price=p_p, customer_id=custo_id)
prod_tab.save()
i=i+1
return HttpResponse('Your Record Has been Saved')
except Exception as e:
return HttpResponse(e)
return render(request, 'invoice_system/order.html')