We Keep Coding

Erratic sort order for hybrid_property sum expression in Flask-Admin using SQLAlchemy

Given the following models, I would like to add a sortable column to Flask Admin that shows all upcoming payments for an Organisation:
organisation_payments = db.Table(
'organisation_payments',
db.Column('payment_id', MyUUID(), db.ForeignKey('payment.id')),
db.Column('organisation_id', MyUUID(), db.ForeignKey('organisation.id'))
)
class Organisation(db.Model):
id = Column(ModelUUID(), primary_key=True)
#hybrid_property
def upcoming_payments(self):
payments = Payment.query.filter(
Payment.organisations.any(id=self.id),
Payment.status == 'active'
).all()
return sum([payment.amount for payment in payments])
#upcoming_payments.expression
def upcoming_payments(cls):
return select([
func.sum(Payment.amount)
]).where(and_(
organisation_payments.c.organisation_id == cls.id,
Payment.status == 'active'
)).label('upcoming_payments')
class Payment(db.Model):
id = Column(ModelUUID(), primary_key=True)
amount = db.Column(db.Integer())
status = db.Column(db.Unicode(255), default='active')
organisations = db.relationship(
'Organisation',
secondary=organisation_payments,
backref=db.backref('payments', lazy='dynamic')
)
Note that a Payment could theoretically be mapped to multiple Organisations, hence the many-to-many relationship.
I have added upcoming_payments to column_sortable_list in a Flask-Admin ModelView, but when sorting on that column the results are erratic: https://i.stack.imgur.com/VxrAE.png
This is the most minimal version of the code, but I've also tried:
using coalesce to force 0 for rows with no upcoming payments
using as_scalar() in place of .label()
Payment.organisations.any(id=cls.id) in place of organisation_payments.c.organisation_id == cls.id (this produced even more confusing and inconsistent results, and sorting asc/desc made no difference)
The values returned from the regular hybrid_property are correct, as is the result if I run this in the shell:
stmt = select([
func.sum(Payment.amount)
]).where(and_(
Payment.organisations.any(id=org.id),
Payment.status == 'active',
))
res = db.engine.execute(stmt)
res.fetchone()
(4036200L,)
However the result is wildly inaccurate if I run this:
stmt = select([
func.sum(Payment.amount)
]).where(and_(
organisation_payments.c.organisation_id == org.id,
Payment.status == 'active',
))
res = db.engine.execute(stmt)
res.fetchone()
(1204440000L,)
But neither the any() or association table method returns the correct sort order in Flask-Admin. What am I doing wrong here? I feel like I must be missing a distinct or a subquery, but I'm stumped.
UPDATE: All thanks to Ilja Everilä for guiding me to the answer:
#upcoming_payments.expression
def upcoming_payments(cls):
q = select([coalesce(func.sum(Payment.amount), 0)])
q = q.where(and_(
organisation_payments.c.charity_id == cls.id,
Payment.status == 'active',
Payment.deleted.isnot(True)
))
j = Payment.__table__.join(organisation_payments)
q = q.select_from(j)
return q.label('upcoming_donations')

UPDATE: All thanks to Ilja Everilä for guiding me to the answer:
#upcoming_payments.expression
def upcoming_payments(cls):
q = select([coalesce(func.sum(Payment.amount), 0)])
q = q.where(and_(
organisation_payments.c.charity_id == cls.id,
Payment.status == 'active',
Payment.deleted.isnot(True)
))
j = Payment.__table__.join(organisation_payments)
q = q.select_from(j)
return q.label('upcoming_donations')

using another WS as validation Flask/Rest/Mysql

I am trying to build a simple web application with 3 web services. Two of my web services are supposed to validate if a student exist in a course or not. This is done by a simple SELECT-query. My third web service should add a student into a database, but only if the student do exist in the specific course.
This is my validation WS which should return a true/false.
#app.route('/checkStudOnCourse/<string:AppCode>/<string:ideal>', methods= ["GET"])
def checkStudOnCourseWS(AppCode, ideal):
myCursor3 = mydb.cursor()
query3 = ("SELECT studentID FROM Ideal.course WHERE applicationCode = " + "'" + AppCode + "' AND Ideal = " + "'" + ideal + "'")
myCursor3.execute(query3)
myresult3 = myCursor3.fetchall()
if len(myresult3) == 0:
return render_template('Invalid.html')
else:
return jsonify({'Student in course ': True})
Below is regResult which should do a SQL insert into a database. I only want the submit to work if the above result is "True", how can I do that? I know I have not done the INSERT query, but that is not a problem.
What I am unsure about is: How can I only let the submit be be INSERTED if the validation WS is "True".
#app.route('/register', methods=["POST", "GET"])
def regResultat():
if request.method == "POST":
Period = request.form['period']
#ProvNr = request.form['provNr']
Grade = request.form['grade']
Applicationcode = request.form['applicationcode']
#Datum = request.form['datum']
Ideal = request.form['ideal']
CheckStudOnCourse = 'http://127.0.0.1:5000/checkAppCodeWS/'+Applicationcode+'/'+Ideal
CheckStudOnResp = requests.get(CheckStudOnCourse)

At first, such syntax:
if len(myresult3) == 0, can be simplified by if myresult3, because Python evaluates that implicitly to bool.
Secondly, if you once returned from function, there is no need to write an else statement:
if len(myresult3) == 0:
return render_template('Invalid.html') # < -- in case 'True',
# it returns here, otherwise
# function keeps going"""
return jsonify({'Student in course ': True}) # < -- in case 'False', it is returned here
Focusing on your issue, you could do that:
Get your value from ws
CheckStudOnCourse = 'http://127.0.0.1:5000/checkAppCodeWS/'+Applicationcode+'/'+Ideal
CheckStudOnResp = requests.get(CheckStudOnCourse)
Extract json from it:
if result_as_json.status == 200:
result_as_json = CheckStudOnResp.json() # < -- it is now a dict
Do some checks:
if result_as_json.get('Student in course', False): # I highly suggest to use other
# convention to name json keys
# e.g. Student in course ->
# student_exists_in_course
# do your code here

django model building table name wrong

I have a postgresql RDS instance on aws free tier with my ec2 instance. I should have used EB, but didn't really know about it until I had a lot built out and am not excited about starting over on a new ami. I am trying to build my first interaction with the db, have a table there called server_config, created as:
CREATE TABLE SERVER_CONFIGS
(
config_key VARCHAR(63) NOT NULL,
config_value VARCHAR(63) NOT NULL,
UNIQUE (config_key)
);
INSERT INTO SERVER_CONFIGS (config_key,config_value)
VALUES ('ClientMustVerify','TRUE');
INSERT INTO SERVER_CONFIGS (config_key,config_value)
VALUES ('Auditing','FALSE');
COMMIT;
and in my django app, I have:
models.py:
from django.db import models
class serverConfig(models.Model):
config_key = models.CharField(max_length=63)
config_value = models.CharField(max_length=63)
excerpt of view.py:
from limbo.models import serverConfig
def editServer(request):
myConfigs = serverConfig.objects.all()
configHtml = ""
# for item in myConfigs #serverConfig.objects.values()
# configHtml += item.config_key + "\t" + item.config_value + "\n"
if request.method == 'POST':
form = serverForm(request.POST)
if form.is_valid():
integer = form.cleaned_data['int_field']
request.session['integer'] = integer
# call out to limboLogic.py to update values, add them to the session
message = 'The value \'' + str(integer) + '\' has been updated.'
return render(request, 'limboHtml/ServerConfiguration.html', {'form': form, 'SubmitMessage': message, 'CurrentConfigs': myConfigs})
else:
message = 'The server configuration has NOT been updated.' + '\n'
message += ', '.join("%s=%r" % (key,val) for (key,val) in form.errors.iteritems()) + '\n'
# message += ', '.join("%s=%r" % (key,val) for (key,val) in form.non_field_errors.iteritems()) + '\n'
return render(request, 'limboHtml/ServerConfiguration.html', {'form': form, 'SubmitMessage': message, 'CurrentConfigs': myConfigs})
# if a GET (or any other method) we'll create a blank form
try:
del request.session['integer']
except KeyError:
pass
form = serverForm()
return render(request, 'limboHtml/ServerConfiguration.html', {'form': form, 'SubmitMessage': '', 'CurrentConfigs': myConfigs})
error message:
File
"/home/ec2-user/limbo/limboenv/local/lib/python2.7/site-packages/django/db/backends/utils.py",
line 64, in execute
return self.cursor.execute(sql, params) ProgrammingError: relation "limbo_serverconfig" does not exist LINE 1: ...ig_key",
"limbo_serverconfig"."config_value" FROM "limbo_ser...
Why is it appending limbo_ at the start of the table? any way I can change this? should I just use my app name (or is that my project name? they're called the same thing...stupid polls tutorial) in the tables?
Django 1.10 and python 2.7 on a linux ami on ec2

Django creates a table for each of your models. The default name of the table will be app_name + '_' + model_name. If you do not want that to be the table's name, you can override it by specifying db_table in the models's Meta class
Ref: https://docs.djangoproject.com/en/1.10/ref/models/options/#db-table
Suggestion
Please do not use the app with the same name as the project to write your application logic. That app should always contain only the project settings

Sqlalchemy not committing object changes to postgres DB

I am using the following passage of code:
#app.route('/budget_item/<int:budget_id>/edit', methods=['GET', 'POST'])
def budget_item_edit(budget_id):
budget_item = session.query(Budget).filter_by(id=budget_id).one()
print "Start EDIT sequence"
# Return form data from HTML initial load form
elif request.method == 'POST':
budget_amount_reallocated_total = budget_item.budget_amount_reallocated_total
#ORIGINAL BUDGET
if request.form['transaction_type'] == 'Original Budget':
#amount
if request.form['amount'] == "":
amount = 0
else:
amount = float(str(request.form['amount']))
budget_item = Budget(
#created_date = "",
budget_transaction_type = request.form['transaction_type'],
budget_line = request.form['budget_line'],
amount = amount,
description = request.form['description']
#date_received = request.form['date_received']
)
try:
count = 1
while count < 10000:
count += 1
#budget_line
setattr(budget_item,'budget_line'+str(count),request.form['budget_line'+str(count)])
#amount
setattr(budget_item,'amount'+str(count),float(request.form['amount'+str(count)]))
budget_amount_reallocated_total += float(request.form['amount'+str(count)])
setattr(budget_item, 'budget_amount_reallocated_total', budget_amount_reallocated_total)
#description
setattr(budget_item,'description'+str(count), request.form['description'+str(count)])
#date_received
setattr(budget_item,'date_received'+str(count),request.form['date_received'+str(count)])
session.commit()
except:
session.commit()
return redirect(url_for('budget_master'))
else:
print "I'm done! This is not a post request"
This block of code is setup to pass data from an HTML via a POST request an then update a corresponding object in the Postgres DB. I can confirm that the object queried from the DB "budget_item" is being updated by settattr. At the end of the passage, I use commit() to update the object; however, the database doesn't reflect the changes. Just to test to make sure things are flowing, I've tried session.add(budget_item) followed by session.commit() to make sure the connect to the DB is OK. That works. How do i update this budget_item object into the database? Any help is much appreciated.

i think that a simple
budget_item.budget_amount_reallocated_total = budget_amount_reallocated_total
session.add(budget_item)
session.commit()
is the right way to do it

To answer your question, to update the budget_item that already exists in the database you need to update the Budget instance that you retrieved from the database, i.e.
budget_item = session.query(Budget).filter_by(id=budget_id).one()
not the one that you have newly created with:
budget_item = Budget(...)
Here the first budget_item represents the row in the database, so this is the one to update. To that end you can replace the code that creates the second Budget instance with this:
budget_item.budget_transaction_type = request.form['transaction_type']
budget_item.budget_line = request.form['budget_line']
budget_item.amount = amount
budget_item.description = request.form['description']
Once you have finished updating the Budget instance you can call session.commit() to flush it to the database.
As mentioned in my comment to your question, it appears that you are trying to add a large number of additional attributes to budget_item all of which will be ignored by sqlalchemy unless they are defined in the mapping between the Budget instance and the Budget table.

Peewee: filtering by many-to-many relationship

Here's a basic example - posts are owned and liked by users. How to select liked posts for specific user?
import datetime
import peewee
class User(peewee.Model):
name = peewee.CharField(max_length=200)
class Post(peewee.Model):
user = peewee.ForeignKeyField(User)
text = peewee.TextField()
class Like(peewee.Model):
user = peewee.ForeignKeyField(User)
post = peewee.ForeignKeyField(Post)
time = peewee.DateTimeField(default=datetime.datetime.now)
I make a query from perspective of Like model, starting query with Like.select() and join Post model. Is it ok? How could I query from perspective of Post model, i.e. Post.select()...?
Here's a complete sample for convinience:
def check_likes():
user, _ = User.get_or_create(name="John")
post, _ = Post.get_or_create(text="Hello world!", user=user)
like, _ = Like.get_or_create(post=post, user=user)
print("Created: ", user, post, like)
# Get liked posts for user
query_likes = (Like.select()
.where(Like.user == user)
.join(User)
.join(Post))
liked_posts = [like.post for like in query_likes]
print("Liked posts: ", liked_posts)
if __name__ == '__main__':
User.create_table(True)
Post.create_table(True)
Like.create_table(True)
check_likes()
UPD. I ended up with such query at the moment:
def liked_by(user_id):
"""Active posts liked by user."""
return (Post.select(Post, User, Like)
.join(User, on=(Post.user == User.id))
.switch(Post)
.join(Like, peewee.JOIN.LEFT_OUTER,
on=(Like.post == Post.id).alias('like'))
.where(Like.user == user_id)
.order_by(Like.time.desc()))

The query above should work, but you might modify it to read:
Like.select(Like, Post)...
You also need to take care about how you are joining on tables. Use the 'switch' method:
query_likes = (Like.select(Like, Post) # <<< Add like and post
.where(Like.user == user)
.join(User)
.switch(Like) # <<< You need this.
.join(Post))
liked_posts = [like.post for like in query_likes]
You can also do:
Post.select().join(Like).where(Like.user == some_user)