We Keep Coding

Speed up python w/ sqlalchemy function

I have a function that populates a database table using python and sqlalchemy. The function is running fairly slowly right now, taking around 17 minutes. I think the main problem is I am looping through two large sets of data to build the new table. I have included the record count in the code below.
How can I speed this up? Should I try to convert the nested for loop into one big sqlalchemy query? I profiled this function with pycharm but am not sure I fully understand the results.
def populate(self):
"""Core function to populate positions."""
# get raw annotations with tag Org
# returns 11,659 records
organizations = model.session.query(model.Annotation) \
.filter(model.Annotation.tag == 'Org')\
.filter(model.Annotation.organization_id.isnot(None)).all()
# get raw annotations with tags Support or Oppose
# returns 2,947 records
annotations = model.session.query(model.Annotation) \
.filter((model.Annotation.tag == 'Support') | (model.Annotation.tag == 'Oppose')).all()
for org in organizations:
for anno in annotations:
# Org overlaps with Support or Oppose tag
# start and end columns are integers
if org.start >= anno.start and org.end <= anno.end:
position = model.Position()
# set to de-duplicated organization
position.organization_id = org.organization_id
position.disposition = anno.tag
# look up bill_id from document_bill table
document = model.session.query(model.document_bill)\
.filter_by(document_id=anno.document_id).first()
position.bill_id = document.bill_id
position.document_id = anno.document_id
model.session.add(position)
logging.info('org: {}, disposition: {}, bill: {}'.format(
position.organization_id, position.disposition, position.bill_id)
)
continue
logging.info('committing to database')
model.session.commit()

My bets, in order of descending probability:
Autocommit is ON, so you're waiting for disk.
The query inside the loop "document = model.session.query(model.document_bill)...." is slow (use EXPLAIN ANALYZE).
most of the time is actually spent printing logs to the terminal in the inner loop (you should profile)
model.session.add(position) is slow (no idea what that does)
(and this one should really be first) Could a SQL query like INSERT INTO SELECT do this in a couple tens of milliseconds? If so, why make a loop in the application?...

Limit child collections in initial query sqlalchemy

I am building an api which can return children of resources if the user requests it. For example, user has messages. I want the query to be able to limit the number of message objects that are returned.
I found a useful tip aboutl imiting the number of objects in child collections here. Basically, it indicates the following flow:
class User(...):
# ...
messages = relationship('Messages', order_by='desc(Messages.date)', lazy='dynamic')
user = User.query.one()
users.messages.limit(10)
My use case involves returning sometimes large numbers of users.
If I were to follow the advice in that link and used .limit() then I would need to iterate over the entire collection of users calling .limit() on each one. This is much less efficient then, say, using LIMIT in the original sql expression which created the collection.
My question is whether it is possible using declarative to efficiently(N+0) load a large collection of objects while limiting the number of children in their child collections using sqlalchemy?
UPDATE
To be clear, the below is what I am trying to avoid.
users = User.query.all()
messages = {}
for user in users:
messages[user.id] = user.messages.limit(10).all()
I want to do something more like:
users = User.query.option(User.messages.limit(10)).all()

This answer comes from Mike Bayer on the sqlalchemy google group. I'm posting it here to help folks:
TLDR:
I used version 1 of Mike's answer to solve my problem because, in this case, I do not have foreign keys involved in this relationship and so cannot make use of LATERAL. Version 1 worked great, but be sure to note the effect of offset. It threw me off during testing for a while because I didn't notice it was set to something other than 0.
Code Block for version 1:
subq = s.query(Messages.date).\
filter(Messages.user_id == User.id).\
order_by(Messages.date.desc()).\
limit(1).offset(10).correlate(User).as_scalar()
q = s.query(User).join(
Messages,
and_(User.id == Messages.user_id, Messages.date > subq)
).options(contains_eager(User.messages))
Mike's Answer
so you should ignore whether or not it uses "declarative", which has nothing to do with querying, and in fact at first ignore Query too, because first and foremost this is a SQL problem. You want one SQL statement that does this. What query in SQL would load lots of rows from the primary table, joined to the first ten rows of the secondary table for each primary?
LIMIT is tricky because it's not actually part of the usual "relational algebra" calculation. It's outside of that because it's an artificial limit on rows. For example, my first thought on how to do this was wrong:
select * from users left outer join (select * from messages limit 10) as anon_1 on users.id = anon_1.user_id
This is wrong because it only gets the first ten messages in the aggregate, disregarding user. We want to get the first ten messages for each user, which means we need to do this "select from messages limit 10" individually for each user. That is, we need to correlate somehow. A correlated subquery though is not usually allowed as a FROM element, and is only allowed as a SQL expression, it can only return a single column and a single row; we can't normally JOIN to a correlated subquery in plain vanilla SQL. We can however, correlate inside the ON clause of the JOIN to make this possible in vanilla SQL.
But first, if we are on a modern Postgresql version, we can break that usual rule of correlation and use a keyword called LATERAL, which allows correlation in a FROM clause. LATERAL is only supported by modern Postgresql versions, and it makes this easy:
select * from users left outer join lateral
(select * from message where message.user_id = users.id order by messages.date desc limit 10) as anon1 on users.id = anon_1.user_id
we support the LATERAL keyword. The query above looks like this:
subq = s.query(Messages).\
filter(Messages.user_id == User.id).\
order_by(Messages.date.desc()).limit(10).subquery().lateral()
q = s.query(User).outerjoin(subq).\
options(contains_eager(User.messages, alias=subq))
Note that above, in order to SELECT both users and messages and produce them into the User.messages collection, the "contains_eager()" option must be used and for that the "dynamic" has to go away. This is not the only option, you can for example build a second relationship for User.messages that doesn't have the "dynamic" or you can just load from query(User, Message) separately and organize the result tuples as needed.
if you aren't using Postgresql, or a version of Postgresql that doesn't support LATERAL, the correlation has to be worked into the ON clause of the join instead. The SQL looks like:
select * from users left outer join messages on
users.id = messages.user_id and messages.date > (select date from messages where messages.user_id = users.id order by date desc limit 1 offset 10)
Here, in order to jam the LIMIT in there, we are actually stepping through the first 10 rows with OFFSET and then doing LIMIT 1 to get the date that represents the lower bound date we want for each user. Then we have to join while comparing on that date, which can be expensive if this column isn't indexed and also can be inaccurate if there are duplicate dates.
This query looks like:
subq = s.query(Messages.date).\
filter(Messages.user_id == User.id).\
order_by(Messages.date.desc()).\
limit(1).offset(10).correlate(User).as_scalar()
q = s.query(User).join(
Messages,
and_(User.id == Messages.user_id, Messages.date >= subq)
).options(contains_eager(User.messages))
These kinds of queries are the kind that I don't trust without a good test, so POC below includes both versions including a sanity check.
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
import datetime
Base = declarative_base()
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
messages = relationship(
'Messages', order_by='desc(Messages.date)')
class Messages(Base):
__tablename__ = 'message'
id = Column(Integer, primary_key=True)
user_id = Column(ForeignKey('user.id'))
date = Column(Date)
e = create_engine("postgresql://scott:tiger#localhost/test", echo=True)
Base.metadata.drop_all(e)
Base.metadata.create_all(e)
s = Session(e)
s.add_all([
User(id=i, messages=[
Messages(id=(i * 20) + j, date=datetime.date(2017, 3, j))
for j in range(1, 20)
]) for i in range(1, 51)
])
s.commit()
top_ten_dates = set(datetime.date(2017, 3, j) for j in range(10, 20))
def run_test(q):
all_u = q.all()
assert len(all_u) == 50
for u in all_u:
messages = u.messages
assert len(messages) == 10
for m in messages:
assert m.user_id == u.id
received = set(m.date for m in messages)
assert received == top_ten_dates
# version 1. no LATERAL
s.close()
subq = s.query(Messages.date).\
filter(Messages.user_id == User.id).\
order_by(Messages.date.desc()).\
limit(1).offset(10).correlate(User).as_scalar()
q = s.query(User).join(
Messages,
and_(User.id == Messages.user_id, Messages.date > subq)
).options(contains_eager(User.messages))
run_test(q)
# version 2. LATERAL
s.close()
subq = s.query(Messages).\
filter(Messages.user_id == User.id).\
order_by(Messages.date.desc()).limit(10).subquery().lateral()
q = s.query(User).outerjoin(subq).\
options(contains_eager(User.messages, alias=subq))
run_test(q)

If you apply limit and then call .all() on it, you will get all objects once and it will not get objects one by one , causing performance issue that you mentioned.
simply apply limit and get all objects.
users = User.query.limit(50).all()
print(len(users))
>>50
Or for child objects / relationships
user = User.query.one()
all_messages = user.messages.limit(10).all()
users = User.query.all()
messages = {}
for user in users:
messages[user.id] = user.messages.limit(10).all()

So, I think you'll need to load the messages in a second query and then later associate with your users somehow.
The following is database dependent; as discussed in this question, mysql does not support in queries with limits, but sqlite at least will parse the query. I didn't look at the plan to see if it did a good job.
The following code will find all the message objects you care about. You then need to associate them with users.
I've tested this to confirm that it produces a query sqlite can parse; I have not confirmed that sqlite or any other database does the right thing with this query.
I had to cheat a bit and use the text primitive to refer to the outer user.id column in the select because SQLAlchemy kept wanting to include an additional join to users in the inner select subjquery.
from sqlalchemy import Column, Integer, String, ForeignKey, alias
from sqlalchemy.sql import text
from sqlalchemy.orm import Session
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key = True)
name = Column(String)
class Message(Base):
__tablename__ = 'messages'
user_id = Column(Integer, ForeignKey(User.id), nullable = False)
id = Column(Integer, primary_key = True)
s = Session()
m1 = alias(Message.__table__)
user_query = s.query(User) # add any user filtering you want
inner_query = s.query(m1.c.id).filter(m1.c.user_id == text('users.id')).limit(10)
all_messages_you_want = s.query(Message).join(User).filter(Message.id.in_(inner_query))
To associate the messages with users, you could do something like the following assuming your Message has a user relation and your user objects have a got_child_message method that does whatever you like for this
users_resulting = user_query.all() #load objects into session and hold a reference
for m in all_messages_you_want: m.user.got_child_message(m)
Because you already have the users in the session and because the relation is on User's primary key, m.user resolves to query.get against the identity map.
I hope this helps you get somewhere.

#melchoirs answer is the best. I basically putting this here for futureselves
I played around with the above stated answer, and it works, I needed it more so to limit the number of associations returned before passing into a Marshmallow Serializer.
Some issues for clarification:
the subquery runs per association, hence it finds the corresponding date to base off properly
think about the limit/offset as give me 1 (limit) record starting at the next X (offset). Hence what is the Xth oldest record, and then in the main query it gives everything back from that. Its damn smart
It appears that if the association has less than X records, it returns nothing, as the offset is past the records, and henceforth the main query does not return a record.
Using the above as a template, I came up with the below answer. The initial query/count guard is due to the issue that if the associated records are less than the offset, nothing is found. In addition, I needed to add an outerjoin in the event that there are no associations either.
At the end, I found this query to be a bit or ORM voodoo, and didn't want to go that route. I instead exclude the histories from the device serializer, and require a second history lookup using the device ID. That set can be paginated and makes everything a bit cleaner.
Both methods work, it just comes down to the why you'll need to do the one query versus a couple. In the above, there was probably business reasons to get everythng back more efficiently with the single query. For my use case, readability, and convention trumped the voodoo
#classmethod
def get_limited_histories(cls, uuid, limit=10):
count = DeviceHistory.query.filter(DeviceHistory.device_id == uuid).count()
if count > limit:
sq = db.session.query(DeviceHistory.created_at) \
.filter(DeviceHistory.device_id == Device.uuid) \
.order_by(DeviceHistory.created_at.desc()) \
.limit(1).offset(limit).correlate(Device)
return db.session.query(Device).filter(Device.uuid == uuid) \
.outerjoin(DeviceHistory,
and_(DeviceHistory.device_id == Device.uuid, DeviceHistory.created_at > sq)) \
.options(contains_eager(Device.device_histories)).all()[0]
It then behaves similar to a Device.query.get(id) but Device.get_limited_histories(id)
ENJOY

Ordering objects by rating accounting for the number or ratings

I'm trying to do something similar to the first response in this SO question: SQL ordering by rating/votes, where resources may be rated (one rating per user per resource), but when ordering the resources based on their ratings, any resources with fewer than X separate ratings will appear below those with X or more.
I'm implementing this in Django and I'd very much prefer to avoid the use of raw query and keep within the Django model and query framework.
So far, this is what I have:
data = []
data_top = Resource.objects.all().annotate(rating=Avg('resourcerating__rating'),rate_count=Count('resourcerating')).exclude(rate_count__lt=settings.ORB_RESOURCE_MIN_RATINGS).order_by(order_by)
for d in data_top:
data.append(d)
data_bottom = Resource.objects.all().annotate(rating=Avg('resourcerating__rating'),rate_count=Count('resourcerating')).exclude(rate_count__gte=settings.ORB_RESOURCE_MIN_RATINGS).order_by(order_by)
for d in data_bottom:
data.append(d)
This all functions and returns the ordering by rating as I need, however, it doesn't feel very efficient - what with running 2 queries and looping over the results of each.
Is there a better way I can code this, either in a single query, or at least avoiding looping though each query set?
Any help much appreciated.

from itertools import chain
main_query = Resource.objects.all().annotate(rating=Avg('resourcerating__rating'),rate_count=Count('resourcerating'))
data_top_query = main_query.exclude(rate_count__lt=settings.ORB_RESOURCE_MIN_RATINGS).order_by(order_by)
data_bottom_query = main_query.exclude(rate_count__gte=settings.ORB_RESOURCE_MIN_RATINGS).order_by(order_by)
data = list(chain(data_top_query, data_bottom_query))
Using itertools.chain is faster than looping each list and appending elements one by one
Also, the querysets will get evaluated when list is called on them (as they don't hit the database till then)
FYI, the above will hit the db twice when evaluated.

You're currently querying twice and iterating twice, but you can cut it down to one and one easily-just query for the items ordered by rating, then iterate like this:
data_top = []
data_bottom = []
data = Resource.objects.all().annotate(rating=Avg('resourcerating__rating'),rate_count=Count('resourcerating')).order_by(order_by)
for d in data:
if data.rate_count >= settings.ORB_RESOURCE_MIN_RATINGS:
data_top.append(d)
else:
data_bottom.append(d)
data = data_top + data_bottom
This can also be done with the query only, by creating another aggregate column which contains the value rate_count < settings.ORB_RESOURCE_MIN_RATINGS (return 0 for values above or at the threshold, 1 for below) and sorting on (new_column, rating). Pretty sure this would require some custom SQL, but perhaps someone else knows otherwise.

How to add a row number column to an SqlAlchemy query?

I would like to have the row number as a column of my queries. Since I am using MySql, I cannot use the built-in func.row_number() of SqlAlchemy. The result of this query is going to be paginated, therefore I would like to keep the row number before the split happen.
session.query(MyModel.id, MyModel.date, "row_number")
I tried to use an hybrid_property to increment a static variable inside MyModel class that I reset before my query, but it didn't work.
#hybrid_property
def row_number(self):
cls = self.__class__
cls.row_index = cls.row_index + 1
return literal(self.row_index)
#row_number.expression
def row_number(cls):
cls.row_index = cls.row_index + 1
return literal(cls.row_index)
I also tried to mix a subquery with this solution :
session.query(myquery.subquery(), literal("#rownum := #rownum + 1 AS row_number"))
But I didn't find a way to make a textual join for (SELECT #rownum := 0) r.
Any suggestions?
EDIT
For the moment, I am looping on the results of the paginated query and I am assigning the calculated number from the current page to each row.

SQLAlchemy allows you to use text() in some places, but not arbitrarily. I especially cannot find an easy/documented way of using it in columns or joins. However, you can write your entire query in SQL and still get ORM objects out of it. Example:
query = session.query(Foobar, "rownum")
query = query.from_statement(
"select foobar.*, cast(#row := #row + 1 as unsigned) as rownum"
" from foobar, (select #row := 0) as init"
)
That being said, I don't really see the problem with something like enumerate(query.all()) either. Note that if you use a LIMIT expression, the row numbers you get from MySQL will be for the final result and will still need to have the page start index added. That is, it's not "before the split" by default. If you want to have the starting row added for you in MySQL you can do something like this:
prevrow = 42
query = session.query(Foobar, "rownum")
query = query.from_statement(sqlalchemy.text(
"select foobar.*, cast(#row := #row + 1 as unsigned) as rownum"
" from foobar, (select #row := :prevrow) as init"
).bindparams(prevrow=prevrow))
In this case the numbers will start at 43 since it's pre-incrementing.

How can I make my code more efficient?

I have a list of tuples that contains a tool_id, a time, and a message. I want to select from this list all the elements where the message matches some string, and all the other elements where the time is within some diff of any matching message for that tool.
Here is how I am currently doing this:
# record time for each message matching the specified message for each tool
messageTimes = {}
for row in cdata: # tool, time, message
if self.message in row[2]:
messageTimes[row[0], row[1]] = 1
# now pull out each message that is within the time diff for each matched message
# as well as the matched messages themselves
def determine(tup):
if self.message in tup[2]: return True # matched message
for (tool, date_time) in messageTimes:
if tool == tup[0]:
if abs(date_time-tup[1]) <= tdiff:
return True
return False
cdata[:] = [tup for tup in cdata if determine(tup)]
This code works, but it takes way too long to run - e.g. when cdata has 600,000 elements (which is typical for my app) it takes 2 hours for this to run.
This data came from a database. Originally I was getting just the data I wanted using SQL, but that was taking too long also. I was selecting just the messages I wanted, then for each one of those doing another query to get the data within the time diff of each. That was resulting in tens of thousands of queries. So I changed it to pull all the potential matches at once and then process it in python, thinking that would be faster. Maybe I was wrong.
Can anyone give me some suggestions on speeding this up?
Updating my post to show what I did in SQL as was suggested.
What I did in SQL was pretty straightforward. The first query was something like:
SELECT tool, date_time, message
FROM event_log
WHERE message LIKE '%foo%'
AND other selection criteria
That was fast enough, but it may return 20 or 30 thousand rows. So then I looped through the result set, and for each row ran a query like this (where dt and t are the date_time and tool from a row from the above select):
SELECT date_time, message
FROM event_log
WHERE tool = t
AND ABS(TIMESTAMPDIFF(SECOND, date_time, dt)) <= timediff
That was taking about an hour.
I also tried doing in one nested query where the inner query selected the rows from my first query, and the outer query selected the time diff rows. That took even longer.
So now I am selecting without the message LIKE '%foo%' clause and I am getting back 600,000 rows and trying to pull out the rows I want from python.

The way to optimize the SQL is to do it all in one query, instead of iterating over 20K rows and doing another query for each one.
Usually this means you need to add a JOIN, or occasionally a sub-query. And yes, you can JOIN a table to itself, as long as you rename one or both copies. So, something like this:
SELECT el2.date_time, el2.message
FROM event_log as el1 JOIN event_log as el2
WHERE el1.message LIKE '%foo%'
AND other selection criteria
AND el2.tool = el1.tool
AND ABS(TIMESTAMPDIFF(SECOND, el2.datetime, el1.datetime)) <= el1.timediff
Now, this probably won't be fast enough out of the box, so there are two steps to improve it.
First, look for any columns that obviously need to be indexed. Clearly tool and datetime need simple indices. message may benefit from either a simple index or, if your database has something fancier, maybe something fancier, but given that the initial query was fast enough, you probably don't need to worry about it.
Occasionally, that's sufficient. But usually, you can't guess everything correctly. And there may also be a need to rearrange the order of the queries, etc. So you're going to want to EXPLAIN the query, and look through the steps the DB engine is taking, and see where it's doing a slow iterative lookup when it could be doing a fast index lookup, or where it's iterating over a large collection before a small collection.

For tabular data, you can't go past the Python pandas library, which contains highly optimised code for queries like this.

I fixed this by changing my code as follows:
-first I made messageTimes a dict of lists keyed by the tool:
messageTimes = defaultdict(list) # a dict with sorted lists
for row in cdata: # tool, time, module, message
if self.message in row[3]:
messageTimes[row[0]].append(row[1])
-then in the determine function I used bisect:
def determine(tup):
if self.message in tup[3]: return True # matched message
times = messageTimes[tup[0]]
le = bisect.bisect_right(times, tup[1])
ge = bisect.bisect_left(times, tup[1])
return (le and tup[1]-times[le-1] <= tdiff) or (ge != len(times) and times[ge]-tup[1] <= tdiff)
With these changes the code that was taking over 2 hours took under 20 minutes, and even better, a query that was taking 40 minutes took 8 seconds!

I made 2 more changes and now that 20 minute query is taking 3 minutes:
found = defaultdict(int)
def determine(tup):
if self.message in tup[3]: return True # matched message
times = messageTimes[tup[0]]
idx = found[tup[0]]
le = bisect.bisect_right(times, tup[1], idx)
idx = le
return (le and tup[1]-times[le-1] <= tdiff) or (le != len(times) and times[le]-tup[1] <= tdiff)