Trying to Convert SQL query to SQLAlchemy query - python

I'm trying to translate raw SQL into an sqlalchemy query, but so far I'm getting different results. The results I get from my attempt at sqlalchemy is missing Property objects (I get back a tuple with (ListingCalendarDays, None, PricingData)), so I think something is missing in the translation.
Original query:
result = session.execute("""SELECT p.id as property_id,
p.home_code,
c.listing_id,
c.calendar_date,
c.available,
ab.price
FROM listing_calendar_days c
LEFT JOIN properties p
ON (p.id::integer = c.listing_id and p.id is not null and p.id <> '')
LEFT JOIN pricing_data ab
ON c.listing_id = ab.listing_id
AND c.calendar_date = ab.price_date
WHERE c.calendar_date >= '%s'
AND c.calendar_date <= '%s' ;""" % ( now.isoformat(), end.isoformat()))
My Attempt:
query = (session.query( ListingCalendarDays, Properties, PricingData )
.outerjoin(Properties,
and_(Properties.id == str(ListingCalendarDays.listing_id),
Properties.id != None))
.outerjoin(PricingData,
and_(ListingCalendarDays.listing_id == PricingData.listing_id,
ListingCalendarDays.calendar_date == PricingData.price_date))
.filter(ListingCalendarDays.calendar_date.between( now, end )))
result = query.all()


You can print out the query that SQLAlchemy generates by doing
print(query)
In this particular case, your join condition for Properties is wrong. str(ListingCalendarDays.listing_id) produces the literal string "ListingCalendarDays.listing_id", not the column listing_id as casted to varchar. For a cast, you need the cast() construct:
cast(ListingCalendarDays.listing_id, String)
In addition, you're also missing the p.id <> '' condition.
Putting it all together:
and_(Properties.id == cast(ListingCalendarDays.listing_id, String),
Properties.id != None,
Properties.id != "")

Related

How to resolve SQLAlchemy Union Throwing Error

I'm using SQL Alchemy(Python, SQLServer) Union on two queries. It throws me the below error. Please help me in resolving it.
query1 = db.query(Employee.LastName).filter(Employee.Age == 30).all()
query2 = db.query(Employee.LastName).filter(Employee.Salary > 25000).all()
query3 = union(query1, query2).all()
**"SELECT construct for inclusion in UNION or other set construct expected, got [('Joseph',),('Alan',),('Joseph',)]."**
Also tried the below query and it throws the below error
query1 = db.query(Employee.LastName).filter(Employee.Age == 30).all()
query2 = db.query(Employee.LastName).filter(Employee.Salary > 25000).all()
query3 = query1.union(query2).all()
**"'list' object has no attribute 'union'"**

Remove the .all() from the first two queries, it turns the queries into lists, but you want to pass Query instances to union.
query1 = db.query(Employee.LastName).filter(Employee.Age == 30) # <- Query
query2 = db.query(Employee.LastName).filter(Employee.Salary > 25000) # <- Query
result = query1.union(query2).all() # <- List

SQL query f string formatting in Python script

I have been trying to apply an f string formatting for the colname parameter inside the SQL query for a script I am building, but I keep getting a parse exception error.
def expect_primary_key_have_relevant_foreign_key(spark_df1, spark_df2, colname):
'''
Check that all the primary keys have a relevant foreign key
'''
# Create Temporary View
spark_df1.createOrReplaceTempView("spark_df1")
spark_df2.createOrReplaceTempView("spark_df2")
# Wrap Query in spark.sql
result = spark.sql("""
select df1.*
from spark_df1 df1
left join
spark_df2 df2
f"on trim(upper(df1.{colname})) = trim(upper(df2.{colname}))"
f"where df2.{colname} is null"
""")
if result == 0:
print("Validation Passed!")
else:
print("Validation Failed!")
return result

I found the solution, the f goes before the triple quotes """ as:
# Wrap Query in spark.sql
result = spark.sql(f"""
select df1.*
from spark_df1 df1
left join
spark_df2 df2
on trim(upper(df1.{colname})) = trim(upper(df2.{colname}))
where df2.{colname} is null
""")

Getting SQL syntax error while using case when in sqlalchemy

I am using case when in sqlAlchemy as shown below:
abc = "%abc%"
def = "%def%"
proj1 = "%project1%"
proj2 = "%project2%"
case_condition = case([
(text('FPro.status = "ON" and Ab.name.like (''' + abc + ''') and F.project.like (''' + proj1 ''')'''), 'value1'),
(text('FPro.status = "ON" and Ab.name.like (''' + abc + ''') and F.project.like (''' + proj2 + ''')'''), 'value2'),
(text('FPro.status = "OFF" and Ab.name.like (''' + def + ''') and F.project.like (''' + abc + ''')'''), 'value3')]).label (deriver_vals)
query = db.session.query(F)\
.join(FPro, F.id == FPro.f_id)\
.join(Ab, Ab.id == F.ab_id).with_entities(FPro.f_id, case_condition,
F.f_name, F.proj,
FPro.status).subquery()
main_query = db.session.query(Tags).join(query, Tags.tag == query.c.derived_vals).\
with_entities(query.c.f_id.label('f_id'), query.c.derived_vals.label('derived_vals'), Tags.id.label('tag_id')).all()
The above code generates a sql statement like below:
SELECT anon_1.f_id AS f_id, anon_1.derived_vals AS derived_vals, fw_tags.id AS tag_id
FROM fw_tags INNER JOIN (SELECT fpro.f_id AS f_id, CASE WHEN FPro.status = "ON" and Ab.name.like (%%abc%%) and F.project.like (%%proj1%%) THEN %(param_1)s WHEN FPro.status = "ON" and Ab.name.like ( %%abc%%) and Flow.project.like ( %%proj2%%) THEN %(param_2)s WHEN FPro.status = "ON" and Ab.name.like (%%def%%) and F.project.like (%%proj1%%) THEN %(param_3)s END AS derived_vals, F.f_name , F.proj AS project,
FROM F INNER JOIN FPro ON f.id = Fpro.f_id INNER JOIN Ab ON Ab.id = F.ab_id) AS anon_1 ON fw_tags.tag = anon_1.derived_vals]
This is exactly the query I want but I am getting below error while executing the script which contains above code:
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(%abc%) and F.project.like (%proj1%) THEN 'value1' WHEN FPro' at line 2")
I am guessing the error is with the '%' getting appended, but I am not sure, so any help regarding why this error is occuring or what can be done to prevent the % getting added will be appreciated.
Is there a way to do this without using text or literal()?

In standard SQL "ON" is a delimited/quoted identifier, not a (text) literal, identifying a column, table, or some other schema object. Use single quotes instead: 'ON'. Some DBMS have modes that allow using double quotes for literals, or even attempt to infer the meaning from context, but perhaps it is not a good idea to get into the habit.
In Python 'FPro.status ...''' is the concatenation of 2 string literals, 'FPro.status ...' and '', not a single literal with an escaped single quote in it.
Please do not concatenate or otherwise manually format values to SQL queries, it is error prone as you have found out. It should be obvious from the generated SQL that the values are concatenated as is, without proper quoting, and so produce the incorrect statement. The correct way to pass values to (raw) SQL queries is to use placeholders, or in case of SQLAlchemy, use the SQL Expression Language.
Using placeholders:
abc = "%abc%"
def_ = "%def%"
proj1 = "%project1%"
proj2 = "%project2%"
# Using **raw SQL** fragments
case_condition = case([
(text("FPro.status = 'ON' AND Ab.name LIKE :abc AND F.project LIKE :proj1"), 'value1'),
(text("FPro.status = 'ON' AND Ab.name LIKE :abc AND F.project LIKE :proj2"), 'value2'),
(text("FPro.status = 'OFF' AND Ab.name LIKE :def_ AND F.project LIKE :abc"), 'value3')
])
query = db.session.query(F)\
.join(FPro, F.id == FPro.f_id)\
.join(Ab, Ab.id == F.ab_id)\
.with_entities(
FPro.f_id,
case_condition.label('derived_vals'),
F.f_name,
F.proj,
FPro.status)\
.subquery()
main_query = db.session.query(Tags)\
.join(query, Tags.tag == query.c.derived_vals)\
.with_entities(
query.c.f_id.label('f_id'),
query.c.derived_vals.label('derived_vals'),
Tags.id.label('tag_id'))\
.params(abc=abc, def_=def_, proj1=proj1, proj2=proj2)\
.all()
Using the expression language:
from sqlalchemy import and_
abc = "%abc%"
def_ = "%def%"
proj1 = "%project1%"
proj2 = "%project2%"
# Using SQLAlchemy SQL Expression Language DSL **in Python**
case_condition = case([
(and_(FPro.status == 'ON', Ab.name.like(abc), F.project.like(proj1)), 'value1'),
(and_(FPro.status == 'ON', Ab.name.like(abc), F.project.like(proj2)), 'value2'),
(and_(FPro.status == 'OFF', Ab.name.like(def_), F.project.like(abc)), 'value3')
])
query = db.session.query(F)\
.join(FPro, F.id == FPro.f_id)\
.join(Ab, Ab.id == F.ab_id)\
.with_entities(
FPro.f_id,
case_condition.label('derived_vals'),
F.f_name,
F.proj,
FPro.status)\
.subquery()
main_query = db.session.query(Tags)\
.join(query, Tags.tag == query.c.derived_vals)\
.with_entities(
query.c.f_id.label('f_id'),
query.c.derived_vals.label('derived_vals'),
Tags.id.label('tag_id'))\
.all()

SQLAlchemy Joining with subquery issue

I am trying to translate SQL into SQLAlchemy. The SQL version of the query I want is as follows:
SELECT * from calendarEventAttendee
JOIN calendarEventAttendanceActual ON calendarEventAttendanceActual.id = calendarEventAttendee.attendanceActualId
LEFT JOIN
(SELECT bill.id, bill.personId, billToEvent.eventId FROM bill JOIN billToEvent ON bill.id = billToEvent.billId) b
ON b.eventId = calendarEventAttendee.eventId AND b.personId = calendarEventAttendee.personId
WHERE b.id is NULL
My SQLAlchemy query is as follows:
query = db.session.query(CalendarEventAttendee).join(CalendarEventAttendanceActual)
sub_query = db.session.query(Bill, BillToEvent).join(BillToEvent, BillToEvent.billId == Bill.id).subquery()
query = query.outerjoin(sub_query, and_(sub_query.Bill.personId == CalendarEventAttendee.personId, Bill.eventId == CalendarEventAttendee.eventId))
results = query.all()
I am getting an error AttributeError: 'Alias' object has no attribute 'Bill'
If I adjust the SQLAlchemy query to the following:
sub_query = db.session.query(Bill, BillToEvent).join(BillToEvent, BillToEvent.billId == Bill.id).subquery()
query = query.outerjoin(sub_query, and_(sub_query.Bill.personId == CalendarEventAttendee.personId, sub_query.BillToEvent.eventId == CalendarEventAttendee.eventId))
results = query.all()
I get an error AttributeError: Bill
Any help would be appreciated, thanks!

Once you call subquery(), there is no access to objects, but only to columns via .c.{column_name} accessor.
Do the following for sub_query instead: load only the columns you need in order to avoid any name collisions:
sub_query = db.session.query(
Bill.id, Bill.personId, BillToEvent.eventId
).join(BillToEvent, BillToEvent.billId == Bill.id).subquery()
Then in your query use column names with .c.column_name:
query = query.outerjoin(
sub_query, and_(
sub_query.c.personId == CalendarEventAttendee.personId,
sub_query.c.eventId == CalendarEventAttendee.eventId)
)
results = query.all()

How can I get the youngest objects from SQLAlchemy?

Each row in my table has a date. The date is not unique. The same date is present more than one time.
I want to get all objects with the youngest date.
My solution work but I am not sure if this is a elegent SQLAlchemy way.
query = _session.query(Table._date) \
.order_by(Table._date.desc()) \
.group_by(Table._date)
# this is the younges date (type is date.datetime)
young = query.first()
query = _session.query(Table).filter(Table._date==young)
result = query.all()
Isn't there a way to put all this in one query object or something like that?

You need a having clause, and you need to import the max function
then your query will be:
from sqlalchemy import func
stmt = _session.query(Table) \
.group_by(Table._date) \
.having(Table._date == func.max(Table._date)
This produces a sql statement like the following.
SELECT my_table.*
FROM my_table
GROUP BY my_table._date
HAVING my_table._date = MAX(my_table._date)
If you construct your sql statement with a select, you can examine the sql produced in your case using. *I'm not sure if this would work with statements query
str(stmt)

Two ways of doing this using a sub-query:
# #note: do not need to alias, but do in order to specify `name`
T1 = aliased(MyTable, name="T1")
# version-1:
subquery = (session.query(func.max(T1._date).label("max_date"))
.as_scalar()
)
# version-2:
subquery = (session.query(T1._date.label("max_date"))
.order_by(T1._date.desc())
.limit(1)
.as_scalar()
)
qry = session.query(MyTable).filter(MyTable._date == subquery)
results = qry.all()
The output should be similar to:
# version-1
SELECT my_table.id AS my_table_id, my_table.name AS my_table_name, my_table._date AS my_table__date
FROM my_table
WHERE my_table._date = (
SELECT max("T1"._date) AS max_date
FROM my_table AS "T1")
# version-2
SELECT my_table.id AS my_table_id, my_table.name AS my_table_name, my_table._date AS my_table__date
FROM my_table
WHERE my_table._date = (
SELECT "T1"._date AS max_date
FROM my_table AS "T1"
ORDER BY "T1"._date DESC LIMIT ? OFFSET ?
)

Categories