I'm trying to convert an sql query into an sqlalchemy one and I'm having trouble with translating a "where not exists(select...)" part.
Here's the base query that I wish to convert to sqlalchemy:
select sd.address,
sd.colour_code,
s.id as deactivated_id,
s.type,
'http://site/groups/' || c.group_id || '/user/' || c.subject_id as URL
from sensors s
join contracts c on s.contract_id=c.id
join sensordevices sd on sd.id=s.device_id
where s.begin_time != null and
s.end_time != null and
c.end_date != and
not exists (select * from sensors sb
where sb.begin_time != null and --b properly entered
sb.end_time == null and -- b active
sb.contract_id==s.contract_id and
s.type==sb.type and
s.end_time<=sb.begin_time)
;
This is where I left the code(erroneous code) after my last tries:
s = aliased(SensorInstance)
sb = aliased(SensorInstance)
sd = aliased(SensorDevice)
c = aliased(Contract)
info = db.session.query(sd.address,sd.colour_code,s.id,s.role,c.group_id,c.resident_id)\
.filter(s.contract_id==c.id)\
.filter(s.device_id==sd.id)\
.filter(s.begin_time != None)\
.filter(s.end_time != None)\
.filter(c.end_date != None)\
.filter(sd.device_type == device_type)\
.filter(sd.address == device_address)\
.filter(not_(and_(sb.begin_time != None,
sb.end_time == None,
sb.role==s.role,
sb.contract_id==s.contract_id,
s.end_time<=sb.begin_time).exists())).first()
Which produces a 'Comparator' object has no attribute 'exists' which is a reasonable error.
Another thing that I have tried is this (with the same aliases as above):
info = db.session.query(sd.address,sd.colour_code,s.id,s.role,c.group_id,c.resident_id)\
.filter(s.contract_id==c.id)\
.filter(s.device_id==sd.id)\
.filter(s.begin_time != None)\
.filter(s.end_time != None)\
.filter(c.end_date != None)\
.filter(sd.device_type == device_type)\
.filter(sd.address == device_address)\
.filter(~exists().where(and_(sb.begin_time != None,
sb.end_time == None,
sb.role==s.role,
sb.contract_id==s.contract_id,
s.end_time<=sb.begin_time).exists())).first()
Which produces a sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) SELECT * with no tables specified is not valid LINE 3: ...sordevices_1.address = 4278262373 AND NOT (EXISTS (SELECT *
error. (This I tried to do based on the solution provided here
Does anyone have any pointers or a solution on how to fix this?
Thank you very much!
Without being able to test because I dont have access to data I believe the second approach can work but as the error says you have a malformed subquery. You are making a select without specifying the table (or object in this cas of orm) where the subquery operates on.
Basically I think you mees to make a subquery of the form:
subquery = ~db.session.query(sd.address,sd.colour_code,s.id,s.role,c.group_id,c.resident_id)\
.filter(sb.begin_time is not None,
sb.end_time is None,
sb.role==s.role,
sb.contract_id==s.contract_id,
s.end_time<=sb.begin_time).exists()
For more details see here this thread
Related
I have a python code that create sql query in order to allow the user to filter the required data from a database.
Until now I am able to create a query using function.
The problem is that the logical operators or/and are the same for all fields
Query: name.str.contains('') or nickname.str.contains('') or mother_name.str.contains('') or first_nationality == 'None'
what i want is to be able to create different logical operator for each field like this:
Query: name.str.contains('') or nickname.str.contains('') and mother_name.str.contains('') or first_nationality == 'None'
the image below show the user input and the constructed query
code:
import streamlit as st
import pandas as pd
#SQL pckgs
import pyodbc
def build_query(input_values, logical_op, compare_op):
query_frags = []
for k, v in input_values.items():
if v['dtype'] == list:
query_frag_expanded = [f"{v['db_col']} {compare_op} '{val}'" for val in v['value']]
query_frag = f' {logical_op} '.join(query_frag_expanded)
elif v['dtype'] == int or v['dtype'] == float:
query_frag = f"{v['db_col']} {compare_op} {v['dtype'](v['value'])}"
elif v['dtype'] == str:
query_frag = f"{v['db_col']}.str.contains('{v['dtype'](v['value'])}')"
else:
query_frag = f"{v['db_col']} {compare_op} '{v['dtype'](v['value'])}')"
query_frags.append(query_frag)
query = f' {logical_op} '.join(query_frags)
return query
def configure_query():
c1, c2, _ = st.columns([1,1,2])
with c1:
logical_op = st.selectbox('Logical operator', options=['and', 'or'], index=1)
with c2:
compare_op = st.selectbox('Comparator operator', options=['==', '>', '<', '<=', '>='], index=0)
return logical_op, compare_op
logical_op, compare_op = configure_query()
query = build_query(input_values, logical_op, compare_op)
Take a look at SQLAlchemy
I've never used pyodbc to be honest and I'm not sure if you have any constraint that makes you use it. I normally use Sqlalchemy to create a connection to a database (my case is PostgreSQL) and then I use pd.read_sql(QUERY,con) where con is the connection I've created with Sqlalchemy and QUERY is a string that represents the sql query.
If you do that you would be able to easily create a function that receives the conditions you want to add and pass it to the query text.
What I want is to get the entities from the ShipProperties table and boil down information from the ShipPropertiesVisibility table to get simple entities can_see and can_edit stating whether the user can see or edit the property, respectively.
Basically something like this (where I am trying to use python methods from the same module) except this is not working syntactically:
def get_properties(ship_internal_id, user_id, company_id):
ship_properties = db.session.query(ShipProperties,
_can_see(ShipPropertiesVisibility.visible
if ShipPropertiesVisibility is not None else None) \
.label("can_see"),
_can_edit(ShipAccess.view_only, ShipPropertiesVisibility.editable
if ShipPropertiesVisibility is not None else None) \
.label("can_edit"))
...
return ship_properties
def _can_see(visible):
return visible is None or visible is True
def _can_edit(ship_access_view_only, editable):
return ship_access_view_only is False \
and (editable is None or editable is True)
I also tried the following since my _can_see and _can_edit functions are just simple combinations of and and or operators:
ship_properties = db.session.query(ShipProperties,
func.or_(ShipPropertiesVisibility.visible == None,
ShipPropertiesVisibility.visible == True) \
.label("can_see"),
func.and_(ShipAccess.view_only == False,
or_(ShipPropertiesVisibility.editable == None,
ShipPropertiesVisibility.editable == True)) \
.label("can_edit"))\
...
But with this I get the following error:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "or"
LINE 1: ...groups, properties.private AS properties_private, or(ship_pr...
I also looked into hybrid properties and methods, but I don't think those will work since the ShipPropertiesVisibility entity can be None (meaning the property can both be seen and edited by the user) in which case I don't believe I could call a method or property. Correct me if I'm wrong.
Am I close to a solution and just missing the correct syntax or is this a profound error in thinking, and I should rather be looking into e.g. transforming the sqlalchemy result into a view model, or something else?
I was able to find a way to use the boolean sql operators by utilizing the generic .op() method.
ship_properties = db.session.query(ShipProperties, Properties,
((ShipPropertiesVisibility.visible == None) \
.op("OR")(ShipPropertiesVisibility.visible)) \
.label("can_see"),
(ShipAccess.view_only == False)
.op("AND")((ShipPropertiesVisibility.editable == None) \
.op("OR")(ShipPropertiesVisibility.editable)) \
.label("can_edit"))\
I have following three tables.
User:
[
user_id
first_name
last_name
]
Picture:
[
filename
picture_type // eg: painting, photograph
painter
]
Comment
[
user_id
filename
comment
]
I am trying to query all filenames that was not reviewed by current user.
following line returns all the filenames that was reviewed and commented in the given picture_type
session.query(Picture.filename).outerjoin(Comment).filter(
Picture.filename == Comment.filename,
Picture.picture_type == 'photograph'
).all()
, and the following line returns all the filenames in the in the given picture_type
session.query(Picture.filename).outerjoin(Comment).filter(
Picture.picture_type == 'photograph'
).all()
I was expecting following line would return filenames not reviewed in the in the given picture_type, but it returns an empty list
session.query(Picture.filename).outerjoin(Comment).filter(
Picture.filename != Comment.filename,
Picture.picture_type == 'photograph'
).all()
Am I doing something wrong here? What am I missing here?
i think your code have syntax error. are you missed a dot ? you wrote:
session.query(Picture.filename)outerjoin(Comment).filter(
Picture.filename == Comment.filename,
Picture.picture_type == 'photograph'
).all()
but it should be like this:
session.query(Picture.filename).outerjoin(Comment).filter(
Picture.filename == Comment.filename,
Picture.picture_type == 'photograph'
).all()
other sections also has this problem
I researched Exclusive Join and figured out the problem.
It should have been None == Comment.filename, not Picture.filename != Comment.filename.
I fixed the code like below code and it is working now.
session.query(Picture.filename).outerjoin(Comment).filter(
None == Comment.filename,
Picture.picture_type == 'photograph'
).all()
The problem was the resulting table of left outer join would not have any row that meet left_field != right_field condition. The resulting rows would meet conditions left_field == right_field or None == right_field since the missing value in the right table would be marked as null in the resulting table.
Below link gave me good learnings about this subject.
http://www.datasavantconsulting.com/roland/sql_excl.html
Thanks all for trying to help.
However, I will still be open to better suggestions!!!
I want to use outerjoin operation on a subquery and also include values from the subquery also.
My code
q_responses = session.query(Candidate, CandidateProfile)
.join(CandidateProfile, CandidateProfile.candidate_id == Candidate.id)
subq = (session.query(AppAction.candidate_id, Activity.archived)\
.join(Activity, and_(AppAction.candidate_id == Activity.candidate_id,
Activity.archived == 1)))\
.subquery("subq")
responses = q_responses.outerjoin(subq, Candidate.id == subq.c.candidate_id).all()
So I get the result in this format
(Candidate, CandidateProfile)
But I also want to include the archived value from subquery in the result.
By reading many relevant posts from the internet, I have tried
add_entity(subq.c.archived)
with_entities
add_column
select_from
But all those have resulted in some error.
Please help me out.
Please share your error for when you try add_column. The code below should work just fine (assuming that it does work without like which contains add_column):
responses = (
q_responses
.add_column(subq.c.archived) # #new
.outerjoin(subq, Candidate.id == subq.c.candidate_id)
).all()
Also you could have created a query straight away with this column included:
subq = (
session.query(AppAction.candidate_id, Activity.archived)
.join(Activity, and_(AppAction.candidate_id == Activity.candidate_id,
Activity.archived == 1))
).subquery("subq")
q_responses = (
session.query(Candidate, CandidateProfile, subq.c.archived)
.join(CandidateProfile, CandidateProfile.candidate_id == Candidate.id)
.outerjoin(subq, Candidate.id == subq.c.candidate_id)
).all()
Say I have a query object
query = session.query(SomeModel) \
.filter(SomeModel.foo == 'bar') \
.filter(SomeModel.active == True)
Can you modify the object list prior to executing the query, maintaining all the same filters
i.e.
.change_query(SomeModel.id, SomeModel.name)
so that the resulting query object is:
session.query(SomeModel.id, SomeModel.name) \
.filter(SomeModel.foo == 'bar') \
.filter(SomeModel.active == True)
Wow already found the answer. Will leave this open in case people are searching for this:
query = query.with_entities(SomeModel.id, SomeModel.name)