I have the next PostgreSQL query:
WITH sub_query AS (
SELECT vi.idvalfield,
vi.value::text
FROM valueint_value AS vi
UNION
SELECT vt.idvalfield,
vt.value::text
FROM valuetext_value AS vt
)
SELECT
sq.idvalfield,
sq.value
FROM sub_query AS sq
JOIN valuefield AS vf ON vf.idvalfield = sq.idvalfield
JOIN event e on vf.idevent = e.idevent
WHERE NOT (e.idevent || array[]::uuid[]) && (SELECT array_agg(e.idevent) AS id_event
FROM sub_query AS sq
JOIN valuefield AS vf ON vf.idvalfield = sq.idvalfield
JOIN event e on vf.idevent = e.idevent
WHERE (idtable = 41 AND sq.value = 222)
OR (idtable = 43 AND sq.value = 18)
);
I described the construction of WITH. The number of tables in UNION is dynamic:
from sqlalchemy.dialects import postgresql
from sqlalchemy import or_, and_
from sqlalchemy import cast, Table, Text
from sqlalchemy.dialects.postgresql import array_agg, array, ARRAY, UUID
models_view = [
session.query(
model.c.idvalfield.label('id_value'),
cast(model.c.value, Text).label('value')
).filter(model.c.idvalfield.in_(id_fields))
for model, id_fields in model_values.items()
]
cte_union_view = models_view[0].union_all(*models_view[1:]).cte()
Described a subquery in WHERE:
filtered_event = session.query(array_agg(Event.idevent))\
.select_from(cte_union_view)\
.join(Valuefield, cte_union_view.c.id_value == Valuefield.idvalfield)\
.join(Event, Event.idevent == Valuefield.idevent)\
.filter(or_(and_(Valuefield.idtable == 41, cte_union_view.c.value == '222'),
and_(Valuefield.idtable == 43, cte_union_view.c.value == '18'))).subquery()
Described the main request:
event_all = session.query(cte_union_view)\
.join(Valuefield, cte_union_view.c.id_value == Valuefield.idvalfield)\
.join(Event, Event.idevent == Valuefield.idevent).all()
But I can't describe the selection condition in any way:
WHERE NOT (e.idevent || array[]::uuid[]) && (SELECT array_agg(e.idevent)
I try:
filter(cast([Event.idevent + array([])], ARRAY(UUID)).in_(filtered_event))
and
filter(cast([array([Event.idevent]) + array([])], ARRAY(UUID)).in_(filtered_event))
But I don't know how to describe it
Found a solution. Maybe it will be useful to someone.
Converting a UUID to an array of UUID's:
(e.idevent || array[]::uuid[])
you can do this using literal_column:
literal_column('ARRAY[]::uuid[]').op('||')(Event.idevent)
And now the whole `WHERE ' block can be described like this:
.filter(uuid_event_arr.notin_(filtered_event)).all()
But, in fact, it was easier to rewrite the subquery in WHERE, without the 'array_agg ()' function. This in turn makes it easier to build a query using alchemy:
filtered_event = session.query(Event.idevent)\
.select_from(cte_union_view)\
.join(Valuefield, cte_union_view.c.id_value == Valuefield.idvalfield)\
.join(Event, Event.idevent == Valuefield.idevent)\
.filter(and_(Valuefield.idtable == 41, cte_union_view.c.v_text == '222'))
views_value = session.query(cte_union_view)\
.join(Valuefield, cte_union_view.c.id_value == Valuefield.idvalfield)\
.join(Event, Event.idevent == Valuefield.idevent)\
.filter(Event.idevent.in_(filtered_event)).all()
You can use also: array(tuple_(Event.idevent), type_=UUID)
Related
I am trying to use python to use a parametrized query through a list. This is the following code:
loan_records =['604150062','604150063','604150064','604150065','604150066','604150067','604150069','604150070']
borr_query = "select distinct a.nbr_aus, cast(a.nbr_trans_aus as varchar(50)) nbr_trans_aus, c.amt_finl_item, case when a.cd_idx in (-9999, 0) then null else a.cd_idx end as cd_idx, a.rate_curr_int, case when a.rate_gr_mrtg_mrgn = 0 then null else a.rate_gr_mrtg_mrgn end as rate_gr_mrtg_mrgn, a.rate_loln_max_cap, case when a.rate_perdc_cap = 0 then null else a.rate_perdc_cap end as rate_perdc_cap from db2mant.i_lp_trans a left join db2mant.i_lp_trans_borr b on a.nbr_aus = b.nbr_aus and a.nbr_trans_aus = b.nbr_trans_aus left join db2mant.i_lp_finl_item c on a.nbr_aus = c.nbr_aus and a.nbr_trans_aus = c.nbr_trans_aus where a.nbr_trans_aus in (?) and c.cd_finl_item = 189"
ODS.execute(borr_query, loan_records)
#PML.execute(PML_SUBMN_Query, (first_evnt, last_evnt, x))
ODS_records = ODS.fetchall()
ODS_records = pd.DataFrame(ODS_records, columns=['nbr_aus', 'nbr_trans_aus', 'amt_finl_item', 'cd_idx', 'rate_curr_int', 'rate_gr_mrtg_mrgn', 'rate_loln_max_cap', 'rate_perdc_cap'])
When I try to run this code: this is the following error message:
error message
As above, I'm trying to use pd.read_sql to query our mysql database, and getting an error for double/single quotes.
When I remove the % operators from the LIKE clause (lines 84-87) the query runs, but these are needed. I know I need to format the strings but I don't know how within such a big query.
Here's the query:
SELECT
s.offer_id,
s.cap_id,
vi.make,
vi.model,
vi.derivative,
i.vehicle_orders,
s.lowest_offer,
CASE
WHEN f.previous_avg = f.previous_low THEN "n/a"
ELSE FORMAT(f.previous_avg, 2)
END as previous_avg,
f.previous_low,
CASE
WHEN ( ( (s.lowest_offer - f.previous_avg) / f.previous_avg) * 100) = ( ( (s.lowest_offer - f.previous_low) / f.previous_low) * 100) THEN "n/a"
ELSE CONCAT(FORMAT( ( ( (s.lowest_offer - f.previous_avg) / f.previous_avg) * 100), 2), "%")
END as diff_avg,
CONCAT(FORMAT( ( ( (s.lowest_offer - f.previous_low) / f.previous_low) * 100), 2), "%") as diff_low,
s.broker,
CASE
WHEN s.in_stock = '1' THEN "In Stock"
ELSE "Factory Order"
END as in_stock,
CASE
WHEN s.special IS NOT NULL THEN "Already in Specials"
ELSE "n/a"
END as special
FROM
( SELECT o.id as offer_id,
o.cap_id as cap_id,
MIN(o.monthly_payment) as lowest_offer,
b.name as broker,
o.stock as in_stock,
so.id as special
FROM
offers o
INNER JOIN brands b ON ( o.brand_id = b.id )
LEFT JOIN special_offers so ON ( so.cap_id = o.cap_id )
WHERE
( o.date_modified >= DATE_ADD(NOW(), INTERVAL -1 DAY) OR o.date_created >= DATE_ADD(NOW(), INTERVAL -1 DAY) )
AND o.deposit_value = 9
AND o.term = 48
AND o.annual_mileage = 8000
AND o.finance_type = 'P'
AND o.monthly_payment > 100
GROUP BY
o.cap_id
ORDER BY
special DESC) s
INNER JOIN
( SELECT o.cap_id as cap_id,
AVG(o.monthly_payment) as previous_avg,
MIN(o.monthly_payment) as previous_low
FROM
offers o
WHERE
o.date_modified < DATE_ADD(NOW(), INTERVAL -1 DAY)
AND o.date_modified >= DATE_ADD(NOW(), INTERVAL -1 WEEK)
AND o.deposit_value = 9
AND o.term = 48
AND o.annual_mileage = 8000
AND o.finance_type = 'P'
AND o.monthly_payment > 100
GROUP BY
o.cap_id ) f ON ( s.cap_id = f.cap_id )
LEFT JOIN
( SELECT a.cap_id as cap_id,
v.manufacturer as make,
v.model as model,
v.derivative as derivative,
COUNT(*) as vehicle_orders
FROM
( SELECT o.id,
o.name as name,
o.email as email,
o.date_created as date,
SUBSTRING_INDEX(SUBSTRING(offer_serialized, LOCATE("capId", offer_serialized) +12, 10), '"', 1) as cap_id
FROM moneyshake.orders o
WHERE o.name NOT LIKE 'test%'
AND o.email NOT LIKE 'jawor%'
AND o.email NOT LIKE 'test%'
AND o.email NOT LIKE '%moneyshake%'
AND o.phone IS NOT NULL
AND o.date_created > DATE_ADD(NOW(), INTERVAL -1 MONTH)
) a JOIN moneyshake.vehicles_view v ON a.cap_id = v.id
GROUP BY
v.manufacturer,
v.model,
v.derivative,
a.cap_id) i ON ( f.cap_id = i.cap_id )
INNER JOIN
( SELECT v.id as id,
v.manufacturer as make,
v.model as model,
v.derivative as derivative
FROM moneyshake.vehicles_view v
GROUP BY v.id ) vi ON s.cap_id = vi.id
WHERE
( ( s.lowest_offer - f.previous_low ) / f.previous_low) * 100 <= -15
GROUP BY
s.cap_id
Thanks!
That error occurs then the DBAPI layer (e.g., mysqlclient) natively uses the "format" paramstyle and the percent sign (%) is misinterpreted as a format character instead of a LIKE wildcard.
The fix is to wrap the SQL statement in a SQLAlchemy text() object. For example, this will fail:
import pandas as pd
import sqlalchemy as sa
engine = sa.create_engine("mysql+mysqldb://scott:tiger#localhost:3307/mydb")
sql = """\
SELECT * FROM million_rows
WHERE varchar_col LIKE 'record00000%'
ORDER BY id
"""
df = pd.read_sql_query(sql, engine)
but simply changing the read_sql_query() call to
df = pd.read_sql_query(sa.text(sql), engine)
will work.
I'm having a problem with SqlAlchemy and a group_by clause. See the SqlAlchemy query below.
I've got a SqlAlchemy query that includes a group_by clause and it's raising an exception, '(cx_Oracle.DatabaseError) ORA-00979: not a GROUP BY expression'. However, when I get the SQL generated by the SqlAlachemy query, and run that manually, the query works fine.
I'm not sure how to figure out what's wrong with the group_by clause. How can I debug this problem and figure out what I can do to fix it?
# create shorthand aliases
b = db.aliased(Batch)
bs = db.aliased(BatchingStatus)
bp = db.aliased(BatchPress)
bst = db.aliased(BatchState)
bit = db.aliased(BatchItem)
bin = db.aliased(BatchInput)
bpri = db.aliased(BatchPriority)
lcu = db.aliased(LCUser)
s = db.aliased(SubBatch)
w = db.aliased(WorkType)
ptw = db.aliased(LCProductToWorkType)
ctp = db.aliased(LCCategoryToProduct)
c = db.aliased(LCCategory)
# for correlated subquery
subq = (
db.session.query(ctp.product_name)
.join(c, c.category_id == ctp.category_id)
.filter(func.lower(c.category_path) == category)
.filter(ctp.active == 1)
)
# start of problem query
q = db.session.query(
b.batch_signature.label('batch_signature'),
b.batch_num,
b.created_date.label('created_date'),
bst.code.label('batch_state'),
func.min(bin.promise_date).label('due_out'),
bs.job_status,
bp.press_id.label('press_id'),
bp.description.label('press_description'),
bp.code.label('press_code'),
bp.active.label('press_active'),
func.listagg(bin.item_id, ',').within_group(bin.item_id).label('subbatches'),
bs.item_count.label('item_count'),
bs.product.label('product'),
bpri.code.label('priority'),
ptw.display_format.label('product_display_format'),
c.display_name.label('category_display_name'),
lcu.coalesce_first_name,
lcu.coalesce_last_name,
lcu.coalesce_email,
) \
.join(bs, (bs.batch_signature == b.batch_signature) & (bs.press_id == b.press_id)) \
.join(bp, bp.press_id == b.press_id) \
.join(bst, bst.state_id == b.state_id) \
.join(bit, bit.batch_id == b.batch_id) \
.join(bin, bin.batch_input_id == bit.batch_input_id) \
.join(bpri, bpri.priority_id == bin.priority_id) \
.join(lcu, lcu.username == bs.actor) \
.join(s, s.subbatchno == func.to_char(bin.item_id)) \
.join(w, w.worktypeenum == s.worktypeenum) \
.join(ptw, ptw.worktypeenum == w.worktypeenum) \
.join(ctp, ctp.category_to_product_id == ptw.category_to_product_id) \
.join(c, c.category_id == ctp.category_id) \
.filter(bs.product.in_(subq)) \
.filter(b.state_id <= 200) \
.group_by(
b.batch_signature,
b.batch_num,
b.created_date,
bst.code,
bs.job_status,
bp.press_id,
bp.description,
bp.code,
bp.active,
bs.item_count,
bs.product,
bpri.code,
ptw.display_format,
c.display_name,
lcu.coalesce_first_name,
lcu.coalesce_last_name,
lcu.coalesce_email,
) \
.order_by('batch_signature', 'batch_num', 'created_date')
try:
retval = q.all()
except Exception as e:
print e
The above doesn't show the models, some of which have #hybrid_property/#.expression methods, like the lcu.coalesce_first_name columns, which are an attempt to hid the #func.coalesce code that I thought was causing the group_by problems.
For a simple select, pagination works as implemented here:
mheader_dict = dict(request.headers)
no_of_pgs = 0
if 'Maxpage' in mheader_dict.keys():
max_per_pg = int(mheader_dict['Maxpage'])
else:
max_per_pg = 100
page_no = int(request.headers.get('Pageno', type=int, default=1))
offset1 = (page_no - 1) * max_per_pg
s = select[orders]
if s is not None:
s = s.limit(max_per_pg).offset(offset1)
rs = g.conn.execute(s)
Conn is the connection object above
When text is used in the select statement, How to specify the limit?.How to rectify in below:
s1 = text('select d.*, (select array(select localities.name from localities, localities_boys where localities.id = localities_boys.locality_id and localities_boys.boy_id = d.id and localities_boys.boy_id is not null )) from delivery_boys d order by d.id;')
page_no = int(request.headers.get('Pageno', type=int, default=1))
offset1 = (page_no - 1) * max_per_pg
s1 = s1.limit(max_per_pg).offset(offset1)
rs1 = g.conn.execute(s1)
If s1 = s1.compile(engine) is used, it returns sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2 object which doesn't have limit functionality
How to convert sqlalchemy.sql.elements.TextClause to sqlalchemy.sql.selectable.Select using sqlalchemy core 1.0.8 to solve the above?
using sqlalchemy core v. 1.0.8, python 2.7,flask 0.12
Converted the TextClause to Select as :
s1 = select([text('d.*, (select array(select localities.name from localities, localities_boys where localities.id = localities_boys.locality_id and localities_boys.boy_id = d.id and localities_boys.boy_id is not null)) from delivery_boys d')])
Hence able to use limit and offset on the generated Select object
I need filter using not in but in two fields.
q = db_session.query(Necessidade, WFLeilao, BidHeader, BidItemPrice, func.sbprecobruto(BidItemPrice.bid_number,BidItemPrice.line_number, Necessidade.identportal, type_=Float))
q = q.join(WFLeilao, and_(Necessidade.numeroportal == WFLeilao.leilao, Necessidade.numeroitemportal == WFLeilao.itemleilao))
q = q.join(BidHeader, and_(BidHeader.bid_number == BidItemPrice.bid_number))
q = q.join(BidItemPrice, and_(BidItemPrice.auction_header_id == WFLeilao.leilao, BidItemPrice.auction_line_number == WFLeilao.itemleilao, BidItemPrice.bid_number == WFLeilao.lance, BidItemPrice.line_number == WFLeilao.itemlance))
subquery = db_session.query(ItfRetornoPedido.num_leilao_superbuy, ItfRetornoPedido.num_item_leilao_superbuy).filter_by(status_comprador=1).filter_by(acao='I').filter_by(empresa='NK').subquery()
q = q.filter(~(WFLeilao.leilao,Wfleilao.itemleilao).in_(subquery))
In oracle is possible, a similar example:
Select *
from table_a
where (leilao, itemleilao) not in
(Select num_leilao_superbuy, num_item_leilao_superbuy
from table_b
where empresa = 'NK')
Is it possible?
I found a solution using tuple_
q = q.filter(~tuple_(WFLeilao.leilao, WFLeilao.itemleilao).in_(subquery))
you can chain the query:
q = q.filter(~(WFLeilao.leilao.in_(subquery))) \
.filter(~(Wfleilao.itemleilao.in_(subquery)))