I have this raw SQL statement that I would like to use for both Core and ORM:
SELECT * FROM `buffet` WHERE `roomId` = '864495034004835' AND `recordMasa` > '1514600000' AND `recordMasa` < '1514900000' AND `recordMasa` mod 10 = 0 LIMIT 0,10000000000;
Please do let me know how can I add to my existing code below to include the modulo function:
select_statement = select([Buffet]).where(and_(
Buffet.recordMasa > arguments['startDate'],
Buffet.recordMasa < arguments['endDate'],
Buffet.roomId == arguments['ident']
))
rooms = conn.execute(select_statement).fetchall()
what about the modulo operator from python?
select_statement = select([Buffet]).where(and_(
Buffet.recordMasa > arguments['startDate'],
Buffet.recordMasa < arguments['endDate'],
Buffet.roomId == arguments['ident'],
Buffet.recordMasa % 10 == 0,
))
Related
I have a task in which I must update a database on another server. As my options are limited I'm using python to do the update.
However I have this error:
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near ')'. (102) (SQLExecDirectW)")
My code is this:
first I create a select and then use it in the update
> query_dwt = "SELECT [cdcliente]\
> ,[nmcontato]\
> ,[cddepartamento]\
> ,[nmcargo] \
> ,[dsemail]\
> ,[cdlingua]\
> ,[nrcpfcnpj]\
> ,[cdcargo]\
> ,[cdcontatosuperior]\
> ,[idativo]\
> ,[cdcidade]\
> ,[dsendereco]\
> ,[dscomplemento]\
> ,[nmbairro]\
> ,[nrcep]\
> ,[nrcelular]\
> ,[dtnascimento]\
> ,[idbloqueado]\
> ,[cdlocalidade]\
> ,[nrmatricula]\
> ,[nmskin]\
> FROM [dw].[d_Qualitor_ad_contato_RH] WITH (NOLOCK)\
> WHERE cdcliente = 9402\
> AND (cdcontato = 38584 OR cdcontato = 22320 OR cdcontato = 37284);"
Second I use the select created to bring the information from the table to update the desired table
> query_qltr = """UPDATE ad\
> SET\
> ad.nmcontato = PR.nmcontato\
> ,ad.cddepartamento = PR.cddepartamento\
> ,ad.nmcargo = PR.nmcargo\
> ,ad.dsemail = PR.dsemail\
> ,ad.cdlingua = PR.cdlingua\
> ,ad.nrcpfcnpj = PR.nrcpfcnpj\
> ,ad.cdcargo = PR.cdcargo\
> ,ad.cdcontatosuperior = PR.cdcontatosuperior\
> ,ad.idativo = PR.idativo\
> ,ad.cdcidade = PR.cdcidade\
> ,ad.dsendereco = PR.dsendereco\
> ,ad.dscomplemento = PR.dscomplemento\
> ,ad.nmbairro = PR.nmbairro\
> ,ad.nrcep = PR.nrcep\
> ,ad.nrcelular = PR.nrcelular\
> ,ad.dtnascimento = PR.dtnascimento\
> ,ad.idbloqueado = PR.idbloqueado\
> ,ad.cdlocalidade = PR.cdlocalidade\
> ,ad.nrmatricula = PR.nrmatricula\
> ,ad.nmskin = PR.nmskin\
> FROM dbo.ad_contato ad\
> INNER JOIN ({}) PR\
> ON ad.cdcontato = PR.cdcontato\
> AND ad.cdcliente LIKE '9402';""".format(OpenSqlDatabaseConnection.execute_query(query_dwt,'target-db-conn-str'))
>
> OpenSqlDatabaseConnection.execute_query(query_qltr,'rdn-db-clt-sql-06a-inssql01-qualitor-prd-jdbc-conn-string-01')
I'm sure it's something simple but I can't figure it out.
Solution:
1-Extract the data in the first database and insert it into a dataframe.
def select_dw_qualitor_ad_contato():
query_dwt = "SELECT [cdcliente]\
,[cdcontato]\
,[nmcontato]\
,[cddepartamento]\
,[nmcargo] \
,[dsemail]\
,[cdlingua]\
,[nrcpfcnpj]\
,[cdcargo]\
,[cdcontatosuperior]\
,[idativo]\
,[cdcidade]\
,[dsendereco]\
,[dscomplemento]\
,[nmbairro]\
,[nrcep]\
,[nrcelular]\
,[dtnascimento]\
,[idbloqueado]\
,[cdlocalidade]\
,[nrmatricula]\
,[nmskin]\
FROM [dw].[d_Qualitor_ad_contato_RH] WITH (NOLOCK)\
WHERE cdcliente = 9402\
AND (cdcontato = 38584\
OR cdcontato = 22320\
OR cdcontato = 37284\
OR cdcontato = 36139\
OR cdcontato = 41035\
OR cdcontato = 38819);"
return pd.read_sql(query_dwt,OpenSqlDatabaseConnection.connection('target-db-conn-str'),
parse_dates={"date_column": {"errors": "ignore"}})
2- Update second query row by row with dataframe
def update_qualitor_table():
dfdw = QueriesLists.select_dw_qualitor_ad_contato()
end = len(dfdw)
for i,line in enumerate(dfdw):
if i < end:
df = dfdw.iloc[i]
QueriesLists.update_database(df)
else:
break
3- Query SQL using update command using dataframe
def update_database(df):
query_qltr = "UPDATE [dbo].[ad_contato]\
SET [nmcontato] = CAST('{0}' AS VARCHAR(200))\
,[cddepartamento] = CAST('{1}' AS INT)\
,[nmcargo] = CAST('{2}' AS VARCHAR (50))\
,[dsemail] = CAST('{3}' AS VARCHAR(200))\
,[cdlingua] = CAST('{4}' AS INT)\
,[nrcpfcnpj] = CAST('{5}' AS VARCHAR(20))\
,[cdcargo] = CAST('{6}' AS INT)\
,[cdcontatosuperior] = CAST('{7}' AS INT)\
,[idativo] = CAST('{8}' AS VARCHAR(1))\
,[dsendereco] = CAST('{9}' AS VARCHAR(200))\
,[dscomplemento] = CAST('{10}' AS VARCHAR(200))\
,[nmbairro] = CAST('{11}' AS VARCHAR(40))\
,[nrcep] = CAST('{12}' AS VARCHAR(9))\
,[dtnascimento] = CAST('{13}' AS DATETIME) \
,[idbloqueado] = CAST('{14}' AS VARCHAR(1))\
,[cdlocalidade] = CAST('{15}' AS INT)\
,[nrmatricula] = CAST('{16}' AS VARCHAR(20))\
WHERE [cdcontato] = CAST('{17}' AS INT)\
AND [cdcliente] = 9402;\
".format(str(df[2]) #nmcontato
,int(df[3]) #cddepartamento
,str(df[4]) #nmcargo
,str(df[5]) #dsemai
,int(df[6]) #cdlingua
,str(df[7]) #nrcpfcnpj
,int(df[8]) #cdcargo
,int(df[9]) #cdcontasuperior
,str(df[10]) #idativo
,str(df[12]) #dsendereco
,str(df[13]) #dscomplemento
,str(df[14]) #nmbairro
,str(df[15]) #nrcep
,pd.to_datetime(df[17]) #datetime
,str(df[18]) #idbloqueado
,int(df[19]) #cdlocalidade
,str(df[20]) #nrmatricula
,int(df[1])
)
OpenSqlDatabaseConnection.execute_query(query_qltr,'rdn-db-clt-sql-06a-inssql01-qualitor-prd-jdbc-conn-string-01')
As above, I'm trying to use pd.read_sql to query our mysql database, and getting an error for double/single quotes.
When I remove the % operators from the LIKE clause (lines 84-87) the query runs, but these are needed. I know I need to format the strings but I don't know how within such a big query.
Here's the query:
SELECT
s.offer_id,
s.cap_id,
vi.make,
vi.model,
vi.derivative,
i.vehicle_orders,
s.lowest_offer,
CASE
WHEN f.previous_avg = f.previous_low THEN "n/a"
ELSE FORMAT(f.previous_avg, 2)
END as previous_avg,
f.previous_low,
CASE
WHEN ( ( (s.lowest_offer - f.previous_avg) / f.previous_avg) * 100) = ( ( (s.lowest_offer - f.previous_low) / f.previous_low) * 100) THEN "n/a"
ELSE CONCAT(FORMAT( ( ( (s.lowest_offer - f.previous_avg) / f.previous_avg) * 100), 2), "%")
END as diff_avg,
CONCAT(FORMAT( ( ( (s.lowest_offer - f.previous_low) / f.previous_low) * 100), 2), "%") as diff_low,
s.broker,
CASE
WHEN s.in_stock = '1' THEN "In Stock"
ELSE "Factory Order"
END as in_stock,
CASE
WHEN s.special IS NOT NULL THEN "Already in Specials"
ELSE "n/a"
END as special
FROM
( SELECT o.id as offer_id,
o.cap_id as cap_id,
MIN(o.monthly_payment) as lowest_offer,
b.name as broker,
o.stock as in_stock,
so.id as special
FROM
offers o
INNER JOIN brands b ON ( o.brand_id = b.id )
LEFT JOIN special_offers so ON ( so.cap_id = o.cap_id )
WHERE
( o.date_modified >= DATE_ADD(NOW(), INTERVAL -1 DAY) OR o.date_created >= DATE_ADD(NOW(), INTERVAL -1 DAY) )
AND o.deposit_value = 9
AND o.term = 48
AND o.annual_mileage = 8000
AND o.finance_type = 'P'
AND o.monthly_payment > 100
GROUP BY
o.cap_id
ORDER BY
special DESC) s
INNER JOIN
( SELECT o.cap_id as cap_id,
AVG(o.monthly_payment) as previous_avg,
MIN(o.monthly_payment) as previous_low
FROM
offers o
WHERE
o.date_modified < DATE_ADD(NOW(), INTERVAL -1 DAY)
AND o.date_modified >= DATE_ADD(NOW(), INTERVAL -1 WEEK)
AND o.deposit_value = 9
AND o.term = 48
AND o.annual_mileage = 8000
AND o.finance_type = 'P'
AND o.monthly_payment > 100
GROUP BY
o.cap_id ) f ON ( s.cap_id = f.cap_id )
LEFT JOIN
( SELECT a.cap_id as cap_id,
v.manufacturer as make,
v.model as model,
v.derivative as derivative,
COUNT(*) as vehicle_orders
FROM
( SELECT o.id,
o.name as name,
o.email as email,
o.date_created as date,
SUBSTRING_INDEX(SUBSTRING(offer_serialized, LOCATE("capId", offer_serialized) +12, 10), '"', 1) as cap_id
FROM moneyshake.orders o
WHERE o.name NOT LIKE 'test%'
AND o.email NOT LIKE 'jawor%'
AND o.email NOT LIKE 'test%'
AND o.email NOT LIKE '%moneyshake%'
AND o.phone IS NOT NULL
AND o.date_created > DATE_ADD(NOW(), INTERVAL -1 MONTH)
) a JOIN moneyshake.vehicles_view v ON a.cap_id = v.id
GROUP BY
v.manufacturer,
v.model,
v.derivative,
a.cap_id) i ON ( f.cap_id = i.cap_id )
INNER JOIN
( SELECT v.id as id,
v.manufacturer as make,
v.model as model,
v.derivative as derivative
FROM moneyshake.vehicles_view v
GROUP BY v.id ) vi ON s.cap_id = vi.id
WHERE
( ( s.lowest_offer - f.previous_low ) / f.previous_low) * 100 <= -15
GROUP BY
s.cap_id
Thanks!
That error occurs then the DBAPI layer (e.g., mysqlclient) natively uses the "format" paramstyle and the percent sign (%) is misinterpreted as a format character instead of a LIKE wildcard.
The fix is to wrap the SQL statement in a SQLAlchemy text() object. For example, this will fail:
import pandas as pd
import sqlalchemy as sa
engine = sa.create_engine("mysql+mysqldb://scott:tiger#localhost:3307/mydb")
sql = """\
SELECT * FROM million_rows
WHERE varchar_col LIKE 'record00000%'
ORDER BY id
"""
df = pd.read_sql_query(sql, engine)
but simply changing the read_sql_query() call to
df = pd.read_sql_query(sa.text(sql), engine)
will work.
I have the next PostgreSQL query:
WITH sub_query AS (
SELECT vi.idvalfield,
vi.value::text
FROM valueint_value AS vi
UNION
SELECT vt.idvalfield,
vt.value::text
FROM valuetext_value AS vt
)
SELECT
sq.idvalfield,
sq.value
FROM sub_query AS sq
JOIN valuefield AS vf ON vf.idvalfield = sq.idvalfield
JOIN event e on vf.idevent = e.idevent
WHERE NOT (e.idevent || array[]::uuid[]) && (SELECT array_agg(e.idevent) AS id_event
FROM sub_query AS sq
JOIN valuefield AS vf ON vf.idvalfield = sq.idvalfield
JOIN event e on vf.idevent = e.idevent
WHERE (idtable = 41 AND sq.value = 222)
OR (idtable = 43 AND sq.value = 18)
);
I described the construction of WITH. The number of tables in UNION is dynamic:
from sqlalchemy.dialects import postgresql
from sqlalchemy import or_, and_
from sqlalchemy import cast, Table, Text
from sqlalchemy.dialects.postgresql import array_agg, array, ARRAY, UUID
models_view = [
session.query(
model.c.idvalfield.label('id_value'),
cast(model.c.value, Text).label('value')
).filter(model.c.idvalfield.in_(id_fields))
for model, id_fields in model_values.items()
]
cte_union_view = models_view[0].union_all(*models_view[1:]).cte()
Described a subquery in WHERE:
filtered_event = session.query(array_agg(Event.idevent))\
.select_from(cte_union_view)\
.join(Valuefield, cte_union_view.c.id_value == Valuefield.idvalfield)\
.join(Event, Event.idevent == Valuefield.idevent)\
.filter(or_(and_(Valuefield.idtable == 41, cte_union_view.c.value == '222'),
and_(Valuefield.idtable == 43, cte_union_view.c.value == '18'))).subquery()
Described the main request:
event_all = session.query(cte_union_view)\
.join(Valuefield, cte_union_view.c.id_value == Valuefield.idvalfield)\
.join(Event, Event.idevent == Valuefield.idevent).all()
But I can't describe the selection condition in any way:
WHERE NOT (e.idevent || array[]::uuid[]) && (SELECT array_agg(e.idevent)
I try:
filter(cast([Event.idevent + array([])], ARRAY(UUID)).in_(filtered_event))
and
filter(cast([array([Event.idevent]) + array([])], ARRAY(UUID)).in_(filtered_event))
But I don't know how to describe it
Found a solution. Maybe it will be useful to someone.
Converting a UUID to an array of UUID's:
(e.idevent || array[]::uuid[])
you can do this using literal_column:
literal_column('ARRAY[]::uuid[]').op('||')(Event.idevent)
And now the whole `WHERE ' block can be described like this:
.filter(uuid_event_arr.notin_(filtered_event)).all()
But, in fact, it was easier to rewrite the subquery in WHERE, without the 'array_agg ()' function. This in turn makes it easier to build a query using alchemy:
filtered_event = session.query(Event.idevent)\
.select_from(cte_union_view)\
.join(Valuefield, cte_union_view.c.id_value == Valuefield.idvalfield)\
.join(Event, Event.idevent == Valuefield.idevent)\
.filter(and_(Valuefield.idtable == 41, cte_union_view.c.v_text == '222'))
views_value = session.query(cte_union_view)\
.join(Valuefield, cte_union_view.c.id_value == Valuefield.idvalfield)\
.join(Event, Event.idevent == Valuefield.idevent)\
.filter(Event.idevent.in_(filtered_event)).all()
You can use also: array(tuple_(Event.idevent), type_=UUID)
I have a condition in which I have joined a table with itself. For getting result rows I need to have two separate attributes in the result rows each of which contain model items.
order_sq1: Order = self._query_from(Order). \
filter(Order.status == OrderStatus.REGISTERED.value,
Order.dispatch_date + Order.dispatch_time > current_time,
Order.who_added_role != UserRole.SHIPPER.value).subquery('Order1')
order_sq2: Order = self._query_from(Order). \
filter(Order.status == OrderStatus.REGISTERED.value,
Order.dispatch_date + Order.dispatch_time > current_time,
Order.who_added_role != UserRole.SHIPPER.value).subquery('Order2')
query = self._query_from(order_sq1).join(
order_sq2,
and_(order_sq1.c.tracking_code != order_sq2.c.tracking_code,
func.ST_DistanceSphere(order_sq1.c.destination_location,
order_sq2.c.source_location) < 0.15 *
order_sq1.c.distance
, func.ST_DistanceSphere(order_sq2.c.destination_location,
order_sq1.c.source_location) < 0.15 *
order_sq1.c.distance
, between(((order_sq1.c.dispatch_date + order_sq1.c.dispatch_time) -
(order_sq2.c.dispatch_date + order_sq2.c.dispatch_time) -
(order_sq1.c.distance / 60) * timedelta(hours=1)),
timedelta(hours=2),
timedelta(hours=20))
, order_sq1.c.source_region_id != order_sq1.c.destination_region_id
, order_sq2.c.source_region_id != order_sq2.c.destination_region_id
, order_sq1.c.vehicle_type == order_sq2.c.vehicle_type
)).add_columns(order_sq2)
I need the result in the form of (Order1, Order2) fields, each as Order class.
I tried many ways with no success!
I need filter using not in but in two fields.
q = db_session.query(Necessidade, WFLeilao, BidHeader, BidItemPrice, func.sbprecobruto(BidItemPrice.bid_number,BidItemPrice.line_number, Necessidade.identportal, type_=Float))
q = q.join(WFLeilao, and_(Necessidade.numeroportal == WFLeilao.leilao, Necessidade.numeroitemportal == WFLeilao.itemleilao))
q = q.join(BidHeader, and_(BidHeader.bid_number == BidItemPrice.bid_number))
q = q.join(BidItemPrice, and_(BidItemPrice.auction_header_id == WFLeilao.leilao, BidItemPrice.auction_line_number == WFLeilao.itemleilao, BidItemPrice.bid_number == WFLeilao.lance, BidItemPrice.line_number == WFLeilao.itemlance))
subquery = db_session.query(ItfRetornoPedido.num_leilao_superbuy, ItfRetornoPedido.num_item_leilao_superbuy).filter_by(status_comprador=1).filter_by(acao='I').filter_by(empresa='NK').subquery()
q = q.filter(~(WFLeilao.leilao,Wfleilao.itemleilao).in_(subquery))
In oracle is possible, a similar example:
Select *
from table_a
where (leilao, itemleilao) not in
(Select num_leilao_superbuy, num_item_leilao_superbuy
from table_b
where empresa = 'NK')
Is it possible?
I found a solution using tuple_
q = q.filter(~tuple_(WFLeilao.leilao, WFLeilao.itemleilao).in_(subquery))
you can chain the query:
q = q.filter(~(WFLeilao.leilao.in_(subquery))) \
.filter(~(Wfleilao.itemleilao.in_(subquery)))