I have an error, when SQLAlchemy produced wrong SQL query, but I can't determine conditions.
I use Flask-SQLAlchemy and initially it's a just MyModel.query and it represented by simple SELECT with JOINs. But when .limit() method is applied, it transforms and uses subquery for fetch main objects and only then apply JOINs. The problem is in ORDER BY statement, which remains the same and ignores the subquery definition.
Here's example and I've simplify select fields:
-- Initially
SELECT *
FROM customer_rates
LEFT OUTER JOIN seasons AS seasons_1 ON seasons_1.id = customer_rates.season_id
LEFT OUTER JOIN users AS users_1 ON users_1.id = customer_rates.customer_id
-- other joins ...
ORDER BY customer_rates.id, customer_rates.id
-- Then .limit()
SELECT anon_1.*, *
FROM (
SELECT customer_rates.*
FROM customer_rates
LIMIT :param_1) AS anon_1
LEFT OUTER JOIN seasons AS seasons_1 ON seasons_1.id = anon_1.customer_rates_season_id
LEFT OUTER JOIN users AS users_1 ON users_1.id = anon_1.customer_rates_customer_id
-- other joins
ORDER BY customer_rates.id, customer_rates.id
And this query gives following error:
ProgrammingError: (psycopg2.ProgrammingError) missing FROM-clause entry for table "customer_rates"
The last line in query should be:
ORDER BY anon_1.customer_rates_id
The code, that produces this queries is a part of large application. I've tried to implement this from scratch in a small flask application, But I can't reproduce it. In small application it always uses a JOIN.
So I need to know, when SQLAlchemy decides to use subquery.
I use python 2.7 and PostgreSQL 9
The answer is pretty straightforward. It uses subquery when it joined table has many-to-one relations with queried model. So for producing correct number of results it limits the queried rows in the subquery
Related
I have this subquery:
aliasBilling = aliased(Billing)
subQueryReversesBilling = db.session.query(BillingReversal).select_from(BillingReversal).with_entities(BillingReversal.reverses_billing_id).filter(BillingReversal.billing_id == aliasBilling.id).subquery()
Which generates:
SELECT billing_reversal.reverses_billing_id
FROM billing_reversal, billing AS billing_1
WHERE billing_reversal.billing_id = billing_1.id
How to get rid of that join(comma style) I didn't specified as it makes a join for the full table and every record. Apparently it is assuming I need it as it is in the filtering part(where clause), but I take that table from the bigger query I have later on. That is why it is a subquery. What I need is this:
SELECT billing_reversal.reverses_billing_id
FROM billing_reversal
WHERE billing_reversal.billing_id = billing_1.id
The cross join is coming from the aliased Billing table in the filter clause.
If BillingReversal and Billing are linked with a relationship (and a foreign key), you should leverage this relationship and join instead of filter.
subQueryReversesBilling = (
db.session.query(BillingReversal)
.join(Billing, BillingReversal.billing)
.with_entities(BillingReversal.id)
.subquery()
)
I was following a tutorial to make my first Flask API (https://medium.com/#dushan14/create-a-web-application-with-python-flask-postgresql-and-deploy-on-heroku-243d548335cc) I did it but now I want to do queries more custom with SQLAlchemy and PostgreSQL. My question is how I could do something like this:
query = text("""SELECT enc.*, persona."Persona_Nombre", persona."Persona_Apellido", metodo."MetEnt_Nombre", metodo_e."MetPag_Descripcion"
FROM "Ventas"."Enc_Ventas" AS enc
INNER JOIN "General"."Persona" AS persona ON enc."PersonaId" = persona."PersonaId"
INNER JOIN "Ventas"."Metodo_Entrega" AS metodo ON enc."MetodoEntregaId" = metodo."MetodoEntregaId"
INNER JOIN "General"."Metodo_Pago" AS metodo_e ON enc."MetodoPagoId" = metodo_e."MetodoPagoId"
INNER JOIN "General"."Estatus" AS estado ON enc. """)
but with SQLAlchemy in order to use the models that I created previously. Thanks in advance for any answer!!
Edit:
The columns that I wish to see at the final result are: enc.*, persona."Persona_Nombre", persona."Persona_Apellido", metodo."MetEnt_Nombre", metodo_e."MetPag_Descripcion"
I really wish I could share more info but sadly I can't at the moment.
Doing this from the ORM layer, you would reference model names (I match the names of your query above, but I'm sure some of the model/table names are off - now adjusted slightly).
Now revised to include the specific columns you only want to see (note that I ignore your SQL aliases, ORM layer handles the actual query construction):
selection = session.query(Enc_Ventas, Persona.Persona_Nombre, Persona.Persona_Apellido, Metodo_Entrega.MetEnt_Nombre, Metodo_Pago.MetPag_Descripcion).\
join(Persona, Enc_Ventas.PersonaId == Persona.PersonaId).
join(Metodo_Entrega, Enc_Ventas.MetodoEntregaId == Metodo_Entrega.MetodoEntregaId).\
join(Metodo_Pago, Enc_Ventas.MetodoPagoId == Metodo_Pago.MetodoPagoId).\
join(Estatus).all()
Referencing the selection collection would be by iteration through the rows of tuples. A more robust and stable solution would be to transform each output row into a dict.
Otherwise, by including whole models, the collection of rows returned can be individually accessed by referencing as dot notation the model names in the query().
If you need further access to the columns in the related tables, use the ORM technique of .options(joinedload(myTable)), which in a single database query will bring in those additional columns, using the relationship name, also as dot notation.
You also need to define sqlalchemy relationships within your models for this to work, as well as defining the underlying SQL foreign keys.
Much more detail and/or a more specific question is needed to help further, imo.
I have been using the Django Rest Framework, and part of the ORM does the following query as part of a generic object list endpoint:
`SELECT COUNT(*) AS `__count`
FROM `album`
INNER JOIN `tracks`
ON (`album`.`id` = `tracks`.`album_id`)
WHERE `tracks`.`viewable` = 1`
The API is supposed to only display albums with tracks that are set to viewable, but with a tracks table containing 50 million rows this is query never seem to complete and hangs the endpoint's execution.
All columns referenced are indexed, so I do not know why this is taking so long to execute. If there are any potential optimisations that I might have not considered please let me know.
For this query:
SELECT COUNT(*) AS `__count`
FROM `album` INNER JOIN
`tracks`
ON (`album`.`id` = `tracks`.`album_id`)
WHERE `tracks`.`viewable` = 1`;
An index on tracks(viewable, album_id) and album(id) would help.
But, in all likelihood a join is not needed, so you can do:
select count(*)
from tracks
where viewable = 1;
For this the index on tracks(viewable) will be a big help.
I would like to know wether there is a way to combine joining two tables, and retrieving the two entities only with their relevant columns.
I want to avoid doing a thing such select * after joining two tables and getting only column id from Table A and column address from table B.
Working with python2.7
I heard of sqlalchemy feature called with_entities, but not sure how can it be mixed with this kind of join, lets work on this example from sqlalchemy tutorial:
u, a = session.query(User, Address).\
filter(User.id==Address.user_id).\
filter(Address.email_address=='jack#google.com').\
first():
Check out Query.join()
id, address = session.query(A.id, B.address).\
join(B, B.user_id == A.id).\
filter(A.email_address=='jack#google.com').\
first()
This can be done using a join or outerjoin depending on the use case, joins can be implicit or explicit. The .join second argument is the explicit join statement.
I have a two sqlalchemy query objects (q1 and q2) - they both belong to the same table and I want to be able to intersect the two queries. Since my database is MySql, q1.intersect(q2) throws sql syntax error. Is there a way to perform intersect MySql queries in sqlalchemy? My research pointed at using subqueries, aliases and left joins but all these solutions are native sql queries. I am looking for a sqlalchemy syntax.
Query:
q1 = Model1().query().filter(Model1.description.ilike(%aws%))
q2 = Model1().query().filter(Model1.tags.ilike(%cloud%))
I want to return q1.intersect(q2)
Also, what I have specified here as queries is just one of the cases of a broader set. I have a function which takes in an operator (and/or) and two operands(sql alchemy query objects, q1 and q2) which can be different and complex for different function calls. In this case, I cannot do a nested filter. I need to work with just q1 and q2.
For this simple case, you could just use two filters in the same query
results = db.query(Model1).filter(
Model1.description.ilike('%aws'),
Model1.tags.ilike('%cloud%')
)
This would return the same results as an intersect.
With two separate queries:
stmt = q2.subquery()
results = q1.outerjoin(stmt, Model1.id==stmt.c.id).filter(stmt.c.id != None)