I need to write a complex query, which retrieves a lot of data from a bunch of tables. Basically I need to find all instances of the models
Customer
Payment
Invoice
where relationships intersect in a specific way. In SqlAlchemy, I would be able to do something like
for c, p, i in session.query(Customer, Payment, Invoice).\
filter(User.id==Payment.customer_id).\
filter(Invoice.id==Payment.invoice_id).\
filter(Payment.date==...).\
filter(Customer.some_property==...)
all():
# Do stuff ...
This would allow me to set several constraints and retrieve it all at once. In Django, I currently do something stupid like
customers = Customer.objects.filter(...)
payments = Payment.objects.filter(customer=customer)
invoices = Invoice.objects.filter(customer=customer, payment_set=payments)
Now, we already have three different queries (some details are left out to keep it simple). Could I reduce it to one? Well, I could have done something like
customers = Customer.objects.filter(...).prefetch_related(
'payments', 'payments__invoices'
)
but now I have to traverse a crazy tree of data instead of having it all laid out neatly in rows, like with SqlAlchemy. Is there any way Django can do something like that? Or would I have to drop through to custom SQL directly?
After reading up on different solutions, I have decided to use SqlAlchemy on top of my Django models. Some people try to completely replace the Django ORM with SqlAlchemy, but this almost completely defeats the purpose of using Django, since most of the framework relies on the ORM.
Instead, I use SqlAlchemy simple for querying the tables defined by the Django ORM. I follow a recipe similar to this
# Setup sqlalchemy bindings
import sqlalchemy as s
from sqlalchemy.orm import sessionmaker
engine = s.create_engine('postgresql://<user>:<password>#<host>:<port>/<db_name>')
# Automatically read the database tables and create metadata
meta = s.MetaData()
meta.reflect(bind=engine)
Session = sessionmaker(bind=engine)
# Create a session, which can query the tables
session = Session()
# Build table instances without hardcoding tablenames
s_payment = meta.tables[models.Payment()._meta.db_table]
s_allocation = meta.tables[models.Allocation()._meta.db_table]
s_customer = meta.tables[models.Customer()._meta.db_table]
s_invoice = meta.tables[models.Invoice()._meta.db_table]
report = session.query(s_payment.c.amount, ...).all()
There is room for a few improvements on this recipe, e.g. it is not very elegant to create an empty instance of Django models in order to find their table name, however, with a few lines of code, I get the full flexibility of SqlAlchemy without compromising with the Django ORM layer. This means both can live happily alongside each other.
One caveat is that SqlAlchemy will not use the same connection as the Django ORM, which means that the view of things may not appear consistent if I use both approaches in the same context. This won't be a problem for me though, since I just want to read a bunch of data from the database.
Related
Our current project relies heavily on SQL Alchemy for table creation/data insertion. We would like to switch to timescaledb's hypertables, but it seems the recommended way to create hypertables is by executing a
create_hypertable
command. I need to be able to dynamically create tables, and so manually doing this for every table created is not really an option. One way of handling the conversion is to run a python script sending psycopg2 commands to convert all newly-created tables into hypertables, but this seems a little clumsy. Does timescaledb offer any integration with SQL Alchemy with regards to creating hypertables?
We currently do not offer any specific integrations with SQL Alchemy (broadly or specifically for creating hypertables). We are always interested in hearing new feature requests, so if you wanted to post your issue/use case on our Github it would help us keep better track of it for future work.
One thing that might work for your use case is to create an event trigger that executes on table creation. You'd have to check that it's in the correct schema since TimescaleDB creates its own chunk tables dynamically and you don't want to have them converted to hypertables.
See this answer for more info on event triggers:
execute a trigger when I create a table
Here is a practical example of using event trigger to create a hyper table:
from sqlalchemy import Column, Integer, DateTime, event, DDL, orm
Base = orm.declarative_base()
class ExampleModel(Base):
__tablename__ = 'example_model'
id = Column(Integer, primary_key=True)
time = Column(DateTime)
event.listen(
ExampleModel.__table__,
'after_create',
DDL(f"SELECT create_hypertable('{ExampleModel.__tablename__}', 'time');")
)
I am using SQL Alchemy ORM and have some classes/tables each of which may have some custom queries. Let's say, I want to add to table Fruit the filtering possibility called with_seed giving me only fruits with seeds, and to table Cutlery the filtering method is_sharp giving me only sharp cutlery. I want to define these filters as extensions to the Query object, and I want to use them in the same transaction:
def delete_sharp_cutlery_and_seedy_fruits(session_factory):
session = session_factory()
session.query(Fruit).with_seed().delete(synchronize_session='fetch')
session.query(Cutlery).is_sharp().delete(synchronize_session='fetch')
session.commit()
Is this possible?
This is related to the question here. But the solution there requires different sessions to be created for the different query classes.
You can pass session to Query constructor
CustomQuery(entities=[Fruit], session=session).with_seed().delete(synchronize_session='fetch')
What are the pros and cons of manually creating an ORM for an existing database vs using database reflection?
I'm writing some code using SQLAlchemy to access a pre-existing database. I know I can use sqlalchemy.ext.automap to automagically reflect the schema and create the mappings.
However, I'm wondering if there is any significant benefit of manually creating the mapping classes vs letting the automap do it's magic.
If there is significant benefit, can SQLAlchemy auto-generate the python mapping classes like Django's inspectdb? That would make creating all of the declarative base mappings much faster, as I'd only have to verify and tweak rather than write from scratch.
Edit:
As #iuridiniz says below, there are a few solutions that mimic Django's inspectdb. See Is there a Django's inspectdb equivalent for SQLAlchemy?. The answers in that thread are not Python3 compatible, so look into sqlacodegen or flask-sqlacodegen if you're looking for something that's actually maintained.
I see a lot of tables that were created with: CREATE TABLE suppliers
AS (SELECT * FROM companies WHERE 1 = 2 );, (a poor man's table copy), which will have no primary keys. If existing tables don't have primary keys, you'll have to constantly catch exceptions and feed Column objects into the mapper. If you've got column objects handy, you're already halfway to writing your own ORM layer. If you just complete the ORM, you won't have to worry about whether tables have primary keys set.
I'm trying to get the columns for a MySQL table whether in a string or a list format.
Since I defined the table through my main app, I could use dir(table_name) to get a list of attributes but those contain private attributes and other built-in attributes like "query" or "query_class". Filtering these would be possible but I'm trying to find an easier way to get the columns without going through the attribute route.
Between reading the flask-sqlalchemy documentation and sqlalchemy documentation, I noticed that using table_name.c will work for sqlalchelmy. Is there an equivalent for flask-sqlalchemy?
Flask-SQLAlchemy does nothing to the SQLAlchemy side of things, it is just a wrapper to make it easier to use with Flask along with some convenience features. You can inspect orm objects the same way you normally would. For example, inspecting a model returns its mapper.
m = db.inspect(MyModel)
# all orm attributes
print(list(m.all_orm_descriptors.keys()))
# just the columns
print(list(m.columns.keys()))
# etc.
Lets say I created a product database system for different departments of my company. Each department has its own PostgreSQL-databse-instance for various reasons. The schemata of the databases are the same, however the data in them is not. For each of these systems a Python application exists that does some business logic (irrelevant). Each Python app accesses its and only its databases through SQLAlchemy.
I want to create a Supervisior-System that can access all data in all of these databases (readthrough functionality).
Here is an example of what I think about:
Can I do that with SQLAlchemy? If so, what is the best approach for that kind of problem?
Sure you can do that with SQLAlchemy.
All you need to do is create different connection engines, each with their own session maker. Nothing in SQLAlchemy limits you to only one database at a time.
engines = []
sessions = []
for dbconninfo in databases:
engine = create_engine(dbconninfo)
engines.append(engine)
sessions.append(sessionmaker(bind=engine)())
You can use each session to run queries, result objects are attached to the session that produced them, so that changes flow back to the correct database. Do study the session documentation in detail, to see what happens if you were to merge an object from one session into another, for example.