Starting from an existing (SQLite) database with foreign keys, can SQLAlchemy automatically build relationships?
SQLAlchemy classes are automatically created via __table_args__ = {'autoload': True}.
The goal would be to easily access data from related tables without having to add all the relationships one by one by hand (i.e. without using sqlalchemy.orm.relationship() and sqlalchemy.orm.backref).
[Update] As of SQLAlchemy 0.9.1 there is Automap extension for doing that.
For SQLAlchemy < 0.9.0 it is possible to use sqlalchemy reflection.
SQLAlchemy reflection loads foreign/primary keys relations between tables. But doesn't create relations between mapped classes. Actually reflection doesn't create mapped classes for you - you have to specify mapped class name.
Actually I think that reflection support for loading foreign keys is a great helper and time saving tool. Using it you can build a query using joins without need to specify which columns to use for a join.
from sqlalchemy import *
from sqlalchemy import create_engine, orm
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
metadata = MetaData()
Base = declarative_base()
Base.metadata = metadata
db = create_engine('<db connection URL>',echo=False)
metadata.reflect(bind=db)
cause_code_table = metadata.tables['cause_code']
ndticket_table = metadata.tables['ndticket']
sm = orm.sessionmaker(bind=db, autoflush=True, autocommit=True, expire_on_commit=True)
session = orm.scoped_session(sm)
q = session.query(ndticket_table,cause_code_table).join(cause_code_table)
for r in q.limit(10):
print r
Also when I was using reflection to run queries to existing database - I had to define only mapped classes names, table bindings, relations, BUT there were no need to define table columns for these relations.
class CauseCode(Base):
__tablename__ = "cause_code"
class NDTicket(Base):
__tablename__ = "ndticket"
cause_code = relationship("CauseCode", backref = "ndticket")
q = session.query(NDTicket)
for r in q.limit(10):
print r.ticket_id, r.cause_code.cause_code
Overall SQLAlchemy reflection is already powerful tool and save me time, so adding relations manually is a small overhead for me.
If I would have to develop functionality that will add relations between mapped objects using existing foreign keys, I would start from using reflection with inspector. Using get_foreign_keys() method gives all information required to build relations - referred table name, referred column name and column name in target table. And would use this information for adding property with relationship into mapped class.
insp = reflection.Inspector.from_engine(db)
print insp.get_table_names()
print insp.get_foreign_keys(NDTicket.__tablename__)
>>>[{'referred_table': u'cause_code', 'referred_columns': [u'cause_code'], 'referred_schema': None, 'name': u'SYS_C00135367', 'constrained_columns': [u'cause_code_id']}]
As of SQLAlchemy 0.9.1 the (for now experimental) Automap extension would seem to do just that: http://docs.sqlalchemy.org/en/rel_0_9/orm/extensions/automap.html
Related
I am trying to take a simple database schema in Oracle and migrate it to a mssql database. There are other ways I can do this but my first thought was to utilize SQLAlchemy's automap and create_all functionality to do it pretty much instantaneously.
Unfortunately when I attempt to do so I run into some conversion errors:
Input:
from sqlalchemy.ext.automap import automap_base
from custom_connections import connect_to_oracle, connect_to_mssql
Base = automap_base()
oracle_engine = connect_to_oracle()
mssql_engine = connect_to_mssql()
Base.prepare(oracle_engine, reflect=True, schema = ‘ORACLE_MAIN_DB’)
Base.metadata.create_all(mssql_engine)
(Note that the connect_to functions are custom functions which return sqlalchemy engines. Currently they just return engines with base settings.)
Output:
CompileError: (in table 'acct', column 'acctnbr'): Compiler <sqlalchemy.dialects.mssql.base.MSTypeCompiler object at 0x00000268E8FF6DA0> can't render element of type <class 'sqlalchemy.dialects.oracle.base.NUMBER'>
The issue is that while Sqlalchemy is converting most types to sqlalchemy types when mapping the Base, it doesn't do the same with Oracle NUMBER types. I attempted a similar trick using alembic autogeneration off the automapped Base, but the Oracle NUMBER types caused issues there as well.
Given all the power behind it, I would have thought Sqlalchemy would be able to handle this without any issues. Is there a technique or setting I could use when running this code which would cause it to convert all types to their Sqlalchemy equivalent when mapping the base instead of just most types?
This is described in the SQLAlchemy docs about reflection:
from sqlalchemy import MetaData, Table, create_engine
from sqlalchemy import event
mysql_engine = create_engine("mysql://scott:tiger#localhost/test")
metadata_obj = MetaData()
my_mysql_table = Table("my_table", metadata_obj, autoload_with=mysql_engine)
# my_table should already exist in your db, this is the table you want to convert
#event.listens_for(metadata_obj, "column_reflect")
def genericize_datatypes(inspector, tablename, column_dict):
column_dict["type"] = column_dict["type"].as_generic()
my_generic_table = Table("my_table", metadata_obj, autoload_with=mysql_engine)
# generic table will be a generic version of my_table
sqlite_eng = create_engine("sqlite:///test.db", echo=True)\
my_generic_table.create(sqlite_eng)
Say, there is a movie.py, which contians the table definition for movie and base like
# base.py
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
engine = create_engine('postgresql://usr:pass#localhost:5432/sqlalchemy')
Session = sessionmaker(bind=engine)
Base = declarative_base()
# move.py
from sqlalchemy import Column, String, Integer, Date
from base import Base
class Movie(Base):
__tablename__ = 'movies'
id = Column(Integer, primary_key=True)
title = Column(String)
release_date = Column(Date)
def __init__(self, title, release_date):
self.title = title
self.release_date = release_date
And insert some queries like
# coding=utf-8
from datetime import date
from base import Session, engine, Base
from movie import Movie
# 2 - generate database schema
Base.metadata.create_all(engine)
# 3 - create a new session
session = Session()
# 4 - create movies
bourne_identity = Movie("The Bourne Identity", date(2002, 10, 11))
furious_7 = Movie("Furious 7", date(2015, 4, 2))
pain_and_gain = Movie("Pain & Gain", date(2013, 8, 23))
# 5 - persists data
session.add(bourne_identity)
session.add(furious_7)
session.add(pain_and_gain)
# 10 - commit and close session
session.commit()
session.close()
Is there a way I could restore the old data I have inserted, if I got a new definition for my movie table (add more columns to the move.py) ?
Did you mean to ask "how do I update the database schema of an existing database?". That's called "schema migration". There are a number of ways of attacking that. The most basic way is to have SqlAlchemy work in the other direction...have it generate its schema metadata from an existing database, instead of creating metadata and then having SqlAlchemy build a database from that. This is called Reflection. You'd do this, then issue individual commands to update your database schema. In doing this, you'd have to allow for what is going to happen to the existing rows in your table as you make these changes. You would still use your domain object definition (the Movie object), but you wouldn't use create_all(). create_all() ignores any tables that already exist.
In reality, this gets complex quickly, and so you usually want to use a formal schema migration strategy, and probably a support package for doing so. SqlAlchemy's own documentation recommends two packages for doing so. See this page:
https://docs.sqlalchemy.org/en/latest/core/metadata.html
Scroll down a bit to the "Altering Schemas through Migrations" section.
Someone may have more to offer you in terms of how to do this manually, without a migration package. I've always used such a package for any task where I wasn't willing to blow away my data and start from scratch whenever my schema changed.
Another option I've seen used is to export all your data, have SqlAlchemy build a fresh, empty database, and then import your existing data back into that new database. You would set up appropriate defaults for the new fields that won't exist in the incoming data. You'll be doing this thing with setting defaults for missing columns no matter how you choose to attack this problem.
I am using Flask-SQLAlchemy to recreate tables in an existing SQL Server database. The usual declarative base wasn't working out so I am trying to use the automap base.
My DB has a table called 'orders' and I am trying to query the data:
db = SQLAlchemy(app)
db.Model.metadata.reflect(bind=db.engine)
Base = automap_base()
Base.prepare(db.engine, reflect=True)
db.session.query(Base.classes.orders).all()
However, an ArgumentError is thrown:
ArgumentError: orders.orders and back-reference
orders.orders_collection are both of the same
direction symbol('ONETOMANY'). Did you mean to set remote_side on the
many-to-one side ?
The SQLAlchemy doc tells me that the latter orders.orders_collection is automatically created. What could I do to make sure the creation of this collection doesn't lead to this ArgumentError?
New to sqlalchemy and somewhat novice with programing and python. I had wanted to query a table. It seems I can use the all() function when querying but cannot filter without creating a class.
1.) Can I filter without creating a class and using the declarative api? Is the filtering example stated below incorrect?
2.) When would it be appropriate to use declarative api in sqlalchemy and when would it not be appropriate?
import sqlalchemy as sql
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import sessionmaker
db = sql.create_engine('postgresql://postgres:password#localhost:5432/postgres')
engine = db.connect()
meta = MetaData(engine)
session = sessionmaker(bind=engine)
session = session()
files = Table('files',meta,
Column('file_id',Integer,primary_key=True),
Column('file_name',String(256)),
Column('query',String(256)),
Column('results',Integer),
Column('totalresults',Integer),
schema='indeed')
session.query(files).all() #ok
session.query(files).filter(files.file_name = 'test.json') #not ok
If you want to filter by a Table construct, it should be:
session.query(files).filter(files.c.file_name == 'test.json')
You need to create mapped classes if you want to use the ORM features of SQLAlchemy. For example, with the code you currently have, in order to do an update you have to do
session.execute(files.update().values(...))
As opposed to:
file = session.query(File).first()
file.file_name = "new file name"
session.commit()
The declarative API happens to be the easiest way of constructing mapped classes, so use it if you want to use the ORM.
Filter using declarative api this way:
session.query(files).filter(files.file_name == 'test.json').all()
You can also use raw sql queries (docs).
Whether using declarative api or not may depend on your queries complexity, because sometimes sqlalchemy doesn't optimize them right way.
How close can I get to defining a model in SQLAlchemy like:
class Person(Base):
pass
And just have it dynamically pick up the field names? anyway to get naming conventions to control the relationships between tables? I guess I'm looking for something similar to RoR's ActiveRecord but in Python.
Not sure if this matters but I'll be trying to use this under IronPython rather than cPython.
It is very simple to automatically pick up the field names:
from sqlalchemy import Table
from sqlalchemy.orm import MetaData, mapper
metadata = MetaData()
metadata.bind = engine
person_table = Table(metadata, "tablename", autoload=True)
class Person(object):
pass
mapper(Person, person_table)
Using this approach, you have to define the relationships in the call to mapper(), so no auto-discovery of relationships.
To automatically map classes to tables with same name, you could do:
def map_class(class_):
table = Table(metadata, class_.__name__, autoload=True)
mapper(class_, table)
map_class(Person)
map_class(Order)
Elixir might do everything you want.
AFAIK sqlalchemy intentionally decouples database metadata and class layout.
You may should investigate Elixir (http://elixir.ematia.de/trac/wiki): Active Record Pattern for sqlalchemy, but you have to define the classes, not the database tables.